Wireshark Plugin for TLS
I am currently running Wireshark 4.4.6 and option for F5 TLS in Analyze > Enabled Protocols is missing. I have been trying to locate plugins to install but not having any luck. The problem is when I do captures encrypted traffic, the TLS info is missing, and I am unable to locate keylog data to decrypt. Do anyone know any plugin locations or alternative steps to decrypt without using the irule option? tcpdump -nn -s0 -i 0.0:nnnp -w /shared/tmp/Cxxxx_tcpdump_12_Jun_05_59_40_F05.com.pcap --f5 ssl host XXX.XXX.XXX.XXX
Running Wireshark captures from F5 BIG-IP
My colleague, Simon Kowallik, recently posted something really cool to our internal message boards. It started with the question: “Have you ever wanted to run captures with Wireshark on BIG-IP?” Answer: Yes, for like twelve years I wanted to do this! In the post below, Simon shows us how to use the packet tracing tool Wireshark (or any other tool that reads pcaps from tcpdump) directly with BIG-IP using only some slight of hand. Anyway, I thought this was so awesome that it deserved wider audience so here it is, republished with Simon’s permission. Have fun! Posted by Simon Kowallik in on Jul 7, 2013 9:02:38 AM We actually can do that without installing X, wireshark and hundreds of libraries on BIG-IP. Which is not an option anyway. 🙂 There are a few things we need: SSH access to the BIG-IP, bash or tmsh is fine Proper SSH client on our Desktop, eg. OpenSSH or alternatives (putty & plink) The trick is to launch an ssh session without a login shell and run tcpdump through it on the remote system making tcpdump write raw packets to STDOUT while piping it to our local wireshark reading from STDIN. Here are two examples: cygwin on Windows # ssh -l root 192.168.1.245 "tcpdump -w - -s0 -pi 0.0 tcp or udp or icmp" | /cygdrive/c/Program\ Files/Wireshark/Wireshark.exe -k -i - Linux # ssh -l root 192.168.1.245 "tcpdump -w - -s0 -pi 0.0 tcp or udp or icmp" | /usr/bin/wireshark -k -i - Windows CMD with plink (download from putty homepage): plink.exe -l root -pw default 192.168.1.245 "tcpdump -w - -s0 -pi 0.0 tcp or udp or icmp" | "c:\Program Files\Wireshark\wireshark.exe" -k -i - I think you can figure out how it works. If not, here are a few hints: Tcpdump's option -w with - as an argument writes to STDOUT instead of a file Wireshark's -i option reads from an interface, - as an argument makes STDIN the interface. STDIN/STDOUT is represented by - on most platforms. Caveats Tcpdump does buffer the output when writing to a file (our STDOUT in our case), which unfortunately means it might take some time until we can see the traffic in wireshark. Tcpdump offers options to influence the buffering however this is not implemented in our version of Libpcap (tested on 11.4HF1). This is especially annoying if we want to capture low volume traffic. What we could do is capturing icmp echo requests+replies additionally to the traffic we are interested in, and remove them again with the wireshark display filter. Then start a ping to push the interesting packets to wireshark faster. Words of warning You are piping the whole packet capture through ssh, so make sure you define your tcpdump filter reasonable, otherwise bad things might happen.Packet Analysis with Scapy and tcpdump: Checking Compatibility with F5 SSL Orchestrator
In this guide I want to demonstrate how you can use Scapy (https://scapy.net/) and tcpdump for reproducing and troubleshooting layer 2 issues with F5 BIG-IP devices. Just in case you get into a finger-pointing situation... Starting situation This is a quite recent story from the trenches: My customer uses a Bypass Tap to forward or mirror data traffic to inline tools such as IDS/IPS, WAF or threat intelligence systems. This ByPass Tap offers a feature called Network Failsafe (also known as Fail-to-Wire). This is a fault tolerance feature that protects the flow of data in the event of a power outage and/or system failure. It allows traffic to be rerouted while the inline tools (IDS/IPS, WAF or threat intelligence systems) are shutting down, restarting, or unexpectedly losing power (see red line named Fallback in the picture below). Since the ByPass Tap itself does not have support for SSL decryption and re-encryption, an F5 BIG-IP SSL Orchestrator shall be introduced as an inline tool in a Layer 2 inbound topology. Tools directly connected to the Bypass Tap will be connected to the SSL Orchestrator for better visibility. To check the status of the inline tools, the Bypass Tap sends health checks through the inline tools. What is sent on one interface must be seen on the other interface and vice versa. So if all is OK (health check is green), traffic will be forwarded to the SSL Orchestrator, decrypted and sent to the IDS/IPS and the TAP, and then re-encrypted and sent back to the Bypass Tap. If the Bypass Tap detects that the SSL Orchestrator is in a failure state, it will just forward the traffic to the switch. This is the traffic flow of the health checks: Target topology This results in the following topology: Problem description During commissioning of the new topology, it turned out that the health check packets are not forwarded through the vWire configured on the BIG-IP. A packet analysis with Wireshark revealed that the manufacturer uses ARP-like packets with opcode 512 (HEX 02 00). This opcode is not defined in the RFC that describes ARP (https://datatracker.ietf.org/doc/html/rfc826), the RFC only describes the opcodes Request (1 or HEX 00 01) and Reply (2 or HEX 00 02). NOTE: Don't get confused that you see ARP packets on port 1.1 and 1.2. They are not passing through, the Bypass Tap is just send those packets from both sides of the vWire, as explained above. The source MAC on port 1.1 and 1.2 are different. Since the Bypass Tap is located right behind the customer's edge firewall, lengthy and time-consuming tests on the live system are not an option, since it would result in a massive service interruption. Therefore, a BIG-IP i5800 (the same model as the customer's) was set up as SSL Orchestrator and a vWire configuration was build in my employers lab. The vWire configuration can be found in this guide (https://clouddocs.f5.com/sslo-deployment-guide/chapter2/page2.7.html). INFO: For those not familiar with vWire: "Virtual wire … creates a layer 2 bridge across the defined interfaces. Any traffic that does not match a topology listener will pass across this bridge." Lab Topology The following topology was used for the lab: I build a vWire configuration on the SSL Orchestrator, as in the customer's environment. A Linux system with Scapy installed was connected to Interface 1.1. With Scapy TCP, UDP and ARP packets can be crafted or sent like a replay from a Wireshark capture. Interface 1.3 was connected to another Linux system that should receive the ARP packets. All tcpdumps were captured on the F5 and analyzed on the admin system (not plotted). Validating vWire Configuration To check the functionality of the F5 and the vWire configuration, two tests were performed. A replay of the Healthcheck packets from the Bypass Tap and a test with RFC-compliant ARP requests. Use Scapy to resend the faulty packets First, I used Wireshark to extract a single packet from packet analysis we took in the customer environment and saved it to a pcap file. I replayed this pcap file to the F5 with Scapy. The sendp() function will work at layer 2, it requires the parameters rdpcap (location of the pcap file for replay) and iface (which interface it shall use for sending). webserverdude@tux480:~$ sudo scapy -H WARNING: IPython not available. Using standard Python shell instead. AutoCompletion, History are disabled. Welcome to Scapy (2.5.0) >>> sendp(rdpcap("/home/webserverdude/cusomter-case/bad-example.pcap"),iface="enp0s31f6") . Sent 1 packets. This test confirmed the behavior that was observed in the customer's environment. The F5 BIG-IP does not forward this packet. Use PING and Scapy to send RFC-compliant ARP packets To create RFC-compliant ARP requests, I first sent an ARP request (opcode 1) through the vWire via PING command. As expected, this was sent through the vWire. To ensure that this also works with Scapy, I also resent this packet with Scapy. >>> sendp(rdpcap("/home/webserverdude/cusomter-case/good-example.pcap"),iface="enp0s31f6") . Sent 1 packets. In the Wireshark analysis it can be seen that this packet is incoming on port 1.1 and then forwarded to port 1.3 through the vWire. Solving the issue with the help of the vendor It became evident that the BIG-IP was dropping ARP packets that failed to meet RFC compliance, rendering the Bypass Tap from this particular vendor seemingly incompatible with the BIG-IP. Following my analysis, the vendor was able to develop and provide a new firmware release addressing this issue. To verify that the issue was resolved in this firmware release, my customer's setup, the exact same model of the Bypass Tap and a BIG-IP i5800, were deployed in my lab, where the new firmware underwent thorough testing. With this approach I could test the functionality and compatibility of the systems under controlled conditions. In this Wireshark analysis it can be seen that the Healthcheck packets are incoming on port 1.1 and then forwarded to port 1.3 through the vWire (marked in green) and also the other way round, coming in on port 1.3 and then forwarded to port 1.1 (marked in pink). Also now you can see that the packet is a proper gratuitous ARP reply (https://wiki.wireshark.org/Gratuitous_ARP). Because the Healthcheck packets were not longer dropped by the BIG-IP, but were forwarded through the vWire the Bypass Tap subsequently marked the BIG-IP as healthy and available. The new firmware resolved the issue. Consequently, my customer could confidently proceed with this project, free from the constraints imposed by the compatibility issue.
Wireshark not displaying application data for tcpdump using ssldump
Hello everyone I have been testing SSLdump and I have ran into what seems to be a Wireshark problem but I'm not sure. I have added a custom Client SSL Profile to exclude Diffie-Hellman algorithms using the following Cipher Option: NATIVE:!DH:!EDH:!DHE:!ADH:!ECDHE I have also adjusted the Cache Size to 0 sessions and Cache Timeout to 1 seconds so that we do not cache anything. During the SSL Handshake we select the TLS_RSA_WITH_AES_256_CBC_SHA256 and when running the SSLdump command I get entries in the PMS log AND I can see decrypted data. When I launch Wireshark and check the tcpdump + load the PMS I do not see any difference at all. When I check the follow SSL Stream I can see the decrypted data that I saw in the SSLdump. But the thing is I want to see the packets in the packet list so I can follow the SYN/ACK packets with the GET requests. But I do not see any GET requests at all. I noticed that when I have not added the PMS key I do not have any packet that states "Application Data" and I believe here is the problem. Here is how it looks when reviewing an F5 technician doing it: No PMS: With PMS: Here is how my output looks (No PMS): The output I can see in my ssldump is this: 1 10 1476792858.7953 (0.0006) C>SV3.3(336) application_data --------------------------------------------------------------- GET / HTTP/1.1 Accept: text/html, application/xhtml+xml, */* Accept-Language: sv-SE User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko Accept-Encoding: gzip, deflate Host: [Hidden] Connection: Keep-Alive Cache-Control: no-cache So there are application data but Wireshark does not want to display it in the Packet List. I'm currently running: * Wireshark - Version 1.12.5 (v1.12.5-0-g5819e5b from master-1.12) * F5: 12.1.0 Build: 0.0.1434 I'm running the exact same tcpdump command as the F5 engineer. You guys got any idea on how to display the packets in the Packet List?
Wireshark F5 Plugin - Unable to locate file
The last couple of hours I've spent attempting to install the F5 plugin for Wireshark. The directions I've followed are on this DevCentral article. Here's where I am at: 1) Downloaded the Wireshark source tarball. 2) Extracted the file. Step 3 says to extract the files in the F5 package, but there is no F5 package there. I have a wireshark-plugin.f5ethtrailer.bin.1.11.zip file I downloaded from DevCentral, but that's not what is being asked for I don't believe. There's a comment in the Notes section that says the following: When compiling on Windows, you need to pretty much build the entire WS distro due to the way Windows handles DLLs. For Linux (and I believe Mac, but I’m not sure), you can get the sources all setup, add in the plugin source, run configure and then run make only in the plugins/f5ethtrailer directory This makes me believe I need to uninstall and reinstall Wireshark entirely. Is that so? Not sure what I'm doing wrong here. Any help would be appreciated!The Core Addendum: CTF Walkthrough
On the debut of my new show The Core on Tuesday, I got to share a few, but not all of the Wireshark exercises I had planned. In honor of Cybersecurity Awareness Month, I thought it appropriate to frame the activity in a capture the flag (CTF) format as is often offered at security conferences and increasingly as well through online learning platforms. All the exercises, my approach to solving them, and the answers are below. I worked from two packet captures from my local test BIG-IP system. Task 1.A - How many arp requests for 10.0.3.100? My approach to most tasks is to start with a wide net and work toward the narrow. So for this first one, I apply a generic arp filter. I could count from there, as you can see that there is at least one matching in packet 477, but instead, I can narrow the filter to the arp destination of the IP asked for. And with that, I have a final answer for this task of 5. Task 1.B - How many ping requests? For this task, following directions is necessary, as the request is for ping requests, not all pings. There could be even more icmp traffic at any given time, but this capture only has the request/response types. Our task is request only, which has an icmp type of 8. Final answer: 7 Task 1.C - How many multicast DNS queries for an "Elgato Key Light Air"? Wrapping up the first task focus, this can be accomplished one of two ways, with a generic mdns protocol filter or the dns.qry.name filter and then the contains operator for that string. Both results are the same, and the final answer here is 13. Task 2 - What’s the MAC address for the IP host 10.0.4.20? This one is pretty straight forward as well. I just need to find the IP address with the ip.addr filter and match the IP as source or destination to the MAC address, but this can be refined to ip.dst or ip.src so I only need to consider one. I use ip.dst in this case. Final answer: the mac address is 00:0c:29:b4:5c:90. Task 3 - What are the resolved names for the 00:0c:29:21:ad:4d and ff:ff:ff:ff:ff:ff MAC addresses? If you look at the images above in Task 1A, you'll notice that the MAC addresses in the Source/Destination columns are raw. To enable name resolution, I need to change the preferences in Wireshark. And now I can see the resolved MAC names. Final answer: resolved MAC names are VMware_21:ad:4d and Broadcast (duh...), respectively. Task 4.A - What percentage of packets in the capture are handled by TMM0? This task is our first look at using the f5ethtrailer filter list, incorporating the f5ethtrailer.tmm filter. Note that in Wireshark versions of yesteryear you had to compile this into a custom Wireshark build yourself. But from v2.6+, it's included and at least in my latest version, also enabled by default. Final answer: 77.1% of the packets in this capture are handled by TMM0. Task 4.B - What number of packets are handled by TMM1? This task only needs an adjustment of the tmm number. I included it just for the purpose of being attentive to what's asked. When you're in a competition, sometimes you only get one attempt to answer, so make sure you don't fall into the trap of providing the information in the wrong format. Final answer: 148 packets. Task 5 - For a traffic flow to ip 10.0.4.20 on port 8081, what’s the request verb and the value of the X-Server-Disabled header in the response? This one I combine two filters, ip.addr and tcp.port 8081 to find the flow or flows for this address/port pair and then look at the TCP stream of the first flow and hope for the best. That selection reveals the text-based flow of HTTP messages in this TCP flow. Final answer: request verb is HEAD and the X-Server-Disabled header value is true. Task 6 - What’s the BIG-IP version and tcpdump command that was run to generate this packet capture This has an easy cheat in that you just need to look at the first packet in the capture, but the goal is to identify the filter f5fileinfo.cmdline. Final answer: BIG-IP version is 15.1.2.1 (build 0.0.10) and the tcpdump command was "-ni 0.0 -w /var/tmp/f5tcpdump1.pcap -s0". Note that even though I didn't specify even the low F5 details in this capture that a limited subset of that information is provided anyway, such as the BIG-IP system info shown here as well as the TMM core shown in tasks above. Task 7 - Limit the capture to only tcp ports 80, 443, 8080, 8081 OR tcp resets? This task is an easy grab from the same filter as in task 6, but I wanted to make mention of the difference of capture filters and display filters. Capture filters reduce the number of packets that a) the system has to log, reducing the impact to system performance during capture and b) the analyst has to comb through and process. These are both net positives with the assumption that you are not restricting the capture past the point where you lose the necessary data to properly capture the problem you're trying to find and solve. Display filters are only limiting what you are looking at after the capture is already taken. The data set itself is not changed. Final answer: The capture filter is ‘(tcp[13] & 4!=0) or (tcp port 80 or 443 or 8080 or 8081)’. Note that the TCP flags are in offset byte 13 in the TCP header, and the RST flag is in the 2**2 position (or 4) of the byte. Also note in that tcpdump command the --f5 n,ssl:v. More on that later. Task 8 - How many GET requests can you see against tcp port 443 This is another multi-filter combination, using http and tcp.port. Hmmm, that seems fishy..or does it? Would you expect to see HTTP traffic in cleartext to TCP port 443, which is normally SSL traffic and thus encrypted? Interesting. The final answer, for now, is 0. Task 9 - How do we decrypt the HTTPS traffic and once done, repeat task 8? In task 7 I noted the --f5 ssl flag on the tcpdump. When you enable the database key tcpdump.sslprovider and use that flag when taking a capture, the key material will be included in your capture (WARNING!! Do not share/store this file without understanding the repercussions!) so you can decrypt the traffic for analysis. On the Wireshark side since I did include this in this second capture, I use the f5ethtrailer.keylog filter to find all the keylog entries, which need to be copied out of the capture to a file and then referenced back in the Wireshark settings. I show one below, but note that all should be done to make the whole capture decrypted. This can be done auto-magically with tshark as shown at the bottom of this lab document on Clouddocs. I take that value and copy it into a file (and the other entries as well, but just showing this one for brevity). Then I open Wireshark's preferences and select protocols and then TLS and select that file. And now I'll repeat task 8. And BINGO! Final answer: 5 GET requests. Task 10 - What are the virtual server names for the destinations 10.0.3.50:80 and 10.0.3.50:443, respectively? For this final task in this CTF walkthrough, I'll need the f5ethtrailer.vip filter to narrow down the packets I need to reference, and then a combination as well of ip.dst and tcp.port filters. First, port 80. And secondly port 443. Final answer: /Common/testvip for port 80, /Common/nerdlife_ssl_vip for port 443. Conclusion And there we have it! A successful journey through a variety of tasks analyzing a couple BIG-IP packet captures. As I mentioned during the show, I had a teacher in a packet analysis 20 years ago who encouraged 15 minutes a day looking at captures to keep those skills sharp. I didn't, and he was right. Use it or lose, people!SSL Forward Proxy Explained using Wireshark
Related articles: SSL Legacy Renegotiation vs Secure Renegotiation Explained using Wireshark Summary The idea behind this feature is to allow BIG-IP to sniff into SSL connections to any Internet destination that goes through it whilst preserving client's trust of the destination. When this feature is enabled, BIG-IP uses a pre-installed CA set under proxy-ca-cert (CA Certificate in the GUI) that is also trusted by client's browser and BIG-IP pretends to be the Internet destination by re-signing SSL certificates on the fly with the aforementioned CA and using public key from CA configured under cert on BIG-IP. The client trusts the connection and is unaware of BIG-IP's presence. The most common use case is Secure Web Gateway (SWG), although neither APM nor SWG are needed to deploy SSL Forward Proxy functionality. LTM is enough. Lab Scenario Lab test results: Client opens connection with BIG-IP and sends Client Hello. BIG-IP immediately opens connection with web server and completes SSL handshake. BIG-IP creates unique certificate with following information and sends it over via Server Hello: Serial number: random Issuer: copied from default.crt Validity: notBefore (copied from server1.crt), notAfter: 30 days counting from time it was generated (configurable via cert-lifespan or Certificate Lifespan in the GUI) Subject: copied from server1.crt Public Key: copied from ltm2.CA.crt All X.509 extensions allowed in cert-extensions-include are also copied from server1.crt to unique custom certificate Client-side handshake completes successfully using above dynamically generated custom certificate just created by BIG-IP and data exchange works as expected. How SSL Forward Proxy works on Wireshark Actual capture used for below explanation is attached to this article below (and here ssl-forward-sample-3.pcap ) Upon Client Hello sent by Client BIG-IP rushes to open new connection and completes server-side SSL handshake: Server-side handshake completes on frames 17-18 and and BIG-IP immediately resumes and completes Client-side handshake. Note: The reason why BIG-IP waits for server-side handshake to complete first is because BIG-IP uses server-side certificate information to generate a custom certificate that will be sent to Client on client-side. The most interesting bit here is BIG-IP's Certificate message sent on frame 20 to Client as it contains the custom Certificate generated by BIG-IP: We can see there are 2 Certificates: CA Certificate (proxy-ca-cert) is the Certificate Authority (CA) that signs the newly created custom certificate and is supposed to be trusted by Client to confirm BIG-IP's custom (dynamically generated) certificate can be trusted too. We can immediately work out that the 2nd certificate is default.crt just by matching its serial number which is unique for each issuer: However, the first certificate looked a mystery to me until I found where all the pieces came from. The serial number looks randomly generated to me as it doesn't match (not even close to any of the certificates above): The Issuer field, which shows information about who signed the certificate, shows that default.crt was the one who signed BIG-IP's custom certificate: Certificate generation date (notBefore) is copied from server1.crt but expiry date (notAfter) is calculated based on cert-lifespan (Certificate Lifespan in the GUI) which by default is 30 days added to time custom certificate was generated: Subject fields, which shows information about the current certificate, is copied and identical to information on server1.crt: And finally, the public key is copied from cert field (ltm2CA.crt). Notice they are the same: That's enough information for now.Understanding HTTP/2 Profile's Frame Size option on BIG-IP
Quick Intro The Overview of the BIG-IP HTTP/2 profile article on AskF5 I created a while ago describes all the HTTP/2 profile options but sometimes we need to test things out ourselves to grasp things at a deeper level. In this article, I'm going to show how Frame Size option sets specifically only the maximum size of HTTP/2 DATA message's payload in bytes and what happens when we change this value on Wireshark. Think of it as a quick walkthrough to give us a deeper understanding of how HTTP/2 works as we go through. The Topology It's literally a client on 10.199.3.135 and a virtual server with HTTP + HTTP/2 profile applied with the default settings: Testing Frame Size Option Here I've tried to modify frame-size to an invalid value so we can see the valid range: Let's set the frame-size to 1024 bytes: I have curl installed in my client machine and this is the command I used: If we just filter for http2 on Wireshark, we should see the negotiation phase (SETTINGS) as well as request (GET) and response (200 OK) headers in their specific message type (HEADERS). However, our focus here is on DATA message type as seen below: I've now added a new column (Length) to include the length of DATA messages so we can easily see how Frame Size settings affect DATA length. Here's how we create such filter: I've further renamed it to HTTP2 DATA Length but you've got the point. If we list only DATA messages, we can see that the payload of HTTP/2 DATA message type will not go beyond 1024 bytes: Wireshark confirms that HTTP/2 headers + DATA payload of frame 26 is 1033 bytes but DATA payload-only is 1024 bytes as seen below: We can then confirm that only payload counts for frame-size configuration on BIG-IP. I hope you enjoyed the above hands-on walk-through.TCP Internals: 3-way Handshake and Sequence Numbers Explained
In this article, I will explain and show you what really happens during a TCP 3-way handshake as captured by tcpdump tool. We'll go deeper into details of TCP 3-way handshake (SYN, SYN/ACK and ACK) and how Sequence Numbers and Acknowledgement Numbers actually work. Moreover, I'll also briefly explain using real data how TCP Receive Window and Maximum Segment Size play an important role in TCP connection. As a side note, I will not touch TCP SACK and TCP Timestamps this time as they should be covered in a future article about TCP retransmissions. FYI, the TCP capture was generated by a simple HTTP GET request to BIG-IP to get hold of a file on /cgi-bin/ directory called script.pl using HTTP/1.1 protocol: BIG-IP then responds with HTTP/1.1 200 OK with the requested data. This is not very relevant as we'll be looking at TCP layer but it's good to understand the capture's context to fully understand what's going on. This is what a TCP 3-way handshake looks like on Wireshark: As we can see, the first 3 packets are exchanged less than 1 second apart from each other. The IN/OUT portion of Info field on BIG-IP's capture tells us if the packet is coming IN or being sent OUT by BIG-IP (as capture was taken on BIG-IP). As this is a slightly more in-depth explanation of TCP internals, I am assuming you know at least what a TCP 3-way handshake is conceptually. The TCP SYN, SYN/ACK and ACK Segments We can see that first packet is [SYN], second one is [SYN/ACK] and last one is [SYN/ACK] as displayed on Wireshark. The Info section as a whole only shows the summary of the most relevant fields copied from the TCP header. It is just enough to make us understand the context of the TCP segment. Let's now have a look what these fields mean with the exception of SACK_PERM and TSval. When we double click on the [SYN] packet below, we find the same information again in the actual TCP header: The most important thing to understand here is that [SYN], [SYN/ACK] and [ACK] are all part of the Flags header above. They're just 1's and 0's. When SYN flag is enabled (i.e its value is 1), the receiving end (in this case BIG-IP) should automatically understand that someone (my client PC in this case) is trying to establish a TCP connection. The response from BIG-IP (SYN/ACK) is an acknowledgement to the SYN packet and therefore it has both SYN and ACK flags set to 1. Client's last response is just an ACK as seen below: As per RFC, both sides should now assume a TCP connection is established. For plain-text HTTP/1.1 protocol, there should now be a GET request in another layer as a payload of (or encapsulated by) TCP layer. If our traffic it is protected by TLS then TLS layer should come first as the payload of TCP layer and HTTP would be the payload of TLS layer. Does it make sense? That's how things work in the real world. TCP Sequence numbers A side note, Wireshark shows that our first SYN segment's Sequence number is 0 (Seq=0): It also shows that it is relative sequence number but this is not the real TCP sequence number. Wireshark automatically zeroes it for you to make it easier to visualise and/or troubleshoot. In reality, the real sequence number is a much longer number that is calculated by your OS using current time and other random parameters for security purposes. This is how we see the real sequence number in Wireshark: Now back to business. Some people say if Client sends a TCP segment to BIG-IP, BIG-IP's ACK should be client's sequence number + 1 right? Wrong! Instead of +1 it should be + number of bytes last received from peer or +1 if SYN or FIN segments. To clarify, here's the full Flow Graph of our capture using relative sequence numbers to make it easier to grasp (.135 = Client and .143 = BIG-IP): On 4th segment above (PSH, ACK - Len: 93), client sends TCP segment with Seq = 1 and TCP payload data length (comprised of HTTP layer) of 93 bytes. In this case, BIG-IP's response is not ACK = 2 (1 + 1) as some might think. Instead, BIG-IP responds with whatever client's last Sequence number was plus number of bytes last received. As last sequence number was 1 and client also sent a TCP payload of 93 bytes, then ACK is 94! This is the most important concept to grasp for understanding sequence numbers and ACKs. SEQs and ACKs only increment when there is a TCP payload involved (by the number of bytes). SYN, FIN or ZeroWindow segments count as 1 byte for SEQs/ACKs. I added a full analysis using real TCP SEQs/ACKs to an Appendix section if you'd like to go deeper into it. For the moment let's shift our attention towards TCP Receive Window. TCP Receive Window and Maximum Segment Size (MSS) During 3-way handshake, the Receive Window (Window size value on Wireshark) tells each side of the connection the maximum receiving buffer in bytes each side can handle: So it's literally like this (read red lines first please): [1] → Hey, BIG-IP! My receiving buffer size is 29200 bytes. That means, you can initially send me up to 29200 bytes before you even bother waiting for an ACK from me to send further data. [2] → This should be the same as [1], unless Window Scale TCP Option is active. Window Scale should be the subject of a different article but I briefly touch it on [3]. [3] → Original TCP Window Size field is limited to 16 bits so maximum buffer size is just 65,535 bytes which is too little for today's speedy connections. This option extends the 16-bit window to 32-bit window but because BIG-IP did not advertise Window Scale option for this connection, it is disabled as both sides must support it for it to be used. [4] → Hey, client! My receiving buffer size is 4380 bytes. That means, you can initially send me up to 4328 bytes before you even bother waiting for an ACK from me to send further data. The reason why the word initially is underlined on [1] and [3] is because Window size typically changes during the connection. For example, client's initial window size is 29200 bytes, right? This means that if it receives 200 bytes from BIG-IP it should go down to 2900 bytes. Easy, eh? But that's not what always happens in real life. In fact, in our capture it's the opposite! Bytes in flight column shows the data BIG-IP (*.143) is sending in bytes to our client (*.135) that has not yet been acknowledged. I've added a column with Window Size value to make it easier to spot how variable this field is: It is the OS TCP Flow control implementation that dictates the Receive Window size taking into account the current "health" of its TCP stack and of course your configuration. Yes, in many cases, especially in the middle of a connection, the Window Size does decrease based on amount of data received/buffered so our first explanation also makes sense! How does BIG-IP know that client has freed up it's buffer again? As we can see above, when Client ACKs the receipt of BIG-IP's data, it also informs the size of its buffer in the Window Size value field. That's how BIG-IP knows how much data it can send to Client before it receives another ACK. What about the Maximum Segment Size? Each side also displays a TCP Option - Maximum Segment size of 1460 bytes. This informs the maximum size of the TCP payload each side can send at a time (per TCP segment). Looking at the picture above, BIG-IP sent 334 bytes of TCP payload to client, right? In theory, this could've been up to 1460 bytes as it's also within client's initial buffer of 29200 bytes. So apart from informing each other about the maximum buffer, the maximum size of TCP segment is also informed. TCP Len vs Bytes in Flight Column (BIF) If we look at our last picture, we can see that whatever is in Len field matches what's in our BIF column, right? Are they the same? No! Len shows the current size of TCP payload (excluding the size of TCP header). Remember that TCP payload in this case is the whole HTTP portion that our TCP segment is carrying. Bytes in flight is not really part of TCP header but that's something Wireshark adds to make it easier for us to troubleshoot. It just means the number of bytes sent that have not yet been acknowledged by receiver. In our capture, data is acknowledged immediately so both Len and BIF are the same. I've picked a different capture here where there are 3 TCP segments sent with no acknowledgement so BIF column increments for each unacknowledged data segment but goes back to zero as soon as an ACK is received by receiver: Notice that BIF values now differ from TCP payload (the equivalent to Len in Info column). That's it for now. The next article would be about TCP retransmission. Appendix - Going in depth into TCP sequence numbers! Here's a full explanation about what actually takes place on TCP layer from the point of view of BIG-IP: Just follow along from [1] to [10]. That's it.Hello, DevCentral! New F5 DevCentral member saying hi (DevOps)
Hey guys, I'd like to introduce myself as a new F5 DevCentral team member and to say I'm very happy to be part of this outstanding community. I've been working for F5 (based in the UK) for over 4 years and mostly for our Engineering department where I gained experience with a variety of technology: SSL/TLS, HTTP, HTTP/2, OSPF/BGP, Cloud and DevOps. I'm specifically more focused on DevOps content here so feel free to let me know if there is any topic you'd like to be covered or any ideas suggestion. Also, don't hesitate to reach out to me ! :) Cheers. Rodrigo
