decrypted tcpdump capture without using an iRule and without using tshark
Last week I attended the Wireshark Foundation’s SharkFest in Warsaw. While there I raised a question with core developer Stig Bjørlykke that’s been bothering me for some time: why go through all the hassle of using tshark when the needed data is already present in the pcap? There must be a smarter way to do this from within Wireshark — and there is. Although the solution described in https://my.f5.com/manage/s/article/K31793632 and Mohamed_Ahmed_Kansoh’s post are useful, and Jason Rahm’s script (https://github.com/f5-rahm/pcap_utils/blob/main/TLSv1_3_captures.py) is also helpful, I’m particularly pleased with the Lua-based approach Stig shared with me. I’m happy to pass it along. The Lua script reads session keys from the capture and exports them to a Pre-Master Secret (PMS) log file, using the correct formatting — no tshark, sed, or other external tools required. How to use it: Copy the Lua script into a file and place it in your Wireshark Personal Lua Plugins folder. (You can find that folder via Help → About Wireshark → Folders.) Open the capture you want to decrypt. Make sure it was captured as described in K31793632. Make sure the PMS file option is enabled and points to the desired file (Preferences → Protocols → TLS → Pre-Master Secret log filename). Under the Tools menu you will find the Lua script. Run it — it will display which PMS file is being used and how many keys it found. Click Export, then Close. That’s it — your capture should now be decrypted. -- F5 Keylog Export Wireshark Plugin set_plugin_info({version = "1.0", author = "Stig Bjørlykke <stig@bjorlykke.org>"}) local keylog_field = Field.new("f5ethtrailer.tls.keylog") local keylog_list = "" local keylog_count = 0 local function export_keylog() local keylog_file = get_preference("tls.keylog_file"); if keylog_file and keylog_file ~= "" then io.open(keylog_file, "w"):write(keylog_list):close() redissect_packets() end end local function f5_keylog_export() local tw = TextWindow.new("F5 Keylog Export") local tap = Listener.new("f5ethtrailer") tw:add_button("Export", function() export_keylog() end) tw:set_atclose(function () tap:remove() end) function tap.packet() for _, keylog in ipairs({keylog_field()}) do keylog_list = keylog_list .. keylog.value .. "\n" keylog_count = keylog_count + 1 end end function tap.draw() local keylog_file = get_preference("tls.keylog_file"); if keylog_file and keylog_file ~= "" then tw:set("TLS keylog file: " .. keylog_file .. "\n") tw:append("Press Export to write keys to file.\n\n") tw:append("Found " .. keylog_count .. " keylog entries.") else tw:set("No TLS keylog file specified in Preferences -> Protocols -> TLS\n\n") end end function tap.reset() keylog_list = "" keylog_count = 0 tw:clear() end retap_packets() end register_menu("F5 Keylog Export", f5_keylog_export, MENU_TOOLS_UNSORTED)
TCP Internals: 3-way Handshake and Sequence Numbers Explained
In this article, I will explain and show you what really happens during a TCP 3-way handshake as captured by tcpdump tool. We'll go deeper into details of TCP 3-way handshake (SYN, SYN/ACK and ACK) and how Sequence Numbers and Acknowledgement Numbers actually work. Moreover, I'll also briefly explain using real data how TCP Receive Window and Maximum Segment Size play an important role in TCP connection. As a side note, I will not touch TCP SACK and TCP Timestamps this time as they should be covered in a future article about TCP retransmissions. FYI, the TCP capture was generated by a simple HTTP GET request to BIG-IP to get hold of a file on /cgi-bin/ directory called script.pl using HTTP/1.1 protocol: BIG-IP then responds with HTTP/1.1 200 OK with the requested data. This is not very relevant as we'll be looking at TCP layer but it's good to understand the capture's context to fully understand what's going on. This is what a TCP 3-way handshake looks like on Wireshark: As we can see, the first 3 packets are exchanged less than 1 second apart from each other. The IN/OUT portion of Info field on BIG-IP's capture tells us if the packet is coming IN or being sent OUT by BIG-IP (as capture was taken on BIG-IP). As this is a slightly more in-depth explanation of TCP internals, I am assuming you know at least what a TCP 3-way handshake is conceptually. The TCP SYN, SYN/ACK and ACK Segments We can see that first packet is [SYN], second one is [SYN/ACK] and last one is [SYN/ACK] as displayed on Wireshark. The Info section as a whole only shows the summary of the most relevant fields copied from the TCP header. It is just enough to make us understand the context of the TCP segment. Let's now have a look what these fields mean with the exception of SACK_PERM and TSval. When we double click on the [SYN] packet below, we find the same information again in the actual TCP header: The most important thing to understand here is that [SYN], [SYN/ACK] and [ACK] are all part of the Flags header above. They're just 1's and 0's. When SYN flag is enabled (i.e its value is 1), the receiving end (in this case BIG-IP) should automatically understand that someone (my client PC in this case) is trying to establish a TCP connection. The response from BIG-IP (SYN/ACK) is an acknowledgement to the SYN packet and therefore it has both SYN and ACK flags set to 1. Client's last response is just an ACK as seen below: As per RFC, both sides should now assume a TCP connection is established. For plain-text HTTP/1.1 protocol, there should now be a GET request in another layer as a payload of (or encapsulated by) TCP layer. If our traffic it is protected by TLS then TLS layer should come first as the payload of TCP layer and HTTP would be the payload of TLS layer. Does it make sense? That's how things work in the real world. TCP Sequence numbers A side note, Wireshark shows that our first SYN segment's Sequence number is 0 (Seq=0): It also shows that it is relative sequence number but this is not the real TCP sequence number. Wireshark automatically zeroes it for you to make it easier to visualise and/or troubleshoot. In reality, the real sequence number is a much longer number that is calculated by your OS using current time and other random parameters for security purposes. This is how we see the real sequence number in Wireshark: Now back to business. Some people say if Client sends a TCP segment to BIG-IP, BIG-IP's ACK should be client's sequence number + 1 right? Wrong! Instead of +1 it should be + number of bytes last received from peer or +1 if SYN or FIN segments. To clarify, here's the full Flow Graph of our capture using relative sequence numbers to make it easier to grasp (.135 = Client and .143 = BIG-IP): On 4th segment above (PSH, ACK - Len: 93), client sends TCP segment with Seq = 1 and TCP payload data length (comprised of HTTP layer) of 93 bytes. In this case, BIG-IP's response is not ACK = 2 (1 + 1) as some might think. Instead, BIG-IP responds with whatever client's last Sequence number was plus number of bytes last received. As last sequence number was 1 and client also sent a TCP payload of 93 bytes, then ACK is 94! This is the most important concept to grasp for understanding sequence numbers and ACKs. SEQs and ACKs only increment when there is a TCP payload involved (by the number of bytes). SYN, FIN or ZeroWindow segments count as 1 byte for SEQs/ACKs. I added a full analysis using real TCP SEQs/ACKs to an Appendix section if you'd like to go deeper into it. For the moment let's shift our attention towards TCP Receive Window. TCP Receive Window and Maximum Segment Size (MSS) During 3-way handshake, the Receive Window (Window size value on Wireshark) tells each side of the connection the maximum receiving buffer in bytes each side can handle: So it's literally like this (read red lines first please): [1] → Hey, BIG-IP! My receiving buffer size is 29200 bytes. That means, you can initially send me up to 29200 bytes before you even bother waiting for an ACK from me to send further data. [2] → This should be the same as [1], unless Window Scale TCP Option is active. Window Scale should be the subject of a different article but I briefly touch it on [3]. [3] → Original TCP Window Size field is limited to 16 bits so maximum buffer size is just 65,535 bytes which is too little for today's speedy connections. This option extends the 16-bit window to 32-bit window but because BIG-IP did not advertise Window Scale option for this connection, it is disabled as both sides must support it for it to be used. [4] → Hey, client! My receiving buffer size is 4380 bytes. That means, you can initially send me up to 4328 bytes before you even bother waiting for an ACK from me to send further data. The reason why the word initially is underlined on [1] and [3] is because Window size typically changes during the connection. For example, client's initial window size is 29200 bytes, right? This means that if it receives 200 bytes from BIG-IP it should go down to 2900 bytes. Easy, eh? But that's not what always happens in real life. In fact, in our capture it's the opposite! Bytes in flight column shows the data BIG-IP (*.143) is sending in bytes to our client (*.135) that has not yet been acknowledged. I've added a column with Window Size value to make it easier to spot how variable this field is: It is the OS TCP Flow control implementation that dictates the Receive Window size taking into account the current "health" of its TCP stack and of course your configuration. Yes, in many cases, especially in the middle of a connection, the Window Size does decrease based on amount of data received/buffered so our first explanation also makes sense! How does BIG-IP know that client has freed up it's buffer again? As we can see above, when Client ACKs the receipt of BIG-IP's data, it also informs the size of its buffer in the Window Size value field. That's how BIG-IP knows how much data it can send to Client before it receives another ACK. What about the Maximum Segment Size? Each side also displays a TCP Option - Maximum Segment size of 1460 bytes. This informs the maximum size of the TCP payload each side can send at a time (per TCP segment). Looking at the picture above, BIG-IP sent 334 bytes of TCP payload to client, right? In theory, this could've been up to 1460 bytes as it's also within client's initial buffer of 29200 bytes. So apart from informing each other about the maximum buffer, the maximum size of TCP segment is also informed. TCP Len vs Bytes in Flight Column (BIF) If we look at our last picture, we can see that whatever is in Len field matches what's in our BIF column, right? Are they the same? No! Len shows the current size of TCP payload (excluding the size of TCP header). Remember that TCP payload in this case is the whole HTTP portion that our TCP segment is carrying. Bytes in flight is not really part of TCP header but that's something Wireshark adds to make it easier for us to troubleshoot. It just means the number of bytes sent that have not yet been acknowledged by receiver. In our capture, data is acknowledged immediately so both Len and BIF are the same. I've picked a different capture here where there are 3 TCP segments sent with no acknowledgement so BIF column increments for each unacknowledged data segment but goes back to zero as soon as an ACK is received by receiver: Notice that BIF values now differ from TCP payload (the equivalent to Len in Info column). That's it for now. The next article would be about TCP retransmission. Appendix - Going in depth into TCP sequence numbers! Here's a full explanation about what actually takes place on TCP layer from the point of view of BIG-IP: Just follow along from [1] to [10]. That's it.HTTP/2 Protocol in Plain English using Wireshark
1. Quick Intro Some people find it easier to do a "test drive" first to learn how a new protocol works in its simplest form and only then read the RFC. It turns out Wireshark is a perfect tool for me to do just that. It's a simple test and here's the topology: I'll just issue a HEAD request and later on a GET request and we'll see how it looks like on Wireshark. For more info about HTTP/2 profile and HTTP/2 protocol itself you can read the article I published on AskF5 and Jason's DevCentral article: What is HTTP Part X - HTTP/2. 2. Confirmation of which protocol will be used The packet capture taken below was the result of the following curl command issued from my ubuntu Linux client (I could've used a browser instead): Note: 10.199.3.44 is my virtual server with HTTP/2 profile applied. Here's packet capture (in case you want to follow along): http2-test-v1.zip HTTP/2 is negotiated during SSL handshake in Application Layer Protocol Negotiation (RFC 7301) SSL extension like this: Client says which protocol(s) it supports and server responds which one it picked (in this case it's HTTP/2!). 3. Negotiation of HTTP/2 Parameters Think of it as something that has to take place like Client Hello and Server Hello in SSL for example. Server side (BIG-IP in this case) sends SETTINGS frame which counts as confirmation that HTTP/2 is being used plus any flow control configuration we want our peer to honour: Client sends Magic frame to confirm HTTP/2 is being used and then SETTINGS with its requirements for the connection. Yes, Magic frame is always the same. Still curious about Magic frame? Read https://tools.ietf.org/html/rfc7540#section-3.5. End-points are also supposed to ACK the receipt of SETTINGS frame from the other peer and the way they do it is by responding with another empty SETTINGS frame with ACK flag set: 4. Exchanging data Connection-wise we're all set now. For HTTP/2 GET/HEAD requests there is a specific frame type called HEADERS which as the name implies carries HTTP/2 header information. If there was payload it would be carried inside DATA frame type but as this is just a HEAD request then no DATA frame follows. 5. Appendix A - Other common frame types 5.1 WINDOW_UPDATE There are other common frame types and in my capture the one that came up was WINDOW_UPDATE. If you look at section 3 above we see that Client advised BIG-IP that its Initial Window Size was 1073741824. WINDOW_UPDATE just adjusted this value to 1073676289: This is HTTP/2 flow control in action. 5.2 DATA in another test (http2-v2.zip) I used HTTP/2 GET request instead of HEAD and requested more data which comes in through DATA frame type: End Stream flag is false in all DATA messages except for the last one. It signals when there is more data as well as the last DATA frame. 5.3 GOAWAY In a subsequent test (http2-connection-idletimeout-1.zip) I set Connection Idle Timeout in HTTP/2 profile to 1 to force BIG-IP sending GOAWAY frame to close down connection after 1 second of idle connection. After last piece of data is sent by BIG-IP to client (frame #39), BIG-IP waits 1 second and sends GOAWAY frame which initiates the shutdown of HTTP/2 connection. GOAWAY messages always contains Promised-Stream-ID which tells the client what is the last Stream ID it processed. A new Stream ID is typically created for every new HTTP request (via HEADERS message). We can see that a new HTTP request slipped in on frame #46 but ignored as connection had already been closed on BIG-IP's side.Exploring Kubernetes API using Wireshark part 1: Creating, Listing and Deleting Pods
Related Articles: Exploring Kubernetes API using Wireshark part 2: Namespaces Exploring Kubernetes API using Wireshark part 3: Python Client API Quick Intro This article answers the following question: What happens when we create, list and delete pods under the hood? More specifically on the wire. I used these 3 commands: I'll show you on Wireshark the communication between kubectl client and master node (API) for each of the above commands. I used a proxy so we don't have to worry about TLS layer and focus on HTTP only. Creating NGINX pod pcap: creating_pod.pcap (use http filter on Wireshark) Here's our YAML file: Here's how we create this pod: Here's what we see on Wireshark: Behind the scenes, kubectl command sent an HTTP POST with our YAML file converted to JSON but notice the same thing was sent (kind, apiVersion, metadata, spec): You can even expand it if you want to but I didn't to keep it short. Then, Kubernetes master (API) responds with HTTP 201 Created to confirm our pod has been created: Notice that master node replies with similar data with the additional status column because after pod is created it's supposed to have a status too. Listing Pods pcap: listing_pods.pcap (use http filter on Wireshark) When we list pods, kubectl just sends a HTTP GET request instead of POST because we don't need to submit any data apart from headers: This is the full GET request: And here's the HTTP 200 OK with JSON file that contains all information about all pods from default's namespace: I just wanted to emphasise that when you list a pod the resource type that comes back is PodList and when we created our pod it was just Pod. Remember? The other thing I'd like to point out is that all of your pods' information should be listed under items. All kubectl does is to display some of the API's info in a humanly readable way. Deleting NGINX pod pcap: deleting_pod.pcap (use http filter on Wireshark) Behind the scenes, we're just sending an HTTP DELETE to Kubernetes master: Also notice that the pod's name is also included in the URI: /api/v1/namespaces/default/pods/nginx ← this is pods' name HTTP DELETE just like HTTP GET is pretty straightforward: Our master node replies with HTTP 200 OK as well as some json file with all the info about the pod, including about it's termination: It's also good to emphasise here that when our pod is deleted, master node returns JSON file with all information available about the pod. I highlighted some interesting info. For example, resource type is now just Pod (not PodList when we're just listing our pods).Understanding IPSec IKEv2 negotiation on Wireshark
Related Articles: Understanding IPSec IKEv1 negotiation on Wireshark 1 The Big Picture There are just 4 messages: Summary: IKE_SA_INIT: negotiate security parameters to protect the next 2 messages (IKE_AUTH) Also creates a seed key (known as SKEYSEED) where further keys are produced: SK_e (encryption): computed for each direction (one for outbound and one for inbound) to encrypt IKE_AUTH messages SK_a (authentication): computed for each direction (one for outbound and one for inbound) to hash (using HMAC) IKE_AUTH messages SK_d (derivation): handed to IPSec to generate encryption and optionally authentication keys for production traffic IKE_AUTH: negotiates security parameters to protect production traffic (CHILD_SA) More specifically, the IPSec protocol used (ESP or AH - typically ESP as AH doesn't support encryption), the Encryption algorithm (AES128? AES256?) and Authentication algorithm (HMAC_SHA256? HMAC_SHA384?). 2 IKE_SA_INIT First the Initiator sends a Security Association —> Proposal —> Transform, Transform... payloads which contains the required security settings to protect IKE_AUTH phase as well as to generate the seed key (SK_d) for production traffic (child SA): In this case here the Initiator only sent one option for Encryption, Integrity, Pseudo-Random Function (PRF) and Diffie Hellman group so there are only 4 corresponding transforms but there could be more. Responder picked the 4 available security options also confirmed in Security Association —> Proposal —> Transform, Transform… payloads as seen above. 3 IKE_AUTH These are immediately applied to next 2 IKE_AUTH messages as seen below: The above payload is Encrypted using SK_e and Integrity-protected using SK_a (these keys are different for each direction). The first IKE_AUTH message negotiates the security parameters for production traffic (child SAs), authenticates each side and informs what is the source/destination IP/Port that is supposed to go through IPSec tunnel: Now, last IKE_AUTH message sent by Responder confirms which security parameters it picked (Security Association message), repeats the same Traffic Selector messages (if correctly configured) and sends hash of message using pre-master key (Authentication message) Note that I highlighted 2 Notify messages. The INITIAL_CONTACT signals to Initiator that this is the only IKE_SA currently active between these peers and if there is any other IKE_SA it should be terminated in favour of this one. The SET_WINDOW_SIZE is a flow control mechanism introduced in IKEv2 that allows the other side to send as many outstanding requests as the other peer wants within the window size without receiving any message acknowledging the receipt. From now on, if additional CHILD_SAs are needed, a message called CREATE_CHILD_SA can be used to establish additional CHILD_SAs It can also be used to rekey IKE_SA where Notification payload is sent of type REKEY_SA followed by CREATE_CHILD_SA with new key information so new SA is established and old one is subsequently deleted.Packet Analysis with Scapy and tcpdump: Checking Compatibility with F5 SSL Orchestrator
In this guide I want to demonstrate how you can use Scapy (https://scapy.net/) and tcpdump for reproducing and troubleshooting layer 2 issues with F5 BIG-IP devices. Just in case you get into a finger-pointing situation... Starting situation This is a quite recent story from the trenches: My customer uses a Bypass Tap to forward or mirror data traffic to inline tools such as IDS/IPS, WAF or threat intelligence systems. This ByPass Tap offers a feature called Network Failsafe (also known as Fail-to-Wire). This is a fault tolerance feature that protects the flow of data in the event of a power outage and/or system failure. It allows traffic to be rerouted while the inline tools (IDS/IPS, WAF or threat intelligence systems) are shutting down, restarting, or unexpectedly losing power (see red line named Fallback in the picture below). Since the ByPass Tap itself does not have support for SSL decryption and re-encryption, an F5 BIG-IP SSL Orchestrator shall be introduced as an inline tool in a Layer 2 inbound topology. Tools directly connected to the Bypass Tap will be connected to the SSL Orchestrator for better visibility. To check the status of the inline tools, the Bypass Tap sends health checks through the inline tools. What is sent on one interface must be seen on the other interface and vice versa. So if all is OK (health check is green), traffic will be forwarded to the SSL Orchestrator, decrypted and sent to the IDS/IPS and the TAP, and then re-encrypted and sent back to the Bypass Tap. If the Bypass Tap detects that the SSL Orchestrator is in a failure state, it will just forward the traffic to the switch. This is the traffic flow of the health checks: Target topology This results in the following topology: Problem description During commissioning of the new topology, it turned out that the health check packets are not forwarded through the vWire configured on the BIG-IP. A packet analysis with Wireshark revealed that the manufacturer uses ARP-like packets with opcode 512 (HEX 02 00). This opcode is not defined in the RFC that describes ARP (https://datatracker.ietf.org/doc/html/rfc826), the RFC only describes the opcodes Request (1 or HEX 00 01) and Reply (2 or HEX 00 02). NOTE: Don't get confused that you see ARP packets on port 1.1 and 1.2. They are not passing through, the Bypass Tap is just send those packets from both sides of the vWire, as explained above. The source MAC on port 1.1 and 1.2 are different. Since the Bypass Tap is located right behind the customer's edge firewall, lengthy and time-consuming tests on the live system are not an option, since it would result in a massive service interruption. Therefore, a BIG-IP i5800 (the same model as the customer's) was set up as SSL Orchestrator and a vWire configuration was build in my employers lab. The vWire configuration can be found in this guide (https://clouddocs.f5.com/sslo-deployment-guide/chapter2/page2.7.html). INFO: For those not familiar with vWire: "Virtual wire … creates a layer 2 bridge across the defined interfaces. Any traffic that does not match a topology listener will pass across this bridge." Lab Topology The following topology was used for the lab: I build a vWire configuration on the SSL Orchestrator, as in the customer's environment. A Linux system with Scapy installed was connected to Interface 1.1. With Scapy TCP, UDP and ARP packets can be crafted or sent like a replay from a Wireshark capture. Interface 1.3 was connected to another Linux system that should receive the ARP packets. All tcpdumps were captured on the F5 and analyzed on the admin system (not plotted). Validating vWire Configuration To check the functionality of the F5 and the vWire configuration, two tests were performed. A replay of the Healthcheck packets from the Bypass Tap and a test with RFC-compliant ARP requests. Use Scapy to resend the faulty packets First, I used Wireshark to extract a single packet from packet analysis we took in the customer environment and saved it to a pcap file. I replayed this pcap file to the F5 with Scapy. The sendp() function will work at layer 2, it requires the parameters rdpcap (location of the pcap file for replay) and iface (which interface it shall use for sending). webserverdude@tux480:~$ sudo scapy -H WARNING: IPython not available. Using standard Python shell instead. AutoCompletion, History are disabled. Welcome to Scapy (2.5.0) >>> sendp(rdpcap("/home/webserverdude/cusomter-case/bad-example.pcap"),iface="enp0s31f6") . Sent 1 packets. This test confirmed the behavior that was observed in the customer's environment. The F5 BIG-IP does not forward this packet. Use PING and Scapy to send RFC-compliant ARP packets To create RFC-compliant ARP requests, I first sent an ARP request (opcode 1) through the vWire via PING command. As expected, this was sent through the vWire. To ensure that this also works with Scapy, I also resent this packet with Scapy. >>> sendp(rdpcap("/home/webserverdude/cusomter-case/good-example.pcap"),iface="enp0s31f6") . Sent 1 packets. In the Wireshark analysis it can be seen that this packet is incoming on port 1.1 and then forwarded to port 1.3 through the vWire. Solving the issue with the help of the vendor It became evident that the BIG-IP was dropping ARP packets that failed to meet RFC compliance, rendering the Bypass Tap from this particular vendor seemingly incompatible with the BIG-IP. Following my analysis, the vendor was able to develop and provide a new firmware release addressing this issue. To verify that the issue was resolved in this firmware release, my customer's setup, the exact same model of the Bypass Tap and a BIG-IP i5800, were deployed in my lab, where the new firmware underwent thorough testing. With this approach I could test the functionality and compatibility of the systems under controlled conditions. In this Wireshark analysis it can be seen that the Healthcheck packets are incoming on port 1.1 and then forwarded to port 1.3 through the vWire (marked in green) and also the other way round, coming in on port 1.3 and then forwarded to port 1.1 (marked in pink). Also now you can see that the packet is a proper gratuitous ARP reply (https://wiki.wireshark.org/Gratuitous_ARP). Because the Healthcheck packets were not longer dropped by the BIG-IP, but were forwarded through the vWire the Bypass Tap subsequently marked the BIG-IP as healthy and available. The new firmware resolved the issue. Consequently, my customer could confidently proceed with this project, free from the constraints imposed by the compatibility issue.Exploring Kubernetes API using Wireshark part 2: Namespaces
Related Articles: Exploring Kubernetes API using Wireshark part 1: Creating, Listing and Deleting Pods Exploring Kubernetes API using Wireshark part 3: Python Client API Quick Intro Using kubectl command is pretty useful: When you execute the above command, kubectl sends a GET request to /api/v1/namespaces/default/pods: Kubernetes master node replies with a JSON file containing all pods (along with their info) that belong to namespace 'default'. In this article, I'm going to explain what Kubernetes namespaces are by showing you real HTTP traffic reaching Kubernetes master node. I've removed the TLS complexity by using proxy so we can just focus on HTTP headers only. Understanding namespaces Initially, I'd say just memorise that /api/v1 is like the root directory of Kubernetes master node's API where client is going to retrieve all sorts of information. Have you noticed the namespaces in /api/v1/namespaces/default/pods? default just happens to be the namespace that our pods listed here belong to. Think of namespaces for Kubernetes as virtual Kubernetes clusters just like Virtual Machines (VMs) for OS. We can have identical objects with same name that belong to different namespaces and therefore are isolated from each other from the point of view of the API. Creating a new custom namespace I can create a new namespace like this using kubectl command: I can then create the same identical pods from default namespace in rodrigo's namespace. Let's see what happened under the hood when I typed the above command. When we create a new namespace, kubectl sends an HTTP POST request Kubernetes master node: pcap: creating-rodrigo-namespace.pcap The kubectl client then sends a JSON file like this in the POST request: Then, Kubernetes Master responds with HTTP 201 Created message and another JSON file with all newly created namespace's info: I've described some of the JSON info that came back from API just out of curiosity. Note that many different objects are 'namespaced', i.e. they belong to a namespace. Others like nodes are namespace-independent. I used pods as an example here to explain namespaces as pods are most popular and well-known object in Kubernetes world. Keeping 2 identical pods in 2 namespaces Let me create a new NGINX pod in the new namespace: Ops! We need to specify that we're creating the same pod in the new namespace we've just created, otherwise it defaults to default namespace where the nginx pod already exists: It now worked. Let's list only pods from rodrigo's namespace only with kubectl: When we capture the above request on Wireshark, we now see that our GET request to Kubernetes Master now uses rodrigo's namespace so we're now listing only pods from rodrigo namespace only : We also have this same exact pod using same name in default namespace. Remember? Deleting my custom namespace Now, let's delete our pod: And that's the API call under the hood (an HTTP DELETE request to complete path of namespace - just like we're deleting a folder): pcap: deleting-namespace.pcap Listing pods from all namespaces If you're curious about how the URL would look like when we list pods from all namespaces with kubectl: The answer is this: This request will list all pods from all namespaces. Troubleshooting Namespaces Remember I mentioned the finalizer attribute? When I was creating this article and I tried to delete the my custom namespace (rodrigo), it got stuck in Terminating state: Initially I thought it was just Google Cloud slowness but 40 minutes? That's a lot. So I suspected it could be because of finalize attribute and googled it so found that it was a bug and here's the solution: Retrieve namespace's JSON declaration to temporary file: Delete kubernetes keyword from finalizers attribute: Now send a PUT request to API and the JSON file above: Then, when I looked back it was finally gone:SSL Legacy Renegotiation vs Secure Renegotiation Explained using Wireshark
Related articles: SSL Forward Proxy Explained using Wireshark Quick Intro This is just a quick but in-depth look into SSL/TLS Renegotation and Secure Renegotiation. I'll just quickly show you how legacy and secure negotiation work in TLS/SSL. Renegotiation takes place in the same TCP connection. Do not confuse with Session Resumption/Reuse which takes place in subsequent TCP connections. Here's the topology I used to test this: 1. Legacy Renegotiation First there is a full SSL handshake: Notice that on Frame 6 (Server Hello) BIG-IP offers a Session ID: However, we do not use this session ID in renegotiation. Why? Because renegotiation means we want to renegotiate security parameters again and reusing session ID we would be reusing the same security parameters. Also, renegotiation takes place over the same TCP connection, so if client is the one that starts renegotiation we see a straight Client Hello still over the same connection above and full handshake takes place: If it's BIG-IP (server-side) who is willing to trigger renegotiation then we see a Hello Request message still over same TCP connection followed by full handshake: That's it. This is legacy Renegotiation. Note: Do not confuse Renegotiation with Session Reuse/Resumption. In Session Reuse a new TCP connection is open and Client sends a Session ID from a previous session so that same security parameters are reused. 2. Secure Renegotiation - The Add on! Secure renegotiation is exactly the same as above with the addition of SSL renegotiation_info extension described in RFC5746. Note: The only reason for this extension is to avoid man-in-the-middle attack where session is hijacked and attacker tries to renegotiate new session using client's handshake information. This extension saves some information from initial handshake that must be provided upon renegotiation which attacker wouldn't have. If we click on first Client Hello we see renegotiation_info extension along with other extensions in Client Hello message: Note: Instead of renegotiation_info extension there is also the option to add TLS_EMPTY_RENEGOTIATION_INFO_SCSV to Cipher Suites list and that means the same thing, i.e. we (or client/server) support Secure Renegotiation. First message is always blank just to indicate Client supports Secure Renegotiation. Server also signals its support in Server Hello: At the end of every SSL handshake there is a Finished message sent by both Client and BIG-IP: If we click on Finished message from Client, more specifically on Verify Data field (assuming it is decrypted) we will see a 12 bytes hash in hexadecimal: This client-side hash (d5 b7 01 35 b3 d2 d7 2a 54 0e 24 f0) is the result of hash of all handshake messages exchanged at this point mixed with previously negotiated master secret and a mathematical function to make it more secure (random). This allows BIG-IP to validate the integrity of the entire handshake. In BIG-IP's Finished message we can also see the same Verify Data message which its own hash which in turn will also allow client to validate the integrity of the entire handshake: But why is it important to know that? Because in the next handshake, renegotiation_info (found within Client Hello sent by Client) contains Verify Data portion it sent in previous Finished message (from previous handshake). Then BIG-IP sends its hash concatenated with client in Verify Data portion. Therefore, it is unlikely an attacker could have obtained these values because Finished message is always encrypted. Let's confirm values match on renegotiation_info on Client Hello sent afterwards in the same TCP connection (frame 1931): On BIG-IP's side it is the concatenation of Client's Verify Data and BIG-IP's Verify data (frame 1965): That's it.SSL Forward Proxy Explained using Wireshark
Related articles: SSL Legacy Renegotiation vs Secure Renegotiation Explained using Wireshark Summary The idea behind this feature is to allow BIG-IP to sniff into SSL connections to any Internet destination that goes through it whilst preserving client's trust of the destination. When this feature is enabled, BIG-IP uses a pre-installed CA set under proxy-ca-cert (CA Certificate in the GUI) that is also trusted by client's browser and BIG-IP pretends to be the Internet destination by re-signing SSL certificates on the fly with the aforementioned CA and using public key from CA configured under cert on BIG-IP. The client trusts the connection and is unaware of BIG-IP's presence. The most common use case is Secure Web Gateway (SWG), although neither APM nor SWG are needed to deploy SSL Forward Proxy functionality. LTM is enough. Lab Scenario Lab test results: Client opens connection with BIG-IP and sends Client Hello. BIG-IP immediately opens connection with web server and completes SSL handshake. BIG-IP creates unique certificate with following information and sends it over via Server Hello: Serial number: random Issuer: copied from default.crt Validity: notBefore (copied from server1.crt), notAfter: 30 days counting from time it was generated (configurable via cert-lifespan or Certificate Lifespan in the GUI) Subject: copied from server1.crt Public Key: copied from ltm2.CA.crt All X.509 extensions allowed in cert-extensions-include are also copied from server1.crt to unique custom certificate Client-side handshake completes successfully using above dynamically generated custom certificate just created by BIG-IP and data exchange works as expected. How SSL Forward Proxy works on Wireshark Actual capture used for below explanation is attached to this article below (and here ssl-forward-sample-3.pcap ) Upon Client Hello sent by Client BIG-IP rushes to open new connection and completes server-side SSL handshake: Server-side handshake completes on frames 17-18 and and BIG-IP immediately resumes and completes Client-side handshake. Note: The reason why BIG-IP waits for server-side handshake to complete first is because BIG-IP uses server-side certificate information to generate a custom certificate that will be sent to Client on client-side. The most interesting bit here is BIG-IP's Certificate message sent on frame 20 to Client as it contains the custom Certificate generated by BIG-IP: We can see there are 2 Certificates: CA Certificate (proxy-ca-cert) is the Certificate Authority (CA) that signs the newly created custom certificate and is supposed to be trusted by Client to confirm BIG-IP's custom (dynamically generated) certificate can be trusted too. We can immediately work out that the 2nd certificate is default.crt just by matching its serial number which is unique for each issuer: However, the first certificate looked a mystery to me until I found where all the pieces came from. The serial number looks randomly generated to me as it doesn't match (not even close to any of the certificates above): The Issuer field, which shows information about who signed the certificate, shows that default.crt was the one who signed BIG-IP's custom certificate: Certificate generation date (notBefore) is copied from server1.crt but expiry date (notAfter) is calculated based on cert-lifespan (Certificate Lifespan in the GUI) which by default is 30 days added to time custom certificate was generated: Subject fields, which shows information about the current certificate, is copied and identical to information on server1.crt: And finally, the public key is copied from cert field (ltm2CA.crt). Notice they are the same: That's enough information for now.Understanding HTTP/2 Profile's Frame Size option on BIG-IP
Quick Intro The Overview of the BIG-IP HTTP/2 profile article on AskF5 I created a while ago describes all the HTTP/2 profile options but sometimes we need to test things out ourselves to grasp things at a deeper level. In this article, I'm going to show how Frame Size option sets specifically only the maximum size of HTTP/2 DATA message's payload in bytes and what happens when we change this value on Wireshark. Think of it as a quick walkthrough to give us a deeper understanding of how HTTP/2 works as we go through. The Topology It's literally a client on 10.199.3.135 and a virtual server with HTTP + HTTP/2 profile applied with the default settings: Testing Frame Size Option Here I've tried to modify frame-size to an invalid value so we can see the valid range: Let's set the frame-size to 1024 bytes: I have curl installed in my client machine and this is the command I used: If we just filter for http2 on Wireshark, we should see the negotiation phase (SETTINGS) as well as request (GET) and response (200 OK) headers in their specific message type (HEADERS). However, our focus here is on DATA message type as seen below: I've now added a new column (Length) to include the length of DATA messages so we can easily see how Frame Size settings affect DATA length. Here's how we create such filter: I've further renamed it to HTTP2 DATA Length but you've got the point. If we list only DATA messages, we can see that the payload of HTTP/2 DATA message type will not go beyond 1024 bytes: Wireshark confirms that HTTP/2 headers + DATA payload of frame 26 is 1033 bytes but DATA payload-only is 1024 bytes as seen below: We can then confirm that only payload counts for frame-size configuration on BIG-IP. I hope you enjoyed the above hands-on walk-through.
