Decrypting BIG-IP Packet Captures without iRules

[JRahm]

In this final article focused on taking and decrypting BIG-IP packet captures, I take the advice of MVPers @Nikoolayy1 and @Juergen_Mang by losing the iRules and instead utilizing the system database key that allows you to embed the session keys in the tcpdump capture as it's capturing. This by default includes the TLSv1.3 sessions missing from my iRule solution in the last article (though it is possible still with a more robust key capture in the iRules to include them there as well.) All this is covered in the live stream sessions, where I walk through converting the iRule solution to the database key. Read on below for the final solution.

Live Session - Nearing the Final Solution

Live Session - Completion

Solution Details

The script in its entirety is here in the github repo, but I'll cover the imporant updates here. First, the tcpdump string that we modify in the script needs an update to incorporate the f5 flag:

TCPDUMP_BASH_STRING = """timeout -s SIGKILL CAP_SECS tcpdump  -s0 -nni 0.0:nnnp --f5 ssl:v VIRTUAL_IP -w /shared/images/autocap_DATESTRING.pcap"""

 I'm still saving to /shared/images to be able to use the rest worker node for downloads from that directory. Next, because the database key exports session keys, you only want that key active for troubleshooting. Do make sure that's handled automatically, I created a function to call right before and after the capture:

def toggle_sslprovider(bigip, state):
    data = {'value': state}
    bigip.modify('/mgmt/tm/sys/db/tcpdump.sslprovider', data)
    print(f'\tDatabase key tcpdump.sslprovider has been {state}d...continuing.')

Running the tcpdump itself only required a couple cosmetic naming changes so I won't include that function here. In the previous solution, the session keys were in log files and had to be culled and stored in a file. That function, create_keyfile, is not needed here so it's been removed. Instead, we need to extract the keys from the tcpdump file and then store them in their own file. I created the function extract_keys to do this.

def extract_keys(tcpdump_file):
    tshark_process = subprocess.run(["tshark",
                                     "-r", f"{tcpdump_file}",
                                     "-Y", "f5ethtrailer.tls.keylog",
                                     "-Tfields",
                                     "-e", "f5ethtrailer.tls.keylog"],
                                    stdout=subprocess.PIPE,
                                    stderr=subprocess.STDOUT)
    keyfile = open('session_keys.pms', 'w')
    subprocess.run(["sed",
                    "s/,/\\n/g"],
                   input=tshark_process.stdout,
                   stdout=keyfile,
                   stderr=subprocess.STDOUT)
    keyfile.close()
    print('\tExtracted keys file: session_keys.pms...continuing.')

 It was this function that I got hung up on in the first live session. Piping and redirecting stdout to a file was not something that python's subprocess was in love with in the way that I was attempting. I solved that in the second live session linked above. The decryption function required only a cosmetic change. The final work was just to make sure the workflow of calling the functions was correct:

    vip_name, duration, filters = user_responses()

    br = instantiate_bigip(int(duration))

    toggle_sslprovider(br, 'enable')
    tcpdump_file = run_tcpdump(br, duration, vip_name, filters)
    toggle_sslprovider(br, 'disable')

    download_files(br, tcpdump_file)
    delete_files(br, tcpdump_file)

    extract_keys(tcpdump_file)
    decrypt_capture(tcpdump_file)

And with that, it worked a treat! Using the database key also significantly reduced the complexity of the solution, which I'm always a fan of.

Conclusion

The solution is final, but I'll refactor the code a little to eliminate the need to manage in separate scripts whether it's a case or not, whether you want a qkview or not, etc. I'll converge all of into a single scripts with command line options to enable/disable different features.

And regarding finality, this is only the beginning of what we can automate now that we have a decrypted capture. What might we be able to pull out of the data to make your job easier before you even launch Wireshark? What might we be able to do with that data to format your Wireshark profile upon launch? Hmm...stay tuned for details!

[Tofunmi]

This looks very promising! I am definitely trying this out in my lab.

[Austin_Geraci]

Nice work Jason - this is good stuff 👍

[lnxgeek]

Dammit @JRahm now you have ruined my entire weekend - I just had to get a Bash version working 🤣

I don't know how you do it but these ideas are awesome and I just love the challenge trying to keep up with your Python (some would might call it an addiction 😋).

The overall concept wasn't the difficult part, getting curl to consume the json payload was! I wasn't able to create a one-liner with the timeout + tcpdump parameters. Either curl complained about the "--f5" option inside the json body or the BigIP complained about it not being a proper formated json, so I gave up and put the body in a file instead. If you have an idea on how to make the right sequence of escaping, please let me know.

During my scritpting sessions I added the option to fetch a VS in another partition and some error catching around different scenarios. You could probably put more catching in there but I'm out of weekend 😆

I also played with the idea of extracting the VS' as a list and then make it a selection but if you have thousands of services this wouldn't scale well. So, I settled for some checking if the return value indead was an IP address.

#!/bin/bash

# Utility functions. It takes two parameters, first is the text and the second is the default value.
# If no default value is supplied it will loop until something is entered.
function prompt_user {
    read -p "${1} (default: [${2}]): " input
    echo "${input:-${2}}"
}

function prompt_bigip_address {
    read -p "BIG-IP address [default: https://10.1.1.10]: " BIGIP
    # Assign a default value if nothing is entered
    BIGIP="${BIGIP:-https://10.1.1.10}"
    # Make sure that the address is prefixed with https if only an IP address is entered
    BIGIP="https://${BIGIP#https://}"
}

function prompt_credentials {
    read -p "BIG-IP username [default: admin]: " USER
    USER="${USER:-admin}"
    read -s -p "BIG-IP password: " PASS
    echo ""
}

function get_token {
     # Authenticate and get auth token
    TOKEN=$(curl -skf -H "Content-Type: application/json" -d '{"username":"'$USER'","password":"'$PASS'","loginProviderName":"tmos"}' "$BIGIP/mgmt/shared/authn/login" | grep -oP '(?<="token":")[^"]+')

}

function delete_token {
    # Delete token
    curl -sk -H "X-F5-Auth-Token: $TOKEN" -X DELETE "$BIGIP/mgmt/shared/authz/tokens/$TOKEN"| jq -r  '.|{"Deleted token": .token}'    
}

function toggle_ssl_provider {
    # Toggle the "sys db tcpdump.sslprovider" variable
    local value=${1}
    curl -sk -X PATCH -H "Content-Type: application/json" -H "X-F5-Auth-Token: $TOKEN" "$BIGIP/mgmt/tm/sys/db/tcpdump.sslprovider" -d '{"value": "'$value'"}'
}

function download_file {
    # Download the pcap file from the filesystem "/share/images/encrypt_autocap_*.pcap"
    local file="${1}" message="${2}"
    curl -skf -H "X-F5-Auth-Token: $TOKEN" "$BIGIP/mgmt/cm/autodeploy/software-image-downloads/$file" -o "$file"
    if [ "$?" -eq 0 ]; then
      echo -e "\t$message"
    else
      echo -e "\tDOWNLOAD FAILED!!!"
    fi
}

function delete_file {
    # Delete the pcap file from the filesystem "/share/images/encrypt_autocap_*.pcap"
    local file="${1}" message="${2}"
    curl -skf -X POST -H "X-F5-Auth-Token: $TOKEN" -H "Content-Type: application/json" "$BIGIP/mgmt/tm/util/unix-rm" -d '{"command": "run", "utilCmdArgs": "/shared/images/'$file'"}'
    if [ "$?" -eq 0 ]; then
      echo -e "\t$message"
    else
      echo -e "\tDELETE FAILED!!!"
    fi
}

# Main function
function main {
    # Get user inputs
    partition=$(prompt_user "Which partition: " "Common")
    
    vip_name=""
    while [[ -z $vip_name ]]; do
        vip_name=$(prompt_user "Virtual name" "")
    done
    
    duration=$(prompt_user "Duration in seconds for capture: " "30")
    filters=$(prompt_user "Capture filters in addition to vip [ex. \"and (port 80 or port 443)\"]: ")
       
    
    # Prep variables for execution
    datestring=$(date +%Y%m%d-%H%M%S)
    tcpdump_file="autocap_$datestring.pcap"
    tcpdump_file_encrypt="encrypt_$tcpdump_file"
    tcpdump_file_decrypt="decrypt_$tcpdump_file"
    pms_file="pms_$datestring.key"
    payload="payload.json"
    
    # Regular expression to match an IP address
    IP_REGEX='^((25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})\.){3}(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})$'

    # Get the IP of the virtual server
    virtual_ip=$(curl -skf -H "X-F5-Auth-Token: $TOKEN" "$BIGIP/mgmt/tm/ltm/virtual/~$partition~$vip_name" | jq -r '.destination | split(":") | .[0]'|awk -F/ '{print $3}')
    # Check if virtual IP is a valid IP address
    if [[ ! $virtual_ip =~ $IP_REGEX ]]; then
      echo "Error: virtual IP '$virtual_ip' is not a valid IP address - Did you spell the VS correctly?"
      exit 1
    fi
    
    # Build command execution string and dump it into a payload file as a json array to be read by curl
    tcpdump_bash_string=(timeout -s SIGKILL $duration tcpdump -s0 -nni 0.0:nnnp --f5 ssl:v host $virtual_ip $filters -w /shared/images/$tcpdump_file_encrypt)
    echo "{\"command\":\"run\",\"utilCmdArgs\":\"-c '${tcpdump_bash_string[@]}'\"}" > $payload

    # Run tcpdump
    echo -e "\tStarting tcpdump...please reproduce your issue now."
    output=$(curl -sk -X POST -H "Content-Type: application/json" -H "X-F5-Auth-Token: $TOKEN" "$BIGIP/mgmt/tm/util/bash" -d @$payload)

    #curl -skf -X POST -H "Content-Type: application/json" -H "X-F5-Auth-Token: $TOKEN" "$BIGIP/mgmt/tm/util/bash" -d '{"command":"run","utilCmdArgs":"-c "'${tcpdump_bash_string[@]}'""}'
    sleep 5
    echo -e "\ttcpdump complete...continuing."
    

    # Download tcpdump capture from BIG-IP
    echo -e "\nDownloading tcpdump capture from BIG-IP..."
    download_file $tcpdump_file_encrypt "\tDownload complete!"
    
    # Delete tcpdump capture on BIG-IP
    echo -e "\nDeleting tcpdump capture on BIG-IP..."
    delete_file $tcpdump_file_encrypt "\tDeletion complete!"
    
    # Extract session keys from tcpdump capture
    echo -e "\nExtracting session keys from tcpdump capture..."
    tshark -r $tcpdump_file_encrypt -Y f5ethtrailer.tls.keylog -Tfields -e f5ethtrailer.tls.keylog > $pms_file
    echo -e "\tSession keys extracted!"
    
    # Create a decrypted tcpdump capture from the encrypted capture + session keys file
    echo -e "\nCreating decrypted tcpdump capture..."
    editcap --inject-secrets tls,$pms_file $tcpdump_file_encrypt $tcpdump_file_decrypt
    echo -e "\tDecrypted tcpdump capture created!"

    echo -e "\nCleanup..."
    rm -f $payload $pms_file
    echo -e "\nComplete!"
}

prompt_bigip_address
prompt_credentials
get_token
main
delete_token

 I'm open to feedback and ideas. This concept of remote "tcpdumping" and decrypting is extremely useful and I can see a lot of applications for it.

[Henning_juul_Hi]

Great job !

[JRahm]

@lnxgeek that's awesome! Always love to see alternate versions so community members have choices in implementation.