Tools and facilities to troubleshoot HTTP/3 over QUIC with the BIG-IP system

Introduction

This article is for engineers who are troubleshooting issues related to HTTP/3 over QUIC as you deploy this new technology on your BIG-IP system. As you perform your troubleshooting tasks, the BIG-IP system provides you a set of useful tools along with other third party software to identify the root cause of issues and even tune HTTP/3 performance to maximize your system's potential.

Overview of HTTP/3 and QUIC

HTTP/3 is the next version of the HTTP protocol after HTTP/2. The most significant change in HTTP/3 from its predecessors is that it uses the UDP protocol instead of TCP. HTTP/3 uses a new Internet transport protocol, QUIC uses streams at the transport layer and provides TCP-like congestion control and loss recovery. One major improvement QUIC provides is it combines the typical 3-way TCP handshake with TLS 1.3's handshake. This improves the time required to establish a connection. Hence, you may see QUIC as providing the functions previously provided by TCP, TLS, and HTTP/2 as shown in the following diagram:

For an overview of HTTP/3 over QUIC with the BIG-IP system, refer to K60235402: Overview of the BIG-IP HTTP/3 and QUIC profiles.

Available tools and facilties

Beginning in BIG-IP 15.1.0.1, HTTP/3 over QUIC (client-side only) is available as an experimental feature on the BIG-IP system.

Beginning in BIG-IP 16.1.0, BIG-IP supports QUIC and HTTP/3.

In addition to that feature, there are tools and facilities that are available to help you troubleshoot issues you might encounter.

1. Install an HTTP/3 command line client
2. Use a browser that supports QUIC
3. Review statistics on your BIG-IP system
4. Enable QUIC debug logging on the BIG-IP system
5. Perform advanced troubleshooting with Packet Tracing
6. Use the NetLog feature from the Chromium Project to capture a NetLog dump.
7. Use the tcpdump command and Wireshark to capture and analyze traffic.
8. Use the qlog trace system database key on the BIG-IP system.

Important: For BIG-IP versions prior to 16.1.0 that are in the experimental stages, it is important that you note in your troubleshooting, the version of the ietf draft that your client and server implements. For example, in the Hello packets between the client and server, version negotiation is performed to ensure that client and server agree to a QUIC version that is mutually supported. In BIG-IP 15.1.0.1, the HTTP/3 and QUIC profiles in the BIG-IP system are experimental implementations of draft-ietf-quic-http-24 and draft-ietf-quic-transport-24 respectively. You need to consider this when configuring the Alt-Svc header in HTTP/3 discovery. For some browsers such as those from the Chromium project, Chrome canary, Microsoft Edge canary and Opera, when starting these browsers from the command line, you need to provide the QUIC version it implements. For example, for Chrome canary, you run the following command: chrome.exe --enable-quic --quic-version=h3-25. Only implementations of the final, published RFC can identify themselves as h3. Until such an RFC exists, implementations must not identify themselves using the h3 string.

1. Install an HTTP/3 command line client

Keep in mind that HTTP/3 over QUIC runs on UDP instead of TCP. By default, browsers always initiate a connection to the server using the traditional TCP handshake which will not work with a QUIC server listening for UDP packets. You therefore need to configure HTTP/3 discovery on your BIG-IP system. This can be done by using the HTTP Alternative Services concept which can be implemented either by inserting the Alt-Svc header or via DNS as a HTTPSVC DNS resource record. To insert the Alt-Svc header, refer to K16240003: Configuring HTTP/3 discovery for BIG-IP virtual server. As you troubleshoot your HTTP/3 discovery implementation, you can use a command line tool that does not come with the overhead of HTTP/3 discovery. Following are two popular tools that you can install on your client system:

• The picoquic client
• The curl client where you have the option to use either the ntcp2 or quiche software libraries.

2. Use a browser that supports QUIC

At this time, browsers by default, still do not support QUIC and do not send the server UDP packets to establish a QUIC connection. The following browsers which are in development support it:

• Firefox Nightly
• Chrome Canary (Chromium Project)
• Microsoft Edge Canary (Chromium Project)
• Opera (Chromium Project)

Note that for browsers from the Chromium project, you need to specify the QUIC ietf version that the browser supports when you launch it. For example, for Chrome, run the following command: chrome.exe --enable-quic --quic-version=h3-25.

In most browsers today, you can quickly view the HTTP information exchanged by using the built-in developer tool. To open the tool, select F12 after your browser opens and access any site that supports QUIC. Select the Network tab and under the Protocol column, look at h3-<draft_version> . If the Protocol column is not there, you may have to right click the toolbar to add it.

Note: Only implementations of the final, published RFC can identify themselves as h3. Until such an RFC exists, implementations must not identify themselves using the h3 string.

Click the name of the HTTP request and you can see that the site returns the Alt-Svc header indicating that it supports HTTP/3 with its ietf draft version.

3. Review statistics on your BIG-IP system

The statistics facility on the BIG-IP system displays the system's QUIC traffic processing. On the Configuration utility, go to Local Traffic > Virtual Servers. Select the name of your virtual server and select the Statistics tab. In the Profiles section, select the HTTP/3 and QUIC profiles associated with the virtual server.

Alternatively, you can view the statistics from the TMOS shell (tmsh) utility using the following command syntax:

• tmsh show ltm profile http3 <http3_profile_name>
• tmsh show ltm profile quic <quic_profile_name>

4. Enable QUIC debug logging on the BIG-IP system

You can use the sys db variable tmm.quic.log.level to adjust the verbosity of the QUIC log level to the /var/log/ltm file. Type the following command to see the list of values.

tmsh list sys db tmm.quic.log.level value-range

sys db tmm.quic.log.level {

   default-value "Critical"

   value-range "Critical Error Warning Notice Info Debug"

}

For example: tmsh modify sys db tmm.quic.log.level value debug

5. Perform advanced troubleshooting with Packet Tracing

a. Use the NetLog feature from the Chromium Project to capture a NetLog dump.

NetLog is an event logging mechanism for Chrome’s network stack to help debug problems and analyze performance not just for HTTP/3 over QUIC traffic but also HTTP/1.1 and HTTP/2. This feature is available only in browsers from the Chromium project, such as Google Chrome, Opera and Microsoft Edge. The feature provides detailed client side logging including SSL handshake and HTTP content without having to perform decryption or run any commands on your BIG-IP system.

To start capturing, open a new tab on your browser and go to, for example, chrome://net-export (Chromium only). For a step by step guide, refer to How to capture a NetLog dump. Once you have your NetLog dump, you can view and analyze it by navigating to netlog-viewer (Chromium only). To analyze QUIC traffic, on the left panel, select Events. In the Description column, identify the URL you requested. For QUIC SSL handshake events, select QUIC_SESSION. For HTTP content, select URL_REQUEST.  For example, in the following NetLog dump, the connection failed at the beginning because the client and server could not negotiate a common QUIC version.

b. Use the tcpdump command and Wireshark to capture and analyze traffic.

The tcpdump command and Wireshark are both essential tools when you need to examine any communication at the packet level. To generate captures and the SSL secrets required to decrypt them, follow the procedure in K05822509: Decrypting HTTP/3 over QUIC with Wireshark. Keep your Wireshark version updated at all times as Wireshark's ability to decode QUIC packets continue to evolve as we speak.

c. Use the qlog traces on the BIG-IP system.

The BIG-IP qlog trace facility provides you another tool to troubleshoot QUIC communications. By enabling a database variable, the system logs packets and other events to /var/log/trace<TMM_number>.qlog files. qlog is a standardized structured logging format for QUIC and is basically a well-defined JSON file with specified event names and associated data that allows you to use tools like qvis for visualization. Note that the payload is not logged. The qlog trace files are compliant to the IETF schema specified in draft-marx-qlog-main-schema-01.

To capture and analyze qlog trace files on your BIG-IP system, perform the following procedure:

• Capturing qlog trace files on the BIG-IP system

1. Login to the BIG-IP command line.
2. Enable qlogging by typing the following command:
3. tmsh modify sys db quic.qlogging value enable
4. Reproduce the issue you are troubleshooting by initiating QUIC traffic to your virtual server.
5. Disable qlogging by typing the following command:
6. tmsh modify sys db quic.qlogging value disable

Note: This step will log required closing json content to the trace files, terminate the trace logging gracefully and is required before you view the files.

• Sanitizing the qlog trace files

Before loading the trace files onto a graphical visualization tool, you first need to sanitize the json content. The tools attempt to fix some common json errors but there may be cases where you need to manually correct some json syntax errors by adding closing braces or commas.

Note: Knowledge of different json constructs such as objects, arrays and members may be required when you fix the json files.

You can use any of the available online tools such as the following:

• Json Formatter
• Fixjson
• freeformatter

Important: Even as the payload information such as IP addresses, or HTTP content are not included in the trace files, you should exercise caution when uploading content to online tools. F5 is not responsible for the privacy and security of your data when you use the third party software listed in this procedure.

Alternatively, you can download and install any of the following command line tools on your client device:

• jsonlint-php
• jsonlint-py
• jsonlint-cli
• Loading and analyzing the qlog trace files with a visualization tool

When you have sanitized your json trace files, upload them to a visualization tool for analysis. For example, you can use the following tool available for free.

• qvis QUIC and HTTP/3 visualization toolsuite

The visualization tool can provide you graphical representations of the sequence of messages, congestion information and qlog stats for troubleshooting. For example, the following screenshot, illustrates a sequence diagram of the SSL handshake.

Important: Even as payload information such as IP addresses, or HTTP content are not included in the trace files, you should exercise caution when uploading content to online tools. F5 is not responsible for the privacy and security of your data when you use the third party software listed in this procedure.

Summary

As you use the tools described in this article, you notice that each one helps you troubleshoot issues at the different OSI layers. The built-in developer tools in each browser provide a quick and easy way to view HTTP content but do not let you see the details of the protocol, as do the NetLog tool and Wireshark. However, viewing qlog traces on the qvis graphical tool provides you high level trends and statistics that packet captures do not show. Using the appropriate tool with the right troubleshooting methodology maximizes the potential of HTTP/3 and QUIC for your organization.