A Closer Look at mTLS and the Default Server in F5 NGINX

When you connect to an HTTPS site, your browser (or other client) typically sends a Server Name Indication (SNI)—the hostname it wants to reach—during the TLS handshake. This lets F5 NGINX figure out which cryptographic settings to use for that specific hostname. In other words, SNI influences:

• Which x509 certificate is sent to the client
• Which cryptographic algorithms are offered
• Which session ticket key is used for encrypting and decrypting session tickets
• Which session cache is active, if you’re using caching
• Which Certificate Authority (CA) is checked when you require mutual TLS (mTLS)

If the client doesn’t provide SNI or if the hostname doesn’t match any of your configured server_name directives, F5 NGINX defaults to a “fallback” setup—usually called the default server. That means:

• The default server’s certificate, ciphers, ticket key, and session cache get used automatically.
• If you haven’t explicitly marked any server block as default_server, F5 NGINX chooses the first block for that listen socket in your configuration as the fallback.

Here’s the crucial detail: once the TLS handshake finishes and a certificate has been selected (for example, from the default server), F5 NGINX will still examine the HTTP Host header for request routing. If it specifies a different domain matching another server block, the request is forwarded there at the HTTP layer. From the client’s perspective, however, the originally cryptographic settings remain in effect, because the TLS negotiation is already complete.

 

Single Server Block

In the simplest configuration, only one server block listens for TLS connections on a given IP address and port:

server {
    listen 443 ssl;
    server_name example.com;
    ssl_certificate      /etc/ssl/certs/example.com.crt;
    ssl_certificate_key  /etc/ssl/private/example.com.key;
    ssl_client_certificate /etc/ssl/certs/ca.crt;
    ssl_verify_client on;

    # Additional configuration, such as locations and logging
}

 

In this example:

• If the client provides SNI matching example.com, F5 NGINX presents /etc/ssl/certs/example.com.crt and verifies the client certificate against /etc/ssl/certs/ca.crt.
• If the client does not provide SNI, this same server block still handles the request because there are no other blocks to consider; the same certificate and CA (ca.crt) apply.
• Once authenticated, the client proceeds under the cryptographic settings of this single server block. With only one server block present, there is no additional routing or fallback scenario to manage.

 

Multiple Server Blocks on the Same IP/Port

When multiple server blocks listen on the same IP address and port, F5 NGINX uses SNI to determine which server block should handle the request. If no matching SNI is found, requests fall back to the server marked with default_server. As previously stated, if the default_server is not explicitly defined, F5 NGINX will use the first server block in the configuration as the fallback.

# example.com and the default server (first in config)
server {
    listen 443 ssl;
    server_name example.com;
    ssl_certificate      /etc/ssl/certs/example.com.crt;
    ssl_certificate_key  /etc/ssl/private/example.com.key;
    ssl_client_certificate /etc/ssl/certs/ca_A.crt;
    ssl_verify_client      on;
    # Additional configuration, such as locations and logging

}

# www.example.com
server {
    listen 443 ssl;
    server_name www.example.com;
    ssl_certificate      /etc/ssl/certs/www.example.com.crt;
    ssl_certificate_key  /etc/ssl/private/www.example.com.key;
    ssl_client_certificate /etc/ssl/certs/ca_B.crt;
    ssl_verify_client      on;

    # Additional configuration, such as locations and logging
}

 

In this example:

• If the client provides SNI matching example.com, the first server block’s certificate (example.com.crt) and CA settings (ca_A.crt) are used.
• If the client provides SNI matching www.example.com, the second server block’s certificate (www.example.com.crt) and CA settings (ca_B.crt) are used.
• If the client does not provide SNI (or provides an unmatched server name), the first server block (example.com) acts as the default. Its certificate (example.com.crt) and CA (ca_A.crt) apply for the TLS handshake.
• After TLS is established under the default server, if the HTTP Host header is www.example.com, F5 NGINX routes the request to the second server block for application-level processing. However, the TLS session—including which certificate and CA were used—remains with the default server’s settings. This means the second server’s client certificate configuration (ca_B.crt) is not involved in re-validating the client, since no new TLS handshake occurs.

 

Recommendations

The fallback behavior mentioned above might not fit all use cases. If it poses a risk or doesn’t align with your security needs, consider reconfiguring F5 NGINX (e.g., setting up a stub default server or applying tighter mTLS rules) to restrict or eliminate this fallback path.

Defining a Default Server (or Stub Default Server)
It is highly recommended to define a default server in F5 NGINX. If you do not want to allow fallback for clients without valid SNI, you can set up a stub default server (configuration example below).

A stub default server, as shown below, ensures that unmatched SNI (or no SNI) connections are rejected at the handshake level, preventing unintended fallback to a less restrictive configuration. Perhaps most importantly, it does not contain any client authentication configuration directives, forcing client authentication to occur in the most specific server blocks. In the example below I have added `ssl_verify_client off;` for illustrative purposes, however the setting of `off` is the default. Note: ssl_reject_handshake appeared in nginx version 1.19.4. For versions prior to that, one can define a server that simply returns the special 444 HTTP response code.

Authorization Checks in All Server Blocks
Even with a stub default server, all server blocks should implement authorization checks if they serve sensitive content. Because requests may be forwarded from the default server to a non-default server after decryption, every server block must enforce its own mTLS policies and check variables such as $ssl_client_s_dn or $ssl_client_i_dn (if you rely on client certificates) to ensure consistent and robust security across your deployment.

# explicit default server
server {
    listen 443 ssl default_server;
    ssl_reject_handshake on;
    ssl_verify_client off;
    ssl_protocols TLSv1.2 TLSv1.3;
}

# example.com
server {
    listen 443 ssl;
    server_name example.com;
    ssl_certificate      /etc/ssl/certs/example.com.crt;
    ssl_certificate_key  /etc/ssl/private/example.com.key;
    ssl_client_certificate /etc/ssl/certs/ca_A.crt;
    ssl_verify_client      on;

    # Check subject DN
    if ($ssl_client_s_dn !~ "CN=TrustedClient_A,O=MyOrg") {
        return 403;
    }
    # Check issuer DN (this may not be necessary for all deployments)
    if ($ssl_client_i_dn !~ "CN=TrustedCA_A,O=MyOrg") {
        return 403;
    }
    # Additional configuration, such as locations and logging
}

# This server block handles requests for www.example.com
server {
    listen 443 ssl;
    server_name www.example.com;
    ssl_certificate      /etc/ssl/certs/www.example.com.crt;
    ssl_certificate_key  /etc/ssl/private/www.example.com.key;
    ssl_client_certificate /etc/ssl/certs/ca_B.crt;
    ssl_verify_client      on;
    
    # Check subject DN
    if ($ssl_client_s_dn !~ "CN=TrustedClient_B,O=MyOrg") {
        return 403;
    }
    # Check issuer DN (this may not be necessary for all deployments)
    if ($ssl_client_i_dn !~ "CN=TrustedCA_B,O=MyOrg") {
        return 403;
    }
    # Additional configuration, such as locations and logging
}

Conclusion

In summary, working with multiple server blocks, SNI, and mTLS can make F5 NGINX setups more complex. Knowing precisely which server block handles the TLS handshake—particularly when there is no SNI or an unmatched name—helps maintain the desired security posture. Careful attention to these details keeps certificates and policies consistent for all client connections.