Request Resubmit on timeout

Problem this snippet solves:

In some cases, a BIG-IP can be sending a request to a pool member that is "stuck" at the application layer; effectively the server is inoperative. While it would be ideal to detect inoperative pool members with health monitoring, occasionally the failures are intermittent.

This iRule allows a user-defined timeout value after which it will "give up" on the originally selected server and then resubmit the request (including payload) to a new server. In order to do this, it must disable the HTTP profile on the connection and so when it resubmits the request, it inserts a Connection: close header so that the resubmitted request will be the last HTTP request for that connection.

It also detects a timeout on the retried request and if a timeout is detected, it will send a 504 Gateway Timeout error.

The current version of the iRule is built for HTTPS on both sides (client-ssl and server-ssl profile on the virtual server) but could be adapted for non SSL on either or both sides relatively easily.

How to use this snippet:

Enable iRule on Virtual Server (that has HTTP profile, client-ssl and server-ssl profiles enabled) and set the timeout value (default is 5 seconds) for HTTP request.

Code :

when RULE_INIT {
    # timeout in milliseconds (1000ms = 1 second)
    set static::response_timeout 5000
    set static::retry_debug 1
}
 
when HTTP_REQUEST {
    set retrycount 0
    if {$static::retry_debug}{
        log "Request Received for [HTTP::uri]"
    }
    set initialrequest_id [\
        after $static::response_timeout {
            if {$static::retry_debug}{
                log "Timeout $static::response_timeout elapsed without server response. [clock seconds]"
                LB::detach
                LB::reselect
                HTTP::disable
                LB::connect
                incr retrycount
                if {$static::retry_debug}{
                    log "Request variable contains: $newrequest"
                }
            }
        }\
    ]
    set request [HTTP::request]
    set newrequest [findstr $request "" 0 "\r\n\r\n"]
    if {$static::retry_debug}{
        log "Did findstr - oldrequest: $request - newrequest: $newrequest"
    }
    append newrequest "\r\nConnection: close"
    append newrequest [findstr $request "\r\n\r\n" 0]
    if {$static::retry_debug}{
        log "Inserted Connection Close header - originalrequest: $request - newrequest: $newrequest"
    }
    if {[HTTP::header exists "Content-Length"]}{
        HTTP::collect [HTTP::header "Content-Length"]
    }
}
 
when SERVERSSL_HANDSHAKE {
    log "Server Connected: Retry Count: $retrycount"
    if {$retrycount}{
        if {$static::retry_debug}{
            log "Going to send HTTP request data to new server"
        }
        SSL::respond $newrequest
        if {$static::retry_debug}{
            log "Collecting Response Data from Retry and setting new timeout timer"
        }
        set retryrequest_id [\
            after $static::response_timeout {
                if {$static::retry_debug}{
                    log "Double Timeout - Initial Request timed out and Retry timed out"
                }
                clientside { SSL::respond "HTTP/1.0 504 Gateway Timeout\r\nServer: BIG-IP\r\nConnection: close\r\n\r\n" }
                #clientside { TCP::release }
                clientside { TCP::close }
            }\
        ]
        SSL::collect
        #SSL::release
        #TCP::release
        #HTTP::enable
    } elseif {$retrycount > 1} {
        if {$static::retry_debug} {
            log "Exceeded retry count"
        }
    } else {
        if {$static::retry_debug}{
            log "Not a retry; request: $newrequest"
        }
    }
   
}
 
when SERVERSSL_DATA {
    if {$retrycount}{
        if {$static::retry_debug}{
            log "Response Data from Retried Request - Inserting Connection: close header"
            log "Canceling retryrequest_id"
        }
        after cancel $retryrequest_id
        set response [SSL::payload]
        set newresponse [findstr $response "" 0 "\r\n\r\n"]
        if {$static::retry_debug}{
            log "Did findstr - origresponse: $response - newresponse: $newresponse"
        }
        append newresponse "\r\nConnection: close"
        append newresponse [findstr $response "\r\n\r\n" 0]
        SSL::payload replace 0 [string length $response] $newresponse
        SSL::release
        TCP::close
        clientside { TCP::close }
   }
}
 
when HTTP_REQUEST_DATA {
    if {$static::retry_debug}{
        log "Got some payload data for $request"
    }
    append newrequest [HTTP::payload]
    if {$static::retry_debug}{
        log "Newrequest with Payload: $newrequest"
    }
    HTTP::release
}
 
when HTTP_RESPONSE {
    if {[info exists initialrequest_id]}{
        if {$static::retry_debug}{
            log "Canceling after timeout with id $initialrequest_id because server responded prior to timeout"
        }
        after cancel $initialrequest_id
    }
}

Tested this on version: