Zero Downtime IIS Code Deployment

We have a production pool with 4 nodes. Our code deployment process is to:

 

1. Enable source address affinity persistence on the production VS.
2. Set node 1 to "Forced Offline".
3. Poll node 1 periodically for active connections, waiting until connections=0 before continuing.
4. Stop IIS on node 1.
5. Copy new code to node 1.
6. Start IIS on node 1.
7. Set node 1 to "Enabled" in the pool.
8. Repeat steps 2 - 7 for nodes 2, 3, and 4.
9. Disable persistence.

This works pretty well, except we run into an issue on step 3. Let's say step 3 is currently operating on the final node 4. At this point, nodes 1 - 3 have the new code, while node 4 still has the old code. We have HTTP keep-alive configured on IIS with a 120 second timeout, so these outstanding active connections may take up to two minutes to drain.

 

The problem is that web browsers typically spin up multiple connections in parallel to maximize performance. This means that an existing active connection may hit old code on node 4, while any other newly spooled up connections from the same client will hit newly deployed code on nodes 1 - 3. If there are incompatibilities between code versions (say with dependencies between .JS and .CSS files) our application may present strange side effects to the end-user for a few moments before the keep-alive connection to node 4 expires.

 

One thing I tried is to set the node to "Disabled (Only persistent or active connections allowed)" but the problem with that is we have highly active connections that will pass traffic continuously all day long, resulting in a node whose connections will never fully drain.

 

Is there something with my process / BIG-IP configuration I'm missing? I can't see a way around this problem because no matter how you approach it, at some point you have to force one or more nodes offline to deploy new code, and those clients will for a few moments hit old code and new code simultaneously, resulting in compatibility issues. Any ideas?