IIS, dotNet, and App pool recycling

You could absolutely do that with iControl. The process would be something like this

 

 

1. Modify pool member to disable any new connections with the set_session_enabled_state() method passing in STATE_DISABLED for the associated pool member(s).

 

2. Use the PoolMember's get_statistics() methods to query the CURRENT_CONNECTIONS statistic until that value drops down to zero.

 

3. Issue the remote call to your windows box to reset IIS.

 

4. Re-enable new sessions by passing STATE_ENABLED to the set_session_enabled_state() method.

 

 

I've wrote a tech tip titled "Graceful Server Shutdown" that you could use as a basis to start with (it's with our PowerShell library which should integrate well with IIS controls).

 

 

http://devcentral.f5.com/Tutorials/TechTips/tabid/63/articleType/ArticleView/articleId/254/iControl-Apps--04--Graceful-Server-Shutdown.aspx

 

 

You'll just have to add the IIS hooks in there.

 

 

Hope this helps...

 

Thanks, Joe.

 

I found your posting about graceful Server shutdown just moments after I posted this. I've been using it with great success over the past few days, but it has raised a few questions.

 

I have to admit I don't quite understand the three 'states' that a pool member can be in - the 'Enabled' state is easily understood, as is 'forced offline', however, the 'disabled' state is where I'm tripping up.

 

When we say 'active' connections, is that something that is defined by a connection with data travelling back and forth every so often ? What determines the 'every so often' intervals ?

 

 

In practice, what I see is when I use the script to do a graceful shutdown of my pool members is that the great portion of the connections quickly go away, the connection count drops rapidly, down to the last 6 or 10. These then 'hang on' for L-O-N-G periods. I'd guess I could tune this in a profile setting, at the Virtual Server level ?

 

 

thanks !!

While I agree that an iControl way of disabling new connections to a pool member before planned maintenance is useful, I'm curious why app pool recycles in particular are causing a problem for your team.

 

 

A couple of items that might be worth looking into:

 

- Why is the app pool being recycled so frequently that people notice? I would make sure the default recycle interval of 1740 minutes is disabled in favor of recycling at a specific time or not at all. An app pool that contains stable production apps may not need to be recycled between planned maintenance.

 

- Application pools use overlapping recycles by default so the app is typically not down during the recycle. However, in-process session state will be lost. If loss of session state is the problem and your team owns the apps, your team may consider using one of the other methods available to store session state: http://msdn.microsoft.com/en-us/library/ms178586.aspx

 

 

If loss of session state is in fact the problem, eliminating it by avoiding in-process sessions may also help to minimize the impact of app releases, unplanned app pool recycles (due to memory exhaustion, for example) and other planned maintenance. Good luck!

 

Randy, the three states are documented as follows in the manuals:

 

 

Enabled - All traffic allowed

 

Disabled - Only persistent or active connections allowed

 

Forced Offline - Only active connections allowed

 

 

Enabled is easy, it's on and new connections are allowed.

 

 

Disabled means that no new connections are allowed in but persistent connections (ones that have been assigned through a persistence profile) or currently active ones are allowed.

 

 

Forced Offline means that only currently active connections are allowed (not persistent connections).

 

 

In your situation, the sample code I referenced should put the pool member in "forced offline" mode, but if you are seeing long lived connections, then that seems like there are some persistence entries still in there (or you have some really long lived keep-alive connections to your backend apps.

 

 

-Joe

 

Thanks for the explanation, Joe. Pretty much the way I'd interpreted it.

 

 

What our issue appears to be is 'long-lived' connections. Most recently, I used icontrol to set a pool member into 'forced offline' mode. Connections slowly dwindled (as seen in the PS console- 'Current Connections:1' however, this one connection remained even overnight.

 

 

Reviewing our application logs, we see one client hitting one portion of our API repeatedly, every few seconds.

 

 

 

Seems like the only way to handle something like this would be to set a TCP profile 'timeout' to a very low value (and this is probably not a great idea...)