iRule Maintenance Windows
A fun, but not well known, feature on BIG-IP is the Statistics Profile. This tech tip is the second in a series on how the Statistics Profile and iRules, when working together, can save time, productivity, and your sleep!
The Scenario:
Imagine yourself, Mr./Mrs. IT Manager, working for a company that's business relies on updated content on it's back-end web servers. Your company has a web team that has decided that it's test cycle will always end at 3:00AM (your time) on Saturday morning. So, here you are, arriving at home after a night of wild karaoke and one too many drinks with cheap umbrellas and you get a call from the dev team that they need the site taken off line so they can push out the new content.
In the past, you gave the dev lead the keys to the castle (meaning the admin credentials for your BIG-IP) and he/she somehow thought "rm -rf /" was how you took a virtual server offline. After an emergency restore of the software you vowed to never give that department direct access to your servers - ever... So, now you are stuck with being on call whenever they deem updates are needed. If only, you could provide your dev team access to control taking their application offline while they are doing updates. Oh, and they should also be able to view the current status as well as being able to bring it back online.
Never fear, help is on the way! There are a couple of handy features within iRules that make this problem a thing of the past.
The Setup:
To get things going, you'll need a Virtual Server setup fronting your web application. Since your application is up and running, that step is already taken care of. Next, you'll need to create a Statistics Profile, assign it to your Virtual Server and write an iRule. So, first we'll create the statistics profile...
Create the Statistics Profile
- Login to the BIG-IP Administrative GUI.
- Select the Profiles option under Local Traffic and Virtual Servers
- Select the Statistics from the Other menu.
- Click the Create button
- Enter "maintenance_window" for the Profile name
- Add the following fields: day, start_time, end_time
- Click Update to create the profile
Create the iRule
Now that the virtual server is setup with the statistics profile, you'll need to create an iRule to to add the controlling functions as well as the maintenance window enforcement.
- Select Rules from the Local Traffic/Virtual Servers menu.
- Click the Create button.
- Enter "maintenance_window" for the iRule name and enter the iRule from below into the Definition text box.
- Click Finished to save the iRule
Apply the statistics profile to your virtual server
The statistics profile doesn't do much until you apply it to your virtual server properties. Here's how:
- Select Virtual Servers from the Local Traffic menu.
- Click on your Virtual Server to enter it's properties
- Make sure the Advanced Configuration option is selected, and scroll down to the Statistics Profile option
- Select the previously created profile maintenance_window
- Click the Update Button
- Select the Resources Menu
- Click the Manage Rules button
- Make sure the maintenance_window iRule is in the Enabled list box and click the Finished button.
The iRule
Now for the fun, here's the iRule that will give you peace, tranquility (and a good nights sleep). This iRule will look for commands in the URI to control the maintenance window settings (see the "/usage" section for details).
when HTTP_REQUEST {
set TITLE "iRule Maintenance Window Control"
set PROFILE_NAME "maintenance_window"
# Look for embedded commands to control the maintenance window.
switch -glob [string tolower [HTTP::uri]] {
"/enable*" {
# Extract the maintenance window day, start time, and end time from the URI.
# If not there, return an error message.
set params [split [HTTP::uri] "/"]
if { [llength $params] < 5 } {
HTTP::respond 200 content "
<head><center><title>$TITLE</title>
<body><h1>Usage: http://[HTTP::host]/enable/day_number/start_hour/end_hour</h1>
</body></html>"
} else {
# Update the maintenance window values in the statistics profile.
STATS::set $PROFILE_NAME "day" [lindex $params 2]
STATS::set $PROFILE_NAME "start_time" [lindex $params 3]
STATS::set $PROFILE_NAME "end_time" [lindex $params 4]
HTTP::respond 200 content "
<head><center><title>$TITLE</title>
<body><h1>Maintenance Window enabled</h1>
</body></html>"
}
}
"/disable" {
# By zero'ing out the values in the statistics profile, the maintenance window is not enforced.
STATS::set $PROFILE_NAME "day" 0
STATS::set $PROFILE_NAME "start_time" 0
STATS::set $PROFILE_NAME "end_time" 0
HTTP::respond 200 content "
<head><center><title>$TITLE</title>
<body><h1>Maintenance Window disabled</h1>
</body></html>"
}
"/check" {
# Return the current time and the current settings for the maintenance window
set day [STATS::get $PROFILE_NAME "day"]
set start_time [STATS::get $PROFILE_NAME "start_time"]
set end_time [STATS::get $PROFILE_NAME "end_time"]
set l [split [clock format [clock seconds] -format {%u %H %M}] " "]
set cur_day [lindex $l 0]
set cur_time [expr [expr [lindex $l 1]*100] + [lindex $l 2]]
HTTP::respond 200 content "
<head><center><title>$TITLE</title>
<body><h1>Current Date and time: $cur_day, $cur_time</h1>
<h1>Maintenance Window: $day, $start_time - $end_time</h1>
</body></html>"
}
"/usage" {
# Usage instructions for the commands
HTTP::respond 200 content "
<head><center><title>$TITLE</title>
<body><h1>Usage: http://[HTTP::host]/\[command\]</h1>
<table><tr><td><ul>
<li>/enable/day_of_week_number(1:mon-7:sun)/start_time(hhmm)/end_time(hhmm) - enable maintenance window.</li>
<li>/disable - disable maintenance window.</li>
<li>/check - check if in maintenance window.</li>
<li>/usage - display this usage message</li>
</ul></td></tr></table>
</body></html>"
}
default {
# no secret command entered, so check the maintenance window
# Get the values from the statistics profile.
set day [STATS::get $PROFILE_NAME "day"]
set start_time [STATS::get $PROFILE_NAME "start_time"]
set end_time [STATS::get $PROFILE_NAME "end_time"]
if { ($day ne 0) and ($start_time ne 0) and ($end_time ne 0) } {
# Use the TCL "clock" command to get the current time settings.
set l [split [clock format [clock seconds] -format {%u %H %M}] " "]
set cur_day [lindex $l 0]
# Math right here is a bit overkill. I've changed it to string concatenation.
#set cur_time [expr [expr [lindex $l 1]*100] + [lindex $l 2]]
set cur_time "[lindex $l 1][lindex $l 2]"
if { ($cur_day eq $day) &&
($cur_time >= $start_time) &&
($cur_time <= $end_time) } {
# The current time is in the maintenance window, so either issue a HTTP::redirect here to a
# prettied up page, or you can do as I've done here, and return a HTML response direction
# to the client.
HTTP::respond 200 content "
<head><center><title>$TITLE</title>
<body><h1>This site is down for maintenance, please come back at $end_time ...</h1>
</body></html>"
} else {
# Not in maintenance window, so allow connection to continue to application.
}
}
}
}
}
Conclusion:
So, now you should be all set. While your dev team now has control over their site's availability and the 3:00am calls are a thing of the past...
Published Apr 10, 2007
Version 1.0
DeVon_JarvisNimbostratus
Here is what I found out. The octal concern is not an issue any more, but comparing anything to 0800 (leading zero) seems to do a string compare, which leads to incorrect results.
What I did was to change the clock format string to:
set t_array [split [clock format [clock seconds] -format {%u %k %M}] " "]
The %k gives you the hours in 24 hour clock without any leading zeros. Yay! Just what we wanted!
This way, numeric comparisons prevail and we get just what we wanted.
I also optimized my version to only do these comparisons once each minute, and store the result in a global variable. This saves lots of processing especially on a busy site! No need to check on every request, as the status of the outage window will not change anyway.
If you want my version of the code, let me know and I will submit it.
jdspring_42045Would love to see the final product. Ours also does validation against the keepalives so that once keepalives < 1 it goes to maintenance too. The final version could be used to give user maintenance control and also prevent the "ugly" page cannot be found when all keepalives are down.- How is it that string comparisons will lead to incorrect results. a string compare for 0800 will be less than 0900 and 1430 is greater then 0600. Can you provide an example of two times in HHMM format that will not work for a string comparison as opposed to a math comparison?
- DeVon_JarvisIn my "modified" version, I had dropped the leading zeros to force numeric comparisons, thinking these would be faster. I have added back the leading zeros and you are right, string comparisons work just fine.
Here is my finished solution. I wanted only fixed outage windows, but wanted to allow more than one, so you will see the start and end times are in an array. I also optimozed the rule by only checking for the outage condition every n (currently 60) seconds. This made a huge difference on performance.
Let me know if you have any suggestions on how to improve it!
when RULE_INIT {
day 1=mon 7=sun, Times are HHMM with leading zeros!
These are list of day, start, end time for multiple outage windows
set ::day {7}
set ::start_time {0700}
set ::end_time {0900}
check_interval is in seconds
set ::check_interval 60
working variables
set ::check_time 0
set ::in_outage_window 0
}
when HTTP_REQUEST timing on {
Use the TCL "clock" command to get the current time settings.
set now [clock seconds]
if { $now >= $::check_time } {
set ::check_time [expr $now + $::check_interval]
set raw_time [clock format $now -format {%u %H %M}]
set raw_time [split [clock format [clock seconds] -format {%u %H %M}] " "]
set cur_day [lindex $raw_time 0]
set cur_time "[lindex $raw_time 1][lindex $raw_time 2]"
set temp_outage_window 0
foreach dayidx $::day startidx $::start_time endidx $::end_time {
if { ($cur_day eq $dayidx) &&
($cur_time >= $startidx) &&
($cur_time < $endidx) } {
set temp_outage_window 1
log local0. "In outage"
}
}
if {$temp_outage_window != $::in_outage_window } {
set ::in_outage_window $temp_outage_window
if {$temp_outage_window == 0} {
log local0. "End of outage"
} else {
log local0. "Start of outage"
}
}
}
if { $::in_outage_window} {
HTTP::respond 307 Location http://www.mysite.com/maintenance.html
return 0
}
} - Jatt_Dill_40963Is there any way we can disable this rule by default and enable when required? Looking for some line code in irule
DM_5174I have tried this and it does not work. Is there a way to remotely enable and disable an IRULE from the browser or via a script? This way, the normal IRULE can be replaced by the one that will be used during a maintenance window without having to login to the LTM. Other team member not part of the network group can do this. I think It's kinda dangerous to have members logging into a network device when they have no clue as to how it works.- You could pretty easily write an iControl application to add/remove an iRule from a virtual server's list of resources. Name your language of choice and I can whip something up for you. Perl (script), PowerShell (Script), Java (bytecode), .Net (compiled .exe). Pick your poison and I can get you some code.
-Joe - Looks like I already wrote a perl app to do just that. It will list, add, and remove iRules from a specified Virtual Server. Check out PerlVirtualServerRule in the iControl CodeShare:
http://devcentral.f5.com/wiki/default.aspx/iControl/PerlVirtualServerRule.html
-Joe - Brian_69413I am having trouble with your code in that the STATS profile fields do not seem to hold the values set in them or rather are not consistent. I have added two log statements, one for when /disable is called and I dump the STATS fields, and the second when I run /check. You will see that multiple hits to /check show different STATS field values:
Mar 18 08:13:17 lbprd501dd_v652 info tmm3[2548]: Rule maintenance_window : Stats Reset to 0 - Day: 0, Start_Time: 0, End_Time: 0
Mar 18 08:13:30 lbprd501dd_v652 info tmm1[2546]: Rule maintenance_window : Check Stats - Day: 4, Start_Time: 800, End_Time: 805
Mar 18 08:13:32 lbprd501dd_v652 info tmm[2545]: Rule maintenance_window : Check Stats - Day: 4, Start_Time: 800, End_Time: 805
Mar 18 08:13:33 lbprd501dd_v652 info tmm3[2548]: Rule maintenance_window : Check Stats - Day: 0, Start_Time: 0, End_Time: 0
Mar 18 08:13:35 lbprd501dd_v652 info tmm2[2547]: Rule maintenance_window : Check Stats - Day: 0, Start_Time: 0, End_Time: 0
Mar 18 08:13:35 lbprd501dd_v652 info tmm[2545]: Rule maintenance_window : Check Stats - Day: 4, Start_Time: 800, End_Time: 805
Mar 18 08:13:36 lbprd501dd_v652 info tmm3[2548]: Rule maintenance_window : Check Stats - Day: 0, Start_Time: 0, End_Time: 0
Mar 18 08:13:37 lbprd501dd_v652 info tmm2[2547]: Rule maintenance_window : Check Stats - Day: 0, Start_Time: 0, End_Time: 0
Mar 18 08:13:38 lbprd501dd_v652 info tmm1[2546]: Rule maintenance_window : Check Stats - Day: 4, Start_Time: 800, End_Time: 805
Mar 18 08:13:40 lbprd501dd_v652 info tmm[2545]: Rule maintenance_window : Check Stats - Day: 4, Start_Time: 800, End_Time: 805
Any ideas on this? 
ASAT_6562I have had trouble getting this to work correctly. Even when the Octal number issue is resolved, the evaluation of current_time fails when the time real current time is between ));24 and 00:59. Using set cur_time [clock format [clock scan now] -format %H:%M] solved the issue for us and this is now happy in our production environment and works as expected.
My code from /check
"/check" {
Return the current time and the current settings for the maintenance window
set day [STATS::get $PROFILE_NAME "day"]
if { ($day < 1) } {
set day "No day set"
}
set start_time [clock format [clock scan [STATS::get $PROFILE_NAME "start_time"]] -format %H:%M]
set end_time [clock format [clock scan [STATS::get $PROFILE_NAME "end_time"]] -format %H:%M]
scan [clock format [clock seconds] -format {%A %k %M}] {%s %s %s} cur_day cur_hour cur_minute
set cur_time [clock format [clock scan now] -format %H:%M]
log local0. "\$cur_day: $cur_day, \$cur_hour: $cur_hour, \$cur_minute: $cur_minute, \$cur_time: $cur_time"
HTTP::respond 200 content [subst $::window_check_response]
}
