Sort objects by busy-ness
Problem this snippet solves:
Here's a request we've had for a long time: sort objects by busy-ness. Since there was never a consensus of what was meant by busy or how that information was to be presented, product development gave us an API to create our own custom views. So here's a cool way to watch objects (vips/snats/nats) sorted by busiest. You can sort on each of the parameters (ex. CPS, max conns, tot throughput, etc.). It requires you to also include the "misc_procedures" script as well.
NOTE: This was created to help quickly find the busiest objects on the box. At one point, we were troubleshooting why TMM CPU was so busy and bigtop only showed VIPS. Turned out traffic was traversing a snat instead. Sorting/grepping/sed/awking through bigpipe/tmsh output for all the objects was cumbersome.
ex.
y=`b virtual show`; for i in `echo "$y" | egrep '^ \| \(cur' | awk '{print $8}' | egrep -o '0-9+' | sort -rn | head`; do echo "$y" | egrep -B 2 "^ \| \(cur, max, limit, tot\) = \(0-9+, $i"; done
y=`b snat show`; for i in `echo "$y" | egrep '^\| \(cur' | awk '{print $8}' | egrep -o '0-9+' | sort -rn | head`; do echo "$y" | egrep -B 1 "^\| \(cur, max, limit, tot\) = \(0-9+, $i"; done
Yuck!
DEFAULTS (without any arguments):
- Shows all object types (vips/snats/nats)
- Sorts by Current Connections
- Samples every 10s (as there can be lots of objects which can put a bit of load on mcpd).
- Auto Formats Output
- Shows all records (simply put number on the end for how many objects you would like to see)
See "Usage example below"
RFEs:
I know, I know. There is always room for improvement. Some have already thought about are:
- Be able to only show field specified (instead of ALL the fields) to prevent wrap around on narrow terminals. Will require complete overhaul but need time to noodle on it.
- Add object sub-type (ex. tcp-vip vs. fastL4 vip)
This is version 1.0 so have a few things would like to clean up but basic functionality is there. Getting it to Devcentral asap to leverage the incredible brain trust on here:)
Sample Output:
root@bigip(Active)(tmos.cli.script)# run watch_by_busy ?
usage: watch_by_busy
(virtual|snat|nat|all)
default = all
(object-name|object-type|cps|cur-conns|max-conns|tot-conns|
bits-in|bits-out|tot-bits-in|tot-bits-out|
pkts-in|pkts-out|tot-pkts-in|tot-pkts-out)
default = cur-conns
(gig|meg|kil|raw|auto)
default = auto
number of records to display
default = all records
ex. watch_by_busy
watch_by_busy virtual max-conns auto 10
root@bigip(Active)(tmos.cli.script)# run watch_by_busy
all bits-out gig max-conns object-name pkts-out tot-bits-in tot-pkts-in
auto cps help meg object-type raw tot-bits-out tot-pkts-out
bits-in cur-conns kil nat pkts-in snat tot-conns virtual
root@bigip(Active)(tmos.cli.script)# run watch_by_busy virtual max-conns 15
Gathering stats over interval 10 secs. Please be patient.
Object Oject Conns Conns Conns Conns Bits Bits Tot-Bits Tot-Bits Pkts Pkts Tot-Pkts Tot-Pkts
Name Type CPS Cur Max Tot In Out In Out In Out In Out
------------------------- -------- ------- ------- ------- ------- ------- ------- ------- -------- ------- ------- ------- -------
out virtual 2 5 128 23.47M 0 0 65.11G 2455.13G 0 0 103.49M 208.55M
ps_http virtual 0 0 115 36.28K 0 0 8.57G 666.21G 0 0 26.57M 55.64M
ps_tftp virtual 0 0 90 4.38K 0 0 562.10M 23.19G 0 0 2.19M 2.19M
vpn-fwd-test virtual 0 2 29 6.14K 0 0 410.00M 1.35G 0 0 341.96K 381.49K
b1600-1.es.f5net.com virtual 0 0 26 46.36K 0 0 587.07M 6.17G 0 0 548.34K 738.84K
b3400-1.es.f5net.com virtual 0 0 26 43.09K 0 0 632.98M 6.21G 0 0 648.91K 782.99K
b8800-2.es.f5net.com virtual 0 0 26 42.73K 0 0 625.30M 6.18G 0 0 641.43K 778.51K
em500-2.es.f5net.com virtual 0 0 26 51.93K 0 0 762.37M 7.02G 0 0 779.38K 916.95K
b3400-2.es.f5net.com virtual 0 0 25 43.46K 0 0 636.59M 6.21G 0 0 652.51K 785.42K
b3400-3.es.f5net.com virtual 0 0 25 41.33K 0 0 755.48M 6.21G 0 0 607.76K 736.14K
b3600-1.es.f5net.com virtual 0 0 25 53.14K 0 0 669.71M 6.45G 0 0 632.91K 826.27K
b6900-1.es.f5net.com virtual 0 0 25 48.43K 0 0 612.60M 6.25G 0 0 570.71K 763.19K
b8800-1.es.f5net.com virtual 0 1 25 237.84K 2.73K 3.18K 3.26G 9.30G 1 1 2.35M 1.92M
b8900-3.es.f5net.com virtual 0 0 25 58.93K 0 0 860.26M 6.77G 0 0 740.68K 908.49K
em500-1.es.f5net.com virtual 0 0 25 49.26K 0 0 910.83M 6.13G 0 0 777.24K 1.01M
es_ldap virtual 3 0 15 3.03M 2.90K 1.79K 2.91G 1.79G 6 5 6.02M 4.99M
Code :
cli script watch_by_busy {
proc script::init {} {
#Version 1.0. Have a few things would like to clean up but basic functionality is there.
set ::usage_string "usage: [lindex $tmsh::argv 0] \n \
\t\t \t\t (virtual|snat|nat|all) \n \
\t\t\t\t\t default = all \n\n \
\t\t \t (object-name|object-type|cps|cur-conns|max-conns|tot-conns| \n \
\t\t\t\t\t bits-in|bits-out|tot-bits-in|tot-bits-out| \n \
\t\t\t\t\t pkts-in|pkts-out|tot-pkts-in|tot-pkts-out) \n \
\t\t\t\t\t default = cur-conns \n\n \
\t\t \t\t (gig|meg|kil|raw|auto) \n \
\t\t\t\t\t default = auto \n\n \
\t\t \t number of records to display \n \
\t\t\t\t\t default = all records \n\n \
\t\t ex. [lindex $tmsh::argv 0] \n \
\t\t [lindex $tmsh::argv 0] virtual max-conns auto 10"
set ::object_types { virtual snat nat all }
set ::fields { object-name object-type cps cur-conns max-conns tot-conns\
bits-in bits-out tot-bits-in tot-bits-out\
pkts-in pkts-out tot-pkts-in tot-pkts-out }
set ::formats { gig meg kil raw auto }
set ::padding 2
set ::interval 10
}
proc usage { } {
puts $::usage_string
exit
}
proc process_cmd_args_p { args_default } {
upvar $args_default my_args
foreach { arg } $tmsh::argv {
if { [ lsearch -exact $::object_types $arg ] >= 0 } {
set my_args(object_type) $arg
} elseif { [ lsearch -exact $::fields $arg ] >= 0 } {
set my_args(field) $arg
set my_args(field_index) [ lsearch -exact $::fields $arg ]
} elseif { [ lsearch -exact $::formats $arg ] >= 0 } {
set my_args(format) $arg
} elseif { [ regexp {\d+} $arg ] } {
set my_args(num_display) $arg
} elseif { $arg equals "help" } {
usage
}
}
}
proc get_stats_p { results_l object_type } {
upvar $results_l results
if { $object_type equals "all" } {
set object_types { virtual snat nat }
} else {
set object_types $object_type
}
foreach obj_type $object_types {
set idx 0
set objs [tmsh::get_status / ltm $obj_type raw ]
set count [llength $objs]
while { $idx < $count } {
set object_l {}
set obj [lindex $objs $idx]
if { $obj_type equals "virtual" or $obj_type equals "snat" } {
lappend object_l [tmsh::get_name $obj] \
$obj_type \
[tmsh::get_field_value $obj "clientside.tot-conns"] \
[tmsh::get_field_value $obj "clientside.cur-conns"] \
[tmsh::get_field_value $obj "clientside.max-conns"] \
[tmsh::get_field_value $obj "clientside.tot-conns"] \
[tmsh::get_field_value $obj "clientside.bits-in"] \
[tmsh::get_field_value $obj "clientside.bits-out"] \
[tmsh::get_field_value $obj "clientside.bits-in"] \
[tmsh::get_field_value $obj "clientside.bits-out"] \
[tmsh::get_field_value $obj "clientside.pkts-in"] \
[tmsh::get_field_value $obj "clientside.pkts-out"] \
[tmsh::get_field_value $obj "clientside.pkts-in"] \
[tmsh::get_field_value $obj "clientside.pkts-out"] \
} elseif { $obj_type equals "nat" } {
lappend object_l [tmsh::get_name $obj] \
$obj_type \
[tmsh::get_field_value $obj "serverside.tot-conns"] \
[tmsh::get_field_value $obj "serverside.cur-conns"] \
[tmsh::get_field_value $obj "serverside.max-conns"] \
[tmsh::get_field_value $obj "serverside.tot-conns"] \
[tmsh::get_field_value $obj "serverside.bits-in"] \
[tmsh::get_field_value $obj "serverside.bits-out"] \
[tmsh::get_field_value $obj "serverside.bits-in"] \
[tmsh::get_field_value $obj "serverside.bits-out"] \
[tmsh::get_field_value $obj "serverside.pkts-in"] \
[tmsh::get_field_value $obj "serverside.pkts-out"] \
[tmsh::get_field_value $obj "serverside.pkts-in"] \
[tmsh::get_field_value $obj "serverside.pkts-out"] \
}
lappend results $object_l
incr idx
}
}
}
proc script::run {} {
tmsh::include misc_procedures
#set defaults. Show all object types, sort by cur-conns = field_index 3, auto format and essentially display unlimited objects
array set args { object_type all field cur-conns field_index 3 format auto num_display 10000 }
if { $tmsh::argc > 5 } {
usage
}
process_cmd_args_p args
set fields1 { Object Oject Conns Conns Conns Conns Bits Bits Tot-Bits Tot-Bits Pkts Pkts Tot-Pkts Tot-Pkts }
set fields2 { Name Type CPS Cur Max Tot In Out In Out In Out In Out }
set field_lengths { 20 8 7 7 7 7 7 7 7 7 7 7 7 7 }
set rate_fields { 2 6 7 10 11 }
# result set tables
set results_1_l {}
set results_2_l {}
set delay [expr $::interval * 1000]
#first poll
get_stats_p results_1_l $args(object_type)
puts "Gathering stats over interval $::interval secs. Please be patient."
while { true } {
after $delay
# second poll
get_stats_p results_2_l $args(object_type)
# list for deltas between results_1_l and results_2_l. using tcl lists as much more conducive to sorting
set delta_l [ calculate_delta_list_p $results_1_l $results_2_l $rate_fields ]
# create an array of formated values so can key off name.
# however, in tcl, easier to pass arrays in list form first
set formated_l [ create_formated_array_p $delta_l $args(format) field_lengths ]
array set formated_a $formated_l
tmsh::clear_screen
#Print Headers
puts [ format_line_p $fields1 $field_lengths ]
puts [ format_line_p $fields2 $field_lengths ]
puts [ format_spacer_p $fields2 $field_lengths ]
#Now comes the sorting
set tx 0
foreach object [ lsort -decreasing -real -index $args(field_index) $delta_l ] {
if { $tx <= $args(num_display) } {
#handle first field "object name" differently
set obj_name [ lindex $object 0 ]
puts [ format_line_p $formated_a($obj_name) $field_lengths ]
incr tx
}
}
# use the most recent results as the next previous results
set results_1_l $results_2_l
set results_2_l {}
}
}
proc script::help {} {
tmsh::add_help "$::usage_string"
}
proc script::tabc {} {
lappend tab_types $::object_types $::fields $::formats help
foreach tab_type $tab_types {
foreach tab $tab_type {
tmsh::add_tabc $tab
}
}
}
} Published Mar 10, 2015
Version 1.0