To Comment or Not to Comment?
Lindsay Hill wrote an article on iRules comments on his site that got me thinking about comments in general. As I prepared to just bring forth some insightful commentary on using comments in iRules, the majority of the quotes I found are actually bent against commenting in code. My favorites: Good code is its own best documentation. As you’re about to add a comment, ask yourself, ‘How can I improve the code so that this comment isn’t needed? - Steve McConnell Every time you write a comment, you should grimace and feel the failure of your ability of expression. - Unknown I'm more than a little knotted up over this idea of not commenting the code. For well-maintained source code in large projects, maybe this is a best practice. For scripts like iRules, however, I'm a big fan of commentary. The reality is that in operational environments, people come and go and, well, even the original author might need a refresher course on what the iRule does after a few (weeks?) months go by and an emergency hits at 3am. Clear documentation in the iRule can go a long way to providing clarity on iRule overall and event/section functionality, especially for the operations on-call looking at not only your iRule for the first time, but perhaps AN iRule for the first time. Another reason for using comments over readable code can be found in #6 of Joe's 10 steps to iRules optimization, which states that longer variable names, whereas making the code more readable, actually have memory implications. Now that I've waxed philosophically on whether or not to include comments, how do you actually do it? Well, the formal comment is accomplished with the pound sign. From the Tcl wiki: If A hash character ("#") appears at a point where Tcl is expecting the first character of the first word of a command, then the hash character and the characters that follow it, up through the next newline, are treated as a comment and ignored. The comment character only has significance when it appears at the beginning of a command. # single line comment started with the hash or pound sign. when HTTP_REQUEST { log local0. "Request event has fired" ; # in-line comment. Lead with a semi-colon. } This is the most commonly seen method of commenting. Something Tcl doesn't support natively like most other languages is the multi-line comment. However, it can be done without commenting each line out. There are many workarounds for this, two of which are shown below. Option 1 One way to comment out many lines is to use a false condition. if 0 { set x 13 set y 15 set product [expr {$x * $y}] } Note that using this method will still require all the lines to be valid iRules lines. The syntax validator will allow the previous example as all lines are valid, but will not allow the example below. if 0 { this is a comment and I can put anything here that I want to. } error: [undefined procedure: this][this is a comment and I can] error: [undefined procedure: put][put anything here that I want] error: [undefined procedure: to][to] This is a good option if you want to comment out a large existing section of code quickly to isolate issues in other sections of code. Option 2 Another way to comment many lines of code is to just to wrap them in a variable. No valid code necessary for this option, as evidenced by a fake command in HTTP::nonsense and a pool that does not exist in undefinedpool. This passed the syntax validator just fine. set comment { this is a comment on multiple lines with non-existent commands and objects yet will save just fine if ([HTTP::nonsense] eq "/test") { pool undefinedpool } } The Fine Print Whereas both these methods work, I'd recommend only using them during development and/or troubleshooting exercises. Why? Well, with the formal comment, Tcl ignores all that and it doesn't get compiled down to bytecode. That is not the case for these options. The code in the false condition will never run and the variable may never be called, but both approaches unnecessarily consume system resources. The first option consumes memory and runtime resources, the the second option only memory. Oh, one more thing. Even in these methods, you need to match your brackets or the syntax validator will complain. Note that this is true even with the formal comments as well when strung together. What other approaches to multi-line comments in Tcl/iRules have you seen or used? Drop a #comment below!
iRule [string range...] not chunking data properly
I have an irule (much of which I found here) that is to gather some much needed troubleshooting data I need in regards to the headers and payload I have coming in. The payload is obvisouly too large for a single log line so this was supposed to chunk it into manageable bytes. I had to make some modifications to the original irule which did not work, but now it is working, sort of. It logs the first 900 bytes as it should, then something happens and skips a bunch, and then logs the final bytes of data. I can't understand why it's not grabbing either the proper amount of data or failing to output this second chunk of missing data before making its final loop. iRule is here: when RULE_INIT { # Log debug to /var/log/ltm? 1=yes, 0=no set static::payload_dbg 1 # Limit payload collection to 5Mb set static::max_collect_len 5368709 # Max characters to log locally (must be less than 1024 bytes) # https://clouddocs.f5.com/api/irules/log.html set static::max_chars 900 set static::min_chars 0 } when HTTP_REQUEST { # Only collect POST request payloads if {[HTTP::method] equals "POST"}{ if {$static::payload_dbg}{log local0. "POST request"} # Get the content length so we can request the data to be processed in the HTTP_REQUEST_DATA event. if {[HTTP::header exists "Content-Length"]}{ set content_length [HTTP::header "Content-Length"] } else { set content_length 0 } # content_length of 0 indicates chunked data (of unknown size) if {$content_length > 0 && $content_length < $static::max_collect_len}{ set collect_length $content_length } else { set collect_length $static::max_collect_len } if {$static::payload_dbg}{log local0. "Content-Length: $content_length, Collect length: $collect_length"} foreach aHeader [HTTP::header names] { log local0. "HTTP Request Headers: $aHeader: [HTTP::header value $aHeader]" } #set payload [HTTP::collect $collect_length] HTTP::collect $collect_length } } when HTTP_REQUEST_DATA { # Log the bytes collected if {$static::payload_dbg}{log local0. "Collected [HTTP::payload length] bytes"} # Log the payload locally if {[HTTP::payload length] < $static::max_chars}{ log local0. "Payload=[HTTP::payload]" } else { # Initialize variables set remaining [HTTP::payload] set position 0 set count 1 set bytes_logged 0 # Loop through and log each chunk of the payload while {[string length $remaining] > $static::min_chars}{ #chunk = 899 set chunk [expr {$position + $static::max_chars -1}] log local0. "Chunk: $chunk" log local0. "Position start: $position" # Get the current chunk to log (subtract 1 from the end as string range is 0 indexed) set current [string range $remaining $position $chunk] log local0. "chunk $count=$current" log local0. "Current chunks bytes: [string length $current]" # Add the length of the current chunk to the position for the next chunk incr position [string length $current] # Get the next chunk to log set remaining [string range $remaining $position end] incr count incr bytes_logged $position log local0. "remaining bytes=[string length $remaining], \$position=$position, \$count=$count, \$bytes_logged=$bytes_logged" } if {[string length $remaining]}{ log local0. "chunk $count=$current" incr bytes_logged [string length $remaining] } log local0. "Logged $count chunks for a total of $bytes_logged bytes" } } when HTTP_RESPONSE { foreach aHeader [HTTP::header names] { log local0. "HTTP Request Headers: $aHeader: [HTTP::header value $aHeader]" } } The image below shows what I'm talking about - it even says "Oh I found 3 chunks", but I'm only being presented 2. I added some logging around the position, chunk value (aka end position), and bytes logged to help illustrate that it's clearly skipping over something. Any help would be appreciated! ThanksAdvanced iRules: An Abstract View of iRules with the Tcl Bytecode Disassembler
In case you didn't already know, I'm a child of the 70's. As such, my formative years were in the 80's, where the music and movies were synthesized and super cheesy. Short Circuit was one of those cheesy movies, featuring Ally Sheedy, Steve Guttenberg, and Johnny Five, the tank-treaded laser-wielding robot with feelings and self awareness. Oh yeah! The plot...doesn't at all matter, but Johnny's big fear once reaching self actualization was being disassembled. Well, in this article, we won't dissamble Johnny Five, but we will take a look at disassembling some Tcl code and talk about optimizations. Tcl forms the foundation of several code environments on BIG-IP: iRules, iCall, tmsh, and iApps. The latter environments don't carry the burden of performance that iRules do, so efficiency isn't as big a concern. When we speak at conferences, we often commit some time to cover code optimization techniques due to the impactful nature of applying an iRule to live traffic. This isn't to say that the system isn't highly tuned and optimized already, it's just important not to introduce any more impact than is absolutely necessary to carry out purpose. In iRules, you can turn timing on to see the impact of an iRule, and in the Tcl shell (tclsh) you can use the time command. These are ultimately the best tools to see what the impact is going to be from a performance perspective. But if you want to see what the Tcl interpreter is actually doing from an instruction standpoint, well, you will need to disassemble the code. I've looked at bytecode in some of the python scripts I've written, but I wasn't aware of a way to do that in Tcl. I found a thread on stack that indicated it was possible, and after probing a little further was given a solution. This doesn't work in Tcl 8.4, which is what the BIG-IP uses, but it does work on 8.5+ so if you have a linux box with 8.5+ you're good to go. Note that there are variances from version to version that could absolutely change the way the interpreter works, so understand that this is an just an exercise in discovery. Solution 1 Fire up tclsh and then grab a piece of code. For simplicity, I'll use two forms of a simple math problem. The first is using the expr command to evaluate 3 times 4, and the second is the same math problem, but wraps the evaluation with curly brackets. The command that will show how the interpreter works its magic is tcl::unsupported::disassemble. ## ## unwrapped expression ## ## % tcl::unsupported::disassemble script { expr 3 * 4 } ByteCode 0x0x1e1ee20, refCt 1, epoch 16, interp 0x0x1d59670 (epoch 16) Source " expr 3 * 4 " Cmds 1, src 12, inst 14, litObjs 4, aux 0, stkDepth 5, code/src 0.00 Commands 1: 1: pc 0-12, src 1-11 Command 1: "expr 3 * 4 " (0) push1 0 # "3" (2) push1 1 # " " (4) push1 2 # "*" (6) push1 1 # " " (8) push1 3 # "4" (10) concat1 5 (12) exprStk (13) done ## ## wrapped expression ## ## % tcl::unsupported::disassemble script { expr { 3 * 4 } } ByteCode 0x0x1de7a40, refCt 1, epoch 16, interp 0x0x1d59670 (epoch 16) Source " expr { 3 * 4 } " Cmds 1, src 16, inst 3, litObjs 1, aux 0, stkDepth 1, code/src 0.00 Commands 1: 1: pc 0-1, src 1-15 Command 1: "expr { 3 * 4 } " (0) push1 0 # "12" (2) done Because the first expression is unwrapped, the interpreter has to build the expression and then call the runtime expression engine, resulting in 4 objects and a stack depth of 5. With the wrapped expression, the interpreter found a compile-time constant and used that directly, resulting in 1 object and a stack depth of 1 as well. Much thanks to Donal Fellows on Stack Overflow for the details. Using the time command in the shell, you can see that wrapping the expression results in a wildly more efficient experience. % time { expr 3 * 4 } 100000 1.02325 microseconds per iteration % time { expr {3*4} } 100000 0.07945 microseconds per iteration Solution 2 I was looking in earnest for some explanatory information for the bytecode fields displayed with tcl::unsupported::disassemble, and came across a couple pages on the Tcl wiki, one building on the other. Combining the pertinent sections of code from each page results in this script you can paste into tclsh: namespace eval tcl::unsupported {namespace export assemble} namespace import tcl::unsupported::assemble rename assemble asm interp alias {} disasm {} ::tcl::unsupported::disassemble proc aproc {name argl body args} { proc $name $argl $body set res [disasm proc $name] if {"-x" in $args} { set res [list proc $name $argl [list asm [dis2asm $res]]] eval $res } return $res } proc dis2asm body { set fstart " push -1; store @p; pop " set fstep " incrImm @p +1;load @l;load @p listIndex;store @i;pop load @l;listLength;lt " set res "" set wait "" set jumptargets {} set lines [split $body \n] foreach line $lines { ;#-- pass 1: collect jump targets if [regexp {\# pc (\d+)} $line -> pc] {lappend jumptargets $pc} } set lineno 0 foreach line $lines { ;#-- pass 2: do the rest incr lineno set line [string trim $line] if {$line eq ""} continue set code "" if {[regexp {slot (\d+), (.+)} $line -> number descr]} { set slot($number) $descr } elseif {[regexp {data=.+loop=%v(\d+)} $line -> ptr]} { #got ptr, carry on } elseif {[regexp {it%v(\d+).+\[%v(\d+)\]} $line -> copy number]} { set loopvar [lindex $slot($number) end] if {$wait ne ""} { set map [list @p $ptr @i $loopvar @l $copy] set code [string map $map $fstart] append res "\n $code ;# $wait" set wait "" } } elseif {[regexp {^ *\((\d+)\) (.+)} $line -> pc instr]} { if {$pc in $jumptargets} {append res "\n label L$pc;"} if {[regexp {(.+)#(.+)} $instr -> instr comment]} { set arg [list [lindex $comment end]] if [string match jump* $instr] {set arg L$arg} } else {set arg ""} set instr0 [normalize [lindex $instr 0]] switch -- $instr0 { concat - invokeStk {set arg [lindex $instr end]} incrImm {set arg [list $arg [lindex $instr end]]} } set code "$instr0 $arg" switch -- $instr0 { done { if {$lineno < [llength $lines]-2} { set code "jump Done" } else {set code ""} } startCommand {set code ""} foreach_start {set wait $line; continue} foreach_step {set code [string map $map $fstep]} } append res "\n [format %-24s $code] ;# $line" } } append res "\n label Done;\n" return $res } proc normalize instr { regsub {\d+$} $instr "" instr ;# strip off trailing length indicator set instr [string map { loadScalar load nop "" storeScalar store incrScalar1Imm incrImm } $instr] return $instr } Now that the script source is in place, you can test the two expressions we tested in solution 1. The output is very similar, however, there is less diagnostic information to go with the bytecode instructions. Still, the instructions are consistent between the two solutions. The difference here is that after "building" the proc, you can execute it, shown below each aproc expression. % aproc f x { expr 3 * 4 } -x proc f x {asm { push 3 ;# (0) push1 0 # "3" push { } ;# (2) push1 1 # " " push * ;# (4) push1 2 # "*" push { } ;# (6) push1 1 # " " push 4 ;# (8) push1 3 # "4" concat 5 ;# (10) concat1 5 exprStk ;# (12) exprStk ;# (13) done label Done; }} % f x 12 % aproc f x { expr { 3 * 4 } } -x proc f x {asm { push 12 ;# (0) push1 0 # "12" ;# (2) done label Done; }} % f x 12 Deeper Down the Rabbit Hole Will the internet explode if I switch metaphors from bad 80's movie to literary classic? I guess we'll find out. Simple comparisons are interesting, but now that we're peeling back the layers, let's look at something a little more complicated like a for loop and a list append. % tcl::unsupported::disassemble script { for { $x } { $x < 50 } { incr x } { lappend mylist $x } } ByteCode 0x0x2479d30, refCt 1, epoch 16, interp 0x0x23ef670 (epoch 16) Source " for { $x } { $x < 50 } { incr x } { lappend mylist $x " Cmds 4, src 57, inst 43, litObjs 5, aux 0, stkDepth 3, code/src 0.00 Exception ranges 2, depth 1: 0: level 0, loop, pc 8-16, continue 18, break 40 1: level 0, loop, pc 18-30, continue -1, break 40 Commands 4: 1: pc 0-41, src 1-56 2: pc 0-4, src 7-9 3: pc 8-16, src 37-54 4: pc 18-30, src 26-32 Command 1: "for { $x } { $x < 50 } { incr x } { lappend mylist $x }" Command 2: "$x " (0) push1 0 # "x" (2) loadStk (3) invokeStk1 1 (5) pop (6) jump1 +26 # pc 32 Command 3: "lappend mylist $x " (8) push1 1 # "lappend" (10) push1 2 # "mylist" (12) push1 0 # "x" (14) loadStk (15) invokeStk1 3 (17) pop Command 4: "incr x " (18) startCommand +13 1 # next cmd at pc 31 (27) push1 0 # "x" (29) incrStkImm +1 (31) pop (32) push1 0 # "x" (34) loadStk (35) push1 3 # "50" (37) lt (38) jumpTrue1 -30 # pc 8 (40) push1 4 # "" (42) done You'll notice that there are four commands in this code. The for loop, the x variable evaluation, the lappend operations, and the loop control with the incr command. There are a lot more instructions in this code, with jump pointers from the x interpretation to the incr statement, the less than comparison, then a jump to the list append. Wrapping Up I went through an exercise years ago to see how far I could minimize the Solaris kernel before it stopped working. I personally got down into the twenties before the system was unusable, but I think the record was somewhere south of 15. So...what's the point? Minimal for minimal's sake is not the point. Meet the functional objectives, that is job one. But then start tuning. Less is more. Less objects. Less stack depth. Less instantiation. Reviewing bytecode is good for that, and is possible with the native Tcl code. However, it is still important to test the code performance, as relying on bytecode objects and stack depth alone is not a good idea. For example, if we look at the bytecode differences with matching an IP address, there is no discernable difference from Tcl's perspective between the two regexp versions, and very little difference between the two regexp versions and the scan example. % dis script { regexp {([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})} 192.168.101.20 _ a b c d } ByteCode 0x0x24cfd30, refCt 1, epoch 15, interp 0x0x2446670 (epoch 15) Source " regexp {([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-" Cmds 1, src 90, inst 19, litObjs 8, aux 0, stkDepth 8, code/src 0.00 Commands 1: 1: pc 0-17, src 1-89 Command 1: "regexp {([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9" (0) push1 0 # "regexp" (2) push1 1 # "([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})" (4) push1 2 # "192.168.101.20" (6) push1 3 # "_" (8) push1 4 # "a" (10) push1 5 # "b" (12) push1 6 # "c" (14) push1 7 # "d" (16) invokeStk1 8 (18) done % dis script { regexp {^(\d+)\.(\d+)\.(\d+)\.(\d+)$} 192.168.101.20 _ a b c d } ByteCode 0x0x24d1730, refCt 1, epoch 15, interp 0x0x2446670 (epoch 15) Source " regexp {^(\d+)\.(\d+)\.(\d+)\.(\d+)$} 192.168.101.20 _" Cmds 1, src 64, inst 19, litObjs 8, aux 0, stkDepth 8, code/src 0.00 Commands 1: 1: pc 0-17, src 1-63 Command 1: "regexp {^(\d+)\.(\d+)\.(\d+)\.(\d+)$} 192.168.101.20 _ " (0) push1 0 # "regexp" (2) push1 1 # "^(\d+)\.(\d+)\.(\d+)\.(\d+)$" (4) push1 2 # "192.168.101.20" (6) push1 3 # "_" (8) push1 4 # "a" (10) push1 5 # "b" (12) push1 6 # "c" (14) push1 7 # "d" (16) invokeStk1 8 (18) done % dis script { scan 192.168.101.20 %d.%d.%d.%d a b c d } ByteCode 0x0x24d1930, refCt 1, epoch 15, interp 0x0x2446670 (epoch 15) Source " scan 192.168.101.20 %d.%d.%d.%d a b c d " Cmds 1, src 41, inst 17, litObjs 7, aux 0, stkDepth 7, code/src 0.00 Commands 1: 1: pc 0-15, src 1-40 Command 1: "scan 192.168.101.20 %d.%d.%d.%d a b c d " (0) push1 0 # "scan" (2) push1 1 # "192.168.101.20" (4) push1 2 # "%d.%d.%d.%d" (6) push1 3 # "a" (8) push1 4 # "b" (10) push1 5 # "c" (12) push1 6 # "d" (14) invokeStk1 7 (16) done However, if you look at the time results from these examples, they are very different. % time { regexp {([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})} 192.168.101.20 matched a b c d } 100000 11.29749 microseconds per iteration % time { regexp {^(\d+)\.(\d+)\.(\d+)\.(\d+)$} 192.168.101.20 _ a b c d } 100000 7.78696 microseconds per iteration % time { scan 192.168.101.20 %d.%d.%d.%d a b c d } 100000 1.03708 microseconds per iteration Why is that? Well, bytecode is a good indicator, but it doesn't address the inherent speed of the commands being invoked. Regex is a very slow operation comparatively. And within the regex engine, the second example is a simpler regex to evaluate, so it's faster (though less accurate, so make sure you are actually passing an IP address.) Then of course, scan shows off its optimized self in grand fashion. Was this a useful exercise in understanding Tcl under the hood? Drop some feedback in the comments if you'd like more tech tips like this that aren't directly covering a product feature or solution, but reveal some utility that assists in learning how things tick.
tcl logic in SAML Attribute value field possible?
Hi. We're running BigIP as a SAML IDP. Can I somehow issue tcl logic in a SAML attributes? I'm talking about the Access ›› Federation : SAML Identity Provider : Local IdP Services, editing an object, under SAML Attributes. Based on what's in the memberOf attribute, I need to issue as a value either empty string or "SpecificValue". I am familiar with the %{session.variable} construct, but I don't want to clutter the session with more variables if I can avoid it, as that impacts all sessions using our IDP (30 or so federated services on the same VIP and AP). I tried these two approches: %{ set result {} ; if { [mcget {session.ad.last.attr.memberOf}] contains {| CN=SpecificGroup,OU=Resource groups,OU=Groups,DC=Domain,DC=com |}} { set result {SpecificValue} } ; return $result } expr { set result {} ; if { [mcget {session.ad.last.attr.memberOf}] contains {| CN=SpecificGroup,OU=Resource groups,OU=Groups,DC=Domain,DC=com |}} { set result {SpecificValue} } ; return $result } Expected result: An issued claim with the value "" or "SpecificValue" Actual result: An issued claim with the above code as the value As I mentioned, we've set it up using one VIP that is hosting 30 or so services. We're running 16.1.3.1. They are using the same SSO configuration and there's an iRule triggerd at ACCESS_POLICY_AGENT_EVENT, which does some magic to extract issuer and suchlike, and that helps to make decisions later in the Access Policy. It also populates a few session variables under the session.custom namespace for use in the Access Policy. Additional session variables are being populated in the Access Policy, such as resolved manager and their email address. I have looked briefly at the ASSERT::saml functions, but even if it would bepossible to manipulate that way, I wish to keep this set up as stream lined as possible and with as few new "special cases" in an iRule. So while I appreciate pointers along that route as well, I would first of all like to know if there is a way to do it natively in the SAML attribute value field. And if there are any options I have not yet explored here?Branch evaluation fails with Rule evaluation error: invalid command name "session.logon.last.username"
Hopefully somebody is able to enlighten me, I have the code below in a branch rule expression of an AD Query element. I get an 'Rule evaluation failed with error: invalid command name "session.logon.last.username"' error. The purpose is to translate an email entered to the matching AD logon username from AD. The AD query succeeds (mail=%{session.logon.last.username}). It looks like the variable assign element is not able to change the session.logon.last.username variable. if { [mcget {session.ad.last.queryresult}] == 1 } { session.logon.last.username = mcget {session.ad.last.attr.sAMAccountName}; return 1; }; return 0; Thanks for sharing your thoughts / ideas.
iRule to drop Public IP access to DNS Wide-ip
We have wide-ip "abc01.example.com" which we want only to access from internal user (private ip) Do we have iRule to drop only public ip and allow private ip on that wide-ip? So when nslookup to "abc01.example.com" via public ip, F5 drop and when nslookup to "abc01.example.com" via private ip, F5 allow Thank you
iRule Encode special charaters in URI String
I have an iRule, which is capturing the current URI, and appending it as a variable onto another URL string later on. when HTTP_REQUEST { ... ... set host [HTTP::host] set uri [HTTP::uri] set url "http://internal-address.local.lan/index.html?aup-key=$aupkey&$schema&$host&$uri" } However, if the original URI contains an ampersand itself, it's truncating it at that character so that the full original URI never gets passed on. I'm not overly familiar with TCL, is there an easy way I could encode the uri variable in my code above so that it includes all special characters as well? Many thanks
