iRule [string range...] not chunking data properly
I have an irule (much of which I found here) that is to gather some much needed troubleshooting data I need in regards to the headers and payload I have coming in. The payload is obvisouly too large for a single log line so this was supposed to chunk it into manageable bytes. I had to make some modifications to the original irule which did not work, but now it is working, sort of. It logs the first 900 bytes as it should, then something happens and skips a bunch, and then logs the final bytes of data. I can't understand why it's not grabbing either the proper amount of data or failing to output this second chunk of missing data before making its final loop. iRule is here: when RULE_INIT { # Log debug to /var/log/ltm? 1=yes, 0=no set static::payload_dbg 1 # Limit payload collection to 5Mb set static::max_collect_len 5368709 # Max characters to log locally (must be less than 1024 bytes) # https://clouddocs.f5.com/api/irules/log.html set static::max_chars 900 set static::min_chars 0 } when HTTP_REQUEST { # Only collect POST request payloads if {[HTTP::method] equals "POST"}{ if {$static::payload_dbg}{log local0. "POST request"} # Get the content length so we can request the data to be processed in the HTTP_REQUEST_DATA event. if {[HTTP::header exists "Content-Length"]}{ set content_length [HTTP::header "Content-Length"] } else { set content_length 0 } # content_length of 0 indicates chunked data (of unknown size) if {$content_length > 0 && $content_length < $static::max_collect_len}{ set collect_length $content_length } else { set collect_length $static::max_collect_len } if {$static::payload_dbg}{log local0. "Content-Length: $content_length, Collect length: $collect_length"} foreach aHeader [HTTP::header names] { log local0. "HTTP Request Headers: $aHeader: [HTTP::header value $aHeader]" } #set payload [HTTP::collect $collect_length] HTTP::collect $collect_length } } when HTTP_REQUEST_DATA { # Log the bytes collected if {$static::payload_dbg}{log local0. "Collected [HTTP::payload length] bytes"} # Log the payload locally if {[HTTP::payload length] < $static::max_chars}{ log local0. "Payload=[HTTP::payload]" } else { # Initialize variables set remaining [HTTP::payload] set position 0 set count 1 set bytes_logged 0 # Loop through and log each chunk of the payload while {[string length $remaining] > $static::min_chars}{ #chunk = 899 set chunk [expr {$position + $static::max_chars -1}] log local0. "Chunk: $chunk" log local0. "Position start: $position" # Get the current chunk to log (subtract 1 from the end as string range is 0 indexed) set current [string range $remaining $position $chunk] log local0. "chunk $count=$current" log local0. "Current chunks bytes: [string length $current]" # Add the length of the current chunk to the position for the next chunk incr position [string length $current] # Get the next chunk to log set remaining [string range $remaining $position end] incr count incr bytes_logged $position log local0. "remaining bytes=[string length $remaining], \$position=$position, \$count=$count, \$bytes_logged=$bytes_logged" } if {[string length $remaining]}{ log local0. "chunk $count=$current" incr bytes_logged [string length $remaining] } log local0. "Logged $count chunks for a total of $bytes_logged bytes" } } when HTTP_RESPONSE { foreach aHeader [HTTP::header names] { log local0. "HTTP Request Headers: $aHeader: [HTTP::header value $aHeader]" } } The image below shows what I'm talking about - it even says "Oh I found 3 chunks", but I'm only being presented 2. I added some logging around the position, chunk value (aka end position), and bytes logged to help illustrate that it's clearly skipping over something. Any help would be appreciated! Thanks
Simple iRulesLX JSON rewrite
Problem this snippet solves: Like rewriting the Location field in a redirect, it's sometimes required to rewrite JSON data, returned in an HTTP response. Whilst it would be possible to write using traditional iRules, the task is made more simple (and less risky) by using iRulesLX. In the following example, an HTTP response contains JSON data with fields containing external URLs. These need to be rewritten to an internal URL for the purpose of internal routing. { "firstUrl":"https://some.public.host.com/api/preauth/ABCDEFGHIJKLM", "secondUrl":"https://some.public.host.com/api/documents/{documentId}/discussion" } The concept can be used to rewrite any JSON data, however more complicated JSON containing arrays for example would need to be taken into consideration. How to use this snippet: use the following iRule to call iRulesLX and pass the necessary parameters when CLIENT_CONNECTED { set newHost "internal.host.local" set jsonKeys "firstUrl secondUrl" set rpcHandle [ILX::init "json-parse-plugin" "json-parse-extension"] } when HTTP_RESPONSE { if {[HTTP::header "Content-Type"] eq "application/json"} { HTTP::collect [HTTP::header "Content-Length"] } } when HTTP_RESPONSE_DATA { set payload [HTTP::payload] set result [ILX::call $rpcHandle "setInternalUrl" $payload $jsonKeys $newHost] HTTP::payload replace 0 [HTTP::header "Content-Length"] $result } When used in combination with the iRulesLX code below the host portion of the URIs in the JSON data are rewritten and sent back to the origin by replacing the HTTP payload { "firstUrl":"https://internal.host.local/api/preauth/ABCDEFGHIJKLM", "secondUrl":"https://internal.host.local/api/documents/{documentId}/discussion" } Code : const f5 = require('f5-nodejs'); const url = require('url'); const ilx = new f5.ILXServer(); function setInternalUrl(req, res) { var json = JSON.parse(req.params()[0]); var jsonObj = req.params()[1].split(' '); var newHost = req.params()[2]; for (var i = 0; i < jsonObj.length; i++) { if (typeof json[jsonObj[i]] == "string") { var oldUrl = url.parse(json[jsonObj[i]]); oldUrl.host = newHost; var newUrl = decodeURI(url.format(oldUrl)); json[jsonObj[i]] = newUrl; } else { json = {"error":"unable to rewrite"}; } } res.reply(JSON.stringify(json)); } ilx.addMethod('setInternalUrl', setInternalUrl); ilx.listen(); Tested this on version: 12.1
FTP Proxy , user based forwarding with iRule
Im looking into using a VS as an FTP Proxy towards my backend webserver. Currently i already works for one webserver behind the VS. Im now trying to write an iRule to forward traffic to the right server based on the username the ftp-connection is initiated with. Secondly i need to rewrite the username and remove the web-paramater Example: USER1 connects to with following credential USER1@WEB1 @WEB1 is removed from the credential and forwarded towards pool WEB1 Anybody any ideas how to do this last part ?
Inspect POST Request for Existence of Username Parameter
Is it possible to to check if a username has been provided in a POST request? Could this be done via HTTP::username command or would a HTTP::collect be needed to inspect the payload of the request? Would you be able to provide an example of how this might be done via an Irule.iRule is not working, HTTP::payload ?
Am trying to check http traffic if it has a specified string, and pass it, or block it if not. but this traffic is confined to one network. this is the iRule: when HTTP_REQUEST { if { [IP::addr [IP::client_addr] equals 10.0.0.0/24] } { HTTP::collect 20 set HTTP::payload [string tolower HTTP::payload] if { not ( [HTTP::payload] contains "ivrTransactionID" )} { reject } HTTP::release } }Read or Modify SOCKS payload
Hello Devcentral, even though apparently not much people is playing with the new SOCKS features, here I am with another question on this subject. Can i modify (or read) the SOCKS payload of my clients' requests ? When i dump the full TCP::payload in the CLIENT_DATA event, all i can see is the client's greeting that is part of the SOCKS handshake: 050100 I understand there is a socks-tunnel interface and that the traffic (after the handshake has completed) is likely to be processed at this level but is there a way to read or modify the socks payload from an iRule ?
Modify SOAP To header based on target node
Hi, I have 2 kinds of virtual servers set up. The first has no pools assigned. It takes requests over SSL, terminates the SSL and routes the requests (in plaintext) to wholly different pools based on content. However, the SOAP To header from these requests needs to be modified. The protocol needs to be changed from https to http, the service URL is suffixed with a designator used for services without transport security and the port needs to be changed from 8443 to some other port. It's the "some other port" that is causing me some issues. Following is my iRule on the SSL-terminating, content-based routing virtual server: when HTTP_REQUEST { if { [HTTP::method] eq "POST" } { if { [HTTP::header exists "Content-Length"] } { set content_length [HTTP::header "Content-Length"] } else { set content_length 1048576 } HTTP::collect $content_length } else { HTTP::respond 405 content "Unsupported" Allow "POST" } } when HTTP_REQUEST_DATA { if { [HTTP::method] eq "POST" } { set payload [HTTP::payload] binary scan [sha1 [SSL::cert 0]] H* certHash set newSoapHeader "" if { [info exists certHash] } { append newSoapHeader $certHash } else { append newSoapHeader "cert_hash_retrieval_failure" } append newSoapHeader "" Try to stick the new header in the header collection set numMatches [regsub {.*\<[A-Za-z0-9:]*Header[^\>]*\>} $payload [concat {&} $newSoapHeader] modifiedPayload] if no matches from that, we know there was no header and, therefore, no subsitution. We need to introduce the whole Header block if { $numMatches == 0 } { set contentToInsert "" append contentToInsert $newSoapHeader append contentToInsert "" set numMatches [regsub {.*\<[A-Za-z0-9:]*Envelope[^\>]*\>} $payload [concat {&} $contentToInsert] modifiedPayload] } Empty it of content first HTTP::payload replace 0 [string length [HTTP::payload]] "" Then replace the empty payload HTTP::payload replace 0 0 $modifiedPayload set currentVirtualServer [virtual name] if { [string match *LEGACY* $currentVirtualServer] > 0 } { Replace the HTTP Location Header HTTP::header replace Location [string map { "https://" "http://" ".svc" "_U.svc" } [ HTTP::header Location]] Replace the SOAP To Header set badToHeaderPattern ".svc" set lenBadHeaderPattern [string length $badToHeaderPattern] set betterToHeaderPattern "_U.svc" set offset [string first $badToHeaderPattern [HTTP::payload]] if { $offset >= 0 } { HTTP::payload replace $offset $lenBadHeaderPattern $betterToHeaderPattern set badToHeaderPattern "https://" set lenBadHeaderPattern [string length $badToHeaderPattern] set betterToHeaderPattern "http://" set offset [string first $badToHeaderPattern [HTTP::payload]] if { $offset >= 0 } { HTTP::payload replace $offset $lenBadHeaderPattern $betterToHeaderPattern } NOW REPLACE THE 8443 PORT WITH... um... } } the trailing arguments align with the parts of the regex in parentheses. First is always the whole match, subsequent per parentheses pair regexp {(\<[A-Za-z0-9:]*Organisation[^\>]*\>)([A-Za-z]*)} $modifiedPayload wholeMatch xmlElementMatch organisationNameMatch if { [string length $organisationNameMatch] > 0 } { string map takes a list of replacement pairs (e.g [list needle1 replace1 needle2 replace2... needleN replaceN]) set targetPool [string map [list -D--- --- MULTI $organisationNameMatch RELEASE RELEASE-U DEBUG DEBUG-U LEGACY ""] $currentVirtualServer] pool $targetPool } else { HTTP::respond 400 content "Unknown" } } Ensure that the request is released back to F5 so it may take control of the underlying connection and complete the routing HTTP::release } I'll detail the HTTP_REQUEST if requested, but it's pretty much stock-standard stuff. All the magic is in the HTTP_REQUEST_DATA handling. I know a lot of my SOAP content replacement is really risky and needs better pattern matching/controls so I don't replace anything I shouldn't, so don't focus on that. But after I switch out the https with http in the SOAP To header (really every instance of https in the whole payload), I want to change the port 8443 to another port. The other port, however, is not known until I route it to the pool later (the "pool $targetPool" in the last couple of lines). That routing works just fine and the correct pool gets the requests and processes them. But I can't seem to manipulate the payload any further after I set the new pool. That means I can't, say, get "[LB::server port]" and replace the mentions of 8443 with it. It looks like I don't have it available to manipulate on the LB_SELECTED event either. So, how do I modify the port in the SOAP To Header (also need to modify the HTTP Location header) with a port I will only know after I've re-routed the request with the "pool" keyword?
regex irule without payload change
Hi, I try to write an irule which logs client ip with posted username on a json request. I can able to gather full post payload but that includes the password and other info so i generate a regex to only write the client ip and username on the log file but when i use regex vserver starts to reset connection. If i avoid regex and collect ip and full json payload no problem. Can anyone help me regarding this example? Appriciated. when HTTP_REQUEST { if { [HTTP::method] eq "POST" and [HTTP::uri] starts_with "/api/test/login" } { set log_msg "" set client_ip [IP::remote_addr] set paypay [HTTP::payload] set pay [HTTP::payload] append log_msg "client_ip=$client_ip " append log_msg [regexp {(?<=\{"username":").*?(?=",)} $paypay] append log_msg "$paypay" log local0. $payload log 1.1.1.1 local0. $log_msg } }
iRule to rewrite payload to HTTPS
Hi, I'm working on making a website secure - the site is pretty archaic and has hard coded references to objects under http:// paths. I need to rewrite the payload so any "; references become ";. I've tried two different methods for doing this: The first one was to use the STREAM function. when HTTP_RESPONSE { if { !$manipulate } { return } if { [HTTP::is_redirect] } { HTTP::header replace Location [string map { "http://" "https://" } [HTTP::header Location]] } STREAM::disable Apply stream profile against text responses from the application if { [HTTP::header value Content-Type] contains "text" }{ Look for http:// and replace it with https:// STREAM::expression {@http://@https://@} Enable the stream profile STREAM::enable } } This pretty much worked, but I noticed that each time the URL fired, it didn't modify the content length. This meant the below HTML: http://www.google.com http://www.bbc.co.uk http://www.google.com http://www.bbc.co.uk http://www.google.com http://www.bbc.co.uk http://www.google.com 9876543210 ...correctly had each link amended to https://www.google.com, but ended at 9876 and lost the remaining numbers (ie, one character was removed for each subsitution). A colleague then suggested I tried the HTTP Payload replace function - this seemed to work pretty well on basic HTML (ie, my google/bbc page above loaded perfectly), although it seemed to balk at more complicated pages (ie, large JSPs). I have an error in ltm: May 25 13:19:43 longgos-dc-comm-lb-bip-02a err tmm1[21928]: 01220001:3: TCL error: /Nova/stream_development3 - Out of bounds (line 1) invoked from within "HTTP::payload replace 0 $content_length $newdata " I believe this may be due to the fact our content is encoded as UTF-8 - there is a caveat on the Wiki page for HTTP::payload: Note that the argument will be interpreted as a byte array. If it is actually a UTF-8 string with multibyte characters, the output will not be what you expect. In order to prepare a UTF-8 string for use as input to HTTP::payload replace, you should first run 'binary scan c* throwawayvariable'. But I don't know where to put this directive. Regardless of whether I put it in the HTTP_REQUST, HTTP_RESPONSE or HTTP_RESPONSE_DATA sections, all connections to my VIP fail. Has anyone had any success implementing this sort of irule?
