cancel
Showing results for 
Search instead for 
Did you mean: 
Deb_Allen_18
Historic F5 Account

LTM's built-in stream profile (and related iRules commands) offer a relatively low-overhead method to replace specific strings in the server response data. 

One of the most common uses for stream replacement is to re-write server-referencing links in HTTP payload -- re-write to https when using SSL acceleration, re-write the hostname to the virtual server hostname, or rewrite the the URI to match a customer-facing scheme.  But what if you need to replace multiple strings with the same replacement text, or replace more than one string with the same string (i.e. rewrite URI's for 2 different paths names in the same response)?

Since only one stream operation can be performed on any given response, you might assume it can't be done with the stream profile.  Not so: The stream profile parameters can be configured so a single stream operation will perform multiple replacements against the same data, or to use basic regular expressions to find the strings you need to replace.

Multiple stream replacements using the stream profile
To replace multiple strings in server-returned content using the stream replacement feature, start by creating a custom Stream profile (Local Traffic / Profiles / Other / Stream).   Leave the "Source" field blank.  In the "Target" field, enter each string you'd like to replace followed by the replacement text  in the following format:
 
<d><search1><d><replace1><d><d><search2><d><replace2><d>...

The first character in the field (<d>) defines the delimiter bounding each field for this replacement, and must not appear anywhere else in the target string.  <d> can be any one of the following characters:

.*/-:_?=@,&
    (period, asterisk, forward slash, dash, colon, underscore, question mark, equals, at, comma, ampersand)

So to replace all instances of "string1" with "string2", and all instances of "string3" with "string4", you could use this syntax for the Target field:
 
/<string1>/<string2>/ /<string3>/<string4>/

Multiple stream replacements using an iRule

To perform multiple search & replace operations with an iRule instead of the stream profile settings (very useful if different conditions dictate different replacements), use the same syntax with the STREAM::expression command:

when HTTP_RESPONSE {
     if {[HTTP::status] ==200}{
       STREAM::enable
       STREAM::expression \
            "@A phrase;@is replaced by this phrase:@ @something else@être remplacé par pourtant une plus longue corde des textes dans une autre langue@"
     }
   }


Using regular expressions for stream replacement
Regular expressions (basic regex ONLY) are supported in the search string portion of the Target parameter. Here's an example that looks for patterns matching IP addresses and replaces each one with 0.0.0.0 :

@\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}@0.0.0.0@

To use this approach in an iRule, you'll need to enclose the Target string in { } instead of " " when defining the STREAM::expression parameter:

STREAM::expression {@\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}@0.0.0.0@}

(Thanks to hoolio for this example.)

You can also combine regular expressions and multiple searches in the same Target string:

@\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}@0.0.0.0@ @http://@https://@

More examples
These 2 examples both accomplish the same replacement using different delimiters:

,http://serverX.domain.com/,https://VirtualServer.domain.com/,,http://serverY.domain.com/,https://VirtualServer.domain.com/,
and
@http://serverX.domain.com/@https://VirtualServer.domain.com/@@http://serverY.domain.com/@https://VirtualServer.domain.com/@

(Note that  /  .  and  :  would NOT be appropriate delimiters for this replacement, since they already appear in the Target string.)

To replace embedded email addresses, you can use any delimiter but "." or "@", with the resulting Target string:
&j.karlsson@dontsendmespam.com&jonas@yourhost.com& &d.allen@dontsendmespam.com&deb@yourhost.com&

Caveats & tips

  • LTM v9.1 is required to use the built-in stream profile for multiple replacement strings.
  • LTM v9.2 is required to use the iRules STREAM::expression command for multiple replacement strings.
  • A stream profile must be associated with the virtual server before an iRule calling the Stream profile can be associated with the virtual server.  The default stream profile (with all parameters blank) may be used.
  • The stream profile performs all replacements in single pass, so data changed once will not be changed again in the same operation even if it matches another replacement pattern after the initial replacement.
  • On LTM versions prior to 9.4, if the HTTP content length may change as a result of the stream replacement, you must enable a HTTP profile with Response Chunking set to Re-chunk to accommodate the content length change on the fly. Otherwise, LTM will forward the now-incorrect Content-Length header, which can cause the browser to hang, display partial pages, or otherwise act funky.
  • Since HTTP compression on the server will prevent LTM from reading the data stream and performing the intended replacement, you may need to use LTM's http profile settings to remove the Accept-Encoding header in the request to prevent compression by the server.
  • The stream profile is case sensitive. Basic regex may do the trick if case insensitivity is required.
  • Start, Separate, AND End each find/replace pair with an appropriate delimiter. (There will be 2 delimiters between pairs.)
  • Leave the "Source" field BLANK when configuring the profile.
  • Optional: Insert whitespace between search/replace pairs -- behavior is the same, improves readability. (Do NOT insert extra whitespace within the replacement pairs.)
Comments
Nicolas_MENOUX_
Nimbostratus
Nimbostratus
stream profile is really useful and efficient. Before using that profile, I set up an IRule for replacing HTTP::payload based on script I've found on DevCentral. However, when I bind it to all of my virtual servers, my BigIP became really weird and charts of active sessions became like a jigsaw... I've looked the log and I've seen entries like below :

 

 

Sep 12 18:08:12 lb lacpd[1045]: 01160015:3: tmm transmit failed: Network is unreachable (port 1)

 

Sep 12 18:08:13 lb LOGIN: Re-starting tmm

 

Sep 12 18:08:13 lb lacpd[1045]: 01160015:3: tmm transmit failed: Network is unreachable (port 1)

 

Sep 12 18:08:13 lb bigd: 01060004:5: Monitoring stops - tmm down.

 

Sep 12 18:08:13 tmm tmm[28107]: 01010001:5: default TMM Version 9.2.3.33.1.5 starting

 

Sep 12 18:08:13 lb mcpd[1089]: 01070408:5: Deleting abandoned subscriber connection for TMM_Subscriber

 

Sep 12 18:08:13 lb mcpd[1089]: 01070406:5: Removed publication with publisher id TMM

 

Sep 12 18:08:13 lb mcpd[1089]: 01070409:5: Add a new Subscription for subscriberID TMM_Subscriber and filterType 0xefdff3ff

 

Sep 12 18:08:13 lb mcpd[1089]: 01070404:5: Add a new Publication for publisherID TMM and filterType 0xb130eed9

 

Sep 12 18:08:14 lb lacpd[1045]: 01160016:6: tmm transmit resumed

 

Sep 12 18:08:15 sccp bcm56xxd[363]: 012c0015:6: Link: 4.1 is DOWN

 

Sep 12 18:08:18 sccp bcm56xxd[363]: 012c0015:6: Link: 4.1 is UP

 

Sep 12 18:08:25 lb bigd: 01060003:5: Monitoring starts - tmm up.

 

 

... goshhh really bad...

 

I do not have the explanation of such behaviour but by using stream profile, I do not have such troubles.. If someone ahs already encountered such behaviour, please contact me to see if something was wrong with my script.

 

 

Regards,

 

 

Nicolas
hooleylist
Cirrostratus
Cirrostratus
It might be helpful to note that by default, a stream profile is applied to both request data and response data. So for HTTP, be aware that POST requests with data could be modified if there is a match. You might want to use an iRule so you can specify that the replacement is only done in the HTTP_RESPONSE event.

 

 

Also, streams only modify the data. So if you want to make replacements in the request or response headers you can use 'HTTP::header replace'.

 

 

Aaron
Deb_Allen_18
Historic F5 Account
Thanks for the feedback, guys!

 

 

- Nicolas: HTTP payload replacement using the HTTP::collect command is much more resource intensive than the stream profile. The collect commands buffer the entire response data set in memory, then some operations may expand up to 4x original size due to internal charset conversions, which may cause memory exhaustion under load and force TMM to restart (as seen in your log snip). Since you mention the iRule was applied to several virtual servers, that is likely what you experienced. Good to hear that the stream profile helped solve the problem

 

 

- Aaron: Last time I tested, the stream profile affected only response data. I am checking with Support regarding request data and to find out if there is any version-specific info we should know.

 

 

peace

 

/deb
Nicolas_MENOUX_
Nimbostratus
Nimbostratus
Thanks /deb for your explanations of our BigIP weird behaviour ! Script was ok but memory and CPU could not follow....

 

 

Bonne journée 😉

 

Kirk_Bauer_1018
Nimbostratus
Nimbostratus
I have confirmed that in 9.4.3 HF3 the stream profile does modify the body of HTTP requests (i.e. POST data). Not sure if this was the case in earlier versions.

 

 

Also it appears as though regular expressions can't be entered via the GUI, or if they can it might require careful escaping of characters.
Kirk_Bauer_1018
Nimbostratus
Nimbostratus
I spoke too soon -- in 9.4.3 it does modify the POST body of HTTP requests but it does not modify the content-length header.
brad_11480
Nimbostratus
Nimbostratus
so how do you specify characters that are not visible? I need to replace hex 0a with hex 0d hex 0a (LF with CRLF). i tried :/n:/r/n: but didn't do anything. thanks.

 

John_Gruber_432
Historic F5 Account
What is '(basic only)' when it comes to regex. (?i) - case insensitivity does not seem to work. Can we definitively say what does?
John_Gruber_432
Historic F5 Account
Additional comment..... In the hints and tricks we speak of removing the Accept-Encoding header from the client request to prevent the backend server from compressing the content and making it invisible to the stream profile. Having done this and them attempting to get the BIG-IP to compres the data towards the client with COMPRESS::enable does not seem to work. If I eat the frontend Accept-Encoding I can not seem to get the BIG-IP compression profile to work. Can we post an example of 1) removing the client Accept-Encoding header, 2) stream replacement, 3) COMPRESS::enable for compression towards the client.. showing all three working at once. I could not get this to work.
John_Gruber_432
Historic F5 Account
I'll comment on my own comment.... sorry guys..

 

 

Basic regex means BRE POSIX I'm assuming. I found a reference to BRE commands and they all seem to work decently.

 

 

We need more examples that these. I had to do relatively link rewrites for 26 different HTML object attributes today and it was not obvious by any means.
hooleylist
Cirrostratus
Cirrostratus
Hi jgruber, you might try posting in the iRules section to get more relevant eyes on your post.

 

 

Aaron
tsoi_94459
Nimbostratus
Nimbostratus
Starting in BIG-IP LTM versions 9.3.1 and 9.4.2, the Stream profile must be enabled after the stream expression is first evaluated. This step allows the BIG-IP iRule system to evaluate the expression first before allowing the Stream profile to be enabled. This behavior protects from misconfiguration, which could result in a crash.

 

 

For example:

 

 

when HTTP_RESPONSE {

 

STREAM::expression "@abc@123@"

 

STREAM::enable

 

}

 

 

If the Stream profile is enabled before the expression is evaluated, an error message will be reported to the /var/log/ltm file that appears similar to the following example:

 

 

tmm tmm[1882]: 01220001:3: TCL error: Rule stream_irule - Operation

 

not supported (line 1) invoked from within "STREAM::enable"

 

 

This error message is reported in BIG-IP LTM versions 9.3.1, 9.4.2, and later when the following iRule is used:

 

 

when HTTP_RESPONSE {

 

STREAM::enable

 

STREAM::expression "@abc@123@"

 

}

 

 

http://support.f5.com/kb/en-us/solutions/public/8000/200/sol8207.html
JackF
F5 Employee
F5 Employee

 

Under the section: Multiple stream replacements using an iRule

 

 

The syntax should be:

 

 

when HTTP_RESPONSE {

 

if {[HTTP::status] == 200}{

 

...

 

 

Just wanting to save others time figuring it out 🙂 This article was a big help.
Colin_Walker_12
Historic F5 Account
Good catch @JackF! Updated.

 

 

Colin
Yaniv_Eitan
F5 Employee
F5 Employee

All - This is an old article for old versions.

Please use this one instead to avoid confusions:

https://clouddocs.f5.com/api/irules/STREAM__expression.html

 

Version history
Last update:
‎12-Sep-2007 13:00
Updated by:
Contributors