My issue is that I do not know exact XML node names, like GetFolder or FolderShape. I would like to iterate SOAP body and receive node names to analyze them
My setup uses Layered-VS scenario (LTM_VS forwarding to APM_VS with help of simple iRule). APM_VS is used for Exchange Server publication (EWS protocol mainly). LTM_VS has datasafe and stream profiles attached. There is another iRule also which is doing simple replace string1 with string2 under HTTP_REQUEST event (this iRule is attached to LTM_VS)
Seems that after enabling stream engine with STREAM::enable under HTTP_REQEUST any Content-Length header is replaced with Transfer-Encoding: chunked. According to F5 docs this is normal behavior. Unfortunately this setup breaks EWS protocol
I compared TCPDUMP captures from working and non-working scenario. And found that Exchange server could answer to authentication requests with Transfer-Encoding: chunked but will not answer to EWS SOAP payload after successful authentication. If EWS SOAP payload came with Content-Length header everything is fine
So my question is: how can I use stream profile and unchunk resulted HTTP packet to be sent to Exchange with Content-Length header instead of Transfer-Encoding: chunked?
I tried adding this iRule to APM_VS under HTTP_REQUEST but nothing changed. TCPDUMP showed that Transfer-Encoding: chunked is still present in the packet that goes to Exchange server
STREAM is designed to modify TCP::payload on a stream of "just enough" segments, to find and replace payload data.
This behavior may break the Content-Length mechanic of HTTP requests, if you're going to change the total length of the HTTP payload.
F5 added changes to STREAM / HTTP profile combination, so that classic Content-Length mechanic (length is included in the header) will be turned into a chunked "Transfer-Encoding" mechanic (length is included in each chunk), to support the STREAM'ing on "just enough" HTTP chunks for find and replace operations. In the case that chunked payload gets changed and resulting in a different chunk size, the length field of the currently processed chunk gets adjusted...
But yeah... you only want to log and not adjust anything, isn't it?
AFAIK did F5 not implemented a setting to explicitly turn off this behavior for STREAM / HTTP traffic. But you may clarify this with F5 Support to get sure...
If the behavior can't be turned off, then you may add a 3rd VS-Layer in your chain to STREAM process certain URIs.
The VS_STREAM will only get a TCP and STREAM Profile assigned, but not a HTTP Profile. Setup your expressions during CLIENT_ACCEPTED and then do some fancy stuff on STREAM_MATCHED events. But keep in mind to not change anything... just analyse the TCP stream and log your expressions...
when RULE_INIT priority 1 {
set static::stream_expression "@</t:\[^>\]*>@@"
set static::already_known_xml_nodes ""
}
when CLIENT_ACCEPTED {
virtual VS_APM
STREAM::expression $static::stream_expression
STREAM::enable
}
when STREAM_MATCHED {
if { [STREAM::match] starts_with "</t:" } then {
set temp(node_name) [string range [STREAM::match] 2 end-1]
if { [lsearch $static::already_known_xml_nodes $temp(node_name)] == -1 } then {
log local0.debug "XML Parser: Learned a new node name \"$temp(node_name)\" on TMM=[TMM::cmp_unit]."
append static::stream_expression "@\<$temp(node_name).*\<\/$temp(node_name)\>@@"
lappend static::already_known_xml_nodes $temp(node_name)
} else {
# t: XML Node is already trained...
}
} else {
log local0.debug "XML Parser: [STREAM::match]"
}
STREAM::replace
}
The iRule analyses TCP payload via STREAM and searches for </t:nodename> strings. If a yet unknown </t:nodename> is detected, the iRule will update the STREAM::expression to include a regex pattern for <t:nodename>.*</t:nodemand>. On the very next TCP connection the iRule will start log the just learned XML node name...
<STREAM_MATCHED>: XML Parser: Learned a new node name "t:BaseShape" on TMM=0.
<STREAM_MATCHED>: XML Parser: Learned a new node name "t:DistinguishedFolderId" on TMM=0.
<STREAM_MATCHED>: XML Parser: Learned a new node name "t:BaseShape" on TMM=1.
<STREAM_MATCHED>: XML Parser: Learned a new node name "t:DistinguishedFolderId" on TMM=1.
<STREAM_MATCHED>: XML Parser: <t:BaseShape>IdOnly</t:BaseShape>
<STREAM_MATCHED>: XML Parser: <t:DistinguishedFolderId Id="root">1</t:DistinguishedFolderId>
<STREAM_MATCHED>: XML Parser: <t:BaseShape>IdOnly</t:BaseShape>
<STREAM_MATCHED>: XML Parser: <t:DistinguishedFolderId Id="root">1</t:DistinguishedFolderId>
Thanks. Do I understand your idea correctly? Because you are using expandable static variable it is possible to access found XML node names using other iRules attached to another VS?
Do you have any ideas if we need to find&erase selected XML nodes in HTTP traffic using STREAM profile? And of course we need to return Content-Length header 🙂 Maybe I have to manually do:
1. Remove Transfer-Encoding header
2. Calculate HTTP payload length and insert Content-Length header
The static::variables are just a store for the STREAM expressions. They are maintained on each TMM core independently (for performance reasons) and adjusting the STREAM parsing over the time.
Its more or less just a gimmic to get it the expressions up an running without knowing the node names in advance. Once you figured out the names of the XML nodes you want to parse, you should change the logic to use fixed expression strings for STREAM.
To delete/change an entire [STREAM::match] your could call [STREAM::replace] with additional parameters.
And unfortunately i dont have a working solution to unchunk a HTTP request before its hits your server.
Resolving the chunked encoding would need to [HTTP::collect] the entire request and parse the chunks from the first to the last chunk while stripping the length fields to restore the Content-Length mechanic.
It would be far easier to avoid the chunks to begin with and directly use [HTTP::collect] to parse your XML... 😉
