on 25-Sep-201416:34 - edited on 05-Jun-202322:13 by JimmyPackets
Yesterday, NIST released information on a new network exploitable vulnerability in the GNU Bash shell as demonstrated by vectors involving parts of OpenSSH sshd, the mod_cgi, and mod_cgid modules in the Apache HTTP Server, scripts executed by DHCP clients, and other situations where setting the environment around a bash process occurs. Vulnerability CVE-2014-6271, or Shellshock as everyone is calling it, allows remote attackers to execute arbitrary code on the target systems. Redhat has a great overview in their security blog posting on the topic.
What makes this vulnerability serious is the fact that server-side CGI applications running under the Apache Web Server with mod_cgi or mod_cgid are susceptible.
F5's Jeff Costlow is maintaining an article on the Shellshock vulnerability so make sure you check back to that article for F5's official stance.
The first step to mitigation is to setup a plan to patch the bash shell on all of your systems. In the interum, if you believe that any of your backend servers are vulnerable (if they are Unix-based then they likely are) and you are fronting them with a BIG-IP, below is an iRule solution you can apply to your Virtual Servers that will protect your servers against attacks.
Block-Shellshocked
The Block-Shellshock iRule searches for the pattern "*() {*" in the URI and in the HTTP headers. It is rare that this pattern of characters will occur in a URI or HTTP Header so we believe the number of false-positives will be minimal. If the pattern is found, an audit entry will be written to the system log with the client ip as well as the URI, and optionally the HTTP header, that the attack was detected. I chose to issue a reject on the connection. If you want to be more polite to the hackers, you can substitute the "reject" with a 403 - Forbidden.
when HTTP_REQUEST {
set pattern
"*() \{*"
;
if { [string match $pattern [HTTP::uri]] } {
log local0. "Detected CVE-2014-6271 attack from '[IP::client_addr]' in URI '[HTTP::uri]'";
reject;
} else {
foreach header_name [HTTP::header names] {
foreach header_value [HTTP::header values $header_name] {
if { [string match $pattern $header_value] } {
log local0. "Detected CVE-2014-6271 attack from '[IP::client_addr]' in HTTP Header $header_name = '$header_value'; URI = '[HTTP::uri]'";
reject;
break;
}
}
}
}
}
Block-Shellshock-full
The first iRule gives a more detailed audit if the attack is detected. In the case where that is not a concern and you are more concerned with performance, the HTTP::request method can be used to perform a single search across all of the request header information. This does not cover the POST body, but at this point there is no indication that the exploit is exposed within the body portion of a POST request. This irule reports the client IP along with the requested URI that is the source of the attack.
when HTTP_REQUEST {
if { [string match
"*() \{*"
[HTTP::request]] } {
log local0. "Detected CVE-2014-6271 attack from '[IP::client_addr]'; URI = '[HTTP::uri]'";
reject;
}
}
We will be monitoring this vulnerability over the coming weeks and update the iRule solution if a more optimal one becomes available. Please leave a comment if you have any suggestions. And, make sure you monitor Jeff's article for updates.
Special Considerations
Please be aware that this iRule will reject any connections containing the string "() {" in the URI or header. While we are confident that this is not a valid pattern for standard headers, there could be situations where custom application values contain this string pattern and this iRule would cause issues to those applications. We suggest running the longer iRule and actively monitoring the system log for false positives. If you find a case where your custom application is being blocked, you can customize the first iRule to exclude the header your application relies on.
We have below User-Agent string, I know it does not match your pattern, but very close.
Is it possible that string pattern might reject a valid User-Agent ? RFC does not say you cannot have ( and ) and then {.
By the way, reject is much better than 403 error that I set.
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SIMBAR
={0D6844D1-DC9B-11E1-B086-00A0C6000000}; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.1; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET4.0C; .NET4.0E)
Block-Shellshock-full iRule contains errors in log sctring ($header_value, $header_value) 🙂 These variables do not exist.
I use:
If it is determined that the vulnerability exists in the payload , this iRule will inspect the entire HTTP request for the pattern.
But it reports only UserAgent value (most common attacks) and reject the source of the attack.
when HTTP_REQUEST {
set pattern "*(*)*\{*";
set uri [HTTP::header Host][HTTP::uri];
set UserAgent [HTTP::header "User-Agent"]";
if { [string match $pattern [HTTP::request]] } {
log local0. "Detected CVE-2014-6271 attack from '[IP::client_addr]' in uri = $uri and UserAgent = $UserAgent";
reject;
}
}
Are you sure the 2nd iRule will scan the POST data? The doc on HTTP::request seems to suggest it does not scan POST data.
"Returns the raw HTTP request headers as a string. You can access the request payload (e.g. POST data) by triggering payload collection with the HTTP::collect command and then using HTTP::payload in the HTTP_REQUEST_DATA event."
About this couple of iRules (which btw I've joined into a single one), I´m concerned about it´s impact on performance. I have several web portals with many users on everyone. If I check every request, and for every request I check it´s URI and (even more) every single header, I think this could lead to a very high resources consumption. Any of you have had any issue about performance after enable this irule (on several VS's)? Thanks in advance.
Thanks for the feedback everyone. It seems the most likely false-positive would be in the user agent. What if we modified the iRule to block all headers that match except the user agent. If the UA matches, then empty it out and send it on. That would allow the slim case of UAs that match the pattern through with the downside that the backend app wouldn't have the true UA. Is that acceptable?
@Carlos - As for the impact on performance, I went with a simple string match on the content. Regular expressions are a bit more costly but the glob-style matching in string match is much faster. Even with regular expressions, the impact should be minimal (in the microseconds for a single match). We have customers doing header string tests in production and have not had performance issues reported that I'm aware of.
@Ian - You are correct. HTTP::request only returns the request portion and not the body. I will update the article with that info. Thanks for keeping me honest!
As per the problem description, bash allows arbitrary code execution as mentioned below.
$ env var='() { ignore this;}; echo vulnerable' bash -c /bin/true
If one tries with any other combination other than "() {" , it does not execute the code. In that case, why do we need a regEx kind of check on combination of "(" and ")" and "{" appearing on other combinations?
As mentioned here by others, it might block legitimate users with user-agent headers containing ( ) { scattered in succession.
What is the problem with the solution of custom signature in the ASM – by rejecting “() {“ pattern in header values?
@Shibu - you bring up good points. At the time I wrote the article, we couldn't determine that the exploit was limited to a "() {..." pattern so I took the route of making the search broader to make sure all bases were covered. Since then, it has been suggested that some User-Agent strings may lead to false positives. We are investigating this now. As for ASM, if you are using that product, then a custom signature is a great idea. This solution was geared for customers who are not using ASM and have iRules fronting their servers.
On the User-Agent front, I just ran through all the registered user agent strings for Safari, Opera, Chrome, Firefox, IE and the googlebot and msnbot search crawlers (~4600 different UA strings) and there were zero matches based on the "*(*)*{*" pattern.
@Joe, if there are applications using custom headers and values include this combinations, it will block it. Though I can't comment on the variety of http application out there, it's possible. Other case is custom applications - any http interface they use have their own user agents and it is usually not found in the published, well known lists.
In any case, can you please comment on the use of custom signature in the ASM – by rejecting “() {“ pattern in header values? [for those has ASM].
Great discussion. On the concern of any custom header that might match, Joe's idea of emptying out that header or rewriting that header value would be good enough. Thank Joe for checking out patterns of user agent strings.
So far, we have blocked several requests from Netherland.
In doing some more testing I've believe I can further restrict my match pattern to "*()*{*;*}*". This should exclude most all User-Agent strings as I haven't found any with empty parenthesis followed by curly braces. I also plan on including URI decoding into the URI and header checks to try to make sure someone isn't encoding the values. I haven't been able to recreate the vuln by passing in encoded characters so I'm going to hold out on publishing that until I do. I would really like to hear of any false-positives anyone finds with the original match pattern.
@Joe - per your request for false-positives, we implemented the second iRule and discovered all traffic through our Big-IP was being detected. Here is an example of the URI it detected as an attack: URI = '/ourfirm/offices/national/Mogul%20Winners/_w/Slalom.com%20Project%20Team%20-%20Q3_PNG.jpg'
I'm going to ask the completely noob question, forgive my ignorance. The Management Website is also vulnerable, correct? Is it possible to add this iRule to that site?
@jpluna: An iRule can be applied to a virtual server only.
In case you manage your F5 via network interfaces (not the management interface) it is possible to use a virtual server (SSL-termination & -reencryption) pointing to a node (local self IP to be used for inband management) via iRule. I would avoid to share the same IP address for the virtual server and self IP.
Example below shows the required one-liner:
when HTTP_REQUEST priority 600 { node 10.131.131.174 }
You can add the other iRules to filter the requests.
@kollenh - After more feedback and recent findings on the vulnerability, I've modified the pattern to look for the exact string of characters "() {" within a header. We believe there are no standard cases where this would match outside of a custom application. I've tested against all known user agents for the major browsers with a zero false-positive match. Could you try it out with the latest, more restrictive, pattern? If you are still getting matches, you might want to attempt the first iRule and log out exactly which header was causing triggering the block.
---UPDATE--- After receiving feedback and finding more details on the vulnerability, we've restricted the search pattern to search for containing the string "() {". This should greatly reduce the minimal amount of false positives that were found by custom applications but keep in mind that if you are doing something custom, that this pattern may still match so we suggest you use the longer iRule for initial analysis and auditing. The shorter iRule can be then run if/when performance is something you are concerned about.
just one notice, When talking about IDS/IPS and similar we know that "workarounds" usually happens from the attacking "side". So when Irules does not do any decoding I guess that strings like this will pass the irule and still be vulnerible:
%28%20) %7b%20
as Gary noted, the pattern *(*)*\{* would end up with many false positives.
User agents, as stated, would be enclosed in parenthesis and then anything following it that uses a left curly brace would trigger. Example:
User-Agent: Mozilla/5.0 (Linux; Android 4.4.2; SM-G900T Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.117 Mobile Safari/537.36 Accept-Encoding: gzip,deflate Accept-Language: en-US,en;q=0.8 Cookie: _gat=1; MM_GUID=1ffa80b5-007f-8d02-fbe5-1c15bd314e65; MM_UUID=EEFC18F2-A109-434E-A7C6-5E4F0F400A3D; MM_DP={"bmi":"","gclid":"CIuggZOkicECFSyCMgoddiMAuw"};
..
@CodeZone - When you say your users could no longer connect to the internet, did you mean you applied this to a virtual server for outbound traffic? It should be applied to a virtual that is fronting a set of application servers.
@Beinhard - We tested decoding and I have an iRule ready to go if we find a way to execute this exploit when passing through encoded values. But, in all our testing, the encoded values are passed back directly to the server and then into Bash where it didn't seem to decode them. Header decoding is something that needs to be handled with the application itself as webservers typically don't change their contents. Have you been able to execute on this exploit by passing in encoded parameters in HTTP Headers? As for the URI, we have only been able to replicate the attack through HTTP::headers but we included the check there for good measure but we haven't been able to replicate with the URI.
@brad - that's why I updated the iRule to be a bit more restrictive on the match. After looking at the bash sources, it looks for the literal string so I changed the iRule to do the same.
On yesterday's guru panel, a user asked a question on how to most easily apply this iRule to 1000+ Virtual Servers. I wrote a little iControl perl script this morning that will apply attempt to apply an iRule (stored in a local file) to all the virtuals on your system. Make sure you test this out before using it but hope it helps those with very large configurations. https://clouddocs.f5.com/api/icontrol-soap/PerlVirtualShow.html
@Joe, i missed that you answered me, sorry for late feedback.
I´m also thinking of hex ascii characters, so for example using \x28..... would not match the irule.
I´m also thinking about XSS and infect the clients to the "website" and similar attackvectors.
I think we will see quite a lot of new attackvectors in the nearest future.
So decoding I think needs to be used.
Br.
Beinhard
Ashish, you can check the list of vulnerable versions and currently available hotfixes for those versions in Solution 15629 on AskF5:
http://support.f5.com/kb/en-us/solutions/public/15000/600/sol15629.html
Quick question. I know this irule script will reject and log. However is there a way to set it to only log and not reject? If I simply remove reject; from the script will it monitor instead of using the reject action?
when HTTP_REQUEST {
set pattern "*() \{*";
if { [string match $pattern [HTTP::uri]] } {
log local0. "Detected CVE-2014-6271 attack from '[IP::client_addr]' in URI '[HTTP::uri]'";
reject;
Is the reject; syntax an action command or is it simply a message that the hacker receives? This part is what is throwing me off.
"I chose to issue a reject on the connection. If you want to be more polite to the hackers, you can substitute the "reject" with a 403 - Forbidden."
Thanks!
@David Hwang - if you want to make the script passive, just comment out the "reject" lines by putting a pound symbol (comment) before the reject (or remove the line completely). The reject will send a connection reset to the client. An alternate approach would be to send a HTTP response 403. That's not as brash has terminating the connection. With a terminated connection, the hacker wouldn't be able to know for sure the cause of it and thus not necessarily know if their attack worked or not. By sending a forbidden message, might lead them to try something else. That's what I was getting at with the comment about reject vs. forbidden.
@rgrojas - AFAIK, there is no direct way to send an email from within an iRule. What I would likely suggest is that you issue a log message and then setup a notification with syslog when it catches that message. Here's a F5 solution on setting email notifications on snmp alerts - http://support.f5.com/kb/en-us/solutions/public/3000/600/sol3667.html. Hope this helps...
