A Different Kind of Spamalot
Those familiar with Monty Python might appreciate this dialog in song from the musical... Customer: What have you got? Waitress: Well, there's egg and bacon, egg sausage and bacon Egg and spam Egg, bacon and spam Egg, bacon, sausage and spam Spam, bacon, sausage and spam Spam, egg, spam, spam, bacon and spam Spam, sausage, spam, spam, spam, bacon, spam tomato and spam Spam, spam, spam, egg and spam Spam, spam, spam, spam, spam, spam, baked beans, spam, spam, spam and spam. Customer's Wife: Have you got anything without spam? Waitress: Well, the spam, eggs, sausage and spam That's not got much spam in it Customer's Wife: I don’t want any spam! Lots of spam, that undesirable, loathsome spam. I think we all sympathize with the wife here. Certainly the DevCentral team does, as we’ve been dealing with a very persistent spammer over the last month. It’s nothing new, every site that promotes and encourages community (and many that don’t) has to deal with the lunatic fringe of the marketing world. To that end, we’d like to offer our sincerest apologies for the unsightly commentary in the Q&A section of our community, and the unfortunate side effect of your subscriptions resulting in the spam reaching your inboxes. No one wants to receive that mess, and we are grieved to have been a vessel for this distasteful marketing. But enough words, let’s talk actions. It’s a lot like chess. Matching wits with spammers, countering the queen’s gambit with albin or QG declined. But rest assured that we are not sitting idly by. We have enhanced and added several countermeasures that have extremely dampened the spirits of the evil devil spawn, to the point where his efforts now are but a mere trickle. We’ll continue to monitor, learn, adapt, and grow. In addition to the work we’ve done on our end, there is something you can do as well to help. You can flag posts that are clearly spam (or offensive or inappropriate) as shown in the image below. Given the new counter measures, there is a slight risk that a false positive will occur and something you post might be flagged as spam. If this happens, please notify us immediately at devcentral@f5.com and we’ll work diligently to resolve the issue. Thanks for all you do to make this community great and for seeing us through this spam issue.Blocking Spam with Custom Attack Signatures
I've said it before, I'll say it again: DevCentral is an awesome place to be. Our community is comprised of the best members in the world, and it shows by the way we help each other and care for our collective community. I have a little story to tell (and a solution to share) about the way our community continues to thrive. Earlier this month, DevCentral got hit with a series of unwanted Spam messages...everything from improving love life to black magic potions to astrology secrets. Did these topics catch my interest? Ummm, yeah. Were they appropriate for DevCentral? Probably not. Did we want to get these removed and ensure they never showed up again? Of course. Do I ask myself questions and then answer them myself? Most definitely! Our site is designed to allow (and encourage) open communication among all our members. While we want everyone to share and collaborate, we obviously discourage malicious or just outright annoying posts on our site. So when we got hit with these spam messages, we quickly deleted them. In fact many of you helped identify and report these as spam messages...so thanks! This is a perfect example of that "caring for our community" stuff I was talking about earlier...good times. While we were able to delete these posts, we wanted a more proactive approach that stopped them before they ever hit our site. Spam noun \ˈspam\ : irrelevant or inappropriate messages sent on the Internet to a large number of recipients. As our team thought about different approaches for proactive spam blocking, we considered the trusty BIG-IP Application Security Manager (ASM). If you don't know about all the goodness of the ASM, check out some of my previous articles on this thing...it's amazing and awesome! The ASM comes pre-loaded with ~2,400 attack signatures that block all kinds of malicious activity. It's also flexible enough to allow for custom attack signatures when you run into a unique situations...and this spam situation was a perfect candidate for some custom attack sigs. We needed a way to trigger on a keyword and block any post that included that keyword (or keywords). So, I created a custom attack signature that does just that...it looks for a keyword and blocks any request that has the keyword. One of the reasons the ASM signature was a good solution was that the spam was hitting us from several different IP addresses. We could have easily blocked an IP, but I'm pretty sure Mr. Spam would have found a quick way around that one. Instead, we decided to block based on the content of the messages being posted. BIG-IP ASM Configuration The first step in solving this problem is to create a custom attack signature. Navigate to Security >> Options >> Application Security >> Attack Signatures >> Attack Signature List and you will see a list of all the attack signatures in the ASM (see the screenshot below). Click the "Create" button at the top right part of the screen. Now it's time to create the custom signature! The first thing you do is name your custom signature (don't put any spaces in the name). Then, you can fill out an optional description...I like to do this to remind myself what in the world I was doing when I created this thing. If you keep the "Auto Apply New Signatures Configuration After Create" enabled, then the ASM will automatically apply any new custom signature to all active security policies that have signature sets containing this signature. I kept this enabled for the DevCentral signatures so that I didn't have to go back and reconfigure any signature sets after I created new custom signatures. On the "Systems" setting, I selected "Various systems" because this specific signature was not necessarily tied to any unique operating systems, web servers, etc. For "Attack Type" I chose "Other Application Attacks" but I could have chosen any of the Attack Types listed in the dropdown menu...but, the "Other Application Attacks" made the most sense for this custom signature. The "Rule" is where the meat of the signature is built. The ASM has several different rule options and each one has specific syntax requirements. You can find all the details in the ASM Configuration Guide - Appendix C. I used the "content" rule for my specific signature. The content rule matches when a specified character string is found anywhere in the full content of the request. The string match is case-sensitive, and it must be exact. For example, if you choose content "keyword" then it will not match on "key". You can also use the not character (!) in front of the string if you want the rule to match when it does not find the exact string. In addition, a "nocase" modifier is available, and this simply takes away the case-sensitivity of the rule. I used this in my "content" rule because I wanted to make sure I caught all the keywords regardless of their character case. Here are a few of the rule options available for creating custom signatures: content: matches when a specified character string is found anywhere in the full content of the request uricontent: matches when the specified string is found anywhere in the normalized URI, including the query string headercontent: matches when the specified string is found anywhere in the HTTP request headers valuecontent: matches when the specified string is found anywhere within a specific alpha-numeric parameter pcre: matches if the regular expression found within the slash (/) characters matches As for the "Accuracy" and "Risk" settings, I didn't worry about those too much because my specific security policy blocks on all attack signatures regardless of Accuracy or Risk settings. If your policy is different, you can certainly change these settings as appropriate. Congratulations! You just created a custom ASM attack signature. But wait, there's more. After you create the signature, you have to associate it with a "Signature Set". If you remember from your security policy building days, you actually assign Signature Sets to a security policy, not individual signatures. So, the next step is to create a custom Signature Set. You can get to this screen by navigating to Security >> Options >> Application Security >> Attack Signatures >> Attack Signature Sets and then click "Create" for your new custom Signature Set. I chose the "Filter-based" signature set because I wanted the ASM to automatically choose the correct signatures to add to this set based on the filters I selected. That way, if I needed to create a new custom signature, I could simply apply the correct filters and the new custom signature would automatically get added to my Signature Set (and thus my security policy as well). If you don't want the ASM to do all that work, you can select "Manual" for the Type of Signature Set and then you will manually add signatures as appropriate. For this specific Signature Set, I chose "Other Application Attacks" as my Attack Type and "Various Systems" as my Assigned Systems (you'll notice a resemblance to the settings in the custom signature). I also kept the default settings for Accuracy, Risk, User-Defined, and Update Date (All). If you mess around with these specific settings, be sure to keep the "User-Defined" signatures or else it won't include the beautiful work you just accomplished when creating your custom (User-Defined) attack signature. Once you select all these settings, the ASM will auto-build the list of attack signatures it plans to include in this Signature Set (listed at the bottom of the page). Be sure to check for your custom signature in this list...if it's not there, then go back and check all the settings to ensure you configured everything properly. Now that the custom signature is built and the Signature Set is configured correctly, the last step is to associate the Signature Set with your security policy. Navigate to Security >> Application Security >> Attack Signatures >> Attack Signatures Configuration and you will see the screenshot listed below. Be sure the correct security policy is listed in the "Current edited policy" list and then simply move the newly-created Signature Set from the Available column to the Assigned column. You also have the option of selecting Learn, Alarm, and/or Block on this page. I checked them all (wanted to make sure I blocked this spammer). In addition, be sure the "Signature Staging" box is not enabled if you want to block with this policy. Then click "Save" at the bottom and, as always with the ASM, be sure you click the "Apply Policy" button on the upper/right part of the screen. Once you do all that, the policy is updated and active and you will be blocking all requests that have "keyword" anywhere in them. The final step in my exciting journey was to update the ASM blocking page. You can navigate to Security >> Application Security >> Blocking >> Response Pages and you will see the screenshot below. I updated the AJAX response page (our web app uses json content for Question and Answer pages) with a custom response and then tested it to make sure everything was working properly. Voilà...it worked!! Check out the screenshot below...and send us an email if your post has been incorrectly identified as spam. Well, that's it. I hope you have enjoyed this spamalicious experience as much as I have. In all seriousness, it's really cool to be a part of DevCentral, and it's also really cool to be able to use awesome F5 technology to keep our community free from annoying spammers. Send me an email or comment below if you have any questions about all this. Until next time, remember that the only good spam is the stuff you buy in a can from Hormel...oh, who am I kidding, that stuff is nasty also.
Backscatter: Be part of the solution, not the problem
I read Robert McMilan's article on backscattering with great interest, primarily because my personal account has been a "bounceback" victim for the past couple of weeks. His article contains a great explanation of what backscatter is and why it happens; it's the kind of article I'd send to my friends who are asking about all the bouncebacks they're seeing these days. What is backscatter?backscatter -- bounceback messages from legitimate e-mail servers that have been fooled by the spammers. Spammers like to put fake information in their e-mail messages in order to sneak them past e-mail filters. Because e-mail filters now just delete messages that come from nonexistent domains, the spammers like to make their messages look like they come from real e-mail addresses. That means, if your e-mail address has been published on the Web somewhere, you're a prime candidate for backscattering. The problem is that backscatter is the side-effect of a poorly configured mail server or ineffective SPAM prevention system. It's not something the victim can stop, it has to be addressed by the administrators of those systems which are too easily fooled by spammers. And from the looks of my inbox, that's a whole lotta systems. Robert goes on to suggest this [reconfiguration] as a solution. But the problem would largely disappear if server administrators configured their mail servers to immediately reject mail that is sent to nonexistent users, rather than accepting it and then bouncing it back to the faked addresses. Some ISPs (Internet service providers), AOL for example, have done this and have largely eliminated their role in the problem. This certainly would decrease a lot of the backscatter, but it wouldn't catch it all. You need something a bit more robust and crafty in its determination of who is a legitimate sender of e-mail and who is not. Something that, say, bases its decisions on the reputation of the sender rather than just on the contents of the e-mail. Reputation-based spam prevention works because it doesn't just look at the message, it looks at the sender. In the real world we filter information based on both; you're not likely to listen to tech advice when it's given by a fifteen year old who looks like she should be on the cover of "emo-tech" magazine even if it is the same advice given by a respected member of the tech community. Why shouldn't we treat e-mail the same way, with an eye toward the credibility of the sender? Between correctly configuring mail servers to verify senders - even to the point of requiring authentication if necessary - and taking into consideration the reputation of the sender, we should be able to eliminate 70% of the SPAM out there, if not more. If you don't think keeping backscatter out of my inbox is a good enough reason to implement a reputation-based mail system, then consider this: by preventing SPAM from reaching your mail servers you are saving (a) CPU cycles, (b) storage, and (c) employee's time. That's because a reputation-based system prevents SPAM from passing beyond it and into the realm of the mail servers, which means you don't have to waste storage (7 years of SPAM, anyone?) or processing power on it. And if you can decrease the load on your mail servers by not passing along every piece of SPAM for inspection, you might not need a second or third mail server to handle the load. If you're wondering how such a system saves employee time, consider the results of a recent survey of e-mail monitoring practices by Proofpoint: Someone is reading your e-mail...Proofpoint found that 41% of the largest companies surveyed (those with 20,000 or more employees) reported that they employ staff to read or otherwise analyze the contents of outbound e-mail. 22% of these companies said they employ staff primarily or exclusively for this purpose." Obviously less e-mail means fewer employees needing to dedicate time and energy to reading through employees' e-mail, which saves time and, if you can reduce the staff and redirect those resources elsewhere, maybe money, too. Did I mention that reputation-based mail systems are a great help if you're trying to go green? Cause they are, of course. Reductions in processing of SPAM result in less power consumed by mail servers, storage devices, and AV servers because they're processing fewer messages. Reputation-based systems can't prevent backscatter, but it can prevent the cause of backscatter by not attempting to deliver what are obviously fraudulent e-mail messages in the first place. So be a part of the solution instead of the problem and consider a reputation-based mail system. My inbox will thank you for it. Imbibing: Coffee More on message security: Enhanced Message Security: Slicing SPAM and Other Threats At The Edge F5's Message Security Module (MSM)
