Error 521: Web server is down - Cloudflare generated error
I have an virtual server created with many websites hosted in it. It has one public IP assigned to it and is NATed to a private IP. The websites are accessible via HTTP and HTTPS. However one website uses Cloudflare, there are times that when we are browsing that website we sometimes come across with error 521. Sometimes after refreshing the web page, the error will no longer present itself sometimes the error is still present even after refreshing several times. I have whitelisted the cloudflare IP addresses onto F5 via Network ›› Packet Filters : Rules I am running software version 12.1.3"Always Send Cookie" problems?
Is there a downside to choosing "Always Send Cookie" in an "HTTP Cookie Insert" persistency profile? I am troubleshooting an issue with Cloudflare and a potential issue with my current F5 settings. The below is specifically called out by CF (re: the F5), but I am not 100% that it correlates to the "Always Send Cookie" setting. Per Cloudflare, via https://support.cloudflare.com/hc/en-us/articles/212794707-General-Best-Practices-for-Load-Balancing-with-Cloudflare; // Session cookies section above Cloudflare article If using HTTP cookies to track and bind user sessions to a specific application server at the load balancer, it is best is to configure the load balancer to parse HTTP requests by cookie headers and directing each request to the correct application server even if HTTP requests share the same TCP connection due to keep-alive. For example: F5 BIG-IP load balancers will set a session cookie (if none exists) at the beginning of a TCP connection and then ignore all cookies passed on subsequent HTTP requests made on the same TCP socket. This tends to break session affinity because Cloudflare will send multiple different HTTP sessions on the same TCP connection. (HTTP cookie-based session affinity).
Madness DDoS Bot Analysis
“Madness” is a Denial of Service bot targeting Windows machines. While it constantly evolves its capabilities, it certainly represents the progress attackers made in Layer 7 DDoS expertise. The bot author proudly states that “Madness” is the superior successor and an actual hybrid of the notorious DDoS malware families “BlackEnergy”, “gbot”, “DirtJumper”, “Darkness Optima”, “iBot” and “w3Bot”. Its phone-home architecture is based on polling the C&C server. Once a minute it sends a probe to its operator with information on the infected machine (OS and level of privileges) and the number of attack payloads successfully sent, while the C&C will respond with the current attacking command (base64 encoded). DDoS Capabilities Madness DDoS capabilities consist of network floods, such as UDP and ICMP, however the more interesting part is its diversity of HTTP (Layer 7) floods. We can separate its HTTP floods to those using "low-level" libraries such as Windows Sockets API and those using more "high-level" libraries such as “WinInet” and “Urlmon”, which are also used by “Internet Explorer”. All the floods support up to 10 different URLs which might be used to attack different servers or different URLs on the same server. Going "low-level" lets the attacker make the flood much faster, and not less importantly gives him the flexibility in constructing the HTTP request from head to toe to bypass many DDoS protections. The added value of using "high-level" HTTP libraries is that they automatically handle all HTTP protocol aspects such as request construction, managing TCPconnections, caching, cookies and redirections. Those attacks support both simple "GET" requests and more complex "POST" requests with the ability tocontrol the data being sent in the POST body. While analyzing the threat, we have also noticed a very strange version of a SLOW GET attack, which rather may be called a "HOLD" flood attack. It sends a “GET” request containing only the “Host” header, sleeps for 0.1 second and only then completes the request by sending "\r\n". Bypassing Challenges Another aspect indicating a progress in attackers DDoS capabilities is having built-in techniques for bypassing anti-bot challenges. It starts with a simple “Anti-Cookie” feature for bypassing simple challenges, while mainly relying on parsing the response and responding appropriately in order to resemble a real browser. It appends cookies issued by the server either by the “Set-Cookie” response header or a JavaScript containing “document.cookie=” code. It also parses a specific JavaScript challenge issued by a well-known Russian web-hosting service. The challenge is obfuscated JavaScript which sets a cookie once executed. The bot looks for the strings "Cookie", "realauth=<value>" and "location" in the response and sets the appropriate cookie as a legitimate browser which renders JavaScript would do. However, the highlight is “Madness” ability to bypass more complex JavaScript challenges, specifically targeting “Cloud Flare”, a popular DDoS protection service. The challenge is JavaScript code which makes a certain calculation and sends back the result. If the result is correct, the server will issue an authorization cookie and all the following requests contacting that cookie will be considered legitimate. Instead of trying to simulate a browser, “Madness” author chooses to use a real browser. Once the attacker issues the “cfa” (apparently “Cloud Flare Attack”) command from the C&C, the victim machine runs an Internet Explorer in minimized window, browse the targeted website, performs the JavaScript challenge calculation, as it is a legitimate browser, and gets the authorization cookie. Once the cookie is saved in the browser, the attacker executes one of the “high-level” HTTP floods, specifically the one that uses the “Urlmon” library which is also used by Internet Explorer’s core, automatically appending the authorization cookie to each attacking request. Summary As layer 7 DDoS attacks are being effective, “Madness” provides different flavors of GET/POST floods with a high level of flexibility while integrating more advanced capabilities to bypass security solutions. We expect this bot to be seen more in the wild with upgraded capabilities. One of the lessons we learn from this analysis while protecting our servers is that it is not enough relying on clients simply answering the challenge, as complex as it might be. It also requires a proactive bot defense which has more integrity checks and implements more sophisticated logic around those challenges, as well as other prevention means such as rate limiting, CAPTCHA and behavioral anomaly detection.
