Javascript injecting systems effect on web application end users - a scenario review
Hello! ArvinF is back to share a scenario review where Javascript-injecting systems affected web application end users - web and mobile application. Problem Users are failing to login to a web application protected by BIG-IP ASM/Adv WAF and Shape Security Defense. The site owner notes that the authentication was failing for an unknown reason. There were ASM Support ID noted and an error informing to enable Javascript. Please enable JavaScript to view the page’s content. Your support ID is: xxxxxxxxxxxx Troubleshooting To understand the cause of the authentication failure, we gathered HTTP traffic through a HTTP sniffer. We used httpwatch and gathered HAR (HTTP Archive) files. The site was protected with both on-premise BIG-IP ASM/Adv WAF bot defense and back then, Shape Security Defense (now F5 Distributed Cloud Bot Defense). After the review of the HAR file in httpwatch, the following were noted: ASM blocks a request in a URL related to authentication with a Support ID in the response. There was also javascript code included and it references https[:]//s[.]go-mpulse[.]net/boomerang/. The authentication attempt failed with an error in the HTTP response: ...unable to process your request. Please try again later... BIG-IP ASM/Adv WAF related HTTP cookies from its various features such as Bot Defense Client Side challenges as TSPD_101* cookie was present and other TS cookies, which could also come from Bot defense and DoS profile and security policy configurations. There were also HTTP cookies coming from BIG-IP AVR - f5_cspm cookie was present. Application Visibility and Reporting (AVR) module provides detailed charts and graphs to give you more insight into the performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.). https://clouddocs.f5.com/training/community/analytics/html/index.html https://clouddocs.f5.com/api/irules/AVR_CSPM_INJECTION.html Seeing the javascript code referencing "/boomerang/" included in the ASM blocking response was interesting. Reviewing the HAR file, there were several instances of this "/boomerang/". This finding was inquired with the site owner and they noted that there is another system that is in the path between the end users and their web application - a CDN. The traffic flow is as follows: End user web browser / mobile application >>> CDN >>> FW >>> BIG-IP >>> web application On the BIG-IP Virtual Server that fronts the web application, F5 AVR profile, ASM/Adv WAF Bot defense, and security policy and Shape Security defense iRule are configured. From the F5 side, these were the products with features that may insert Javascript in the client-side response. As part of troubleshooting, to isolate the feature that might be causing the failing authentication for the web application, the bot defense profile was removed from the site's Virtual Server and the Shape Security iRule and AVR profile were left untouched. Site owner noted that the authentication works after this change. Shape Security Defense was implemented using an iRule to protect specific URIs. When the iRule was removed from the Virtual Server and the Bot defense and AVR profile were left on, the VS, Site owner noted that the authentication works after this change. But if both ASM/Adv WAF Bot defense and Shape Security Defense iRule is configured on the VS, the site's authentication fails. Per the site owner, there were no changes in the Bot Defense or Shape Security Defense iRule configurations prior to the incident and that these configurations were in place way before the incident. Site owners shared the findings with their respective internal teams for their review. Resolution Afterwards, Site owner shared that their site now works as expected and authentication works for the web application with no changes done on both ASM/Adv WAF Bot defense and Shape Security Defense iRule on the site's VS. The cause of the authentication failure was undetermined. A theory on the possible cause of the issue was perhaps, there was another system inserting Javascript code in the responses and it might have affected the authentication process of the web application by prevented that portion of the site from loading. Additional Troubleshooting Notes The data gathered during the troubleshooting were the qkview and HTTPWatch capture - HAR files. It would help if a packet capture was taken along with the HTTPWatch capture while the issue was happening to have a full view of the issue. Decrypt the packet capture to observe HTTP exchanges and to correlate it with HTTPWatch capture events. The corresponding BIG-IP ASM/Adv WAF application event logs, Bot Defense or DoS protection logs will also be helpful in the correlation. Having a visual idea on how the Security Policy, Bot Defense or DoS protection profile are configured is also helpful - so its good to have a screenshot of these. It helps in analysis when there is complete data. Gathering the asmqkview with report and traffic data and corresponding ASM and AVR db dumps helps in the analysis. asmqkview -s0 --add-request-log --include-traffic-data -f /var/tmp/`/bin/hostname`_asmqkview_`date +%Y%m%d%H%M%S`.tgz #mysqldump -uroot -p`perl -I/ts/packages -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw` DCC | gzip -9 > /shared/tmp/dcc.dump.gz # mysqldump -uroot -p`perl -I/ts/packages -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw` PLC | gzip -9 > /shared/tmp/plc.dump.gz # mysqldump -uroot -p`perl -I/ts/packages -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw` PRX | gzip -9 > /shared/tmp/prx.dump.gz # mysqldump -uroot -p`perl -I/ts/packages -MPassCrypt -nle 'print PassCrypt::decrypt_password($_)' /var/db/mysqlpw` logdb | gzip -9 > /shared/tmp/logdb.dump.gz It would also help if the systems in the path of the web application are known and whether it has features that may interfere with the features of BIG-IP ASM/Adv WAF or Shape Security Defense. Per the findings, there was a CDN that was injecting javascript code in the HTTP response and it may have contributed to the authentication failure for the end users. Isolate potentially conflicting features by removing one of them one at a time and observe the HTTP responses. Per the reference configuration, BIG-IP ASM/Adv WAF, Shape Security Defense, and BIG-IP AVR worked well prior to the incident. boomerang The injected javascript code noted in the ASM blocking page response was loaded from https[:]//s[.]go-mpulse[.]net/boomerang/. Checking this reference, it was related to https://github.com/akamai/boomerang. boomerang is a JavaScript library that measures the page load time experienced by real users, commonly called RUM (Real User Measurement). It has the ability to send this data back to your server for further analysis. With boomerang, you find out exactly how fast your users think your site is. In BIG-IP, the similar product we have is BIG-IP AVR - Application Visibility and Reporting (AVR) - where it collects "performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.)." Organizations may have specific needs on data that they need to collect from their site/web application and using a customizable solution such as boomerang can help. That's It For Now I hope this scenario review on Javascript-injecting systems effect on web application end users will be helpful on your next troubleshooting and hopefully gives you guidance on what data to gather and look for and troubleshooting options. The F5 SIRT creates security-related content posted here in DevCentral, sharing the team’s security mindset and knowledge. Feel free to view the articles that are tagged with the following: F5 SIRT series-F5SIRT-this-week-in-security TWIS
