SRE
3 TopicsAccelerating Digital Transformation in Banking and Financial Services
Introduction A recent survey from Forrester’s Business Technographics shows that 33% of BFSI tech leaders are currently undertaking a digital transformation within their organizations. That’s 13 points ahead of the average across industries. Still, many enterprises worry that they aren't moving fast enough. For banking and financial services organizations, there is intense pressure to transform their enterprises to remain more competitive in an age of disruption. Evolving regulatory requirements, rapidly advancing technology, increasing customer demands, COVID-19 and competition from fintech’s are all forcing financial services firms to rethink the way they operate. Digital Transformation Challenges This digital transformation imperative requires banking and financial services organizations to improve their technical capabilities. But true transformation demands more than just new technologies. It requires strategic vision and commitment from the top of the organization to rethink and retool its culture, its processes, and its technology. Admittedly, the financial industry has a long history of not collaborating, lack of transparency, and resistance to adaptability, favoring instead confidentiality, siloed organizational structures, and risk aversion. For many years, that heritage enabled financial services firms to succeed. Existing cultural, behavioral, and organizational hurdles can be hard to overcome because they are so entrenched. New processes and technology are also necessary for digital transformation. Traditional development practices are common in the industry and are built on segmented and monolithic team structures that lack the agility required to achieve transformation. Additionally, very few possess the infrastructure and application architectures required to rapidly innovate. The Benefits of an Open Approach Digital transformation is not merely about adopting new technologies but also establishing new cultural practices and ‘ways of working’ within the IT organization.By taking an open approach to architecture, process, and culture, you can transform the way your entire organization operates. Modular architecture To create a more modular environment, banking and financial services institutions will require integration across the entire legacy network, as well as integration with partner systems, networks, and other external services such as Software-as-a-Service (SaaS) solutions. An open and composablearchitecture gives customers access to a growing range of ‘Best of Breed’ technologies from industry leaders, consumable in a frictionless “single-stack” feel. Agile process In the open organization model, collaboration is key. Modern, agile practices establish common goals and empower teams to move forward together. According to the Harvard Business Review article “Reassessing Digital Transformation:The Culture and Process Change Imperative”, financial services were more apt to say that DevOps was important than other industries, and were also more likely to have implemented agile development, project management processes, CI/CD, and DevOps. These new processes are necessary as financial services firms seek faster time to value and leverage microservices to effect this change. Open culture Open organizations are more transparent, inclusive, adaptive, collaborative, and community focused.When you view digital transformation as a continuous process—and emphasize the importance of culture in parallel to, not at the expense of, technology and process— you’re positioning your organization for a successful transformation. Technologies that Enable Digital Transformation The pandemic has accelerated the need for digital transformation in the BFSI segment.Not only have workforces become remote, but person to person contact has become less frequent.Financial organizations have not only had to scale up infrastructure and security to support a remote workforce but have also had to simultaneously scale to support a fully remote customer base. Inherent in this approach is a hybrid cloud strategy that allows the ability to scale up or down resources to meet application needs.Architectural design and practices must also align with these new cloud infrastructures.There is a need to balance the requirements for speed with the absolute necessity for security and availability.There are a few key best practices that BFSI organizations have used to balance these competing demands: ·Establish a foundation of resilience by adopting site reliability engineering (SRE) concepts. ·Rapidly deploy new services quickly based on market demand. ·Consolidated, consistent, and controlled security and access, including identity management, intrusion protection, anti-virus, predictive threat capabilities ·Application performance (response time and latency), on-demand scalability, and disaster recovery and backup •Automation for efficiency and to speed delivery, with consistency in operations and tools, continuous integration and continuous delivery (CI/CD) •System-wide business monitoring, reporting, and alerting. An Open Architecture with F5 and Red Hat Now that we have established the open approach for implementing a financial service platform and the capabilities needed for a successful digital transformation, we can examine the architecture needed to support it. It starts on the path toward site reliability engineering (SRE). In the SRE model, the operations team and the business give developers free rein to deploy new code—but only until the error budget is exceeded. At that point, development stops, and all efforts are redirected to technical debt. As shown in Figure 1, it boils down to 5 areas that an SRE team should be doing to achieve the balancing goal. Figure 1. Enabling SRE Best Practices Together, F5, Red Hat, Elasticsearch, and other ecosystem partners can deliver a suite of technologies to fulfill the extension and transformation of existing architecture to an agile financial service platform. Figure 2. SRE Microservice Architecture with F5, Red Hat, and Elasticsearch The following describes the most fundamental components of Figure 2 in more detail, to enable the SRE best practices: 1.Red Hat OpenShift Container Platform (container PaaS) provides a modular, scalable, cloud-ready, enterprise open-source platform. It includes a rich set of features to build and deploy containerized solutions and a comprehensive PaaS management portal that together extend the underlying Kubernetes platform. 2.Combining BIG-IP and NGINX, this architecture allows SRE to optimize the balance between agility and stability, by implementing blue-green and targeted canary deployment. It’s a good way to release beta features to users and gather their feedback, and test your ideas in a production environment, with reduced risk. 3.BIG-IP combined NGINX Plus also gives SRE the flexibility to adapt to the changing conditions of the application environments, address the needs of NetOps, DevOps, DevSecOps, and app developers 4.ELK is utilized to analyze and visualize application performance through a centralized dashboard. A dashboard enables end-users to easily correlate North-South traffic with East-West traffic for end-to-end performance visibility. 5.F5’s WAF offerings, including F5 Advanced WAF and NGINX App Protect, deployed across hybrid clouds, protect OpenShift clusters against exploits of web application vulnerabilities as well as malware attempting to move laterally. 6.Equally important is integration with Red Hat Ansible that enables the automated configuration of security policy enforcement for immediate remediation. 7.Built intoCI/CD pipeline so that any future changes to the application are built and deployed automatically. Conclusion Digital transformation has been accelerated by the dual challenges of Covid and the emergence of Fintech.Traditional BFSI organizations have had to respond to these enormous challenges by accelerating their deployment timelines and adopting agile processes without compromising security and availability. These practices also dovetail with the greater adoption of microservices architectures that allow for scale up and scale out of application services.F5 & NGINX helps aid this transformation by providing world class performance and security combined with a flexible microservices ADC (NGINX+). This hybrid architecture allows for Kubernetes deployments to become ‘production grade’.1.9KViews3likes0CommentsAdopting SRE practices with F5: Layered Security Policy for North-South Traffic
In an organization with enough maturity in cybersecurity and modern application architectures, there are two different cybersecurity teams that operate the more advanced security policies for the company. NetSecOps and DevSecOps are the two cybersecurity teams in an organization, and they typically have different security requirements. NetSecOps requires a ‘Standardized Application Security Policy'. They aim to block common attacks to the production network with a high level of confidence, resulting in a ‘low-false positive rate,’ at the network level. The OWASP Top 10 threats is a good example here. Moreover, the responsibility of NetSecOps is not limited to stopping basic attack types like the OWASP Top 10, but it also covers more advanced and complicated application-based attacks such as ‘Bot Attacks,’ ‘Fraud Attacks,’ and ‘DDoS Attacks.’ However, when it comes to the ‘Modern-App environment,’ it is not easy for the NetSecOps team to understand the details of the application traffic flow inside the Kubernetes or OpenShift cluster. For this reason, as far as modern applications are concerned, the security policies of NetSecOps often focus more on compliance and audit purposes. However, DevSecOps wants the application-specific security policies for different types of applications to be operating inside their Kubernetes or OpenShift clusters. This is possible since DevSecOps understands how their applications work and they want to apply more optimized security policies for their backend applications. This is why it is sometimes difficult to achieve both security team’s goals with a single security solution. This is why the enterprise needs to deploy two different WAFs to meet the different requirements from both NetSecOps and DevSecOps. This article will cover how two different security teams can achieve their goals with two separate WAF (Web Application Firewall) deployments in the network - F5 Advanced WAF for NetSecOps and NGINX App Protect for DevSecOps. Solution Overview The solution includes two F5 components – F5 Advanced WAF and NGINX App Protect. From a technological point of view, NGINX App Protectutilizes s a subset of F5 Advanced WAF functionality, meaning that their underlying technologies are the same. Each of those WAF components can run with different security policies in order to achieve different goals. In F5 Advanced WAF, NetSecOps can apply the WAF policy for the ‘coarse-grained model’ of security, while DevSecOps adopts the ‘fine-grained model’ with the NAP. In other words, this means that F5 Advanced WAF can be configured with a ‘Negative Policy,’ and NGINX App Protect can be configured with a ‘Positive Policy.’ In our use-case, we assumed that NetSecOps wants to block the OWASP Top 10 threats while DevSecOps has a different 'file accessing' policy for each backend application. The brief architecture is depicted below. Combining F5 Advanced WAF and NGINX App Protect enables layered application security policies to prevent the most complicated and advanced application-based attacks efficiently. This architecture utilizes the following workflow: 1.The F5 Advanced WAF blocks the most commonly used attack types including ‘Command Injection,’ ‘SQL Injection,’ ‘Cross-Site Scripting,’ and ‘Server Side Request Forgery’ attacks. 2.When the attacker tries to access the different files in each application, NGINX App Protect manually specifies the file types that are allowed (or disallowed) in traffic based on the security policies configured by the DevSecOps team. 3.All alert details from F5 Advanced WAF and NGINX App Protect are sent to the ‘Elasticsearch’ for central monitoring purposes. Each of the above workflows will be discussed in the following sections. ·This blog doesn’t include all the required steps to reproduce the use-case in the environment. Please refer to this link for all the required configuration steps. NGINX App Protect provides ‘Application-Specific’ policies NGINX App Protect can provide security protection and controls at the microservice level inside the Kubernetes or OpenShift cluster. The NGINX App Protect can be deployed in the OpenShift cluster as a container image. The NGINX App Protect policy configuration uses the declarative format built on a pre-defined base template. The policy uses the JSON format to represent the policy details. This file can be edited to apply a unique security policy to the NGINX App Protect instance. Once the policy is created, the policy can be attached to the 'nginx.conf' file by referencing the policy file. In this example, we used the ‘nginx_sre.conf’ file as the main configuration file for NGINX and the ‘NginxSRELabPolicy.json’ file represents the NGINX App Protect policy. NginxSRELabPolicy.json: | { "policy": { "name": "SRE_DVWA01_POLICY", "template": { "name": "POLICY_TEMPLATE_NGINX_BASE" }, "applicationLanguage": "utf-8", "enforcementMode": "blocking", "response-pages": [ { "responseContent": "<html><head><title>SRE DevSecOps - DVWA01 - Blocking Page</title></head><body><font color=green size=10>NGINX App Protect Blocking Page - DVWA01 Server</font><br><br>Please consult with your administrator.<br><br>Your support ID is: <%TS.request.ID()%><br><br><a href='javascript:history.back();'>[Go Back]</a></body></html>", "responseHeader": "HTTP/1.1 302 OK\\r\\nCache-Control: no-cache\\r\\nPragma: no-cache\\r\\nConnection: close", "responseActionType": "custom", "responsePageType": "default" } ], "blocking-settings": { "violations": [ { "name": "VIOL_FILETYPE", "alarm": true, "block": true } ] }, "filetypes": [ { "name": "*", "type": "wildcard", "allowed": true, "checkPostDataLength": false, "postDataLength": 4096, "checkRequestLength": false, "requestLength": 8192, "checkUrlLength": true, "urlLength": 2048, "checkQueryStringLength": true, "queryStringLength": 2048, "responseCheck": false }, { "name": "pdf", "allowed": false } ] } } --- The above configuration file shows the NAP policy of application #01, where the DevSecOps team wants to disallow file access to the ‘PDF’ file format. For application #02, the NAP policy is configured to reject the access to the ‘JPG’ file. And the ‘remote logging’ configuration needs to be applied on the NGINX to export the NGINX App Protect's alert details. The below configuration shows how we exported the NGINX App Protect logging details to an external device, Elasticsearch. server { listen 8080; server_name dvwa02-http; proxy_http_version 1.1; real_ip_header X-Forwarded-For; set_real_ip_from 0.0.0.0/0; app_protect_enable on; app_protect_security_log_enable on; app_protect_policy_file "/etc/nginx/NginxSRELabPolicy.json"; app_protect_security_log "/etc/app_protect/conf/log_default.json" syslog:server=your_elk_ip_here; location / { client_max_body_size 0; default_type text/html; proxy_pass http://dvwa02; proxy_set_header Host $host; } Preventing OWASP Top 10 threats in F5 Advanced WAF F5 Advanced WAF is the next-generation WAF solution designed to prevent advanced application-based attacks. It supports 1000+ proven application-level signatures, custom signatures, Machine-Learning based DDoS prevention, Intelligence-based attack mitigation, and Behavioural-based WAF functions. But in this use-case, we focused on the prevention of the OWASP Top 10 attacks, which is only a small part of the F% Advanced WAF attack overall coverage. The important point here is how we can configure the F5 Advanced WAF to apply the WAF's efficient ‘Negative Security’ model. In order to configure the correct F5 Advanced WAF policy, one should follow the procedures below: 1. Go to 'Security' -> 'Application Security' -> 'Security Policies' -> 'Create' 2. Click the security policy that was just created (SRE_DEVSEC_01) ·Click the 'View Learning and Blocking Settings' under the 'Enforcement Mode' menu 3. Expand 'Attack Signatures' and Click 'Change' menu 4. Apply the check box. ·Click 'Close' ->click 'Save' -> click 'Apply Policy' ·Apply the policy to the virtual server. (Please make sure that we're on OCP partition.) 5. 'Local Traffic' -> 'Virtual Servers' -> 'devsecops_http_vs' -> Security -> Policies Please note that the ‘virtual server’ configuration is required in the BIG-IP before proceeding to this step. Configuring custom blocking page for F5 Advanced WAF 1.Click the security policy that was created (SRE_DEVSEC_01) 2.Go to 'Response and Blocking page' -> 'Blocking page default' -> 'Custom response' -> 'Response Body' <html><head><title>SRE DevSecOps Blocking Page</title></head><body><font color=red size=12>F5 Advanced WAF Blocking Page</font><br><br>Please consult with your administrator.<br><br>Your support ID is: <%TS.request.ID()%><br><br><a href='javascript:history.back();'>[Go Back]</a></body></html> Simulating the Attack The following steps show how to simulate the application-based attacks and to see how F5 Advanced WAF and NGINX App Protect can protect the applications efficiently. Preventing OWASP Top 10 Attacks - NetSecOps First, log in to the application through the GUI and go to the ‘Command Injection’ menu. And type the command ‘8.8.8.8 | cat /etc/passwd’ and click the ‘Submit’ button. If F5 Advanced WAF works correctly, you should be able to see the below ‘blocking page’. ·You can find the instructions from the Github link here how to simulate other attack types – SQL Injection, SSRF and XSS. Restrict file accessing based on the application types - DevSecOps 1.Access to application 01 on the browser with URL -> "http://your_app_domain.com/hackable/uploads/" 2.When the ‘PDF’ file is clicked on in this directory, the following blocking screen should be shown. Summary In modern application architectures, security concerns are becoming more serious. WAF is the major security solution available to enterprise applications. The security policy of the WAF has to protect backend applications correctly, but at the same time, it must also ensure legitimate user traffic access to the backend resources without creating issues. This sounds straightforward, but it is not easy to configure the right security policies to achieve both goals simultaneously. When it comes to modern application architectures, it is even more difficult to achieve this goal. Since traditional security teams lack understanding about the application flow inside a Kubernetes or OpenShift environment, it is challenging to apply the required security policies in the WAF to protect the microservices. Due to the nature of their microservices, different applications spin up and down frequently, and security requirements are also changed on a regular basis. The cybersecurity team needs to have a solution that can fit these unique requirements. For NetSecOps, they would require a solution that can have enterprise-level protection features and operational-efficiency for their SOC team. F5 Advanced WAF is designed to efficiently prevent known and unknown types of advanced application-based attacks, while NGINX App Protect easily provides ‘application-specific’ security policies for each application inside the microservice environment. The enterprises can acquire the proper protection for their modern app environment through the combination of F5 Advanced WAF and NGINX App Protect. Please visit the DevCentral GitHub repo and follow the guidelines to try this use-case in your environment.1.3KViews1like1CommentAdopting Site Reliability Engineering with F5
Foreword The role of the Site Reliability Engineering (SRE) is common in cloud first enterprises and becoming more widespreadin traditional IT teams.Here, we would like to kick off this article series to look at the concepts that give SRE shape, outline the primary tools and best practices that make it possible, and explore some common use cases around Continuous Deployment (CD) strategy, visibility and security. While SRE and DevOps share many areas of commonality, there are significant differences between them. DevOps is a loose set of practices, guidelines, and culture designed to break down silos in Development, IT operations, Network, and Security team. DevOps does not tell you how to run operations at a detailed level. On the other hand, SRE, a term pioneered by Google, brings an opinionated framework to the problem of how to run operations effectively. If you think of DevOps as a philosophy, you can argue that SRE implements some of the philosophy that DevOps describes. In a way, SRE implements DevOps practices. After all, SRE only works at all if we have tools and technologies to enable it. Balancing Release Velocity and Reliability SRE aims to find the balance between feature velocity and reliability, which are often treated as opposing goals. Despite the risk of making changes to software, these changes are necessary for the business to succeed. Instead of advocating against change, SRE uses the concept of Service Level Objectives (SLOs) and error budgets to measure the impact of releases on reliability. The goal is to ship software as quickly as possible while meeting the reliability targets the users expect. While there are a wide range of ways an SRE-focused IT team might optimize the balance between agility and stability, two deployment models stand out for their widespread applicability and general ease of execution: Blue-green deployment For SRE, availability is currently the most common SLO. If getting new software to your users and uninterrupted access is truly required, there needs to be engineering work to implement load balancing or fractional release measures like blue-green or canary deployments to minimize any downtime. Recovery is a factor too. The idea behind blue-green deployment is that your blue environment is your existing production environment carrying live traffic. In parallel, you provision a green environment, which is identical to the blue environment other than the new version of your code. As you prepare a new version of your software, deployment and the final stage of testing takes place inthe environment that is not live: in this example, Green (or new OpenShift Cluster). When it's time to deploy, you route production traffic from the blue environment to the green environment. This technique can eliminate downtime due to app deployment. In addition, blue-green deployment reduces risk: if something unexpected happens with your new version on Green, you can immediately roll back by reverting traffic to the original blue environment. When you are looking for manipulating the traffic with more flexibility, reliability, across different clusters, different clouds, or geo locations, this is when F5 DNS Load Balancer Cloud Service comes into the picture. F5 Cloud service GSLB is a SaaS offering. It can provide automatic failover, load balancing across multiple locations, increased reliability by avoiding a single point of failure, and increased performance by directing traffic to the optimal site. This allows SRE to move fast while still maintaining enterprise grade. Targeted Canary deployment Another approach to promote availability for SRE SLO is canary deployment. In some cases, swapping out the entire deployment via a blue-green environment may not be desired. In a canary deployment, you upgrade an application on a subset of the infrastructure and allow a limited set of users to access the new version. This approach allows you to test the new software under a Production-like load, evaluate how well it meets users’ needs, and assess whether new features are profitable. One approachoftenused by Azure DevOps is ring deployment model. Users fall into three general buckets based on their respective different risk profiles: Ring 1 - Canaries who voluntarily test bleeding edge features as soon as they are available. Ring 2 - Early adopters who voluntarily preview releases, considered more refined than the canary bits. Ring 3 - Users who consume the products, after passing through canaries and early adopters. Developer can promote and target new versions of the same application (version 1.2, 1.1, 1.0) to targeted users (ring 1, 2 and 3) respectively, without involving and waiting the infrastructure operations team (NoOps). To identify theuserfor the right version, you maychoose tosimplyuse IP address, authenticate directlyby backend, oradd an authenticationlayerin front of the backend.F5 technologies can helpenable this targeted canary use case: BIG-IP APM in N-S will authenticate and identify users as ring 1, 2 or 3, and inject user identification into HTTP header This identification is passed on to NGINX plus micro-gateway to direct users to the correct microservice versions. Combining BIG-IP and NGINX, this architecture uniquely gives SRE the flexibility to adapt with the ability to define the baseline service control and security (for NetOps or SecOps), while extending controls for more granular and enhanced security to the developer team (for DevOps). The need for observability For SRE, at the heart of implementing SLOs practically is monitoring. You can't understand what you can't see. A classic and common approach to monitoring is to watch for a specific value or condition, and then to trigger an alert when that value is exceeded or that condition occurs. One of the valid monitoringoutputsis logging, which is recorded for diagnosis or forensic purposes. The ELK stack, a collection of three open source projects, namely Elasticsearch, Logstash and Kibana, provides IT project stakeholders the capabilities of multi-system and multi-application log aggregation and analysis. ELKcan beutilized for the analysis and visualization of applicationmetricsthrough a centralized dashboard. With general visibility in place, tracking can be enabled in order to add a level of specificity to what is being observed.Taking advantageofiRuleon BIG-IP,NetOps can generateUUID and insertitinto the HTTP header of every HTTP request packet arriving at BIG-IP. All traffic access logs containing UUIDs,fromBIG-IP and NGINX,are sent to the ELK server, for validation of informationsuch asuser location, response time by user location, response timeetc. Through the dashboard, end-users can easily correlate North-South traffic (processed by BIG-IP) with East-West traffic (processed by NIGNX+ inside cluster), for an end-to-end performance visibility. In turn, tracking performance metrics opens up the possibility of defining service level objectives (SLO). With observability, security is possible Security incident will always occur, and hence it's essential to integrate security into observability. What’s most important is giving reliability engineers the tools so that they can identify the security problem, work around it, and fix it as quickly as possible. Using the right set of tools, you can build custom autogenerated dashboards and tooling to expose the generated information to engineers in a way that makes it much easier to sort through everything and determine the root cause of a security problem. These include things like Kibana dashboard, which allows engineers to investigate incident, apply filters, quickly pinpoint suspicious data traffic and source. In concert with F5 Advanced WAF and NGINX App Protect, SRE can protect applications against software vulnerabilities and common attacks from both inside and outsidemicroservice clusters.UponBIG-IP Advance WAF orNGINX App Protect detect suspicious traffic, it sends alert with details toELK stack, whichwillindex, and processthe data, and thenexecute the pre-defined ‘Ansible Playbook’,to enforce security policy into Kubernetes or NGINX App Protect for immediate remediation. SRE does not only identify but rectify the anomalies by enacting security policy enforcement along the data path.Detect once and protecteverywhere. What’s next? This serves as an introductiontoor the first article of this SRE article series.In the coming articles, we will deep dive into each of the use cases, to showcase the technical details about how we are leveraging F5technologies and capabilitiestohelpSRE bring together DevOps, NetOps, and SecOps to develop the safeguards and implement the best practices. To learn more about developing a business case for SRE in your organization, please reach out toanF5 Business Development. For technical details and additional information, see thisDevCentralGitHub repo.1.1KViews0likes0Comments