Implementing BIG-IP WAF logging and visibility with ELK
Scope This technical article is useful for BIG-IP users familiar with web application security and the implementation and use of the Elastic Stack.This includes, application security professionals, infrastructure management operators and SecDevOps/DevSecOps practitioners. The focus is for WAF logs exclusively.Firewall, Bot, or DoS mitigation logging into the Elastic Stack is the subject of a future article. Introduction This article focusses on the required configuration for sending Web Application Firewall (WAF) logs from the BIG-IP Advanced WAF (or BIG-IP ASM) module to an Elastic Stack (a.k.a. Elasticsearch-Logstash-Kibana or ELK). First, this article goes over the configuration of BIG-IP.It is configured with a security policy and a logging profile attached to the virtual server that is being protected. This can be configured via the BIG-IP user interface (TMUI) or through the BIG-IP declarative interface (AS3). The configuration of the Elastic Strack is discussed next.The configuration of filters adapted to processing BIP-IP WAF logs. Finally, the article provides some initial guidance to the metrics that can be taken into consideration for visibility.It discusses the use of dashboards and provides some recommendations with regards to the potentially useful visualizations. Pre-requisites and Initial Premise For the purposes of this article and to follow the steps outlined below, the user will need to have at least one BIG-IP Adv. WAF running TMOS version 15.1 or above (note that this may work with previous version but has not been tested).The target BIG-IP is already configured with: A virtual Server A WAF policy An operational Elastic Stack is also required. The administrator will need to have configuration and administrative privileges on both the BIG-IP and Elastic Stack infrastructure.They will also need to be familiar with the network topology linking the BIG-IP with the Elastic Search cluster/infrastructure. It is assumed that you want to use your Elastic Search (ELK) logging infrastructure to gain visibility into BIG-IP WAF events. Logging Profile Configuration An essential part of getting WAF logs to the proper destination(s) is the Logging Profile.The following will go over the configuration of the Logging Profile that sends data to the Elastic Stack. Overview of the steps: Create Logging Profile Associate Logging Profile with the Virtual Server After following the procedure below On the wire, logs lines sent from the BIG-IP are comma separated value pairs that look something like the sample below: Aug 25 03:07:19 localhost.localdomainASM:unit_hostname="bigip1",management_ip_address="192.168.41.200",management_ip_address_2="N/A",http_class_name="/Common/log_to_elk_policy",web_application_name="/Common/log_to_elk_policy",policy_name="/Common/log_to_elk_policy",policy_apply_date="2020-08-10 06:50:39",violations="HTTP protocol compliance failed",support_id="5666478231990524056",request_status="blocked",response_code="0",ip_client="10.43.0.86",route_domain="0",method="GET",protocol="HTTP",query_string="name='",x_forwarded_for_header_value="N/A",sig_ids="N/A",sig_names="N/A",date_time="2020-08-25 03:07:19",severity="Error",attack_type="Non-browser Client,HTTP Parser Attack",geo_location="N/A",ip_address_intelligence="N/A",username="N/A",session_id="0",src_port="39348",dest_port="80",dest_ip="10.43.0.201",sub_violations="HTTP protocol compliance failed:Bad HTTP version",virus_name="N/A",violation_rating="5",websocket_direction="N/A",websocket_message_type="N/A",device_id="N/A",staged_sig_ids="",staged_sig_names="",threat_campaign_names="N/A",staged_threat_campaign_names="N/A",blocking_exception_reason="N/A",captcha_result="not_received",microservice="N/A",tap_event_id="N/A",tap_vid="N/A",vs_name="/Common/adv_waf_vs",sig_cves="N/A",staged_sig_cves="N/A",uri="/random",fragment="",request="GET /random?name=' or 1 = 1' HTTP/1.1\r\n",response="Response logging disabled" Please choose one of the methods below.The configuration can be done through the web-based user interface (TMUI), the command line interface (TMSH), directly with a declarative AS3 REST API call, or with the BIG-IP native REST API.This last option is not discussed herein. TMUI Steps: Create Profile Connect to the BIG-IP web UI and login with administrative rights Navigate to Security >> Event Logs >> Logging Profiles Select “Create” Fill out the configuration fields as follows: Profile Name (mandatory) Enable Application Security Set Storage Destination to Remote Storage Set Logging Format to Key-Value Pairs (Splunk) In the Server Addresses field, enter an IP Address and Port then click on Add as shown below: Click on Create Add Logging Profile to virtual server with the policy Select target virtual server and click on the Security tab (Local Traffic >> Virtual Servers : Virtual Server List >> [target virtualserver] ) Highlight the Log Profile from the Available column and put it in the Selected column as shown in the example below (log profile is “log_all_to_elk”): Click on Update At this time the BIG-IP will forward logs Elastic Stack. TMSH Steps: Create profile ssh into the BIG-IP command line interface (CLI) from the tmsh prompt enter the following: create security log profile [name_of_profile] application add { [name_of_profile] { logger-type remote remote-storage splunk servers add { [IP_address_for_ELK]:[TCP_Port_for_ELK] { } } } } For example: create security log profile dc_show_creation_elk application add { dc_show_creation_elk { logger-type remote remote-storage splunk servers add { 10.45.0.79:5244 { } } } } 3. ensure that the changes are saved: save sys config partitions all Add Logging Profile to virtual server with the policy 1.From the tmsh prompt (assuming you are still logged in) enter the following: modify ltm virtual [VS_name] security-log-profiles add { [name_of_profile] } For example: modify ltm virtual adv_waf_vs security-log-profiles add { dc_show_creation_elk } 2.ensure that the changes are saved: save sys config partitions all At this time the BIG-IP sends logs to the Elastic Stack. AS3 Application Services 3 (AS3) is a BIG-IP configuration API endpoint that allows the user to create an application from the ground up.For more information on F5’s AS3, refer to link. In order to attach a security policy to a virtual server, the AS3 declaration can either refer to a policy present on the BIG-IP or refer to a policy stored in XML format and available via HTTP to the BIG-IP (ref. link). The logging profile can be created and associated to the virtual server directly as part of the AS3 declaration. For more information on the creation of a WAF logging profile, refer to the documentation found here. The following is an example of a pa rt of an AS3 declaration that will create security log profile that can be used to log to Elastic Stack: "secLogRemote": { "class": "Security_Log_Profile", "application": { "localStorage": false, "maxEntryLength": "10k", "protocol": "tcp", "remoteStorage": "splunk", "reportAnomaliesEnabled": true, "servers": [ { "address": "10.45.0.79", "port": "5244" } ] } In the sample above, the ELK stack IP address is 10.45.0.79 and listens on port 5244 for BIG-IP WAF logs.Note that the log format used in this instance is “Splunk”.There are no declared filters and thus, only the illegal requests will get logged to the Elastic Stack.A sample AS3 declaration can be found here. ELK Configuration The Elastic Stack configuration consists of creating a new input on Logstash.This is achieved by adding an input/filter/ output configuration to the Logstash configuration file.Optionally, the Logstash administrator might want to create a separate pipeline – for more information, refer to this link. The following is a Logstash configuration known to work with WAF logs coming from BIG-IP: input { syslog { port => 5244 } } filter { grok { match => { "message" => [ "attack_type=\"%{DATA:attack_type}\"", ",blocking_exception_reason=\"%{DATA:blocking_exception_reason}\"", ",date_time=\"%{DATA:date_time}\"", ",dest_port=\"%{DATA:dest_port}\"", ",ip_client=\"%{DATA:ip_client}\"", ",is_truncated=\"%{DATA:is_truncated}\"", ",method=\"%{DATA:method}\"", ",policy_name=\"%{DATA:policy_name}\"", ",protocol=\"%{DATA:protocol}\"", ",request_status=\"%{DATA:request_status}\"", ",response_code=\"%{DATA:response_code}\"", ",severity=\"%{DATA:severity}\"", ",sig_cves=\"%{DATA:sig_cves}\"", ",sig_ids=\"%{DATA:sig_ids}\"", ",sig_names=\"%{DATA:sig_names}\"", ",sig_set_names=\"%{DATA:sig_set_names}\"", ",src_port=\"%{DATA:src_port}\"", ",sub_violations=\"%{DATA:sub_violations}\"", ",support_id=\"%{DATA:support_id}\"", "unit_hostname=\"%{DATA:unit_hostname}\"", ",uri=\"%{DATA:uri}\"", ",violation_rating=\"%{DATA:violation_rating}\"", ",vs_name=\"%{DATA:vs_name}\"", ",x_forwarded_for_header_value=\"%{DATA:x_forwarded_for_header_value}\"", ",outcome=\"%{DATA:outcome}\"", ",outcome_reason=\"%{DATA:outcome_reason}\"", ",violations=\"%{DATA:violations}\"", ",violation_details=\"%{DATA:violation_details}\"", ",request=\"%{DATA:request}\"" ] } break_on_match => false } mutate { split => { "attack_type" => "," } split => { "sig_ids" => "," } split => { "sig_names" => "," } split => { "sig_cves" => "," } split => { "staged_sig_ids" => "," } split => { "staged_sig_names" => "," } split => { "staged_sig_cves" => "," } split => { "sig_set_names" => "," } split => { "threat_campaign_names" => "," } split => { "staged_threat_campaign_names" => "," } split => { "violations" => "," } split => { "sub_violations" => "," } } if [x_forwarded_for_header_value] != "N/A" { mutate { add_field => { "source_host" => "%{x_forwarded_for_header_value}"}} } else { mutate { add_field => { "source_host" => "%{ip_client}"}} } geoip { source => "source_host" } } output { elasticsearch { hosts => ['localhost:9200'] index => "big_ip-waf-logs-%{+YYY.MM.dd}" } } After adding the configuration above to the Logstash parameters, you will need to restart the Logstash instance to take the new logs into configuration.The sample above is also available here. The Elastic Stack is now ready to process the incoming logs.You can start sending traffic to your policy and start seeing logs populating the Elastic Stack. If you are looking for a test tool to generate traffic to your Virtual Server, F5 provides a simpleWAF tester tool that can be found here. At this point, you can start creating dashboards on the Elastic Stack that will satisfy your operational needs with the following overall steps: ·Ensure that the log index is being created (Stack Management >> Index Management) ·Create a Kibana Index Pattern (Stack Management>>Index patterns) ·You can now peruse the logs from the Kibana discover menu (Discover) ·And start creating visualizations that will be included in your Dashboards (Dashboards >> Editing Simple WAF Dashboard) A complete Elastic Stack configuration can be found here – note that this can be used with both BIG-IP WAF and NGINX App Protect. Conclusion You can now leverage the widely available Elastic Stack to log and visualize BIG-IP WAF logs.From dashboard perspective it may be useful to track the following metrics: -Request Rate -Response codes -The distribution of requests in term of clean, blocked or alerted status -Identify the top talkers making requests -Track the top URL’s being accessed -Top violator source IP An example or the dashboard might look like the following:How I did it - "Visualizing Data with F5 TS and the Elastic ELK Stack"
With the F5 BIG-IP and Telemetry Streaming I have the ability to send BIG-IP metrics to a variety of third-party analytics vendors. One of the more popular of these is Elastic. Elastic's ELK Stack, (acronym for Elasticsearch, Logstash, Kibana) provides a platform where I can store, search, analyze and visualize my BIG-IP telemetry data. With said, here's an overview of "How I did it"; integrating and visualizing data with the ELK Stack. P.S. Make sure to stay for the movie. Application Services 3 Extension (AS3) There are several resources, (logging profiles, log publishers, iRules, etc.) that must be configured on the BIG-IP to enable remote logging. I utilized AS3 to deploy and manage these resources. I used Postman to apply a single REST API declaration to the AS3 endpoint. Telemetry Streaming (TS) F5's Telemetry Streaming, (TS) service enables the BIG-IP to stream telemetry data to a variety of third-party analytics providers. Aside from the aforementioned resources, configuring TS to stream to a consumer, (Logstash in this instance), is simply a REST call away. Just as I did for AS3, I utilized Postman to post a single declaration to the BIG-IP. Elastic (ELK) Stack Configuring the ELK stack to receive and ingest BIG-IP telemetry is a fairly simple process. Logstash, (the "L" in ELK) is the data processor I used to ingest data into the stack. To accomplish this, I applied the sample Logstash configuration file. The configuration file specifies, (among other items) the listener port, message format, and the Elasticsearch index naming format. Dashboards Getting telemetry data into Elasticsearch is great but only if you can make use of it. If I'm going to utilize the data, I need to visualize the data; (should probably trademark that). For visualization, i created a couple sample dashboards. The dashboards, (community-supported and perhaps not suitable for framing) report various relevant BIG-IP performance metrics and WAF incident information. F5 BIG-IP Advanced WAF Insights F5 BIG-IP Performance Metrics Check it Out Rather than walk you through the entire configuration, how about a movie? Click on the link (image) below for a brief walkthrough demo integrating F5's BIG-IP with Elastic's ELK stack using F5 Telemetry Streaming. Try it Out Liked what you saw? If that's the case, (as I hope it was) try it out for yourself. Checkout F5's CloudDocs for guidance on configuring your BIG-IP(s) with the F5 Automation Toolchain.The various configuration files, (including the above sample dashboards) used in the demo are available on the GitHub solution repository Enjoy!
Load balancing is key to successful cloud-based (dynamic) architectures
Much of the dialogue today surrounding cloud computing and virtualization is still taking the 50,000 foot view. It's all conceptual; it's all about business value, justification, interoperability, and use cases. These are all good conversations that need to happen in order for cloud computing and virtualization-based architectures to mature, but as is often the case that leaves the folks tasked with building something right now a bit on their own. So let's ignore the high-level view for just a bit and talk reality. Many folks are being tasked, now, with designing or even implementing some form of a cloud computing architecture - usually based around virtualization technology like VMWare (a March 2008 Gartner Research report predicted VMWare would likely hold 85% of the virtualization market by the end of 2008). But architecting a cloud-based environment requires more than just deploying virtual images and walking away. Cloud-based computing is going to require that architects broaden their understanding of the role that infrastructure like load balancers play in enterprise architecture because they are a key component to a successful cloud-based implementation, whether that's a small proof of concept or a complex, enterprise-wide architectural revolution. The goal of a cloud-based architecture is to provide some form of elasticity, the ability to expand and contract capacity on-demand. The implication is that at some point additional instances of an application will be needed in order for the architecture to scale and meet demand. That means there needs to be some mechanism in place to balance requests between two or more instances of that application. The mechanism most likely to be successful in performing such a task is a load balancer. The challenges of attempting to build such an architecture without a load balancer are staggering. There's no other good way to take advantage of additional capacity introduced by multiple instances of an application that's also efficient in terms of configuration and deployment. All other methods require modifications and changes to multiple network devices in order to properly distribute requests across multiple instances of an application. Likewise, when the additional instances of that application are de-provisioned, the changes to the network configuration need to be reversed. Obviously a manual process would be time consuming and inefficient, effectively erasing the benefits gained by introducing a cloud-based architecture in the first place. A load balancer provides the means by which instances of applications can be provisioned and de-provisioned automatically, without requiring changes to the network or its configuration. It automatically handles the increases and decreases in capacity and adapts its distribution decisions based on the capacity available at the time a request is made. Because the end-user is always directed to a virtual server, or IP address, on the load balancer the increase or decrease of capacity provided by the provisioning and de-provisioning of application instances is non-disruptive. As is required by even the most basic of cloud computing definitions, the end user is abstracted by the load balancer from the actual implementation and needs not care about the actual implementation. The load balancer makes one, two, or two-hundred resources - whether physical or virtual - appear to be one resource; this decouples the user from the physical implementation of the application and allows the internal implementation to grow, to shrink, and to change without any obvious affect on the user. Choosing the right load balancer at the beginning of such an initiative is imperative to the success of more complex implementations later. The right load balancer will be able to provide the basics required to lay the foundation for more advanced cloud computing architectures in the future, while supporting even the most basic architectures today. The right load balancer will be extensible. When first implementing a cloud-based architecture you need simple load balancing capabilities, and little more. But as your environment grows more complex there will likely be a need for more advanced features, like layer 7 switching, acceleration, optimization, SSL termination and redirection, application security, and secure access. The right load balancing solution will allow you to start with the basics but be able to easily provide more advanced functionality as you need it - without requiring new devices or solutions that often require re-architecture of the network or application infrastructure. A load balancer is a key component to building out any cloud computing architecture, whether it's just a simple proof of concept or an advanced, provider-oriented implementation. Related articles by Zemanta Managing Virtual Infrastructure Requires an Application Centric Approach Infrastructure 2.0: The Diseconomy of Scale Virus The next tech boom is already underway The Context-Aware Cloud Battle brewing over next-generation private clouds 4 Things You Need in a Cloud Computing InfrastructureIt’s a Trap!
Don’t get caught in the trap of thinking dynamic infrastructure is all about scalability. If it were the case that a “dynamic infrastructure” was focused solely on issues of scalability then I’d have nothing left to write. That problem, the transparent, non-disruptive scaling of applications - in both directions – has already been solved. Modern load balancers handle such scenarios with alacrity. Luckily, it’s not the case that dynamic infrastructure is all about scalability. In fact, that’s simply one facet in a much larger data center diamond named context-awareness. “Fixed, flat, predictable, no-spike workloads” do not need dynamic infrastructure. That’s the claim, anyway, and it’s one I’d like to refute now before it grows into a data center axiom. All applications in a data center benefit from a dynamic infrastructure whether they are cloud, traditional, legacy or on-premise and in many cases off-premise benefit as well (we’ll get around to how dynamic infrastructure can extend control and services to off-premise applications in a future post). So let me sum up: a dynamic infrastructure is about adaptability. It is the adaptable nature of a dynamic infrastructure that gives it the agility necessary to scale up or scale down on-demand, to adjust application delivery policies in real-time to meet the specific conditions in the network, client, and application infrastructure at the time the request and response is being handled. It is the adaptable nature of a dynamic infrastructure to be programmable - to be able to be extended programmatically – to address the unique issues that arise around performance, access, security, and delivery when network, client,and application environments collide in just the right way. The trap here is that one assumes that “fixed, flat, predictable, no-spike workload” applications are being delivered in a vacuum. That they are not delivered over the same network that is constantly under a barrage of traffic from legitimate traffic, from illegitimate traffic, from applications that are variable, bursty, and unpredictable in their resource consumption. The trap is that applications are being delivered to clients who always access the application over the same network, in the same conditions, using the same client. The trap is to assume that dynamic infrastructure is all about scalability and not about adaptability.Amazon Makes the Cloud Sticky
Stateless applications may be the long term answer to scalability of applications in the cloud, but until then, we need a solution like sticky sessions (persistence) Amazon recently introduced “stickiness” to its ELB (Elastic Load Balancing) offering. I’ve written a bit about “stickiness”, a.k.a. what we’ve called persistence for oh, nearly ten years now, before so I won’t reiterate again but to say, “it’s about time.” A description of why sticky sessions is necessary was offered in the AWS blog announcing the new feature: Up until now each Load balancer had the freedom to forward each incoming HTTP or TCP request to any of the EC2 instances under its purview. This resulted in a reasonably even load on each instance, but it also meant that each instance would have to retrieve, manipulate, and store session data for each request without any possible benefit from locality of reference. -- New Elastic Load Balancing Feature: Sticky Sessions What the author is really trying to say is that without “sticky sessions” ELB breaks applications because it does not honor state. Remember that most web applications today rely upon state (session) to store quite a bit of application and user specific data that’s necessary for the application to behave properly. When a load balancer distributes requests across instances without consideration for where that state (session) is stored, the application behavior can become erratic and unpredictable. Hence the need for “stickiness”.Building an elastic environment requires elastic infrastructure
One of the reasons behind some folks pushing for infrastructure as virtual appliances is the on-demand nature of a virtualized environment. When network and application delivery infrastructure hits capacity in terms of throughput - regardless of the layer of the application stack at which it happens - it's frustrating to think you might need to upgrade the hardware rather than just add more compute power via a virtual image. The truth is that this makes sense. The infrastructure supporting a virtualized environment should be elastic. It should be able to dynamically expand without requiring a new network architecture, a higher performing platform, or new configuration. You should be able to just add more compute resources and walk away. The good news is that this is possible today. It just requires that you consider carefully your choices in network and application network infrastructure when you build out your virtualized infrastructure. ELASTIC APPLICATION DELIVERY INFRASTRUCTURE Last year F5 introduced VIPRION, an elastic, dynamic application networking delivery platform capable of expanding capacity without requiring any changes to the infrastructure. VIPRION is a chassis-based bladed application delivery controller and its bladed system behaves much in the same way that a virtualized equivalent would behave. Say you start with one blade in the system, and soon after you discover you need more throughput and more processing power. Rather than bring online a new virtual image of such an appliance to increase capacity, you add a blade to the system and voila! VIPRION immediately recognizes the blade and simply adds it to its pools of processing power and capacity. There's no need to reconfigure anything, VIPRION essentially treats each blade like a virtual image and distributes requests and traffic across the network and application delivery capacity available on the blade automatically. Just like a virtual appliance model would, but without concern for the reliability and security of the platform. Traditional application delivery controllers can also be scaled out horizontally to provide similar functionality and behavior. By deploying additional application delivery controllers in what is often called an active-active model, you can rapidly deploy and synchronize configuration of the master system to add more throughput and capacity. Meshed deployments comprising more than a pair of application delivery controllers can also provide additional network compute resources beyond what is offered by a single system. The latter option (the traditional scaling model) requires more work to deploy than the former (VIPRION) simply because it requires additional hardware and all the overhead required of such a solution. The elastic option with bladed, chassis-based hardware is really the best option in terms of elasticity and the ability to grow on-demand as your infrastructure needs increase over time. ELASTIC STORAGE INFRASTRUCTURE Often overlooked in the network diagrams detailing virtualized infrastructures is the storage layer. The increase in storage needs in a virtualized environment can be overwhelming, as there is a need to standardize the storage access layer such that virtual images of applications can be deployed in a common, unified way regardless of which server they might need to be executing on at any given time. This means a shared, unified storage layer on which to store images that are necessarily large. This unified storage layer must also be expandable. As the number of applications and associated images are made available, storage needs increase. What's needed is a system in which additional storage can be added in a non-disruptive manner. If you have to modify the automation and orchestration systems driving your virtualized environment when additional storage is added, you've lost some of the benefits of a virtualized storage infrastructure. F5's ARX series of storage virtualization provides that layer of unified storage infrastructure. By normalizing the namespaces through which files (images) are accessed, the systems driving a virtualized environment can be assured that images are available via the same access method regardless of where the file or image is physically located. Virtualized storage infrastructure systems are dynamic; additional storage can be added to the infrastructure and "plugged in" to the global namespace to increase the storage available in a non-disruptive manner. An intelligent virtualized storage infrastructure can further make more efficient the use of the storage available by tiering the storage. Images and files accessed more frequently can be stored on fast, tier one storage so they are loaded and execute more quickly, while less frequently accessed files and images can be moved to less expensive and perhaps less peformant storage systems. By deploying elastic application delivery network infrastructure instead of virtual appliances you maintain stability, reliability, security, and performance across your virtualized environment. Elastic application delivery network infrastructure is already dynamic, and offers a variety of options for integration into automation and orchestration systems via standards-based control planes, many of which are nearly turn-key solutions. The reasons why some folks might desire a virtual appliance model for their application delivery network infrastructure are valid. But the reality is that the elasticity and on-demand capacity offered by a virtual appliance is already available in proven, reliable hardware solutions today that do not require sacrificing performance, security, or flexibility. Related articles by Zemanta How to instrument your Java EE applications for a virtualized environment Storage Virtualization Fundamentals Automating scalability and high availability services Building a Cloudbursting Capable Infrastructure EMC unveils Atmos cloud offering Are you (and your infrastructure) ready for virtualization?Cloud Computing: It's the destination, not the journey that is important
How the cloud acts and is used is more important than where it physically resides Cloud computing and SOA suffer from the same lack of prescriptive architectures. They are defined by how they act rather than what they are, or from what they are composed. They are, in a way, existential technology that cannot be confined to a simple architectural diagram but require instead a set of properties or ways of acting in order to be recognized. To over simplify and paraphrase Jean-Paul Sartre's concepts of existentialism, we define ourselves (mankind) through our actions. To apply this to technology is a fairly easy thing: some technology is defined through what it does rather than what it is. Cloud computing is nothing but the way in which an infrastructure deploys and delivers applications. That will surely irritate cloud purists as much as the impure use of object-oriented principles used to annoy me when I was first developing applications. But with age and experience comes wisdom, and the hind-sight to see that there are many roads which lead to the same end. Unlike many philosophical theories, with technology it often is the destination and not the journey that is important. Definition of Cloud Computing The First Principle of Existentialism "Gartner defines cloud computing (hereafter referred to as "cloud") as a style of computing where massively scalable IT-related functions and information are provided as a service across the Internet, potentially to multiple external customers, where the consumers of the services need only care about what the service does for them, not how it is implemented. Cloud is not an architecture, a platform, a tool, an infrastructure, a Web site or a vendor. It is a style of computing. Many architectures can be used to support its implementation and use. For example, it is possible to use cloud in private enterprises to build private clouds, but there is only one public cloud based on the Internet." SOURCE: GARTNER RESEARCH ID: G00157908 28 MAY 2008 Man is nothing but what he makes of himself. SOURCE "ESSAYS IN EXISTENTIALISM" JEAN-PAUL SARTRE 1965 The First Principle of Cloud Computing Cloud computing is nothing but the way in which an infrastructure deploys and delivers applications. Many pundits argue that the "cloud" in "cloud computing" is the Internet, and only the Internet. But it's telling that almost every application architecture diagram offered up uses the same "cloud" 'to represent the network, whether it's internal or external to the organization. That "cloud" represents abstraction, obfuscation, and is meant to show that there is a network responsible for delivering the applications depicted, it's just too complex (or sensitive) to be depicted in a diagram or, as is more often the case, the folks responsible for application infrastructure aren't concerned about the network infrastructure supporting the applications. Which brings us back full circle to the definition of cloud computing, which includes a lack of concern regarding the implementation details of how applications get delivered. As Gartner has posited, the cloud is not an architecture, a platform, a tool, or an infrastructure. It's not a web site, it's not a vendor. "It is a style of computing." It is a deployment model, much in the same way SOA is a style of computing; it is a deployment model and not a prescriptive architecture. It defines itself by how it acts, not what it is. If it is used to deliver applications in a way that is transparent, that does not require the end-user to understand (or concern themselves with) the underlying infrastructure - application and network - then it likely fits under the moniker "cloud computing". If the same principles used by vendors like Amazon, BlueLock, Joyent, and Microsoft are used by organizations to implement a dynamic, scalable on-demand application and network infrastructure, does it really matter where that infrastructure physically resides? If Microsoft deploys an application in its own cloud, in its data center, and then makes use of that cloud for internal organizational applications, is it still cloud computing? Yes, of course it is. So why should it matter if an enterprise does the same thing? It's still cloud computing based on how the infrastructure acts and what it delivers, not where it is or who uses it. What's important is what the cloud infrastructure does. Scalability, transparency, supporting an on-demand computing model. That's what cloud computing is, whether it's implemented as SaaS (Software as a Service) or as IaaS (Infrastructure as a Service) or using automated virtualization solutions within the data center. As long as the infrastructure you build out is capable of providing the benefits of cloud computing: efficiency, scalability, and agility, you're doing cloud computing. That sounds a lot easier than is, because you have to scale while being efficient, and you have to be agile without sacrificing scalability, and you have to do it in a way that the end-user (who may be a developer) doesn't need to know how its implemented. So rather than worry about how, worry about what. At the end of your implementation is your infrastructure agile? Is it scalable (transparently)? Is it efficient? Is it abstracted? Does it support on-demand computing without sacrificing those properties? If it is, then you've reached your cloud computing destination.Caveat Emptor: Be sure to align your goals for cloud computing with provider models before you sign up
Elasticity (adj) the ability of a cloud computing environment to expand or contract automatically on-demand according to real-time computing needs One of the promises of an on-demand cloud computing environment (that's redundant, I think) is the ability to burst resources. Much in the same way that ISPs have long offered contracts that include the ability of the organization to exceed its allotted bandwidth for a fee, it is expected that cloud computing providers offer a mechanism for "bursting resources" that allows an organization to exceed its agreed upon resources for a fee, based on any number of factors such as bandwidth, requests per second, server resources consumed, etc... Surprisingly, some of the providers being grouped under the "cloud computing provider" moniker do not offer the ability to burst resources. Seriously. I was reading through "Who provides what in the cloud" over at InfoWorld and came across this description of a cloud computing provider: Terremark Worldwide: Resource pool for on-demand servers The Terremark Enterprise Cloud is designed to give datacenters an Internet-optimized computing infrastructure. Enterprise Cloud clients buy a dedicated resource pool of processing, memory, storage, and networking, from which they can deploy servers on demand. A Web portal allows server to be dynamically provisioned from a pre-allocated pool of dedicated computer resources. Terremark promises that its cloud servers behave exactly like their physical counterparts, allowing applications to be run without modification. That sounds more like managed hosted services than cloud computing. There's no indication whether you can "burst" resources if you consume all the dedicated resources you have purchased, but the requirement to "buy a dedicated resource pool" pretty much seems to say that you can burst within your resource pool, but you can't burst beyond that limit. Certainly there's always going to be a physical limit on resources in a cloud computing environment. After all,the provider only has so many resources you can use and, just like in your data center, you can't expand beyond the physical limits of your servers. But artificially limiting resource availability based on what you buy seems to run counter to most often cited benefits of cloud computing: scalability and efficiency. If you aren't using all the dedicated resources, isn't that inefficient? If you can't scale beyond your dedicated resources, isn't that limiting your scalability? Yes, it does. But if the reason you're looking at "cloud" computing is to reduce management and maintenance and procurement costs, then a dedicated "cloud" model is probably a good one. It mitigates concerns over the provider having enough compute resources to ensure service level agreements and availability by dedicating resources to your applications, yet provides some flexibility associated with truly elastic cloud computing within that dedicated pool. If, however, you're looking at cloud computing primarily to gain the benefits of scalability and efficient use of resources, then a dedicated cloud computing solution is probably not for you. So make sure you know why you're considering cloud computing before you start shopping around and align those goals by choosing a provider with a model that suits your needs. Otherwise you may find yourself needing an umbrella.
