The Goldfish Effect

When you combine virtualization with auto-scaling without implementing proper controls you run the risk of scaling yourself silly or worse – broke.

You virtualized your applications. You set up an architecture that supports auto-scaling (on-demand) to free up your operators. All is going well, until the end of the month.

Applications are failing. Not just one, but all of them. After hours of digging into operational dashboards and logs and monitoring consoles you find the problem: one of the applications, which experiences extremely heavy processing demands at the end of the month, has scaled itself out too far and too fast for its environment. One goldfish has gobbled up the food and has grown too large for its bowl.

It’s not as crazy an idea as it might sound at first. If you haven’t implemented the right policies in the right places in your shiny new on-demand architecture, you might just be allowing for such a scenario to occur. Whether it’s due to unforeseen legitimate demand or a DoS-style attack without the right limitations (policies) in place to ensure that an application has scaling boundaries you might inadvertently cause a denial of service and outages to other applications by eliminating resources they need.

Automating provisioning and scalability is a Good Thing. It shifts the burden from people to technology, and it is often only through the codification of the processes IT follows in a more static, manual network to scale an application can inefficiencies be discovered and subsequently eliminated. But an easily missed variable in this equation are limitations that were once imposed by physical containment. An application can only be scaled out as far as its physical containers, and no further. Virtualization breaks applications free from its physical limitations and allows it to ostensibly scale out across a larger pool of compute resources located in various physical nooks and crannies across the data center.

But when you virtualized resources you will need to perform capacity planning in a new way. Capacity planning becomes less about physical resources and more about costs and priorities for processing. It becomes a concerted effort to strike a balance between applications in such a way that resources are efficiently used based on prioritization and criticalness to the business rather than what’s physically available. It becomes a matter of metering and budgets and factoring costs into the auto-scaling process.