F5 OpenStack Testing Methodology
Why We Test
The importance of testing in software development cannot be understated. Why testing is important is the subject of hundreds of books, blog posts and PowerPoint presentations. The F5 OpenStack development team has bought into the idea presented in those books, blogs and presentations: that thorough testing improves software quality. In very brief and basic terms, well-executed testing suites provide:
- proof that new features work as expected,
- assurances that new code does not break existing functionality.
How We Test
Almost all current development projects adhere to test strategies that range from a general belief that tests should be automated to following a specific philosophy, like that defined by Behavior Driven Development. The challenge for our team was to choose from that range appropriate practices that will work well for OpenStack development. The limiting issues were:
- Resources: Physical servers and physical networks.
- Orchestration: Automating the deployment of operating systems and OpenStack installations on available servers and networks.
- Time: We have to pay particular attention to how our testing strategy will scale given testing parameters such as base operating systems, OpenStack releases, network configurations, and BIG-IP releases.
We divided our testing efforts into three familiar categories: unit, functional, and integration. Testing in all of these categories is automated, meaning that tests are triggered by specific events, such as code check-ins, and by scheduled nightly regression runs.
Unit Testing
The goals of unit tests are to provide sanity checks for our software. The questions we are ask are simple ones: Can we instantiate classes? Given inappropriate parameters, will our software fail gracefully? Commits into an F5 OpenStack project - in almost all cases - must include unit tests. Unit tests require almost no resources. We do not need an OpenStack cloud and we do not need access to a BIG-IP to run unit tests.
Functional Testing
As the name implies. functionality is the key concern of these tests. Our Functional tests are particularly useful during development. Since these tests require few resources, our developers can quickly set up a testing environment and, likewise, quickly get feedback on the functionality of the code they are writing.
Our functional tests only require access to a BIG-IP. The OpenStack team has developed infrastructure to support functional testing without the use of a deployed OpenStack Cloud. Instead, the OpenStack cloud is essentially mocked to generate messages that would come from an OpenStack system.
Tempest Testing
The OpenStack project Tempest provides a wealth of testing provided by the OpenStack community. The Tempest test are invaluable. Our small OpenStack team would never be able to create the volume and depth of testing that is provided by the Tempest project. Tempest tests are difficult to set up, but the effort is well worth it given the value they deliver. Having seen the that value, we've made writing tests that we can contribute back to the Tempest project part of every feature we develop for OpenStack.
Tempest testing requires the most resources. All Tempest tests are run against a functional OpenStack cloud with F5 plugins installed and at least one integrated BIG-IP. Tempest testing falls into several different categories: integration, API, and end to end functionality.
Our Experience
Well-written tests that run automatically at key moments during development is just part of modern software development, from our point of view. That being said, our success with testing did not come easily. Writing unit tests requires discipline. We had to create infrastructure for our functional tests. Configuring and running Tempest tests was more difficult then we expected.
Our efforts were rewarded. Our unit tests have not only caught problems very early on, when the code was being committed, but those tests also generated vigorous discussion about how the code was structured to begin with. Being able to quickly set up and run our functional tests has sped up our development cycles. And most rewarding of all: Tempest found problems in our code. Problems we did not initially test for. Problems we found early on. Problems we could fix before the code was released to our customers and the OpenStack community.