Start simple and iterate forward. It’s an expression that we know well as a key tenet of DevOPS and of Agile in general. It’s also a concept that applies very well to Quality Assurance. While quality is a key piece of the Continuous Improvement model we find that it is commonly a weak aspect of that cycle for many organizations. Our job today is to help describe a simple path forward by which growth and improvement in this area can be accomplished. The end goal will will be a quality contribution to continuous delivery and integration that matures along with all other focused contributions to the pipeline.
Anyone involved with delivering software features to customers knows the feeling of schedule and delivery pressure. As we iterate through the features that are to be delivered there needs to be attention and effort put into Quality Assurance. This effort ensures that we are validating the quality of the delivered features and enabling technologies at every section of our delivery pipeline. As an organization moves and grows it’s important to ensure that time and attention is given to maturing the QA effort. It is this continued effort and vigilance that will provide the greatest opportunity for the customer to receive the features they want in a timely manner along with a quality experience. As with any process of maturity we need to start at the beginning.
The Big Challenge - Make it Manual
Start simple. The first step of maturing the QA contribution involves engineers responsible for that effort sitting down with the principal feature developers and integration engineers, having a conversation and taking notes. Items that should be captured involve the following:
- Supporting Technologies
- Key values driving the pipeline
- Delivered Features
- Disaster Recovery
- Upgrade and Migration
If this sounds a bit like the process of describing a product at a high level then that’s deliberate. The art of creating and maturing QA as part of a continuous delivery pipeline is for that portion of the engineering team to understand the end to end view of the product. This view helps to identify the focus and effort that will be needed at each stage to test. During the conversation with the principal feature developers it is imperative that QA teams understand both the feature being delivered and the enabling technologies that make that possible. During the conversation with the integration engineers the key details of the infrastructure deployment and configuration must be identified. It is also imperative that notes and documentation be the output from this process. These notes will be key in driving the first stage of testing and the will play a critical role in how the testing process matures over time. The effort to create the first manual testing should cover answers to the following:
- What results should be visible to the user?
- What services should be installed and listening?
- What connections for those services will be present?
- What constraints on access or operation should be in place?
The collection of manual tests that come out of this effort constitutes the first stage of maturity for QA. Deployment of features must be consistent and easy reproducible. So, too, must these qualities apply to the manual tests for those features. As existing features are matured and new features added this process should continue. This continuous improvement effort allows existing tests to be refined and improved over time. As new features are added a process for capturing the what, how and why for QA becomes part of the kickoff pattern for each sprint. This allows the collection of manual tests to be expanded to continue coverage of the features being delivered.
The Baseline - Automating Technical Testing
Manual testing is a great first step as it provides the starting point for validating what has been delivered. Manual testing is, however, expensive, time consuming and is prone to errors from attention to detail that must be present from the engineer on each testing pass. The next stage of maturity for QA involves the first testing automation. This effort needs to focus on technical aspects of the pipeline that are testable. Testing automation should cover the following areas:
- Validation of deployed infrastructure
- Validation of access between infrastructure layers
- Validation of the Services that were deployed
- Validation of the collection, storage and distribution of key values
Where possible each collection of automated tests should define both positive and negative testing to help validate the results being reported. The automation of manual tests does not mean that these tests are being performed automatically with deployment of the pipeline as that comes later. Each test has it’s own collection of automation that has been written to validate that the required result was achieved. Automation allows consistent execution, polling and collection of results to the engineer running the test. Delivered with this automation must be the documentation of both the design and implementation of the testing as well as how to execute the testing. This documentation is a map for both tuning the automation over time as well as a way for a new team member to learn. As existing features are matured and new features introduced expansion of existing testing is needed to ensure full coverage at each stage of the pipeline.
The Intermediate Step - Automating Functional Testing
The third stage of QA Maturity involves automating the functional testing of the delivered features. The ability to define testing automation for delivered functionality requires a solid understanding of the features that are being delivered. In our experience this is an area where the truly talented QA Engineers will shine. Their ability to not only understand the features but the enabling infrastructure allows for better testing to be created. This understanding drives not only what and how to test but the edge cases that should be considered and accounted for in the testing automation. The fruit from this endeavor is testing automation that fully captures the primary and known edge cases for the features being tested as well as the documentation to support the work completed. As with the prior stage of maturity for QA this stage of testing is not driven from the pipeline but is intended to be wrapped in automation that can be run on demand. The automation is designed to provide a consistent and reproducible test coverage for the features that were delivered and present at each stage of the pipeline. As the each new feature is available for delivery through the pipeline there must also be corresponding testing and documentation to cover the new work.
Advanced - Full Testing Automation
The fourth stage of QA maturity within the Continuous Delivery pipeline is the integration of the work completed in the prior stages into the pipeline automation itself. The tremendous effort that was put into the prior stages now pays off. Each section of the deployment pipeline has it’s own set of testing automation to validate success. The results from the testing for each section becomes the green light to trigger the next section’s deployment and testing automation. Proceeding on to the next stage only occurs after the testing automation determines that all required functionality was delivered and is working properly. Testing in this manner is also the most cost-effective, consistent and reliable method. Automation captures all of the scrutiny of the best engineering effort without the ongoing labor to drive the execution and evaluation of each pass. Each section of deployment and testing automation works together to ensure that the customer only receives product and features that has been rigorously tested and proven.
The maturing of QA as part of a continuous integration and delivery pipeline doesn’t happen overnight. It is, however, something that can be achieved with simple iterations. Each iteration moves the overall effort forward and adds a little more maturity to the process. As growth in the current stage of maturity occurs it will enable movement to the next stage. The end result is a mature pipeline where the customer experience with delivered features has truly benefited from our best efforts.