Production release ??? Every 2 weeks ??
Automation does form the backbone of a frequent product release plan. The acceptance criteria of a story is authored by a business analyst and they are reviewed by client. So the decision to automate any of the acceptance criteria is very close to the critical business functionality. We still do a review of scenarios to make sure it falls in a critical business scenarios to be automated. Automated test suite is hooked up in Continuous Integration engine to run with every check in of a developer. So whenever a build comes out of a pipeline (QA Stage) it has passed all the critical business scenarios. This saves a lot of time during QA to focus on specific areas/issues.
We support all leading browsers like IE8, IE9, Chrome, Firefox and Safari. Even though the automation tests do certify that functionally the app is working, however we still need to run a sanity test on multiple browsers to verify the UI elements and there alignment. You need to keep a close tab on IE as they often behave very weird 🙂
Just to publish a snapshot of the automation state project is in:
|Total number of scenarios||206|
|Number of scenarios that can be automated||168|
|Number of scenarios automated||136|
|Number of critical scenarios||114|
|Number of critical scenarios automated||107|
We use cucumber for test automation. Our build time is around 12 minutes for the full regression test suite to run. The above stat gives a lot of confidence to me as a QA and as well as a PM on the team.
Coming onto deployment, we don’t have a separate release management team. We as QA own the delivery of the product and the job of promoting and making the release. Once we have a stable build, we put it on Staging environment for our clients to test it out and give us any feedback. We have a one click deployment as our environment were set up on Heroku.
This further eases the job.