Sheet 1
HASM (Heuristic Automation Strategy Model)
Rev. 2016-06-30
  Dedicated Automation Engineer or Team
    Process Challenges
      Trust
      Automation Need
      The 100% Fallacy
        Eliminate 100% Approach/Mindsets
          Misconceptions about Test Automation
      Mentality/Approach
      Framework Decisions
      Subject Matter Expertise
      Team Knowledge
      Reporting
      Avoid GUI Automation
      Automation is Software Development
      Non-Product Purposes
    Testing Challenges
      Understanding Requirements
      Testing Priority
      Time Constraints
      Complete vs Satisfice
      Regression
      Reuse of Test Scripts
      Automation Conflict
      Project Communication
      Misdirected Focus
      Lack of Testing Skill
      Developer Relationships
      Process Misunderstanding
      Attrition
    Dedicated Automation Team
      Embedded SDETs
  Development
    Coding Practices
      Follow good practices
      Use Object Oriented Methodologies
      Definition of Done
      Awareness
      Manually Test Data/Scripts
      Test Autonomy
      No Blocking Tests
      Separate Repository
      User-Path Tests
      Test should not cross service boundaries.
      Grid Capacity
      Browser Cleanup
      Failure Screenshots
      Avoid uploading or downloading files during testing.
      Element Prioritization
      No 'Sleep'
      Timeout Settings
      Restrict web testing to the Firefox browser.
      Avoid External Dependencies
      Environment Agnostic
      Clean Configs
      Relative Paths
      Camel Casing
      Test Case Documentation
    Tags & Keywords
      Be generous with tags. Tag all the things.
      Populate tests with a "build" or "regression" tag.
        Redefine Build, Smoke and Regression
          Build: Every story goes in here, so that we can properly support Continuous Delivery?
          Regression: All stories that exist but are not part of the current "Build"
          Smoke: AKA Build Acceptance, a subset of checks that cover high-risk and highly-visible areas.
      Author Tag, Team Tag (for future owner reference)
      Tag all tests with the story ID/number.
      Tag bad/obsolete test with "IGNORE" followed by a tag explaining of why
        e.g. 'IGNORE obsolete', 'IGNORE bugBadSQL', 'IGNORE defectRejectedByProductOwner'
      Keywords should be "obvious"
        Good Keyword = loginINVPLus, loginBackEnd, clickBookOutButton
        Bad Keyword = login, clickButton, enterText
      Wiki Taxonomy
    Data Setup and Usage
      Integrated Data
      Independent Data Sets
        Separate Data Example (see note)
        Shared Data Example (see note)
    SLA (Service Level Agreement)
      100% Pass Rate
      Suite Execution Timebox - 1hr
      Case Execution Time - 5min
      Assertion Execution Timebox - 30sec
  Tooling Selection
    What are we currently using?
      Current caveats/pain points?
    What other options are out there?
      Do these have the same caveats/pain points?
    Familiarity
    Which Tool for Which Project?
  Strong Manual Test Cases
    Test Samples by Category
    Test Data Creation And Reliability
    Manual --> Automation
      Turn Manual Tests Into Automation
  Litmus Test
    Can the test sequence of actions be defined?
    Is it possible to automate the sequence of actions?
    Is it possible to “semi-automate” a test?
    Is the behavior of the software under test the same with automation as without?
    Are you testing non-UI aspects of the program?
    Do you need to run the same tests on multiple hardware configurations?
    Select Correct Test Cases for Automation Testing
      RCRCRC (Regression Heuristic)
        Recent
        Core
        Risky
        Configuration
        Repaired
        Chronic
      Define Parameters
        Test case executed with different set of data
        Test case executed with different browser
        Test case executed with different environment
        Test case executed with complex business logic
        Test case executed with different set of users
        Test case Involves large amount of data
        Test case has any dependency
        Test case requires Special data
        Time it takes to write automation
      Identify Modules (Fig 1.0)
      Aggregate Data (Fig 2.0)
      Initial Conclusions (Fig 3.0)
  Achieve Positive Automation ROI
    Attributes that deter ROI
      Purchasing and licensing cost of the tool
      Time to develop the scripts
      Time to maintain the scripts
      Time to analyze the results
        Via human
        Via Machine
      Time and cost to train the team members
      Management overheads
    Calculating ROI: Example (Fig 4.0)
Credit: Connor Roberts, Brian Kurtz
Contributing work by: Vijay Shinde, Mark Meiller and Steven Staton
Follow Connor on Twitter: https://twitter.com/ConnorRoberts
Follow Brian on Twitter: https://twitter.com/kurtz74
Note: Many of the screenshots linked to various items throughout this model are meant to be used as a teaching tool sometimes for how "not" to do things. For example, if you look at some of the ROI summaries, they neglect to account for the amount of time it takes to initially create the automation, and incorrectly start gathering metrics only after their initial run to determine cost/ROI. There is no real ROI in testing unless you are selling testing or testing tool as a service or product.
This is a basic thing to do. Don’t ask your manual testers to indulge in test automation. If you do want them to do test automation, then free them from manual testing work. Test automation is a full time job. For this you need dedicated resources.
Recommended: Build a test automation team consisting at least one automation architect. You can hire multiple automation engineers to work under the guidance of the test automation architect. The number of automation engineers depends on the number and size of your products.
We should trust our automation stuff. We invest people-hours to build an automation suite but still we don’t believe in the test results. Efforts should be given to maintain the scripts. Also we should see that the team who is doing the manual testing of the application, should be involved to automate it as they know their application. Many times, an external team does the automation so the actual development team is not aware of the scripts and finally end up with running the tests manually because they feel following up on the scripts provide too much time for too little value.
Every test team is unique and has an exclusive need for automation. We cannot develop a fix standard, but we can tailor a standard which will suite our need. Because of this reason, automation does require a good support from the management as well as from the development team.
Automating 100% application is a big task, and impossible. It requires proper planning and monitoring and of course, great amounts of time investment. There are lots of permutations and combinations of data, n number of environments with n number of authentication and authorization attributes which needs to be validated and hence requires a strategy to automate. Focus more on automating the right areas that give the most value-add, rather than trying to automate "all" the areas.
You cannot automate everything. Automation means running fewer tests more often. You have to start small by attacking your smoke tests first. Then cover your build acceptance tests. Then move onto your frequently performed tests, then move onto your time taking tests. But make sure every test you automate, it saves time for a manual tester to focus on more important things.
Automation is not here to replace manual testers. Nor it can. It is here to take the repeated work away from manual testers so that they can use their full focus and strength in finding new testing scenarios and bugs.
Automate few tests that are valuable and time savers or difficult to do for manual testers. If you did that, the task of automation is done.
Manual Vs Automation mentality: “We normally automate that which is important and repetitive, but we prefer to test the important functionality manually”. Sounds hypocritical to an extent. But, we should have criteria that will decide which are the important test cases. These criteria can be based on multiple factors like complex business logic, areas which are of more interest to clients, risk prone areas, etc.
Designing the framework is one of the most important facets of automation. Teams should devote relatively more time to develop the framework than to the script. Whenever we develop the automation plan, framework designing should be the main focus. Plan to design the framework. Identify and make a checklist of the items which will form the framework. If the framework is rock solid, scripting and maintaining becomes easy.
Know the application being tested. The tool selection depends heavily on the technologies used in your product. Know your product inside out before starting the automation.
If it is a web application, know the browsers it will support. Know the technologies being used in it. If it is a desktop application, know which language is built upon. What third-party controls are being used in the application. This will help you make the tool selection and future automation easier.
Whenever we think of automation, we immediately jump to learn the programming language or scripting language. Learning these language will definitely help but more emphasis should be on building and developing logic. Automation should not be the responsibility of some handful resources, rather entire team should contribute towards it. This will help not only to enhance the skills of the resources, but also keep them motivated.
Most every tool has a standard to report the test results. To customize it; is a challenging task. Reporting the test results also requires coordination and maintenance which adds to the cost.
When there is an alterative present, avoid GUI automation. GUI automation is always tougher than other types of automated tests. If there is a situation when you can achieve your target by not automating the GUI, but by some other methods like command line inputs, then the best strategy is to avoid GUI automation.
For example, you want to test the installation of the application. The objective is to check whether application installed or not in a particular environment. One approach is to start the installation and click on “Next” button multiple times through your automation tool. It can be tricky, time consuming and it is subject to maintenance if UI changes. The other approach is to initiate the application installation with a batch file giving silent arguments. The application will silently install showing no GUI. The objective will be achieved in less time and in a more reliable manner.
If you develop a quality piece of software, it needs guard rails and good practices. It needs code reviews being consistently performed. It needs a framework or design pattern to be followed. It needs constant maintenance.
Automation is basically software development; therefore, any practices you follow when developing software should also be followed for automation coding as well. Automation Framework should be there. Code Reviews should be done. Bugs from automation should be reported in bug repository. Source Code of automation should be placed under a source control, etc. The more you treat it like software development, the more successful automation will be.
Use Automation for other useful purposes as well. Automation is such a fantastic thing. You can achieve such things from it that you don’t normally think about. Automation is not just about programming a manual test case. Instead, you can use automation to facilitate different operations in your organization.
For example, you can use automation to create master data and setup configurations automatically for manual testers so that they can start their testing as early as possible.
Example: Your company wants to switch your test case management tool. You are moving from Test Rail to Rally. There are about 4000 manual test cases and bugs in Test Rail. Transferring them manually to Rally could take about a month if done manually.
After researching the tools we find that the DB for Test Rail is using MySQL as its repository. For Rally, we find a tool that can read test cases and bugs from an excel file, if they are written in a particular format, and can insert them into Rally. The rest of the story is simple - write a MySQL query to fetch all test cases and bugs and exported them in an Excel File in the specific format and use that tool which reads all test cases and bugs from excel file and inserted them into Rally. The whole process took only a few hours.
Is it possible? No. Long answer: There are millions of test combinations. It’s not possible to test each and every combination both in manual as well as in automation testing. If you try all these combinations you will never ship the product.
When a project or product goes on expanding the regression testing work simply becomes uncontrolled. Pressure to handle the current functionality changes, previous working functionality checks and bug tracking can fall by the wayside.
It does not necessarily require a skilled tester to handle this relationship positively, but it does require someone who is mature and willing to learn. How you complete the work and in what way it is done will require some skill. There are simply hundreds of excuses developers or testers can make when they are do not agree, and usually these are detractors that worsen the problem, rather than helping. Being a good tester is not just about testing ability, but also requires good communication, troubleshooting and analyzing skill.
Some times you just don’t pay proper attention what the company-defined processes are and these are for what purposes. There are some myths in testers that they should only go with company processes even these processes are not applicable for their current testing scenario. This results in incomplete and inappropriate application testing.
The primary purpose of this node is to provide a reliable and meaningful testing suite that can be completely automated as part of our ongoing continuous delivery strategy. It will run as part of our automated build in our CI environments when you move a story to the 'done' state. If a single test fails during the build then the whole build will fail, and the story will not get merged into master.
Writing automated acceptance tests is just like writing code. If you write an automated acceptance test you are no longer a QA team member, but now also an automation engineer. Be intelligent and deliberate with how you write your test.
Writing automated test needs to be part of your definition of done for all new features. For new features always make a task for automated testing, and do not complete a story until that task is done.
When grooming and tasking stories everyone needs to be conscious of what test a story might break. Being proactive will save you a lot of time and headaches. Do not run a build just to see what breaks.
Manually test your test and data scripts before completing a story. This is very important! A build takes a long time to run and it will be a waste of the team's time if you write an untested, invalid test that clogs the build pipeline.
Always write tests that are independent of all other tests; always assume that other test are interacting with your testing environment at the same time.
Never design a test that requires exclusive access to the system.
Test code needs to be independent from the code it's testing. That means that test code should live in a separate repository, and thus will not have any opportunity to make calls to the code base.
Test should only interact with the same things that a user can interact with. For example, with acceptance tests this means the web page. Test should never directly interact with an application's back end, such as databases, crons, or system processes. If for whatever reason your test needs these capabilities, use an API middleware service to perform these interactions.
Do not test something that is not owned and supported by your developer team. Since you cannot support third-party integrations you have no guarantee that third-party functionality will work; a simple maintenance page could stop your test in its tracks, or an updated API could return unanticipated values. Examples of things not to test: DTV, dealertrack.com, Amazon, AAX data sync, and Google services.
Never open more than two browsers per test. In fact, only use a single browser per test case. By keeping the number of open browsers to a minimum, the CI build process can throttle the number of test running in parallel without overwhelming the Selenium grid. If you open lots of browsers and overload the grid your chances of causing instability in the test suite increase resulting in unnecessary test failures.
Always explicitly close your browsers when your test completes, even on failures shut it all down. Selenium doesn't always close browsers correctly and then they just hang around eating memory.
Always take a screenshot on failure, and name it after the test case and link it in the error report.
Find elements in priority order: ID's - > name -> css -> xpath. Avoid using xpath as much as possible because it changes all the time. Developers must use IDs for new features. In addition to making your life easier, it's also better practice and developers should be doing this anyway.
Do not use Sleep (fixed delay) functions. Instead, use functions that will continue execution the instant an element loads (e.g. Robot Framework's waitForElement function). This ensures that test don't wait longer than they need to, also allows you to use a longer timeout in the event of the interment long load times that are common in a CI environment being hammered by a lot of concurrent test.
Proactively manage your timeout settings. Too high of a timeout will cause failures to make the build take a long time, too low of a timeout will cause unnecessary failures.
Restrict web testing to the Firefox browser. Smoke test should be run on both Firefox and Internet Explorer. This requirement is an optimization issue that reflects on the Selenium grid, and following this will improve overall run time for the testing suite.
Do not write test that have dependencies out side of the testing framework and testing code repo, such as third party libs. Do not include any dependencies on vendor! The testing framework should have a means to ensure the testing environment is consistent, such as Robot Framework's use of pip requires_install.txt files.
Do not write test that are dependent and interwoven with an IDE (e.g. RIDE). The build system will be executing the test directly from the command line. Also, do not write test that are dependent on the OS. We have developers on Macs (BSD Unix), on MS Windows, and our CI servers use Redhat Linux. This is especially important if your test causes the browser to open up an OS specific dialog (e.g. saving a file).
Config files are only for environment settings, do not put any data or test specific settings in the config file. You may use other "data config" files instead.
All paths should be relative to the root of the repo.
Do it.
A keyword must have a verb and a noun, to answer the questions what does the keyword do and what does it do it to.
Recommended: A Wiki Taxonomy of all tags and keywords along with directions on how to use them may be useful for multiple purposes (training new hires, handing off projects to a different team, establishing common understandings, prevent duplication/tag misuse, etc.)
The data needed to run any given regression test should be part of the team environment/CI data slice. The two major advantages are that now your data is always in a known state and you will not need to waste any test time setting up and tearing down data. As a result a test need only login and navigate to the beginning of a workflow or use case to be tested to do it's thing, all while knowing with 100% certainty about the state of the application/data.
If a test has the potential to change the state of the data it should have an independent data set. In an ideal world if you have 100 test you should have 100 data sets(dealers, users, inventories) so that no test can change the state of another test. But in the real world this is not very practical, so instead think about how your test will impact the data set that you are using and if you see potential for conflicts make separate data sets.
Example of when you will need separate data sets: One test adds a vehicle, another test runs a report that counts vehicles in inventory. In a vacuum the second test already knows how many vehicles that are in inventory and so assumes how many vehicles should show up on the report, but if during a regression the first test completes before the second then the second test's assumptions are wrong and it will fail.
Example of when you can share data sets: One test checks if common vehicle vin decoding works, another test checks if common vehicle stock number look up works. These tests have a very similar use case, but because they are just look ups and don't change the state of the data they can be ran using the same data set.
100% of the test should pass 100% of the time. If we consistently get 1% failures how much trust in the build can we really have? What if those failures are sometimes for feature A, but other times feature B? How can i be sure that they are legitimate failures or not? If we arbitrarily say 1% failures are OK how confident can we be that we aren't introducing major bugs and defects into the system? It has to be 100% or bust!
Complete test suite runs in one hour or less. Because of our branching and merging strategy we will only ever run a single build at a time; there have been days where we have completed and tagged over a dozen stories. If our test suite takes much longer than an hour, then the build will become a bottle neck in our process.
Any single test case should run in five minutes or less. Anything more is "code smell"; you're either testing too much, testing the wrong thing, or have a bad test. However, five minutes is an absolute upper max. Most test cases should be completed in only a few minutes.
Any single assertion should run in thirty seconds or less.
Selecting the right tool is just the beginning. Some managers have the misconception that if they select a right tool, they can easily automate anything. Beware, automation tools do not give you everything. They make the process easier. But you need skilled resources to complete the process.
Often automation tools are buggy and they get stuck when trying to identify complex objects on the application. The resources you hire, if they are skilled, come up with work around which takes the process forward. Otherwise, if you don’t hire good resources, no tool alone can guarantee automation success.
Select the automation tool which is familiar to your resources. If your resources are familiar with Java and your application to be tested is also developed in Java, then see if there is a value-add in selecting a tool which does not offer Java to write scripts.
Language learning is a high time-taking process. Avoid this learning curve by utilizing tools that offer minimal learning curve, but still get the job done.
Good Automation can benefit from documented test cases. While the context of your team determines if you need documented test cases, nicely written strong cases for mission critical areas (i.e. build acceptance, smoke test, etc.) can save us from automating those test cases which are easy to automate but weak in exposing risk.
Quote from the book Lessons Learned in Software Testing: “Automating without good test design may result in a lot of activity, but little value.”
It is always advisable to first write the test case in manual form. Identify all prerequisites and test data. Write steps in a clear manner and write expected results in front of each step. The objective of one test case should be clear and it should be less dependent on the other test cases. Automation engineers should run this test case manually at least once to clearly decide what objects need to be identified and what will be the flow of navigation. Ask questions with manual testers and the developers who may better understand the code being changed.
This activity sometimes helps to identify bugs even before the automation script is written. The majority of bugs are identified in the test automation development phase rather than in actual execution phase.
If you answer yes to any of these questions (especially multiple), then your test may be a good candidate for automation.
Is it useful to repeat the sequence of actions many times? Examples of this would be Acceptance tests, Compatibility tests, Performance tests, and regression tests.
This may also determine that automation is not suitable for this sequence of actions.
Automating portions of a test can speed up test execution time.
This is an important concern for performance testing.
Almost all non-UI functions can and should be automated tests.
Run ad hoc tests (Note: Ideally every bug should have an associated test case. Ad hoc tests are best done manually. You should try to imagine yourself in real world situations and use your software as your customer would. As bugs are found during ad hoc testing, new test cases should be created so that they can be reproduced easily and so that regression tests can be performed when you get to the Zero Bug Build phase.) An ad hoc test is a test that is performed manually where the tester attempts to simulate real world use of the software product. It is when running ad hoc testing that the most bugs will be found. It should be stressed that automation cannot ever be a substitute for manual testing.
Use the RCRCRC model as a lighthouse, to find what should and should not be automated.
User stories, defect reports and data model changes
The roots of the product
Features that rely on other services or components
Thinking about the environment
Retest previous defects
The areas of the product that frequently have issues
Identify the parameters on which you will base your test case as a candidate for automation.
You can add your own parameters depending on your application.
Break each application into modules. For each module, analyze and try to identify the test cases which should be automated based on the parameters. This list will vary for projects to projects and can also be enhanced to suit your needs.
(See Figure)
Y – Yes
N – No
In a similar way, for all modules, this list can be used to identify the automation candidate test cases.
Consolidate and group the number of test cases for each module shown in the figure.
Figure 2.0 is quite straight forward and self explanatory. We are trying to quantify the details and giving an estimate to finish the testing manually. Keep in mind, these are loose estimates that are just used to inform our decision making.
Once you have identified all the granular level details, you can present them in the below way. We are now progressing to calculate the ROI.
In many cases, the industry calculates ROI over a period of 5 years, but that is not mandatory. Based on the factors outlined in what deters ROI, let's try to elaborate on the calculation of ROI for 5 years. As always, this can be tailored/modified to fit your scenario.
ROI = (Cumulative saving / Investment through automation) * 100
Keep in mind, ultimately this is a surrogate measure and only one variable, that is meant to help inform, but not direct or constrict.