Anatomy of a Test Case Automation Engine

December 24th at 12:00am

Just before leaving for the Holiday break I was able to demo a 1.0 version of the Test case automation engine to the Director of Engineering and lead developers. The reaction was very positive! I'd like to outline what components are used and the general thought process behind this as I may want to refer back at some future point in time.

 

Automation requirements

  • Work across all major versions of MS Windows and *NIX
  • Simplify the creation of Automation scripts
  • Minimize the time required to maintain automated scripts
  • Minimize or eliminate the need to edit the Source Code when adding test scenarios
  • Interact with test case management software (Testopia)

 

Major Components

  • NUnit 2.5.10 (NUnit download page)
  • Bash / Cygwin  (cygwin download page)
    • (Standardizing on Bash and SSH simplifies the engine quite a bit)
  • Plink & Pscp (putty downloads)
  • MS Visual Studio 2010 (Express version available)
  • NET Framework 4.0
    • This comes along with VS 2010
  • .ini configuration files
    • Used to describe automation run and general test case settings
  • Automation directives for test case generation
    • These are entered directly into the testcase management system
  • Tokens for generalized configurations
    • Specified in one of the ini files for easy Test case maintenance
  • Testcase bundles
    • Files, scripts and description html file are stored together

 

Use case scenario:

 

  1. Edit the TestCases.ini file
    1. This file contains a list of test cases that should be run this test run
  2. Add automation hosts to the TestMachines.ini file
    1. Any machine listed here will be used during the automation run
  3. Configure the TestSettings.ini file
    1. Configure high-level settings like username/password, testopia server and return datatype (xml)
    2. Also allows creation user-defined tokens to speed test case editing (explained below)
  4. Open NUnit GUI and load the automation dll
    1. When the DLL is loaded, All automated test cases for your testplan appear in the list
    2. You can set an Automated bit in Testopia to define this.
    3. At this stage the automation directives are processed and a test script is generated.
  5. Click the Run button to start the automation cycle

 

At this point, if any errors come up you will see information in the Errors and Failures and Text Output tabs. The generated test script returns a descriptive message if something goes wrong. Any test case not included in the run shows up as ignored / skipped

 

Visual Studio Project setup:

 

  • DLL / Class Library with a Main class definition
  • Reference to Nunit.framework.dll
  • Classes:
    • TestCase class
    • AutomationHelper static class
    • TestcaseGenerator static class

Class Details:

The Main class is the entry point for NUnit. It contains:

  • Testfixtures, Setup/Teardown and Test methods
  • Each Testplan is setup as a Test with a TestCaseSource method that returns a KeyValuePair with the Testcase ID and Summary. Returing both ID and Summary is helpful, as NUnit will display both per test case in the UI (helps us intuitively know what is being executed at any point in time.
  • The TestCaseSource attached to each TestPlan is generalized so we just have to pass in the TestPlan ID to get the relevant automated test cases.
  • The Test method itself accepts one argument (integer TestCaseID) and uses the generalized test case execution code found in AutomationHelper to perform the test.

The TestCase class contains all the properties associated with a Testopia TestCase. There are no methods here

The AutomationHelper class is the place where tests get executed. It contains:

  • Methods to execute commands or copy files using plink and pscp
  • Generalized test case execution methods for running tests in series or parallel
  • Handles assertions and exceptions resulting from test case execution

The TestCaseGenerator class is the most recent addition to the project. As you can guess by the name it is what actually creates the test cases. It contains:

  • Methods to interact with Testopia, pull down the Testplan XML and return a list of TestCase objects
  • Methods involved in automation script generation. For each testcase object it:
    • Parses the automation directives and creates Script Fragments for each activity
    • Creates an Automation Script by appending the Script Fragments to a Base Script which sets up script variables and does platform/environment detection
    • Generates exception handling code
    • Processes the list of Tokens defined in TestSettings.ini and replaces the token with the user specified value in the script

      Note: This is important as it allows us to specify a Token like $ParentServerIP$ in all our test cases. If we need to change the "parent server" in all cases for a run, we just alter the token and move along. A one-line change rather than hundreds of changes in individual scripts.

    • Replaces all Windows line endings (CRLF) with *NIX endings (LF). This is necessary as Bash scripts need to be \n rather than \r\n
    • Writes the script to the file system

 

Maintaining these Automated tests:

 

  • This sytem is super easy to maintain. It allows QA to update automated test directives in the same place they update test steps and results.
  • Since the automated test is described as a series of automation directives we don't have to much through scripts if we need to re-order / add / delete test steps
  • As there are user definable tokens, configuration changes that affect all test cases can be accomplished by changing a single Token definition in the ini file.

 

You could say that this is the basic philosophy behind the system:

  • Identify the general case
  • Minimize redundancy (DRY)
  • Use configuration files + Parser over custom code

This system is so simple to implement once you can identify your general cases. It will grow with our QA group.

Merry christmas!