Solutions

Functional and Regression Test Automation

Business Problem

Test redesigned systems

The California Department of Justice (CalDOJ) redesigned its Automated Criminal History System (ACHS) and other Criminal Justice Information System (CJIS) applications. To make sure the redesigned systems met all requirements, thorough testing was needed.

Traditional testing approach was manual

Previous maintenance and enhancements made to ACHS and other CJIS applications were tested manually. Each testing effort was essentially a brand new testing project, with little re-use.

Costly overtime

Due to the manual nature of the testing process, each effort required a large number of test staff to work overtime. Over the life of these systems, many millions of dollars were spent for testing.

Inaccuracies

CalDOJ wanted to move from manual testing to automated testing in order to ensure a repeatable and reliable testing process. Many mistakes were made due to the manual nature of the old testing process.

Solution

Automated Testing Tool

The BCII Electronic System for Testing (BEST) application was developed by CloverLeaf to solve the problem of maintaining, executing, and evaluating results of test cases.

Web interface

The secure web interface allows users to upload or enter test case data into test projects, run tests and evaluate test responses in real-time. It also reports on the current state of test projects and test case data sets.

Role-Based authorization

User roles such as project tester, administrator, manager, technical support, etc. may be defined for each test project. Access to automated test features within a test project is granted or denied based on a user's role in that project. Users may have various levels of access to functionality across multiple projects.

Test case data

Test case data many be created and edited manually or may be recorded in Excel spreadsheets using a defined layout and then uploaded into BEST using the Web interface.

Automated results comparison

One of the most time saving features of BEST is the automated results comparison. Once a test case has a baseline, the result of subsequent executions of that test case are programmatically compared against the baseline's results. Pass/Fail status for the test case is updated automatically.

Allowing for dynamic data and/or acceptable differences

Some test result differences are acceptable. Timestamps are a common example. Also, due to enhancements as part of the redesign, some differences are by design. The comparison feature allows for ignoring defined known and accepted differences, allowing testing staff to concentrate their efforts on reviewing failed tests rather than manually reviewing all results to determine pass/fail status.

Automated release testing

During the testing, defect management, and re-testing stages of the software development life cycle, BEST test projects can be automatically called through a web service. The CJIS automated deployment process automatically calls the BEST web service to run regression tests. Test cases marked for regression testing are automatically run and their test results are compared to the baseline results. Pass/Fail status is automatically set.

Automatic test report generation

Once the release test is complete, BEST generates a report that identifies any failed tests. The report is automatically sent via email to the appropriate testing staff.

Sandbox testing

Users may be assigned to individual database schemas in the environments they are testing. This provides each tester with their own "sandbox" and prevents conflicts in testing data and results during test execution.

Automated processes

Various needs were identified for automated population of test case data. One requirement was to support dynamic date values based on the current date plus or minus a variable amount of time. Another requirement was the ability to flag identified data sets to use particular data in a test result as test case data in a subsequent test case.

CloverLeaf Added Value

While BEST was originally developed to support the ACHS Redesign testing effort, CloverLeaf recognized that this automated testing process would be beneficial to CalDOJ for other CJIS application testing efforts. Since all CJIS applications used the same basic architecture and API, BEST was developed to support them all.

Since BEST greatly reduced the time for testing and analysis for ACHS, CalDOJ has used BEST for other CJIS application testing efforts including the California Sex and Arson Registry (CSAR), Wanted Persons (WPS), Automated Firearms (AFS), Stolen Vehicles (SVS/ABS), and Supervised Release File (SRF).

Benefits

• Automates the previous manual processes which were prone to error, labor-intensive, time consuming, and relied heavily on subject matter experts.
• Greatly shortens the time required to process and analyze tests.
• Reduces the likelihood of error due to its automated and repeatable nature.