Dashboards & Metrics: A Clear Window into Real-time Validation

Most important and time consuming piece of any validation stack are test scripts (Installation, Operational, Performance, etc..), test executions and test results.  Validation is all about authoring test scripts, executing and documenting test results.   Whether testing is manual or automated, a lot of data is collected during validation.  Only in rare cases, one will be able to obtain real-time data on test execution, testing deviations or test coverages.  Such information will enable more efficient management of validation efforts and reduce costs.  A well designed system will provide useful dashboards and metrics to project managers as well as validation personnel.  Let us look at some examples.

Requirements to Test Case Heatmaps: 

A real-time heatmap like the one show below, provides the status of each requirement (for eg: approved) as well as priority and associated risk.  This information is color coded for easy visual assimilation.  The heatmap also provides test case coverage:

  1. Number of passed test cases executed per requirement
  2. Number of failed test cases
  3. Number of blocked test cases
  4. Number of executed test cases
  5. Coverage Quality metric


Such information is invaluable to assess the current test sprint status and quality of coverage.

Test Execution Historical Graphs and KPIs:

When multiple testers are plugging away executing scripts, it is useful to have access to historical execution statistics.  Such statistics can be:

  1. Test Case Results per Session
  2. Test Case Results per Day
  3. All Test Case Results

Test Deviation KPIs and Dashboards:

Test deviation always hold the key to where a system is released to production or not.  It will be very important to obtain key deviation statistics during a test sprint.  Such information can include:

  1. Deviation Breakout by Status, Severity or Priority
  2. Deviation per Assignee by Status or by Priority