Verify the consistency of the measures

Owned by Olivier Philippot
Last updated: 04 May 2018 by Solène MOREAU (Deactivated)
1 min read

UP TO DATE

Once you have measured your application, you should verify the consistency of the results (stability, potential errors...).

Step-by-step guide

  1. Run a minimum of iterations for each of your test cases (at least 3 iterations).
  2. On the Test Results view, check that the functional tests are all passed. If not, you can have measurements but you should not be confident about them (it might be that you were not measuring what you thought)!
  3. On the Test Results view, check that all your measures are present.
  4. On the Meter view, control the standard deviation of the measures. If the standard deviation is too high (in that case, the average of the measurse is displayed in red or orange), it means that the measures of the iterations are not stable.
  5. To improve the standard deviation, you can add iterations or deactivate some iterations (in case you think some should be considered as errors). Measures outside confidence range are displayed in color (from yellow to red).
  6. Select the Test Duration metric and verify that you have durations higher than 0.5 second. If not, it means that you have too few measurement points. In that case, you might want to modify your automated tests in order to increase the test time.



These verifications are useful at the beginning of a project but will be optional throughout the life of the project.

You can also choose the strategy consisting of not correcting the standard deviation because you want to have measures that represent the instability of your application (due to network, server...).