Greenspector Documentation has been moved.
Please go to
https://greenspectorstudio.atlassian.net/l/cp/TDhqYDSu

Test Runner

Owned by Olivier Philippot
Last updated: 11 Sept 2024
9 min read
Redirection

Up TO DATE

Test Runner description

General behavior

The Test Runner:

  1. reads the configuration files (config.yml and job.yml);

  2. checks the components are installed;

  3. downloads the components if they are not present or not up to date;

  4. starts the job;

  5. sends the results to the GREENSPECTOR Server or saves them locally when the job is finished.

Job types

With the Test Runner, you will be able to launch 4 different kinds of jobs:

  • APK job to benchmark your Android application.

  • URL job to benchmark your website.

  • CUSTOM job to measure your Android application or web application with your own automated tests.

  • FREERUN job to run a manual measure of your application or website.

Configuration files

You will find below a reference guide for the 2 configuration files used by the Test Runner.

config.yml

Field in configuration file

Whether the field is necessary depending on the type of job (apk, custom or url)

Description

Default or example value

Field in configuration file

Whether the field is necessary depending on the type of job (apk, custom or url)

Description

Default or example value

platform





ALL

The platform on which the tests will run.

mobile

resultPath





ALL

The path where you want the Test Runner to save the results.

./greenspector/measures

componentsPath





ALL

The path where components used by the Test Runner are installed.

$HOME/.greenspector

greenspector







Informations on your GREENSPECTOR profile.





server



ALL

Your GREENSPECTOR Server URL.

https://app.greenspector.com



token



ALL

Your private token registered on GREENSPECTOR Server.

123456789AZERTY

proxy







The proxy may be automatically detected. If not, please configure these fields if you are using a proxy server.

In windows, Test Runner automatically detects  WPAD settings or PAC configuration

In linux or mac, Test Runner automatically detects env variable HTTPS_PROXY, HTTP_PROXY or FTP_PROXY





protocol



ALL

The protocol used by the proxy.

http or https



hostname



ALL

The proxy's URL.





port



ALL

The port used by the proxy server.





user



ALL

The username you are using to connect to the proxy server.





password



ALL

The password associated with this username.



target







Contains the settings of the targeted platform





connection



ALL

The way you are connected to the device.

wifi or usb



serial



ALL

The mobile device id, can be retrieved with adb devices. Use this if you previously set connection value to usb.





ip



ALL

The mobile IP adress on your Wi-Fi network, can be retrieved with adb devices. Use this if you previously set connection value to wifi.





port



ALL

The mobile remote port for adb on your Wi-Fi network. Use this if you previously set connection value to wifi.

5555



disconnectAfterTest



ALL

This is an optional parameter. When using adb wifi, by default Test Runner disconnect adb at the end of the test. To keep the connection at the end, please set disconnectAfterTest to false





paths





The paths to required Android SDK's tools.







adb

ALL

Path to ADB (Android Debug Bridge).

adb





aapt

ALL

Path to AAPT (Android Asset Packaging Tool).

$HOME/Android/Sdk/build-tools/27.0.2/aapt



phantomas





For module http_request in job.yml file.







path

URL

The path to the phantomas executable if not already in your path.

phantomas



hardware





Contains information about your physical measurement tool







address

ALL

USB port which the physical probe is connected to 

/dev/ttyACM0



setupCommands



ALL

You can provide any useful shell commands that will run on the device before each iteration of testing. Use this parameter if your device needs to be prepared before each iteration and you do not want to prepare it with your GDSL ou UI Automator test.

setupCommands:

  - settings put global verifier_verify_adb_installs 0

  - sh /sdcard/myscript.sh

job.yml

Field in configuration file


Whether the field is necessary depending on the type of job (apk, custom or url)

Description

Default or example value

Field in configuration file


Whether the field is necessary depending on the type of job (apk, custom or url)

Description

Default or example value

mode





ALL

The type of job you want to lauch, as explained above.

apk or custom or url or freerunner

greenspector







This block contains details about the GREENSPECTOR application in which you want to save your measures.





application













name

ALL

The name of the application .







version

ALL

The version of the application.



job







The job definition.





url



APKURL

If you previously set mode to url, it is the URL of the website you want to measure.

Otherwise it is an URL pointing to your Android APK.

You can precise a local path to your apk, a public URL to download your apk, a package name if you want to benchmark an already installed apk or a package name if you want to install from the PlayStore before running the test



You can choose an URL :

  • local path : /home/user/app/demo.apk

  • package name : com.microsoft.office.outlook





urls



APKCUSTOM

You can provide a list of APKs to be installed and eventually measured.

You can precise a local path to your apk, a public URL to download your apk, a package name if you want to benchmark an already installed apk or a package name if you want to install from the PlayStore before running the test

When using custom mode at least one of the APK should contain your instrumented tests (if you do not use tests written in Greenspector DSL with the option testsSuites)

for each url you can choose the location :

  • local path : /home/user/app/demo.apk



testPackages



CUSTOM

If you previously set mode to custom, you can provide a list of packages containing your instrumentation test cases.





testsSuites



CUSTOM

If you previously set mode to custom, you can provide groups of tests files writen with the Greenspector DSL.



For example :

testsSuites:
  - name: suite1
    testFiles:
       - "/home/user/test/steps1.testgb"
       - "home/user/test/steps2.testgb"
  - name: suite2
    testFiles:
       - "/home/user/test/steps3.testgb"



monitoredPackages



CUSTOMFREERUN

If you previously set mode to custom, you can provide a list of packages to monitor.





monitoredPackage

(Deprecated)



CUSTOMFREERUN

If you previously set mode to custom, you can provide the package to monitor. 

This field is deprecated, prefer to use monitoredPackages





extras



CUSTOM

Use to transmit extras parameters to your instrumentation with the format key, value

  • You can use predefined parameters :

    • config-skipsetupphone: if "true" then skip the setup configuration that happens before test.

    • config-waitForIdle: if "false" set idle timeout to 0

    • config-screenofftimeout: Set the time of inactivity before the screen goes to sleep (in millisecond)

    • config-screenbrightness: set brightness of the screen. value is between 0 and 255

    • config-gps: if "enabled" then activate location on phone before test

    • config-bluetooth: if "enabled" then activate bluetooth on phone before test

    • config-nfc: if "enabled" then activate NFC on phone before test

  • You can use custom parameters to fill parameters on your GDSL file

  • You can filter your custom UIAutomator tests (no GDSL tests) using those extras parameters.

https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner)





You can use custom parameters like this :

extras:
    PARAMETERNAME: "myValue"
will replace all ${PARAMETERNAME} in the GDSL file by myValue



If you use custom UIAutomator tests (no GDSL tests) you can filter the class you launch by annotation :

extras:
    annotation: com.greenspector.demo.test.MyAnnotation

(this extra parameter will be added to your uiautomator instrumentation like this -e annotation com.greenspector.demo.test.MyAnnotation)



networkMode



APKURLCUSTOM

The networkMode of the test. The phone has to be able to handle the networkMode.

WIFI, 4G, 3G or 2G. Default value is WIFI.

  

scenario



APKURL

Possible value is simple for an APK job. For a URL job, both simple and avance can be used.

APK job / simple scenario (deprecated) :

  • Reference Measure (only the mobile platform) for 20 seconds

  • Application Idle in foreground for 20 seconds

  • Application Idle in background for 20 seconds

URL job / simple scenario (deprecated) :

  • Reference Measure (only the browser) for 20 seconds

  • Website Launch for 20 seconds

  • Website Idle in foreground for 20 seconds

URL job / avance scenario:

  • Reference Measure (only the browser) for 20 seconds

  • Website Launch for 20 seconds

  • Website Idle in foreground for 20 seconds

  • Scrolling the Website for 4 seconds

  • Website Idle in background for 20 seconds

simple (deprecated) or avance



iterations



ALL

How many times you want to run the scenario.





testTimeout



ALL

Timeout for the total duration of the test. Default value is 30m, max value is 60m.

45m30s



stepName



FREERUN

Name of the step





duration



FREERUN

Duration of the test.

1m30s



authentication





If you previously set mode to apk, the Test Runner can handle an authentication form if your application has one.







actions

APK

Set of actions to interact with the authentication form, either by coordinates or by IDs.

There is currently two types of actions: click and text.

- "text;coordinates;300;400;user@mail.com"

- "text;id;session[password];userpassword"
- "click;coordinates;500;900"



browser



URL

If you previously set mode to url, it is the web browser you want to measure on.

Only Google Chrome is supported for now.

chrome



cache



URL

If you previously set mode to url, set this to true if you want the Test Runner to run a second test of your website with data in the browser cache.





online



ALL

Set this to true if you want the Test Runner to automatically send the results of the measures to your GREENSPECTOR Server. Otherwise, set it to false, the results will be saved locally.





modules













http_request

URL

Set this to true to enable analysis of HTTP requests.







android_system_monitoring

APKURLCUSTOM

Set this to true to enable collection of additional system metrics during the job.







hardware_probe

APKURLCUSTOM

Set this to true to enable physical measurement  







wakelock_probe

APKURLCUSTOM

Set this to true to enable wakelock measurement for the first iteration





Command Line Arguments

The Test Runner can take some command line arguments, for now the following are supported:

  • --config anotherConfigFile → to specify an alternative config.yml file;

  • --jobFile anotherJobFile → to specify an alternative job.yml file.

  • --no-update → to prevent automatically update the components used by the Test Runner when you run it (prefer not using this option)



These can be useful when you want to use different configurations for different kinds of tests. For example:

./testrunner --config androidConfig.yml --jobFile androidAPKJob.yml ./testrunner --config androidConfig.yml --jobFile androidURLJob.yml

You can also use these arguments with the update command:

./testrunner --config androidConfig.yml --jobFile androidURLJob.yml update


Command Line Sub Commands

The Test Runner has subcommands:

Update components:

To update the components of GREENSPECTOR and use the last measure tools, you should run the command:



./testrunner update

This command will download the last android-service and testbench-launcher-android in the folder $HOME/.greenspector. These components will be installed on your phone before running a job.

This command is not necessary anymore if you do not use the option --no-update in your command, Test Runner will update these components automatically

Measure with a free run:

The Test Runner allows you to run a measure for a duration of time, while you can use the application you want to monitor. To do so, you should run the following commands:

This command will start a measure with no automated actions on the device so that you can run your manual scenario while monitoring your application.

Test a DSL Command :

The Test Runner allow you to test one command writen with Greenspector DSL for Android

Monitore packages :

You can use the Test Runner to install an application that you can use to run measurements on a device. The operation is similar to free run, except you don't have to use the Test Runner to launch it once it's installed on your device.

Installation

Once you have downloaded the Test Runner, you can install the monitoring application on any device you want (you have to have unlocked developer mode on the device and be connected to it via adb).

Prerequisite

You have to have a config file filled with the right information (url, token, connection information about the device...). You can check the syntax of the file above.

You also have to decide which package are to be monitored. For example, you can monitore com.android.chrome, a specific application pacakge or the android system (com.android.systemui). You can monitore multiple packages.

Command line

Start and stop measure

Once the application is installed, you can click on the icon to open it.

The button "Start measure" starts a measure and closes the application.

The button "Stop measure" stops the measure. A default name will be given if you don't name the measure before stopping it.

Get results in CSV or send them to the web interface

At the end of the measure, you can either retrieve the results in a CSV file, or send them to the web interface. Both operations delete the results from the device.

You have to have created the application you want to send to measures to before.

Limitations

The application has to be reinstalled every time the device is rebooted (to restart the monitoring service).

The application has to be reinstalled if you want to monitor another package (and the new package to monitor has to be part of the install command line.

If you launch a new measure after stopping one, the results from the previous run will be erased and replaced by the new measure.

Results

When a job is finished, if you configured the online flag in your job.yml to true, the Test Runner will automatically send the measures to your GREENSPECTOR Server.

Otherwise, results will be saved locally to the resultPath you set in config.yml, allowing you to send them later using the GREENSPECTOR Command Line Interface.

Related articles