/
Test Runner

Greenspector Documentation has been moved.
Please go to
https://greenspectorstudio.atlassian.net/l/cp/TDhqYDSu

Test Runner

Owned by Olivier Philippot
Last updated: 11 Sept 2024
Redirection

Up TO DATE

Test Runner description

General behavior

The Test Runner:

  1. reads the configuration files (config.yml and job.yml);

  2. checks the components are installed;

  3. downloads the components if they are not present or not up to date;

  4. starts the job;

  5. sends the results to the GREENSPECTOR Server or saves them locally when the job is finished.

Job types

With the Test Runner, you will be able to launch 4 different kinds of jobs:

  • APK job to benchmark your Android application.

  • URL job to benchmark your website.

  • CUSTOM job to measure your Android application or web application with your own automated tests.

  • FREERUN job to run a manual measure of your application or website.

Configuration files

You will find below a reference guide for the 2 configuration files used by the Test Runner.

config.yml

Field in configuration file

Whether the field is necessary depending on the type of job (apk, custom or url)

Description

Default or example value

Field in configuration file

Whether the field is necessary depending on the type of job (apk, custom or url)

Description

Default or example value

platform





ALL

The platform on which the tests will run.

mobile

resultPath





ALL

The path where you want the Test Runner to save the results.

./greenspector/measures

componentsPath





ALL

The path where components used by the Test Runner are installed.

$HOME/.greenspector

greenspector







Informations on your GREENSPECTOR profile.





server



ALL

Your GREENSPECTOR Server URL.

https://app.greenspector.com



token



ALL

Your private token registered on GREENSPECTOR Server.

123456789AZERTY

proxy







The proxy may be automatically detected. If not, please configure these fields if you are using a proxy server.

In windows, Test Runner automatically detects  WPAD settings or PAC configuration

In linux or mac, Test Runner automatically detects env variable HTTPS_PROXY, HTTP_PROXY or FTP_PROXY





protocol



ALL

The protocol used by the proxy.

http or https



hostname



ALL

The proxy's URL.





port



ALL

The port used by the proxy server.





user



ALL

The username you are using to connect to the proxy server.





password



ALL

The password associated with this username.



target







Contains the settings of the targeted platform





connection



ALL

The way you are connected to the device.

wifi or usb



serial



ALL

The mobile device id, can be retrieved with adb devices. Use this if you previously set connection value to usb.





ip



ALL

The mobile IP adress on your Wi-Fi network, can be retrieved with adb devices. Use this if you previously set connection value to wifi.





port



ALL

The mobile remote port for adb on your Wi-Fi network. Use this if you previously set connection value to wifi.

5555



disconnectAfterTest



ALL

This is an optional parameter. When using adb wifi, by default Test Runner disconnect adb at the end of the test. To keep the connection at the end, please set disconnectAfterTest to false





paths





The paths to required Android SDK's tools.







adb

ALL

Path to ADB (Android Debug Bridge).

adb





aapt

ALL

Path to AAPT (Android Asset Packaging Tool).

$HOME/Android/Sdk/build-tools/27.0.2/aapt



phantomas





For module http_request in job.yml file.







path

URL

The path to the phantomas executable if not already in your path.

phantomas



hardware





Contains information about your physical measurement tool







address

ALL

USB port which the physical probe is connected to 

/dev/ttyACM0



setupCommands



ALL

You can provide any useful shell commands that will run on the device before each iteration of testing. Use this parameter if your device needs to be prepared before each iteration and you do not want to prepare it with your GDSL ou UI Automator test.

setupCommands:

  - settings put global verifier_verify_adb_installs 0

  - sh /sdcard/myscript.sh

job.yml

Field in configuration file


Whether the field is necessary depending on the type of job (apk, custom or url)

Description

Default or example value

Field in configuration file


Whether the field is necessary depending on the type of job (apk, custom or url)

Description

Default or example value

mode





ALL

The type of job you want to lauch, as explained above.

apk or custom or url or freerunner

greenspector







This block contains details about the GREENSPECTOR application in which you want to save your measures.





application













name

ALL

The name of the application .







version

ALL

The version of the application.



job







The job definition.





url



APKURL

If you previously set mode to url, it is the URL of the website you want to measure.

Otherwise it is an URL pointing to your Android APK.

You can precise a local path to your apk, a public URL to download your apk, a package name if you want to benchmark an already installed apk or a package name if you want to install from the PlayStore before running the test



You can choose an URL :

  • local path : /home/user/app/demo.apk

  • package name : com.microsoft.office.outlook





urls



APKCUSTOM

You can provide a list of APKs to be installed and eventually measured.

You can precise a local path to your apk, a public URL to download your apk, a package name if you want to benchmark an already installed apk or a package name if you want to install from the PlayStore before running the test

When using custom mode at least one of the APK should contain your instrumented tests (if you do not use tests written in Greenspector DSL with the option testsSuites)

for each url you can choose the location :

  • local path : /home/user/app/demo.apk



testPackages



CUSTOM

If you previously set mode to custom, you can provide a list of packages containing your instrumentation test cases.





testsSuites



CUSTOM

If you previously set mode to custom, you can provide groups of tests files writen with the Greenspector DSL.



For example :

testsSuites:
  - name: suite1
    testFiles:
       - "/home/user/test/steps1.testgb"
       - "home/user/test/steps2.testgb"
  - name: suite2
    testFiles:
       - "/home/user/test/steps3.testgb"



monitoredPackages



CUSTOMFREERUN

If you previously set mode to custom, you can provide a list of packages to monitor.





monitoredPackage

(Deprecated)



CUSTOMFREERUN

If you previously set mode to custom, you can provide the package to monitor. 

This field is deprecated, prefer to use monitoredPackages





extras



CUSTOM

Use to transmit extras parameters to your instrumentation with the format key, value

  • You can use predefined parameters :

    • config-skipsetupphone: if "true" then skip the setup configuration that happens before test.

    • config-waitForIdle: if "false" set idle timeout to 0

    • config-screenofftimeout: Set the time of inactivity before the screen goes to sleep (in millisecond)

    • config-screenbrightness: set brightness of the screen. value is between 0 and 255

    • config-gps: if "enabled" then activate location on phone before test

    • config-bluetooth: if "enabled" then activate bluetooth on phone before test

    • config-nfc: if "enabled" then activate NFC on phone before test

  • You can use custom parameters to fill parameters on your GDSL file

  • You can filter your custom UIAutomator tests (no GDSL tests) using those extras parameters.

https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner)





You can use custom parameters like this :

extras:
    PARAMETERNAME: "myValue"
will replace all ${PARAMETERNAME} in the GDSL file by myValue



If you use custom UIAutomator tests (no GDSL tests) you can filter the class you launch by annotation :

extras:
    annotation: com.greenspector.demo.test.MyAnnotation

(this extra parameter will be added to your uiautomator instrumentation like this -e annotation com.greenspector.demo.test.MyAnnotation)



networkMode



APKURLCUSTOM

The networkMode of the test. The phone has to be able to handle the networkMode.

WIFI, 4G, 3G or 2G. Default value is WIFI.

  

scenario



APKURL

Possible value is simple for an APK job. For a URL job, both simple and avance can be used.

APK job / simple scenario (deprecated) :

  • Reference Measure (only the mobile platform) for 20 seconds

  • Application Idle in foreground for 20 seconds

  • Application Idle in background for 20 seconds

URL job / simple scenario (deprecated) :

  • Reference Measure (only the browser) for 20 seconds

  • Website Launch for 20 seconds

  • Website Idle in foreground for 20 seconds

URL job / avance scenario:

  • Reference Measure (only the browser) for 20 seconds

  • Website Launch for 20 seconds

  • Website Idle in foreground for 20 seconds

  • Scrolling the Website for 4 seconds

  • Website Idle in background for 20 seconds

simple (deprecated) or avance



iterations



ALL

How many times you want to run the scenario.





testTimeout



ALL

Timeout for the total duration of the test. Default value is 30m, max value is 60m.

45m30s



stepName



FREERUN

Name of the step





duration



FREERUN

Duration of the test.

1m30s



authentication





If you previously set mode to apk, the Test Runner can handle an authentication form if your application has one.







actions

APK

Set of actions to interact with the authentication form, either by coordinates or by IDs.

There is currently two types of actions: click and text.

- "text;coordinates;300;400;user@mail.com"

- "text;id;session[password];userpassword"
- "click;coordinates;500;900"



browser



URL

If you previously set mode to url, it is the web browser you want to measure on.

Only Google Chrome is supported for now.

chrome



cache



URL

If you previously set mode to url, set this to true if you want the Test Runner to run a second test of your website with data in the browser cache.





online