Status


colour	Green
title	Up TO DATE

Table of Contents

Test Runner description

General behavior

The Test Runner:

reads the configuration files (config.yml and job.yml);
checks the components are installed;
downloads the components if they are not present or not up to date;
starts the job;
sends the results to the GREENSPECTOR Server or saves them locally when the job is finished.

Job types

With the Test Runner, you will be able to launch 4 different kinds of jobs:

Status
colour Yellow
title APK
job to benchmark your Android application.
Status
colour Blue
title URL
job to benchmark your website.
Status
colour Green
title CUSTOM
job to measure your Android application or web application with your own automated tests.
Status
colour Red
title FREERUN
job to run a manual measure of your application or website.

Configuration files

You will find below a reference guide for the 2 configuration files used by the Test Runner.

config.yml

Field in configuration file Whether the field is necessary depending on the type of job (apk, custom or url) Description Default or example value

platform

Status

title	ALL

The platform on which the tests will run.

mobile

resultPath

Status

title	ALL

The path where you want the Test Runner to save the results.

./greenspector/measures

componentsPath

Status

title	ALL

The path where components used by the Test Runner are installed.

$HOME/.greenspector

greenspector

Informations on your GREENSPECTOR profile.

server

Status

title	ALL

Your GREENSPECTOR Server URL.

https://app.greenspector.com

token

Status

title	ALL

Your private token registered on GREENSPECTOR Server.

123456789AZERTY

proxy

Configure these fields if you are using a proxy server.

protocol

Status

title	ALL

The protocol used by the proxy.

http or https

hostname

Status

title	ALL

The proxy's URL.

port

Status

title	ALL

The port used by the proxy server.

user

Status

title	ALL

The username you are using to connect to the proxy server.

password

Status

title	ALL

The password associated with this username.

target

Contains the settings of the targeted platform

connection

Status

title	ALL

The way you are connected to the device.

wifi or usb

serial

Status

title	ALL

The mobile device id, can be retrieved with adb devices. Use this if you previously set connection value to usb.

ip

Status

title	ALL

The mobile IP adress on your Wi-Fi network, can be retrieved with adb devices. Use this if you previously set connection value to wifi.

port

Status

title	ALL

The mobile remote port for adb on your Wi-Fi network. Use this if you previously set connection value to wifi.

5555

disconnectAfterTest

Status

title	ALL

This is an optional parameter. When using adb wifi, by default Test Runner disconnect adb at the end of the test. To keep the connection at the end, please set disconnectAfterTest to false

paths

The paths to required Android SDK's tools.

adb

Status

title	ALL

Path to ADB (Android Debug Bridge).

adb

aapt

Status

title	ALL

Path to AAPT (Android Asset Packaging Tool).

$HOME/Android/Sdk/build-tools/27.0.2/aapt

phantomas

For module http_request in job.yml file.

path

Status

colour	Blue
title	URL

The path to the phantomas executable if not already in your path.

phantomas

hardware Contains information about your physical measurement tool

address

Status

title	ALL

USB port which the physical probe is connected to

/dev/ttyACM0

setupCommands

Status

title	ALL

You can provide any useful shell commands that will run on the device before each iteration of testing. Use this parameter if your device needs to be prepared before each iteration and you do not want to prepare it with your GDSL ou UI Automator test.

setupCommands:

- settings put global verifier_verify_adb_installs 0

- sh /sdcard/myscript.sh

job.yml

Field in configuration file Whether the field is necessary depending on the type of job (apk, custom or url) Description Default or example value

mode

Status

title	ALL

The type of job you want to lauch, as explained above.

apk or custom or url or freerunner

greenspector This block contains details about the GREENSPECTOR application in which you want to save your measures.

application

name

Status

title	ALL

The name of the application .

version

Status

title	ALL

The version of the application.

job The job definition.

url

Status

colour	Yellow
title	APK

Status

colour	Blue
title	URL

If you previously set mode to url, it is the URL of the website you want to measure.

Otherwise it is an URL pointing to your Android APK.

You can precise a local path to your apk, a public URL to download your apk, a package name if you want to benchmark an already installed apk or a package name if you want to install from the PlayStore before running the test

You can choose an URL :

local path : /home/user/app/demo.apk

public URL : https://dl.greenspector.com/demo/demo.apk

package name : com.microsoft.office.outlook

urls

Status

colour	Yellow
title	APK

Status

colour	Green
title	CUSTOM

You can provide a list of APKs to be installed and eventually measured.

You can precise a local path to your apk, a public URL to download your apk, a package name if you want to benchmark an already installed apk or a package name if you want to install from the PlayStore before running the test

When using custom mode at least one of the APK should contain your instrumented tests (if you do not use tests written in Greenspector DSL with the option testsSuites)

for each url you can choose the location :

local path : /home/user/app/demo.apk

public URL : https://dl.greenspector.com/demo/demo.apk
package name : com.microsoft.office.outlook

testPackages

Status

colour	Green
title	CUSTOM

If you previously set mode to custom, you can provide a list of packages containing your instrumentation test cases.

testsSuites

Status

colour	Green
title	CUSTOM

If you previously set mode to custom, you can provide groups of tests files writen with the Greenspector DSL.

For example :

testsSuites:
- name: suite1
testFiles:
- "/home/user/test/steps1.testgb"
- "home/user/test/steps2.testgb"
- name: suite2
testFiles:
- "/home/user/test/steps3.testgb"

monitoredPackages

Status

colour	Green
title	CUSTOM

Status

colour	Red
title	FREERUN

If you previously set mode to custom, you can provide a list of packages to monitor.

monitoredPackage

(Deprecated)

Status

colour	Green
title	CUSTOM

Status

colour	Red
title	FREERUN

If you previously set mode to custom, you can provide the package to monitor.

This field is deprecated, prefer to use monitoredPackages

extras

Status

colour	Green
title	CUSTOM

Use to transmit extras parameters to your instrumentation with the format key, value

You can use predefined parameters :
- config-skipsetupphone: if "true" then skip the setup configuration that happens before test.
- config-waitForIdle: if "false" set idle timeout to 0
- config-screenofftimeout: Set the time of inactivity before the screen goes to sleep (in millisecond)
- config-screenbrightness: set brightness of the screen. value is between 0 and 255
- config-gps: if "enabled" then activate location on phone before test
- config-bluetooth: if "enabled" then activate bluetooth on phone before test
- config-nfc: if "enabled" then activate NFC on phone before test

You can use custom parameters to fill parameters on your GDSL file

You can filter your custom UIAutomator tests (no GDSL tests) using those extras parameters.

https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner)

You can use custom parameters like this :

extras:
PARAMETERNAME: "myValue"
will replace all ${PARAMETERNAME} in the GDSL file by myValue

If you use custom UIAutomator tests (no GDSL tests) you can filter the class you launch by annotation :

extras:
annotation: com.greenspector.demo.test.MyAnnotation

(this extra parameter will be added to your uiautomator instrumentation like this -e annotation com.greenspector.demo.test.MyAnnotation)

networkMode

Status

colour	Yellow
title	APK

Status

colour	Blue
title	URL

Status

colour	Green
title	CUSTOM

The networkMode of the test. The phone has to be able to handle the networkMode.

WIFI, 4G, 3G or 2G. Default value is WIFI.

scenario

Status

colour	Yellow
title	APK

Status

colour	Blue
title	URL

Possible value is simple for an APK job. For a URL job, both simple and avance can be used.

APK job / simple scenario (deprecated) :

Reference Measure (only the mobile platform) for 20 seconds
Application Idle in foreground for 20 seconds
Application Idle in background for 20 seconds

URL job / simple scenario (deprecated) :

Reference Measure (only the browser) for 20 seconds
Website Launch for 20 seconds
Website Idle in foreground for 20 seconds

URL job / avance scenario:

Reference Measure (only the browser) for 20 seconds
Website Launch for 20 seconds
Website Idle in foreground for 20 seconds
Scrolling the Website for 4 seconds
Website Idle in background for 20 seconds

simple (deprecated) or avance

iterations

Status

title	ALL

How many times you want to run the scenario.

testTimeout

Status

title	ALL

Timeout for the total duration of the test. Default value is 30m, max value is 60m.

45m30s

stepName

Status

colour	Red
title	FREERUN

Name of the step

duration

Status

colour	Red
title	FREERUN

Duration of the test.

1m30s

authentication If you previously set mode to apk, the Test Runner can handle an authentication form if your application has one.

actions

Status

colour	Yellow
title	APK

Set of actions to interact with the authentication form, either by coordinates or by IDs.

There is currently two types of actions: click and text.

- "text;coordinates;300;400;user@mail.com"

- "text;id;session[password];userpassword"
- "click;coordinates;500;900"

browser

Status

colour	Blue
title	URL

If you previously set mode to url, it is the web browser you want to measure on.

Only Google Chrome is supported for now.

chrome

cache

Status

colour	Blue
title	URL

If you previously set mode to url, set this to true if you want the Test Runner to run a second test of your website with data in the browser cache.

online

Status

title	ALL

Set this to true if you want the Test Runner to automatically send the results of the measures to your GREENSPECTOR Server. Otherwise, set it to false, the results will be saved locally.

modules

http_request

Status

colour	Blue
title	URL

Set this to true to enable analysis of HTTP requests.

android_system_monitoring

Status

colour	Yellow
title	APK

Status

colour	Blue
title	URL

Status

colour	Green
title	CUSTOM

Set this to true to enable collection of additional system metrics during the job.

hardware_probe

Status

colour	Yellow
title	APK

Status

colour	Blue
title	URL

Status

colour	Green
title	CUSTOM

Set this to true to enable physical measurement

wakelock_probe

Status

colour	Yellow
title	APK

Status

colour	Blue
title	URL

Status

colour	Green
title	CUSTOM

Set this to true to enable wakelock measurement for the first iteration

Command Line Arguments

The Test Runner can take some command line arguments, for now the following are supported:

--config anotherConfigFile → to specify an alternative config.yml file;
--jobFile anotherJobFile → to specify an alternative job.yml file.
--no-update → to prevent automatically update the components used by the Test Runner when you run it (prefer not using this option)

These can be useful when you want to use different configurations for different kinds of tests. For example:

Code Block
./testrunner --config androidConfig.yml --jobFile androidAPKJob.yml ./testrunner --config androidConfig.yml --jobFile androidURLJob.yml

You can also use these arguments with the update command:

Code Block
./testrunner --config androidConfig.yml --jobFile androidURLJob.yml update

Command Line Sub Commands

The Test Runner has subcommands:

Update components:

To update the components of GREENSPECTOR and use the last measure tools, you should run the command:

Code Block
./testrunner update

This command will download the last android-service and testbench-launcher-android in the folder $HOME/.greenspector. These components will be installed on your phone before running a job.

This command is not necessary anymore if you do not use the option --no-update in your command, Test Runner will update these components automatically

Measure with a free run:

The Test Runner allows you to run a measure for a duration of time, while you can use the application you want to monitor. To do so, you should run the following commands:

Code Block

./testrunner freerunner init # This line will help you generate the configuration file that you will have to update with some details
# edit the generated files (config.yml and job.yml)
./testrunner freerunner start # This line simply starts the measure
# or
./testrunner freerunner start --duration 1m30s --stepName "New Step Name" # You can pass the duration and the step name as flags so that you don't have to update the job file every time you change these values

This command will start a measure with no automated actions on the device so that you can run your manual scenario while monitoring your application.

Test a DSL Command :

The Test Runner allow you to test one command writen with Greenspector DSL for Android

Code Block
# For example, to start google chrome: ./testrunner testdslcommand --command applicationStart,com.android.chrome

Monitore packages :

You can use the Test Runner to install an application that you can use to run measurements on a device. The operation is similar to free run, except you don't have to use the Test Runner to launch it once it's installed on your device.

Installation

Once you have downloaded the Test Runner, you can install the monitoring application on any device you want (you have to have unlocked developer mode on the device and be connected to it via adb).

Prerequisite

You have to have a config file filled with the right information (url, token, connection information about the device...). You can check the syntax of the file above.

You also have to decide which package are to be monitored. For example, you can monitore com.android.chrome, a specific application pacakge or the android system (com.android.systemui). You can monitore multiple packages.

Command line

Code Block

# To install components on the device and monitore Chrome and Android System
./testrunner  --config config.yml monitoring prepare --monitoredPackage  com.android.chrome --monitoredPackage com.android.systemui

Start and stop measure

Once the application is installed, you can click on the icon to open it.

The button "Start measure" starts a measure and closes the application.

The button "Stop measure" stops the measure. A default name will be given if you don't name the measure before stopping it.

Get results in CSV or send them to the web interface

At the end of the measure, you can either retrieve the results in a CSV file, or send them to the web interface. Both operations delete the results from the device.

You have to have created the application you want to send to measures to before.

Code Block

# To retrieve results in a CSV file
./testrunner --config config.yml monitoring getmeasures

# To send reasults to the web interface
./testrunner --config config.yml monitoring sendmeasures -a "MonApplication" -v "version"

Limitations

The application has to be reinstalled every time the device is rebooted (to restart the monitoring service).

The application has to be reinstalled if you want to monitor another package (and the new package to monitor has to be part of the install command line.

If you launch a new measure after stopping one, the results from the previous run will be erased and replaced by the new measure.

Results

When a job is finished, if you configured the online flag in your job.yml to true, the Test Runner will automatically send the measures to your GREENSPECTOR Server.

Otherwise, results will be saved locally to the resultPath you set in config.yml, allowing you to send them later using the GREENSPECTOR Command Line Interface.

Versions Compared

Old Version 44

New Version 45

Key

Test Runner description

General behavior

Job types

Configuration files

config.yml

job.yml

Command Line Arguments

Command Line Sub Commands

Update components:

Measure with a free run:

Test a DSL Command :

Monitore packages :

Installation

Limitations

Results

Related articles

Page Comparison

Versions Compared

Old Version 44

New Version 45

Key

Test Runner description

General behavior

Job types

Configuration files

config.yml

job.yml

Command Line Arguments

Command Line Sub Commands

Update components:

Measure with a free run:

Test a DSL Command :

Monitore packages :

Installation

Limitations

Results

Related articles