UP TO DATE

Test Runner description

General behavior

The Test Runner:

reads the configuration files (config.yml and job.yml);
checks the components are installed;
downloads the components if they are not present or not up to date;
starts the job;
sends the results to the GREENSPECTOR Server or saves them locally when the job is finished.

Job types

With the Test Runner, you will be able to launch 4 different kinds of jobs:

APK job to benchmark your Android application.
URL job to benchmark your website.
CUSTOM job to measure your Android application or web application with your own automated tests.
FREERUN job to run a manual measure of your application or website.

Configuration files

You will find below a reference guide for the 2 configuration files used by the Test Runner.

config.yml

Field in configuration file			Whether the field is necessary depending on the type of job (apk, custom or url)	Description	Default or example value
`platform`			ALL	The platform on which the tests will run.	`mobile`
`resultPath`			ALL	The path where you want the Test Runner to save the results.	`./greenspector/measures`
`componentsPath`			ALL	The path where components used by the Test Runner are installed.	`$HOME/.greenspector`
`greenspector`				Informations on your GREENSPECTOR profile.
	`server`		ALL	Your GREENSPECTOR Server URL.	`https://app.greenspector.com`
	`token`		ALL	Your private token registered on GREENSPECTOR Server.	`123456789AZERTY`
`proxy`				Configure these fields if you are using a proxy server.
	`protocol`		ALL	The protocol used by the proxy.	`http` or `https`
	`hostname`		ALL	The proxy's URL.
	`port`		ALL	The port used by the proxy server.
	`user`		ALL	The username you are using to connect to the proxy server.
	`password`		ALL	The password associated with this username.
`target`				Contains the settings of the targeted platform
	`connection`		ALL	The way you are connected to the device.	`wifi` or `usb`
	`serial`		ALL	The mobile device id, can be retrieved with `adb devices`. Use this if you previously set connection value to `usb`.
	`ip`		ALL	The mobile IP adress on your Wi-Fi network, can be retrieved with `adb devices`. Use this if you previously set connection value to `wifi`.
	`port`		ALL	The mobile remote port for adb on your Wi-Fi network. Use this if you previously set connection value to `wifi`.	`5555`
	`paths`			The paths to required Android SDK's tools.
		`adb`	ALL	Path to ADB (Android Debug Bridge).	`adb`
		`aapt`	ALL	Path to AAPT (Android Asset Packaging Tool).	`$HOME/Android/Sdk/build-tools/27.0.2/aapt`
	`phantomas`			For module `http_request` in job.yml file.
		`path`	URL	The path to the phantomas executable if not already in your path.	`phantomas`
	`hardware`			Contains information about your physical measurement tool
		`address`	ALL	USB port which the physical probe is connected to	`/dev/ttyACM0`
	setupCommands		ALL	You can provide any useful shell commands that will run on the device before each iteration of testing. Use this parameter if your device needs to be prepared before each iteration and you do not want to prepare it with your GDSL ou UI Automator test.	setupCommands: - settings put global verifier_verify_adb_installs 0 - sh /sdcard/myscript.sh

job.yml

Field in configuration file			Whether the field is necessary depending on the type of job (apk, custom or url)	Description	Default or example value
mode			ALL	The type of job you want to lauch, as explained above.	`apk` or `custom` or `url or freerunner`
greenspector				This block contains details about the GREENSPECTOR application in which you want to save your measures.
	application
		name	ALL	The name of the application .
		version	ALL	The version of the application.
job				The job definition.
	url		APKURL	If you previously set `mode` to `url`, it is the URL of the website you want to measure. Otherwise it is an URL pointing to your Android APK. You can precise a local path to your apk, a public URL to download your apk, a package name if you want to benchmark an already installed apk or a package name if you want to install from the PlayStore before running the test	You can choose an URL : local path : /home/user/app`/demo.apk` public URL : https://dl.greenspector.com/demo/demo.apk package name : com.microsoft.office.outlook
	urls		APKCUSTOM	You can provide a list of APKs to be installed and eventually measured. You can precise a local path to your apk, a public URL to download your apk, a package name if you want to benchmark an already installed apk or a package name if you want to install from the PlayStore before running the test When using `custom` mode at least one of the APK should contain your instrumented tests (if you do not use tests written in Greenspector DSL with the option testsSuites)	for each url you can choose the location : local path : /home/user/app/demo.apk public URL : https://dl.greenspector.com/demo/demo.apk package name : com.microsoft.office.outlook
	testPackages		CUSTOM	If you previously set mode to `custom`, you can provide a list of packages containing your instrumentation test cases.
	testsSuites		CUSTOM	If you previously set mode to `custom`, you can provide groups of tests files writen with the Greenspector DSL.	For example : testsSuites: - name: suite1 testFiles: - "/home/user/test/steps1.testgb" - "home/user/test/steps2.testgb" - name: suite2 testFiles: - "/home/user/test/steps3.testgb"
	monitoredPackages		CUSTOMFREERUN	If you previously set mode to `custom`, you can provide a list of packages to monitor.
	monitoredPackage (Deprecated)		CUSTOMFREERUN	If you previously set mode to `custom`, you can provide the package to monitor. This field is deprecated, prefer to use monitoredPackages
	extras		CUSTOM	Use to transmit extras parameters to your instrumentation with the format key, value You can use predefined parameters : config-skipsetupphone: if "true" then skip the setup configuration that happens before test. config-waitForIdle: if "false" set idle timeout to 0 config-screenofftimeout: Set the time of inactivity before the screen goes to sleep (in millisecond) config-screenbrightness: set brightness of the screen. value is between 0 and 255 config-gps: if "enabled" then activate location on phone before test config-bluetooth: if "enabled" then activate bluetooth on phone before test config-nfc: if "enabled" then activate NFC on phone before test You can use custom parameters to fill parameters on your GDSL file You can filter your custom UIAutomator tests (no GDSL tests) using those extras parameters. https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner)	You can use custom parameters like this : extras: PARAMETERNAME: "myValue" will replace all ${PARAMETERNAME} in the GDSL file by myValue If you use custom UIAutomator tests (no GDSL tests) you can filter the class you launch by annotation : extras: annotation: com.greenspector.demo.test.MyAnnotation (this extra parameter will be added to your uiautomator instrumentation like this -e annotation com.greenspector.demo.test.MyAnnotation)
	networkMode		APKURLCUSTOM	The networkMode of the test. The phone has to be able to handle the networkMode.	WIFI, 4G, 3G or 2G. Default value is WIFI.
	scenario		APKURL	Possible value is `simple` for an APK job. For a URL job, both `simple` and `avance` can be used. APK job / simple scenario (deprecated) : Reference Measure (only the mobile platform) for 20 seconds Application Idle in foreground for 20 seconds Application Idle in background for 20 seconds URL job / simple scenario (deprecated) : Reference Measure (only the browser) for 20 seconds Website Launch for 20 seconds Website Idle in foreground for 20 seconds URL job / avance scenario: Reference Measure (only the browser) for 20 seconds Website Launch for 20 seconds Website Idle in foreground for 20 seconds Scrolling the Website for 4 seconds Website Idle in background for 20 seconds	`simple (deprecated)` or `avance`
	iterations		ALL	How many times you want to run the scenario.
	testTimeout		ALL	Timeout for the total duration of the test. Default value is `30m`, max value is `60m`.	`45m30s`
	stepName		FREERUN	Name of the step
	duration		FREERUN	Duration of the test.	`1m30s`
	authentication			If you previously set mode to `apk`, the Test Runner can handle an authentication form if your application has one.
		actions	APK	Set of actions to interact with the authentication form, either by coordinates or by IDs. There is currently two types of actions: `click` and `text`.	`- "text;coordinates;300;400;user@mail.com"` `- "text;id;session[password];userpassword"` `- "click;coordinates;500;900"`
	browser		URL	If you previously set mode to `url`, it is the web browser you want to measure on. Only Google Chrome is supported for now.	`chrome`
	cache		URL	If you previously set mode to `url`, set this to `true` if you want the Test Runner to run a second test of your website with data in the browser cache.
	online		ALL	Set this to true if you want the Test Runner to automatically send the results of the measures to your GREENSPECTOR Server. Otherwise, set it to `false`, the results will be saved locally.
	modules
		http_request	URL	Set this to true to enable analysis of HTTP requests.
		android_system_monitoring	APKURLCUSTOM	Set this to true to enable collection of additional system metrics during the job.
		hardware_probe	APKURLCUSTOM	Set this to true to enable physical measurement
		wakelock_probe	APKURLCUSTOM	Set this to true to enable wakelock measurement for the first iteration

Command Line Arguments

The Test Runner can take some command line arguments, for now the following are supported:

--config anotherConfigFile → to specify an alternative config.yml file;
--jobFile anotherJobFile → to specify an alternative job.yml file.
--no-update → to prevent automatically update the components used by the Test Runner when you run it (prefer not using this option)

These can be useful when you want to use different configurations for different kinds of tests. For example:

./testrunner --config androidConfig.yml --jobFile androidAPKJob.yml 
./testrunner --config androidConfig.yml --jobFile androidURLJob.yml

You can also use these arguments with the update command:

./testrunner --config androidConfig.yml --jobFile androidURLJob.yml update

Command Line Sub Commands

The Test Runner has subcommands:

Update components:

To update the components of GREENSPECTOR and use the last measure tools, you should run the command:

./testrunner update

This command will download the last android-service and testbench-launcher-android in the folder $HOME/.greenspector. These components will be installed on your phone before running a job.

This command is not necessary anymore if you do not use the option --no-update in your command, Test Runner will update these components automatically

Measure with a free run:

The Test Runner allows you to run a measure for a duration of time, while you can use the application you want to monitor. To do so, you should run the following commands:

./testrunner freerunner init # This line will help you generate the configuration file that you will have to update with some details
# edit the generated files (config.yml and job.yml)
./testrunner freerunner start # This line simply starts the measure
# or
./testrunner freerunner start --duration 1m30s --stepName "New Step Name" # You can pass the duration and the step name as flags so that you don't have to update the job file every time you change these values

This command will start a measure with no automated actions on the device so that you can run your manual scenario while monitoring your application.

Test a DSL Command :

The Test Runner allow you to test one command writen with Greenspector DSL for Android

# For example, to start google chrome:
./testrunner testdslcommand --command applicationStart,com.android.chrome

Monitore packages :

You can use the Test Runner to install an application that you can use to run measurements on a device. The operation is similar to free run, except you don't have to use the Test Runner to launch it once it's installed on your device.

Installation

Once you have downloaded the Test Runner, you can install the monitoring application on any device you want (you have to have unlocked developer mode on the device and be connected to it via adb).

Prerequisite

You have to have a config file filled with the right information (url, token, connection information about the device...). You can check the syntax of the file above.

You also have to decide which package are to be monitored. For example, you can monitore com.android.chrome, a specific application pacakge or the android system (com.android.systemui). You can monitore multiple packages.

Command line

# To install components on the device and monitore Chrome and Android System
./testrunner  --config config.yml monitoring prepare --monitoredPackage  com.android.chrome --monitoredPackage com.android.systemui

Start and stop measure

Once the application is installed, you can click on the icon to open it.

The button "Start measure" starts a measure and closes the application.

The button "Stop measure" stops the measure. A default name will be given if you don't name the measure before stopping it.

Get results in CSV or send them to the web interface

At the end of the measure, you can either retrieve the results in a CSV file, or send them to the web interface. Both operations delete the results from the device.

You have to have created the application you want to send to measures to before.

# To retrieve results in a CSV file
./testrunner --config config.yml monitoring getmeasures

# To send reasults to the web interface
./testrunner --config config.yml monitoring sendmeasures -a "MonApplication" -v "version"

Limitations

The application has to be reinstalled every time the device is rebooted (to restart the monitoring service).

The application has to be reinstalled if you want to monitor another package (and the new package to monitor has to be part of the install command line.

If you launch a new measure after stopping one, the results from the previous run will be erased and replaced by the new measure.

Results

When a job is finished, if you configured the online flag in your job.yml to true, the Test Runner will automatically send the measures to your GREENSPECTOR Server.

Otherwise, results will be saved locally to the resultPath you set in config.yml, allowing you to send them later using the GREENSPECTOR Command Line Interface.

Test Runner