Results from testing of master branch on 02-13-2018 (after adding support for functional testing)
Linux Test Box: Ubuntu 17.04 / Linux 4.10.0-42-generic / 32 HT Cores @ 2.1GHz / 64GB RAM
Windows Test Box: Windows 10 Education / V 1709 / OS Build 16299.192 / 32 HT Cores @ 2.1GHz / 64GB RAM
Mac Test Box: OS X 10.11.6 (OS X EI Capitan) / 4 Cores / 16GB RAM
Android Studio: 3.0.1
Android Build Tools: 26.0.2
System Images: Nexus 5X 1920x1080 Intel Atom x86 Goggle APIs images of following APIs. (Device Hash: MD5:bc5032b2a871da511332401af3ac6bb0)
- 19 (API Revision 2)
- 21 (API Revision 2)
- 22 (API Revision 1)
- 23 (API Revision 2)
- 24 (API Revision 1)
- 25 (API Revision 3)
Emulator advanced settings: You can access these at AVD Manager --> Edit this AVD --> Show Advanced Settings
- Graphics: Automatic
- Boot option: Quick boot
- Multi-core CPU: 4
Memory and Storage
- RAM: 1536 MB
- VM heap: 256 MB
- Internal Storage: 800 MB
- SD card: Studio-managed: 100 MB
Functional Test Results in Strict Mode
- Both the vulnerable/benign and secure versions of apps in a benchmark are tested for presence and absence of concerned vulnerability using
- In strict mode, while testing a benchmark, the AVD is rebooted between the execution of vulnerable/benign test and secure test.
- Each cell captures test failures. W, L, and M identify the test failed on Windows, Linux, and Mac, respectively.
- Cells with intermittent failures (i.e., failures cannot be reproduced regularly) are highlighted in yellow. Almost all of these failures are caused by the Android test automation framework not being able find UI objects required to drive testing.
Testing of secure app of OrderedBroadcast-DataInjection-Lean benchmark fails consistently across all three platforms with API 23. When the AVD is rebooted after testing vulnerable/benign apps and before testing secure app, an error dialog pops up stating "Unfortunately, com.android.phone process has stopped."
For API level 21, the apps in all benchmark behave as expected when tested manually. However, during automated testing, the apps show inconsistent behavior and almost always fail with UiObjectNotFoundException.