1. SecureItI Avatar SecureItI
  2. Ghera
  3. android-app-vulnerability-benchmarks

Wiki

Clone wiki

android-app-vulnerability-benchmarks / Results from testing of master branch on 02-13-2018 (after adding support for functional testing)

View History

Results from testing of master branch on 02-13-2018 (after adding support for functional testing)

Setup

Linux Test Box: Ubuntu 17.04 / Linux 4.10.0-42-generic / 32 HT Cores @ 2.1GHz / 64GB RAM

Windows Test Box: Windows 10 Education / V 1709 / OS Build 16299.192 / 32 HT Cores @ 2.1GHz / 64GB RAM

Mac Test Box: OS X 10.11.6 (OS X EI Capitan) / 4 Cores / 16GB RAM

Android Studio: 3.0.1

Android Build Tools: 26.0.2

System Images: Nexus 5X 1920x1080 Intel Atom x86 Goggle APIs images of following APIs. (Device Hash: MD5:bc5032b2a871da511332401af3ac6bb0)

  • 19 (API Revision 2)
  • 21 (API Revision 2)
  • 22 (API Revision 1)
  • 23 (API Revision 2)
  • 24 (API Revision 1)
  • 25 (API Revision 3)

Emulator advanced settings: You can access these at AVD Manager --> Edit this AVD --> Show Advanced Settings

  • Emulated Performance

    • Graphics: Automatic
    • Boot option: Quick boot
    • Multi-core CPU: 4

  • Memory and Storage

    • RAM: 1536 MB
    • VM heap: 256 MB
    • Internal Storage: 800 MB
    • SD card: Studio-managed: 100 MB

Functional Test Results in Strict Mode

  1. Both the vulnerable/benign and secure versions of apps in a benchmark are tested for presence and absence of concerned vulnerability using functional-test.[sh|ps1] script.
  2. In strict mode, while testing a benchmark, the AVD is rebooted between the execution of vulnerable/benign test and secure test.
  3. Each cell captures test failures. W, L, and M identify the test failed on Windows, Linux, and Mac, respectively.
  4. Cells with intermittent failures (i.e., failures cannot be reproduced regularly) are highlighted in yellow. Almost all of these failures are caused by the Android test automation framework not being able find UI objects required to drive testing.

strict.JPG

Observations

  1. Testing of secure app of OrderedBroadcast-DataInjection-Lean benchmark fails consistently across all three platforms with API 23. When the AVD is rebooted after testing vulnerable/benign apps and before testing secure app, an error dialog pops up stating "Unfortunately, com.android.phone process has stopped."

  2. For API level 21, the apps in all benchmark behave as expected when tested manually. However, during automated testing, the apps show inconsistent behavior and almost always fail with UiObjectNotFoundException.

Updated