Clone wiki

android-app-vulnerability-benchmarks / Results from testing of master branch on 05-24-2018 (after switching to Android Studio 3.1.2)

Results from testing of master branch on 05-24-2018 (after switching to Android Studio 3.1.2)

Setup

Linux Test Box: Ubuntu 18.04 LTS / 32 HT Cores @ 2.1GHz / 64GB RAM

Windows Test Box: Windows 10 Education / V 1803 / OS Build 17134.48 / 32 HT Cores @ 2.1GHz / 64GB RAM

Mac Test Box: OS X 10.11.6 (OS X EI Capitan) / 4 Cores / 16GB RAM

Android Studio: 3.1.2

Android Build Tools: 27.0.3

System Images: Nexus 5X 1920x1080 Intel Atom x86 Goggle APIs images of following APIs. (Device Hash: MD5:bc5032b2a871da511332401af3ac6bb0)

  • 22 (API Revision 1)

  • 23 (API Revision 2)

  • 24 (API Revision 1)

  • 25 (API Revision 3)

  • 26 (API Revision 2)

  • 27 (API Revision 1)

Emulator advanced settings: You can access these at AVD Manager --> Edit this AVD --> Show Advanced Settings

  • Emulated Performance

    • Graphics: Automatic
    • Boot option: Quick boot
    • Multi-core CPU: 4
  • Memory and Storage

    • RAM: 1536 MB
    • VM heap: 256 MB
    • Internal Storage: 800 MB
    • SD card: Studio-managed: 100 MB

Results-2.JPG

Naming conventions

Each cell captures test failures. W, L, and M identify the test failed on Windows, Linux, and Mac, respectively.

  1. UiObjectNotFoundException: W1, L1 and M1

  2. Other errors: W0, L0 and M0

Observations

  1. Google API image for API level 27 emulator reboots on its own on Ubuntu 18.04 LTS. Consequently, a large number of tests for API 27 fail with a INSTRUMENTATION_ABORTED error. Corresponding bug report against Android Emulator can be found here.

  2. All the UiObjectNotFoundExceptions are intermittent.

Updated