Results from testing of master branch on 05-24-2018 (after switching to Android Studio 3.1.2)

Setup

Linux Test Box: Ubuntu 18.04 LTS / 32 HT Cores @ 2.1GHz / 64GB RAM

Windows Test Box: Windows 10 Education / V 1803 / OS Build 17134.48 / 32 HT Cores @ 2.1GHz / 64GB RAM

Mac Test Box: OS X 10.11.6 (OS X EI Capitan) / 4 Cores / 16GB RAM

Android Studio: 3.1.2

Android Build Tools: 27.0.3

System Images: Nexus 5X 1920x1080 Intel Atom x86 Goggle APIs images of following APIs. (Device Hash: MD5:bc5032b2a871da511332401af3ac6bb0)

• 22 (API Revision 1)

• 23 (API Revision 2)

• 24 (API Revision 1)

• 25 (API Revision 3)

• 26 (API Revision 2)

• 27 (API Revision 1)

Emulator advanced settings: You can access these at AVD Manager --> Edit this AVD --> Show Advanced Settings

• Emulated Performance

  • Graphics: Automatic
  • Boot option: Quick boot
  • Multi-core CPU: 4

• Memory and Storage

  • RAM: 1536 MB
  • VM heap: 256 MB
  • Internal Storage: 800 MB
  • SD card: Studio-managed: 100 MB

Naming conventions

Each cell captures test failures. W, L, and M identify the test failed on Windows, Linux, and Mac, respectively.

1. UiObjectNotFoundException: W1, L1 and M1

2. Other errors: W0, L0 and M0

Observations

1. Google API image for API level 27 emulator reboots on its own on Ubuntu 18.04 LTS. Consequently, a large number of tests for API 27 fail with a INSTRUMENTATION_ABORTED error. Corresponding bug report against Android Emulator can be found here.

2. All the UiObjectNotFoundExceptions are intermittent.