Can't run tests from NetBeans with custom test runner, though command line works ok

[Rob Smith]

I've created a custom test runner by extending InstrumentationTestRunner to create com.mycom.myapp.test.MyTestRunner.

I believe I've set the instrumentation correctly in the manifest file:

    <instrumentation android:name="com.mycom.myapp.test.MyTestRunner"
                     android:targetPackage="com.mycom.myapp"
                     android:label="Tests for com.mycom.myapp with custom runner"/>

And this works successfully from the adb command line:

adb shell am instrument -w com.mycom.myapp.test/com.mycom.myapp.test.MyTestRunner

However, no tests are run when the Android Test project is run from NetBeans (7.2.1) IDE. From the 'Android development' output, it looks as though the NBAndroid plugin is hard-coded to use android.test.InstrumentationTestRunner as the test runner, regardless of the android:name in the manifest's instrumentation tag.

Package MyAppTest-debug.apk deployed
Starting tests on device emulator-5554
testRunStarted com.mycom.myapp.test, 0
testRunFailed Unable to find instrumentation info for: ComponentInfo{com.mycom.myapp.test/android.test.InstrumentationTestRunner}
testRunEnded 0, {}

I can recreate the same error using the command line:

adb shell am instrument -w com.mycom.myapp.test/android.test.InstrumentationTestRunner

Please can you fix this. Thanks.

Rob

[Radim Kubacki]

You are right - the runner is hard-coded at the moment. Would you mind to create a sample project with a custom runner? It can save some time when implementing this feature.

Another questions: does it make sense to have more runners defined in AndroidManifest.xml? does it make sense to run one test class/method using different runner? The reason for these is to find possible use cases and how to enhance the UI.

[Rob Smith]

Hi Radim,

Thanks for your swift response.

I've created a demo suite to highlight the issue - see email. There are 2 instrumentations in the test project manifest: one for InstrumentationTestRunner and one for MyTestRunner. Just comment/uncomment these tags to switch between them as necessary.

With MyTestRunner enabled, I can successfully run the tests from the command line, though running from NetBeans runs no tests and generates an error - see previous comments.

adb shell am instrument -w com.awayteamsoftware.hellocalc.test/com.awayteamsoftware.hellocalc.test.MyTestRunner

I noticed another couple of issues while making this demo. When creating a new Android test project, the user enters the 'tested project' and 'package name'.

1. The plugin seems to incorrectly add '.test' to the target package name in the test project manifest. No suffix is necessary.

2. It also adds a hardcoded '.tests' to the create the test package name from the target package. This should simply be the name of the current (Android test) package.

I've corrected these manually in the test project manifest, but have left its adb comment line unchanged so you can see the issue for yourself.

I hope this helps. Please let me know if you need further details.

Rob

[Rob Smith]

With regard to your multiple test runners question:

I would like to be able to choose which tests are run - either all, or a single class, or just an individual test.

I believe there are command line options to run subsets of Android tests, but could see no NetBeans controls for these.

The code I'm working on is a port to Android, so I have the luxury of a mature code base, including tests. I realised that my existing (JUnit) test runner would give me simple controls for now, so I implemented my custom test runner to allow this.

I hope this helps.

Rob

[Rob Smith]

Hi Radim,

Just wanted to check that you received my email with the demo project attachment. I've heard no response, and just wanted to confirm.

Many thanks.

Rob

[Radim Kubacki]

Hi, must have missed it. Can you attach it here (there is 'More | Attach files' in the right navigation bar next to edit button). Anyway I was pretty busy last week.

[Rob Smith]

No problem. Here is the demo project, as promised.

Thanks for the swift confirmation.

Rob

[Rob Smith]

A quick update on multiple instrumentations in the manifest:

There is a use case for this when running tests from the emulator/device itself.

1. Install the test package.
2. Start the Dev Tools app (from the Android SDK).
3. Select Instrumentation.
4. Choose the set of tests to run (from the list of manifest instrumentations).

This is unnecessary when running from the command line with adb, as using '-e class <test_class>' or '-e class <test_class>#<test_method>' can run a subset of tests, as mentioned on 26/1/14. However, it may be important for testing in the field, where the development environment is not available.

Hope this helps.

Rob

[Radim Kubacki]

Fixed in 1.227 that is available at http://nbandroid.org/updates/updates.xml (see http://www.nbandroid.org/2014/02/java-7-is-required-now.html). Just set the runner on Project | Run customizer of your test project. It is very simple - values in combobox are not populated yet with avaliable test runners.

[Radim Kubacki]

1.227

[Rob Smith]

Hi Radim,

I can confirm that 1.227 is working. I can see you have more to do, but the basic functionality is a good start.

I'd add a couple of notes for others trying to use this:

1. Set the test runner in File->Project Properties->Run->Instrumentation Runner (Optional)
2. This value can be fully qualified, e.g com.mycom.myapp.test.MyTestRunner, or abbreviated, e.g. .MyTestRunner (note the leading '.')

Many thanks for your support.

Rob