Give an indication which test is currently running/busy indicator

[Lübbe Onken]

I'm currently limited to XE2, so if this proposed feature already works in newer XEs, fine by me :)

I have a set of long running integration tests, which need around 5-15 seconds each to complete. Currently the tree in the test insight window is cleared when the test are running and after the tests are finished, the entire tree with all test results is shown. At the moment I have no indication if the tests are running at all and which test is currently running.

It would be great if the tree would be visible all the time and the current test would be highlighted somehow. The tree could be subdivided into several subtrees:

• Failed tests
• Successful tests
• Tests that haven't run yet
• Other test states (warning, skipped, whatever, ...)

If this is not possible, at least a status message (tests running / idle) would be helpful.

[Stefan Glienke]

The tree is divided into these categories - it's in the group by type tab.

The tree is only cleared when you click "run all tests". When you click "run selected" it does not clear the result but updates them. If no test is selected it runs them all.

For showing progress while the test runs set SendImmediately of the ITestInsightClient to True. This is disabled by default to provide a quicker testrun as sending each result individually takes more time than collecting them all and sending them in one batch at the end.

[Lübbe Onken]

That's how I tried to use TestInsight.

In the group by type tab

• Step 1: run all tests
• Step 2: check a few tests and "run selected tests"

The following happens:

• The tree is always cleared immediately and stays empty while the tests are running
• Only the selected tests are shown in the tree after the tests are run
• There is no indication which test is currently running

Maybe it's XE2 or some IDE plugin getting in the way? From the top of my head I have the following installed:

• DDevExtensions
• Delphi IDE colorizer
• Eurekalog
• FixInsight
• GExperts
• Parnassus Bookmarks
• Nick Rings shortcut finder
• TestInsight

[Stefan Glienke]

Are you using DUnitX? Because that always clears the tests - I don't know why - might have slipped through. Remove line 139 in TestInsight.DUnitX.pas

[Lübbe Onken]

Yes, DUnitX. I uncommented line 139 in TestInsight.DUnitX.pas, but the behaviour is still weird, when I run a subset of tests

1. Run all tests (10 in this case) -> Tree is cleared, all tests pass, fine until now.
2. Select four of the tests and run selected tests ->

• Tree stays open during the run
• No indication which test is currently run (all tests keep their green tick)
• After the tests are finished, 2-3 of the four selected tests are shown in the tree
• The other tests are erased

This behaviour only occurs after commenting out line 139. If the line is there, the tree gets erased at the beginning, but the complete subset of tests is shown afterwards. I can repeat this over and over again and only in one out of ten cases all selected test are shown after the second run (without line 139).

Actually in this case I only have two tests fixtures, which contain one test procedure each, but with different parameters for each fixture, so it adds up to ten tests in total. Is it possible that the same test, being run with different parameters, confuses the test listener somehow?

This also happens when "running continuous tests when idle". If I check some of the tests, the skipped tests plus some others disappear from the tree during the next run. So it's not caused by the manual trigger.

I took a look at TSelectedFilter and noticed the following: in line 95 you set fIgnoreSkippedTests := Length(fSelectedTests) > 0; if I set this to false, the skipped tests show up.

My missing tests from above now appear in the list of skipped tests, so there probably is a bug in TSelectedFilter.Match(...) Stepping through TSelectedFilter.Match(...), I found out why some tests are skipped:

• One of them contains a German Umlaut in the test name
• One of them contains a '/' in the test name

These strings are "escaped" in fselectedtests and "plain text" in test.name, which is why they don't match.

[Stefan Glienke]

When the tests are not cleared it syncs the incoming tests with the existing ones in the tree (it uses the name). Any tests that are not contained in the results are marked as skipped (you might have filtered them out). So changing parameters or the name of the test will cause some "garbage" in the skipped tests node. But that is what clear is for.

[Lübbe Onken]

I was debugging through TestInsight.DUnitX and editing my previous reply while your answer came in. Please re-read :)

Example:

• in test.name '...Schulungssteuergerät'
• in fselectedTests '...Schulungssteuerger\u00E4t'

All my tests have distinct name attributes, but shouldn't you also take the parameter attributes into account when matching tests? Currently they are ignored. I know it's bad practice to use the same name and different parameters, but you know what people are doing ...

[Stefan Glienke]

That's DUnitX issue then for not naming the test in a distinct way. When using my Attributes with DUnit the name of the test includes the values that are passed to that test which makes it distinct.

[Lübbe Onken]

There are two things here:

One is that DUnitX doesn't include both, test name and test parameter attributes into the names of their test cases. Granted, leave that as it is in TestInsight.

The other thing is that the name of a test case which is defined like this:

  [TestFixture]
  TMyTest = class(TBaseTest)
    [Test]
    [TestCase('Gerät/XYZ','List, of, parameters')]
    procedure Doit(P1,P2,P3: string);
  end;

is passed around in two different string encodings and thus is skipped, because the function TSelectedFilter.Match(...) doesn't match it.

test name in:

for s in fSelectedTests 

yields
 s = '...TMyTest.Doit.Ger\u00E4t\/XYZ'

but
  test.Fixture.UnitName + '.' + test.Fixture.Name + '.' + test.Name = '...TMyTest.Doit.Gerät/XYZ'

so this test is always skipped if "run selected tests" is selected

PS: We (sh|c)ould change the subject of this ticket. I found the "fClient.SendImmediately := True" option and now TestInsights behaviour is closer to what I like :)

[Stefan Glienke]

handle json encoding properly (fixed #40)

→ <<cset 7d93227269d1>>

[Stefan Glienke]

Change the method as follows:

function TTestInsightClientBase.ParseTests(const response: string): TArray<string>;
var
  obj: TJSONObject;
  arr: TJSONArray;
  i: Integer;
begin
  obj := TJSONObject.ParseJSONValue(response) as TJSONObject;
  try
    arr := TJSONArray(obj.Pairs[0].JsonValue);
    SetLength(Result, arr.Count);
    for i := 0 to arr.Count - 1 do
      Result[i] := TJSONString(arr.Items[i]).Value;
  finally
    obj.Free;
  end;
end;

In Delphi versions below XE6 use Get instead of Pairs and Size instead of Count

[Lübbe Onken]

Thanks, that fixes the issue. Here's my function for XE2:

function TTestInsightClientBase.ParseTests(const response: string): TArray<string>;
var
  obj: TJSONObject;
  arr: TJSONArray;
  i: Integer;
begin
  obj := TJSONObject.ParseJSONValue(response) as TJSONObject;
  try
    arr := TJSONArray(obj.Get(0).JsonValue);
    SetLength(Result, arr.Size);
    for i := 0 to arr.Size - 1 do
      Result[i] := TJSONString(arr.Get(I)).Value;
  finally
    obj.Free;
  end;
end;