[Buildroot] [PATCH 1/5] support/testing: core testing infrastructure

[Luca Ceresoli]

On 07/03/2017 23:09, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 7 Mar 2017 22:59:35 +0100, Luca Ceresoli wrote:
> 
>>> Are you talking about:
>>>
>>>  * Failures because there is a syntax error in the test script
>>>
>>>  * Failures of the test itself, because the generated Buildroot system
>>>    doesn't behave as expected  
>>
>> Is there a way to understand which kind of failure we had?
> 
> Well the former will give you Python exceptions, the latter will give
> you a failure of the test itself.

Oh, sorry for misreading your original e-mail. Yeah, you're right, they
are clearly different.

I was speaking about the 2nd point. When a test is syntactically OK it
should stay so in the future until it's modified. But the test could
start failing later in time (which is why we want these tests).

> 
>> But my point was mostly about the first part of the process:
>>
>> $ ./support/testing/run-tests  -d ../br-test-dl -o ../br-test-out -a
>> [br-test-out/TestPythonBase/2017-03-03 23:22:46] Starting
>> [br-test-out/TestPythonBase/2017-03-03 23:22:46] Building
>> [br-test-out/TestPythonBase/2017-03-03 23:27:37] Building done
>> [br-test-out/TestPythonBase/2017-03-03 23:27:45] Cleaning up
>> .[br-test-out/TestDropbear/2017-03-03 23:27:46] Starting
>> [br-test-out/TestDropbear/2017-03-03 23:27:46] Building
>> [br-test-out/TestDropbear/2017-03-03 23:29:00] Building done
>> [br-test-out/TestDropbear/2017-03-03 23:29:06] Cleaning up
>> [...]
>> F[br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Starting
>> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:13] Building
>> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Building done
>> [br-test-out/TestRootfsOverlay/2017-03-03 23:37:27] Cleaning up
>> .[br-test-out/TestSquashfs/2017-03-03 23:37:27] Starting
>> [br-test-out/TestSquashfs/2017-03-03 23:37:27] Building
>> [br-test-out/TestSquashfs/2017-03-03 23:39:38] Building done
>> [br-test-out/TestSquashfs/2017-03-03 23:39:41] Cleaning up
>> .[br-test-out/TestJffs2/2017-03-03 23:39:41] Starting
>> [br-test-out/TestJffs2/2017-03-03 23:39:41] Building
>> [br-test-out/TestJffs2/2017-03-03 23:42:40] Building done
>> open input file: No such file or directory
>> [br-test-out/TestJffs2/2017-03-03 23:42:40] Cleaning up
>>
>> There's a character before each "Starting" line, I guess it's nose2
>> reporting the test outcome. It seems to be a '.' for success, 'F' or 'E'
>> for failures. What's the difference between 'E' and 'F'? Can we remove
>> these characters, which interfere with the rest of the output?
> 
> Don't know, we would need to look at the internals of nose2/unittest to
> know what's possible. Not sure how much customization is possible here.
> 
>> Also, the rest of the output is a bit more verbose than I'd like to
>> have. Can we have something like:
>>
>> 23:37:13 TestRootfsOverlay           Starting
>> 23:37:13 TestRootfsOverlay           Building
>> 23:37:27 TestRootfsOverlay           Building done
>> 23:37:27 TestRootfsOverlay           Cleaning up
>> 23:37:13 TestFoo                     Starting
>> 23:37:13 TestFoo                     Building
>> 23:37:27 TestFoo                     Building done
>> 23:37:27 TestFoo                     *** FAILED! ***
> 
> Indeed, would be nice.
> 
> The more I think of it, the less I believe re-using the existing
> "runners" for Python unit tests is useful in our case. It doesn't bring
> much, and prevents doing the customizations we need.

I don't know how much effort it would be to write our own test runner.
It the effort is little, I agree with you.

-- 
Luca