[Buildroot] testing/infra: generating tests for python packages

[Yann E. MORIN]

Arnout, All,

On 2017-09-13 23:04 +0200, Arnout Vandecappelle spake thusly:
> 
> 
> On 13-09-17 18:41, Yann E. MORIN wrote:
> > Ricardo, Thomas, All,
> > 
> > On 2017-09-13 00:29 -0300, Ricardo Martincoski spake thusly:
> >> Thomas mentioned [1] you guys talked about tests for python packages being
> >> generated by the test infra.
> >>
> >> I performed an experiment [2] ignoring where the data is coming from (whether
> >> it comes from the package recipe or from inside the infra) and I start this
> >> thread to follow up the discussion.
> 
>  I didn't look at your solution yet, Ricardo, but here are my high-level
> observations.
> 
> > I also performed an experiment, which failed because I am no python
> > expert, but basically it was something like (in pseudo code because I
> > lost it and it anyway did not work):
> > 
> >     support/testing/tests/package/test_python-modules.py
> > 
> >         for dir, files, _ = os.walk(os.path.join(buildroot_top_dir,'package'):
> >             pkg = os.path.last_component(dir)
> >             for f in files:
> >                 if re.match('{}-test.py'.format(pkg),f):
> >                     import_module(os.path.join(dir,f))
> > 
> > which basically scans the Buildroot package/ subdir to search for files
> > matching the glob 'PKG_NAME-test.py' (with PKG_NAME replaced by the
> > package name, obviously), and imports them one by one.
> 
>  Looks like the right thing to do to me. However, to be consistent with the
> existing tests, the pattern should be test_pkg.py, and the package name should
> go through - to _ conversion.

And to be consistent with the existing .mk and .hash files, the test
file should start with the package name. So, we have two incompatible
consistency requirements. One will have to win and the other to lose.
And I think the best would be to keep the .mk and .hash scheme, because
this is in the package directory.

> > Of course, one may be more inventive with the tests.
> > 
> > And in the end, it would have been all integrated in the current infra.
> > 
> > But alas, that import stuff I could not make to work... :-( Maybe there
> > is a node function to tell it where to load extra tests from?
> 
>  We can just add support/testing to sys.path.

Except that did not work when I tried... :-/

But I also tried setting package/ in sys.path and it did not work
either... Meh, someone will have to teach me some python magic... ;-)

> > An alternative would be to go with per-package test files, like
> > explained above, but have the support/testing/run-test script do the can
> > before calling nose, and for each file if finds, do a symlink ( or a
> > copy) in the testing infra, in a special directory that is git-ignored ?
> 
>  Bwerk, hack...

Maybe, but very, very easy and trivial!

We can even keep the PKG_NAME-test.py (or PKG_NAME.test) and symlink
that as test_PKG_NAME.py to respect the nose naming scheme.

> > Now, completely unrelated to the above, when we want to test each module
> > to ensure they are correct and have the required dependencies, we need
> > to have a configuration for each of the modules, which means doing one
> > build per module we want to test, each build including building the
> > python interpreter. This will make for a very long test campaign,
> > indeed... :-/
> 
>  We don't care that it takes a long time. It's not intended to be run
> sequentially anyway. And in gitlab, all tests run in parallel (but only on 4
> builders IIRC). As long as the tests on gitlab finish within a day or so, we're
> golden.

Yep, but we currently have about 224 python packages, so if each needs
30 minutes to build and run, that's about 5 days to run the full
suite... Just sayin'...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'