[Buildroot] testing/infra: generating tests for python packages

[Ricardo Martincoski]

Yann,

Thomas mentioned [1] you guys talked about tests for python packages being
generated by the test infra.

I performed an experiment [2] ignoring where the data is coming from (whether
it comes from the package recipe or from inside the infra) and I start this
thread to follow up the discussion.

It consist of a simple templating scheme using sed (of course it could be done
by some fancy templating like pystache, jinja, ... but I like sed) and
inheritance for a generic method that tests a command, and it tries to generate
about the same test cases Yegor created [3].

We cannot use TestPython{2,3} classes as-is since nose2 (or one of its
dependencies) would call its test_run method twice. I did not investigate the
reason, just worked around it for now.

Together to testing for exit code 0 to determine Pass/Fail, I added regexes for
the output of the command, maybe it can become handy (for example, hashlib was
just added as dependency to python-cryptography with Python 2 but when it was
missing the test did not fail since the exit code was still 0).

Since .gitlab-ci.yml cannot yet be dynamically generated (see [4] from Arnout
and its linked-list of similar issues reported) and we want to keep separate
results to each test, it looks to me a good place to call the test case
generation would be just before the handling of --list option in run-tests.
This way 'make .gitlab-ci.yml' and 'run-tests --list' always keep working as
expected and also the test case is correctly generated in the docker runner.

It seems to me that besides the package + the test command, some interesting
variables to generate the test cases are:
- if the package supports Python 2, 3 or both;
- if testing for exit code 0 is enough, and the regex to determine pass/fail
  when appropriate;
- the timeout to the command;
- the non-python dependencies of the package (python-slob needs gcc 4.9 so as a
  dirty hack I changed the base config, of course we could try to find a good
  default for all python packages and use it in the TestPython2Package class);

Of course we can draw a line and choose to cover most of, but not all, python
packages and avoid or replace some of these variables by good defaults.

Now concerning the choice between keeping the data in the package recipe or in
the test infra I don't have strong opinion. It looks nice to have it in the
package recipe, but maybe it could be too complicated or even not feasible
using make targets due to the Python2/Python3/both variable. Perhaps externally
parsing package/python-*/{Config.in,*.mk} could do the job.

[1] http://lists.busybox.net/pipermail/buildroot/2017-September/201766.html
[2] https://gitlab.com/RicardoMartincoski/buildroot/commit/41da07c352dc3ad45af5a447cbabf8bef1245f98
[3] http://patchwork.ozlabs.org/patch/811011/
[4] https://gitlab.com/gitlab-org/gitlab-ce/issues/34595

Regards,
Ricardo