[Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages
Arnout Vandecappelle
arnout at mind.be
Sun Sep 6 22:46:27 UTC 2015
Hi Denis,
On 31-08-15 11:59, Denis THULIN wrote:
> This patch series adds a test generation infrastructure for packages to
> Buildroot.
> The generated tests are robotframework tests.
>
> While it is always necessary to test that packages are installed
> correctly and work in an intended way, Buildroot does not currently
> provide any tests of package nor any way to automate the process of
> creating tests depending on your configuration or even share tests you
> have written for packages. You have to write tests manually almost
> everytime you start a new project with buildroot. This can cost a lot
> of time.
Just to be clear: the intention of this series is to make it possible to
include in the buildroot package a way to specify some tests to run on the host
and on the target, where the purpose of these tests is to validate that the
package has been built correctly. So it is not exactly testing the buildroot
infrastructure (like what Thomas is trying to do), and it's not really running
the tests included with the package itself (these are not built and installed),
but something in-between.
However, as far as I can see, this patch series only solves a small part of the
problem. It basically creates and installs a bunch of .robot files, and that's
it, right? You still have to make the robot framework run on these files. And
you still have to express in the robot framework or in jenkins the dependencies
to make sure the tests are first built and then executed. Correct?
In addition, it seems to me that the munging done by the TEST_BUILD macro has
little to offer above simply copying the .robot file from the package directory
to the test directory. But possibly I don't sufficiently understand what's going on.
And finally, buildroot tries not to enforce any particular policy or tool on
the user. So, much as I like robot, requiring the tests to use this specific
framework is definitely a no-go (at least, if it's meant to be used by end
users). Especially since it means that a rootfs with python3 can't be tested...
I'm not entirely sure what the way to go is. It definitely is nice to have some
kind of test description as part of the packages - I like that more than the way
Thomas did it in buildroot-runtime-test, where the buildroot tree is basically
replicated externally for the tests. But the exact location of the test files is
just a minor detail I guess.
Bottom line: I'm not in favour of this series either. Of course, the first
patch adding python-robotframework is still relevant independently.
Regards,
Arnout
>
> This patch series is intented as a solution for that problem.
>
> Why Robotframework Tests:
> Robotframework is a widely used test automation framework that allow
> writing tests in various languages (Python, Java and even C), provides
> useful test librairies and allow writing tests in various syntaxes.
> Robotframework tests are meant to be easy to read by people whom did
> not write the tests, and that can definitly be useful.
>
> List of patches from the series :
> * Patch 1/6 adds a test generation step before install target step.
> * Patch 2/6 adds useful keywords for testing through Telnet and Qemu
> * Patch 3/6 adds menuconfig options to ease the configuration required
> for testing through telnet and Qemu
> * Patch 4/6 adds documentation for patches 1,2 and 3
> * Patch 5/6 adds exemple tests for package python-flask
> * patch 6/6 adds exemple tests for package python (inspired by tests
> from https://github.com/tpetazzoni/buildroot-runtime-test)
>
> Denis THULIN (6):
> python-robotframework: New package
> Adds package test infrastructure
> tests: create variable files through kconfig
> tests: Adds user manual entry
> flask: Adds robotframework tests
> python: Adds tests
>
> Config.in | 6 +
> Makefile | 9 +-
> docs/manual/common-usage.txt | 2 +
> docs/manual/test-infrastructure.txt | 239 +++++++++++++++++++++
> package/Config.in | 1 +
> package/Config.in.host | 1 +
> package/Makefile.in | 1 +
> package/pkg-generic.mk | 20 +-
> package/pkg-test.mk | 94 ++++++++
> package/python-flask/python-flask.mk | 5 +
> .../target_test_material/target_test.robot | 25 +++
> package/python-robotframework/Config.in | 9 +
> package/python-robotframework/Config.in.host | 8 +
> .../python-robotframework.hash | 4 +
> .../python-robotframework/python-robotframework.mk | 16 ++
> package/python/python.mk | 17 ++
> package/python/python.robot | 32 +++
> tests/Config.in | 8 +
> tests/host/resource.robot | 7 +
> tests/target/resource.robot | 1 +
> tests/tests.mk | 20 ++
> 21 files changed, 522 insertions(+), 3 deletions(-)
> create mode 100644 docs/manual/test-infrastructure.txt
> create mode 100644 package/pkg-test.mk
> create mode 100644 package/python-flask/target_test_material/target_test.robot
> create mode 100644 package/python-robotframework/Config.in
> create mode 100644 package/python-robotframework/Config.in.host
> create mode 100644 package/python-robotframework/python-robotframework.hash
> create mode 100644 package/python-robotframework/python-robotframework.mk
> create mode 100644 package/python/python.robot
> create mode 100644 tests/Config.in
> create mode 100644 tests/host/resource.robot
> create mode 100644 tests/target/resource.robot
> create mode 100644 tests/tests.mk
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
More information about the buildroot
mailing list