[Buildroot] [PATCH 00/11] Overwritten file detection, improvements to file listing logic

Thomas Petazzoni thomas.petazzoni at bootlin.com
Thu Apr 30 09:52:37 UTC 2020


Hello,

The initial starting point for this series was that I wanted to detect
when a package overwrites files installed by another package, which is
something that is not compatible with per-package directory support. I
indeed have a complex build configuration which I'm converting to use
TLP, and this build configuration does lots of overwrites, which I
want to detect and fix.

While doing this overwrite detection, I realized that our existing
logic to detect which files are installed by each package is not
reliable: if a target package installs something in $(HOST_DIR), those
files installed in $(HOST_DIR) are not tracked. Getting this list of
files correct is also needed to properly support the
-reinstall,-rebuild,-reconfigure targets in a per-package
configuration (which this series does not address yet).

So this series goes on like this (I'm only highlighting the main
commits, not all preparation commits and details)

 - We create a new "install" step in PATCH 3. It is executed after all
   installation steps have executed. As of PATCH 3, it does nothing
   except creating a stamp file, but it will allow us to do some logic
   once a package is finished installing. Indeed, so far we had no
   place where we could run some logic after all installation steps of
   a package have completed.

 - In PATCH 5, we rework the pkg_size logic that detects files
   installed by each package to rely on the new "install"
   step. Instead of capturing the list of files "before" at the
   beginning of install-target, install-staging, install-images,
   install-host, and the list of files "after" at the end of these
   steps, we capture the list of files "before" at the beginning of
   the configure step, and the list of files "after" during the new
   "install" step.

 - In PATCH 8, we introduce the logic to detect files overwritten by
   other packages, using a simple md5sum hash.

 - In PATCH 9-11, we add test cases for the "check binary
   architecture" functionality and the "overwritten files detection"
   functionality.

This has received some basic testing so far, I'm definitely interested
in getting feedback from people, simply by testing if it works for
them.

All these changes are also available on the Git branch at:

  https://github.com/tpetazzoni/buildroot/tree/detect-overwrite

Best regards,

Thomas Petazzoni

Thomas Petazzoni (11):
  package/pkg-generic.mk: use $(PKG)_NAME in step_pkg_size_after
  package/pkg-generic.mk: drop useless $(1) argument in
    step_pkg_size_{before,after}
  package/pkg-generic.mk: introduce final 'install' step
  package/pkg-generic.mk: create directories upfront in the configure
    step
  package/pkg-generic.mk: rework pkg_size logic with the "installed"
    step
  package/pkg-generic.mk: exclude the staging sub-directory
  package/pkg-generic.mk: move pkg_size_{before,after} and
    check_bin_arch functions
  package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
    HOST_DIR
  support/testing/infra: add log_file_path() function
  support/testing/tests: add test for check_bin_arch
  support/testing/tests: add test for file overwrite detection

 .gitlab-ci.yml                                |   3 +
 package/pkg-generic.mk                        | 146 ++++++++++--------
 package/pkg-utils.mk                          |   1 -
 support/testing/infra/__init__.py             |  13 +-
 .../br2-external/detect-bad-arch/Config.in    |   1 +
 .../detect-bad-arch/external.desc             |   1 +
 .../br2-external/detect-bad-arch/external.mk  |   1 +
 .../package/detect-bad-arch/Config.in         |   4 +
 .../detect-bad-arch/detect-bad-arch.mk        |  15 ++
 .../br2-external/detect-overwrite/Config.in   |   1 +
 .../detect-overwrite/external.desc            |   1 +
 .../br2-external/detect-overwrite/external.mk |   1 +
 .../package/detect-overwrite/Config.in        |   5 +
 .../detect-overwrite/detect-overwrite.mk      |  19 +++
 support/testing/tests/core/test_bad_arch.py   |  19 +++
 .../testing/tests/core/test_file_overwrite.py |  47 ++++++
 16 files changed, 213 insertions(+), 65 deletions(-)
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
 create mode 100644 support/testing/tests/core/test_bad_arch.py
 create mode 100644 support/testing/tests/core/test_file_overwrite.py

-- 
2.25.4



More information about the buildroot mailing list