[Buildroot] [PATCH] infra: add the transient download mechanism
Michael Nazzareno Trimarchi
michael at amarulasolutions.com
Wed Jan 15 22:04:07 UTC 2020
Hi
On Wed, Jan 15, 2020 at 9:37 PM Yann E. MORIN <yann.morin.1998 at free.fr> wrote:
>
> NOTE: NOT TO BE COMMITTED!
>
> This patch is a proof of concept for the transient download mechanism,
> that would help using branches as versions, and keep using the head of
> the branch each time a build started.
>
> A package declares its download as transient with:
> FOO_DOWNLOAD_TRANSIENT = YES
>
Why we can not just have like
PKG_TRANSIENT_OVERRIDE_FILE="ci.mk"
ci.mk
FOO_DOWNLOAD_TRANSIENT = YES
FOO_REVISION="<TAG> e/o BRANCH"
Michael
> this means that the download infra will not use any already downloaded
> archive, and will instead always download it as if missing.
>
> Since the check is done in the download wrapper, we have no TOCTOU race
> in case two bulds would attempt the same transient download: the archive
> is only replaced ato,mically as usual.
>
> So, if the package uses a branch as version, the branch's HEAD at that
> very moment will be redownloaded.
>
> This stil has the drawback that two builds in parallel may get slightly
> different content for the same branch, and the first build could end up
> using the download of the second build:
>
> build-1 build-2
>
> download
> | download
> | |
> save to dl-dir |
> [...] save to dl-dir
> extract
>
> Furthermore, even with a single build, it might get what it expects:
>
> developer-1 developer-2
>
> git push branch
> trigger CI git push branch
> [...]
> download
>
> In that case the build of delopper-1 would get the code of developper-2
> who pushed on the same branch.
>
> For people who are aiming at their feet, we're now providing them with
> a loaded gun. ;-]
>
> Signed-off-by: Yann E. MORIN <yann.morin.1998 at free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni at bootlin.com>
> Cc: Vincent Fazio <vfazio at xes-inc.com>
> Cc: Michael Nazzareno Trimarchi <michael at amarulasolutions.com>
> Cc: Chris Packham <judge.packham at gmail.com>
> Cc: Nicolas Carrier <nicolas.carrier at orolia.com>
> Cc: Adam Duskett <aduskett at gmail.com>
> ---
> docs/manual/adding-packages-generic.txt | 16 +++++++++++++---
> package/busybox/busybox.mk | 1 +
> package/pkg-download.mk | 1 +
> package/pkg-generic.mk | 8 +-------
> support/download/dl-wrapper | 8 +++++---
> 5 files changed, 21 insertions(+), 13 deletions(-)
>
> diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
> index baa052e31c..afffb09bc8 100644
> --- a/docs/manual/adding-packages-generic.txt
> +++ b/docs/manual/adding-packages-generic.txt
> @@ -206,12 +206,14 @@ information is (assuming the package name is +libfoo+) :
> ** a tag for a git tree +LIBFOO_VERSION = v0.1.2+
> +
> .Note:
> -Using a branch name as +FOO_VERSION+ is not supported, because it does
> -not and can not work as people would expect it should:
> +Using a branch name as +FOO_VERSION+, although technically possible,
> +is highly discouraged, because it does not and can not work as people
> +would expect it should:
> +
> 1. due to local caching, Buildroot will not re-fetch the repository,
> so people who expect to be able to follow the remote repository
> - would be quite surprised and disappointed;
> + would be quite surprised and disappointed (but see
> + +LIBFOO_DOWNLOAD_TRANSIENT+, later);
> 2. because two builds can never be perfectly simultaneous, and because
> the remote repository may get new commits on the branch anytime,
> two users, using the same Buildroot tree and building the same
> @@ -342,6 +344,14 @@ not and can not work as people would expect it should:
> submodules when they contain bundled libraries, in which case we
> prefer to use those libraries from their own package.
>
> +* +LIBFOO_DOWNLOAD_TRANSIENT+ can be set to +YES+ or +NO+ (the default).
> + If set to +YES+, the download for that package will be attempted at
> + every build; any already downloaded archive is ignored as if missing.
> + This may help when a branch is specified in +LIBFOO_VERSION+, and the
> + head/tip of the branch is to be built, like a CI pipeline would need
> + for example. This still suffers from the other issues listed above
> + about using branches, though.
> +
> * +LIBFOO_STRIP_COMPONENTS+ is the number of leading components
> (directories) that tar must strip from file names on extraction.
> The tarball for most packages has one leading component named
> diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
> index 6283bc96ea..ee17febe48 100644
> --- a/package/busybox/busybox.mk
> +++ b/package/busybox/busybox.mk
> @@ -9,6 +9,7 @@ BUSYBOX_SITE = http://www.busybox.net/downloads
> BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2
> BUSYBOX_LICENSE = GPL-2.0
> BUSYBOX_LICENSE_FILES = LICENSE
> +BUSYBOX_DOWNLOAD_TRANSIENT = YES
>
> define BUSYBOX_HELP_CMDS
> @echo ' busybox-menuconfig - Run BusyBox menuconfig'
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index de619ba90a..c3a98ce7b0 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -108,6 +108,7 @@ define DOWNLOAD
> -n '$($(2)_BASENAME_RAW)' \
> -N '$($(2)_RAWNAME)' \
> -o '$($(2)_DL_DIR)/$(notdir $(1))' \
> + $(if $($(2)_DOWNLOAD_TRANSIENT),-F) \
> $(if $($(2)_GIT_SUBMODULES),-r) \
> $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
> $(QUIET) \
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 268d999efb..7c2e9db604 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -158,13 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
> @$(call step_start,download)
> $(call prepare-per-package-directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES))
> $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
> -# Only show the download message if it isn't already downloaded
> - $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \
> - if test ! -e $($(PKG)_DL_DIR)/`basename $$p` ; then \
> - $(call MESSAGE,"Downloading") ; \
> - break ; \
> - fi ; \
> - done
> + @$(call MESSAGE,"Downloading") ; \
> $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep))
> $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
> $(Q)mkdir -p $(@D)
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index 3315bd410e..ab22ca7e4f 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -21,11 +21,12 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
>
> main() {
> local OPT OPTARG
> - local backend output hfile recurse quiet rc
> + local backend output hfile recurse quiet rc force
> local -a uris
>
> # Parse our options; anything after '--' is for the backend
> - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
> + force=false
> + while getopts ":c:d:D:o:n:N:H:rf:u:qF" OPT; do
> case "${OPT}" in
> c) cset="${OPTARG}";;
> d) dl_dir="${OPTARG}";;
> @@ -38,6 +39,7 @@ main() {
> f) filename="${OPTARG}";;
> u) uris+=( "${OPTARG}" );;
> q) quiet="-q";;
> + F) force=true;;
> :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
> \?) error "unknown option '%s'\n" "${OPTARG}";;
> esac
> @@ -67,7 +69,7 @@ main() {
> # - matches all its hashes: do not download it again and exit promptly
> # - fails at least one of its hashes: force a re-download
> # - there's no hash (but a .hash file): consider it a hard error
> - if [ -e "${output}" ]; then
> + if ! ${force} && [ -e "${output}" ]; then
> if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
> exit 0
> elif [ ${?} -ne 2 ]; then
> --
> 2.20.1
>
--
| Michael Nazzareno Trimarchi Amarula Solutions BV |
| COO - Founder Cruquiuskade 47 |
| +31(0)851119172 Amsterdam 1018 AM NL |
| [`as] http://www.amarulasolutions.com |
More information about the buildroot
mailing list