[Buildroot] [PATCH 1/1] Add gh site method for GitHub cli

Sat May 17 16:41:30 UTC 2025

On Fri, 2025-05-16 at 23:18 +0200, Arnout Vandecappelle wrote:
> Having something github specific is not very nice... We should then
> also 
> provide the equivalent for gitlab, bitbucket, ...

Thanks for looking into this.

There's a tool https://github.com/gruntwork-io/fetch but I haven't
given it further consideration because it doesn't seem to be packaged
in Fedora or Debian.

> > It is intended as an alternative to pulling with the "git" method,
> > which
> > has the drawback of requiring separate ssh keys
> 
>   I guess you mean that you need to generate ssh keys for CI?

Other people may have different experiences, but mine is that if you're
working within a bigger organization you want to stick to https because
otherwise you'll be constantly fighting the admins about getting port
22 unblocked and keeping it that way.

> > and resulting in
> > different archives depending on the buildroot version,
> 
>   How does gh help with that? The tarballs it produces are not
> reproducible at 
> all... The ones we use happen to be the same all the time because we
> re-request 
> them all the time so they're cached, but there's actually no
> guarantee for that.

I agree that it is an issue, but we have an internal primary set up
that keeps copies of the archives indefinitely.

Out of curiosity, are you sure you're hitting GitHub that often? Even
for rarely used packages cached by sources.buildroot.net?

>From where I stand, it's more of a hypothetical issue as compared to
the practical one that some users are adding packages with "br1" hashes
to our shared external while others add "git4" hashes.

> > and passing an
> > authentication token on the wget command line, which risks leaking
> > the
> > token to third-party servers.
> 
>   Well, that's not really a good reason, there are better ways to do
> that. I've 
> created a documentation patch [1] to explain it and put you in Cc.

Good addition to the documentation, however I've already used all those
tricks and still here we are.

The .netrc method unfortunately does not work because the forges
generally require a token header instead of http basic authentication.
And because GitHub might change things in the future, going with the
official cli tool seems like a relatively safe bet.

Regards,
Stefan

> [1] 
> https://patchwork.ozlabs.org/project/buildroot/patch/20250516211429.763211-1-arnout@rnout.be/
> 
> 
> 
> > 
> > Signed-off-by: Stefan Nickl <Stefan.Nickl at gmail.com>
> > ---
> >   Config.in                   |  4 ++++
> >   package/pkg-download.mk     |  1 +
> >   support/download/dl-wrapper |  2 +-
> >   support/download/gh         | 43
> > +++++++++++++++++++++++++++++++++++++
> >   4 files changed, 49 insertions(+), 1 deletion(-)
> >   create mode 100755 support/download/gh
> > 
> > diff --git a/Config.in b/Config.in
> > index d730f2034b..cb178ee8d2 100644
> > --- a/Config.in
> > +++ b/Config.in
> > @@ -172,6 +172,10 @@ config BR2_HG
> >   	string "Mercurial (hg) command"
> >   	default "hg"
> >   
> > +config BR2_GH
> > +	string "GitHub cli command"
> > +	default "gh"
> > +
> >   config BR2_ZCAT
> >   	string "zcat command"
> >   	default "gzip -d -c"
> > diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> > index cf5959ea95..08b0d420c7 100644
> > --- a/package/pkg-download.mk
> > +++ b/package/pkg-download.mk
> > @@ -111,6 +111,7 @@ define DOWNLOAD
> >   	CVS="$(call qstrip,$(BR2_CVS))" \
> >   	GIT="$(call qstrip,$(BR2_GIT))" \
> >   	HG="$(call qstrip,$(BR2_HG))" \
> > +	GH="$(call qstrip,$(BR2_GH))" \
> >   	LOCALFILES="$(call qstrip,$(BR2_LOCALFILES))" \
> >   	SCP="$(call qstrip,$(BR2_SCP))" \
> >   	SFTP="$(call qstrip,$(BR2_SFTP))" \
> > diff --git a/support/download/dl-wrapper b/support/download/dl-
> > wrapper
> > index 5445aad5a7..a209665b0f 100755
> > --- a/support/download/dl-wrapper
> > +++ b/support/download/dl-wrapper
> > @@ -95,7 +95,7 @@ main() {
> >           backend_urlencode="${uri%%+*}"
> >           backend="${backend_urlencode%|*}"
> >           case "${backend}" in
> > -            git|svn|cvs|bzr|file|scp|hg|sftp) ;;
> > +            git|svn|cvs|bzr|file|scp|hg|gh|sftp) ;;
> >               ftp|ftps) backend="curl" ;;
> >               *) backend="wget" ;;
> >           esac
> > diff --git a/support/download/gh b/support/download/gh
> > new file mode 100755
> > index 0000000000..5c36855ccf
> > --- /dev/null
> > +++ b/support/download/gh
> > @@ -0,0 +1,43 @@
> > +#!/usr/bin/env bash
> > +
> > +# We want to catch any unexpected failure, and exit immediately
> > +set -e
> > +
> > +# Download helper for GitHub CLI, to be called from the download
> > wrapper script
> > +#
> > +# Options:
> > +#   -o FILE     Save into file FILE.
> > +#   -f FILENAME The filename of the tarball to get at URL
> > +#   -u URL      Download file at URL.
> > +#
> > +# Environment:
> > +#   GH        : the gh command to call
> > +
> > +quiet=
> > +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> > +    case "${OPT}" in
> > +    o)  output="${OPTARG}";;
> > +    f)  filename="${OPTARG}";;
> > +    u)  url="${OPTARG}";;
> > +    :)  printf "option '%s' expects a mandatory argument\n"
> > "${OPTARG}"; exit 1;;
> > +    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> > +    esac
> > +done
> > +
> > +shift $((OPTIND-1)) # Get rid of our options
> > +
> > +# Caller needs to single-quote its arguments to prevent them from
> > +# being expanded a second time (in case there are spaces in them)
> > +_gh() {
> > +    if [ -z "${quiet}" ]; then
> > +        printf '%s ' ${GH} "${@}"; printf '\n'
> > +    fi
> > +    _plain_gh "$@"
> > +}
> > +# Note: please keep command below aligned with what is printed
> > above
> > +_plain_gh() {
> > +    eval ${GH} "${@}"
> > +}
> > +
> > +_gh api -H "'Accept: application/vnd.github.raw+json'" "${@}" \
> > +    "'${url}/${filename}'" ">'${output}'"