[Buildroot] [PATCH/autobuild 8/8] autobuild-run: use in-tree toolchain configs
Arnout Vandecappelle
arnout at mind.be
Mon Apr 10 08:49:42 UTC 2017
Ho, turns out I sent this out before amending, so this patch is NOT correct...
I'll quickly send out an update for proper review, but it'll probably need a v3
as well.
Regards,
Arnout
On 09-04-17 22:51, Arnout Vandecappelle (Essensium/Mind) wrote:
> Instead of having the list of toolchain configs as a CSV file, they
> are now maintained within the buildroot tree itself. Therefore,
> autobuild-run must get the toolchain configs from there.
>
> The toolchains-url option is replaced with a toolchains-path option.
> This path is relative to the buildroot directory and defaults to
> support/config-fragments/autobuild. It is possible to override this
> with a relative or absolute path to the directory containing the
> toolchain configs.
>
> The toolchain configs no longer contain the hostarch information. For
> the predefined external toolchains, this is handled by checking after
> the olddefconfig: because of "depends on HOSTARCH=...", the toolchain
> config will have been removed. For the custom external toolchains, we
> instead rely on the URL: it ends with -HOSTARCH.tar.gz.
>
> Since we don't fetch from a URL anymore, the exception handling
> introduced in 5f1016ab0 is no longer needed.
>
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout at mind.be>
> ---
> Clearly, at the moment, the custom external toolchain tarball names
> *don't* end with -HOSTARCH, so the hostarch check doesn't actually
> work. However, I think this is the most elegent solution. Other
> alternatives are:
> - Adding a dummy config option BR2_HOSTARCH=... to the toolchain
> config fragments. This works pretty nicely, because then it will
> be covered automatically by the check that all the toolchain config
> lines are still there. However, this requires manual handling when
> generating the toolchain configs (they are no longer the result of
> "make savedefconfig"). Also, it is not enough for the x86 case
> (i.e. the ctng toolchains): these also work on x86_64 autobuilders,
> so special handling for that case would be needed.
> - As it happens, all br-* toolchains are x86_64, and all ctng
> toolchains are x86, so we could just do string matching on those to
> derive the hostarch. However, that would be too much of a hack for
> my comfort.
>
> Any other ideas are welcome!
>
> If this solution is chosen, I propose to replace
> http://patchwork.ozlabs.org/patch/748187/ with the renamed tarballs
> immediately while applying.
> ---
> scripts/autobuild-run | 62 +++++++++++++++------------------------------------
> 1 file changed, 18 insertions(+), 44 deletions(-)
>
> diff --git a/scripts/autobuild-run b/scripts/autobuild-run
> index d17e089..f920251 100755
> --- a/scripts/autobuild-run
> +++ b/scripts/autobuild-run
> @@ -67,7 +67,7 @@ defaults = {
> '--nice': 0,
> '--pid-file': '/tmp/buildroot-autobuild.pid',
> '--http-url': 'http://autobuild.buildroot.org/submit/',
> - '--toolchains-url': 'http://autobuild.buildroot.org/toolchains/configs/toolchain-configs.csv',
> + '--toolchains-path': 'support/config-fragments/autobuild',
> }
>
> doc = """autobuild-run - run Buildroot autobuilder
> @@ -102,7 +102,8 @@ Options:
> -c, --config CONFIG path to configuration file
> Not set by default.
> -d, --debug Send log output to stdout instead of log file
> - --toolchains-url URL URL of toolchain configuration file
> + --toolchains-path URL Path to the toolchain configuration files
> + (relative to buildroot dir).
>
> Format of the configuration file:
>
> @@ -169,7 +170,7 @@ else:
> encode_str = _identity
>
> MAX_DURATION = 60 * 60 * 8
> -VERSION = 1
> +VERSION = 2
>
> def log_write(logf, msg):
> logf.write("[%s] %s\n" % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), msg))
> @@ -255,44 +256,20 @@ def get_toolchain_configs(**kwargs):
> This function returns an array of toolchain configurations. Each
> toolchain configuration is itself an array of lines of the defconfig.
> """
> - toolchains_url = kwargs['toolchains_url']
> + toolchains_path = kwargs['toolchains_path']
>
> - with urlopen_closing(toolchains_url) as r:
> - l = decode_byte_list(r.readlines())
> - configs = []
> -
> - (_, _, _, _, hostarch) = os.uname()
> - # ~2015 distros report x86 when on a 32bit install
> - if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
> - hostarch = 'x86'
> -
> - for row in csv.reader(l):
> - config = {}
> - url = row[0]
> - config_hostarch = row[1]
> - keep = False
> -
> - # Keep all toolchain configs that work regardless of the host
> - # architecture
> - if config_hostarch == "any":
> - keep = True
> -
> - # Keep all toolchain configs that can work on the current host
> - # architecture
> - if hostarch == config_hostarch:
> - keep = True
> -
> - # Assume that x86 32 bits toolchains work on x86_64 build
> - # machines
> - if hostarch == 'x86_64' and config_hostarch == "x86":
> - keep = True
> -
> - if not keep:
> - continue
> + if os.path.isabs(toolchains_path):
> + toolchains_dir = toolchains_path
> + else:
> + idir = "instance-%d" % kwargs['instance']
> + toolchains_dir = os.path.join(idir, "buildroot", toolchains_path)
>
> - with urlopen_closing(url) as r:
> - config = decode_byte_list(r.readlines())
> - configs.append(config)
> + configs = []
> + for configfile in os.listdir(toolchains_dir):
> + if configfile.endswith('.config'):
> + with open(os.path.join(toolchains_dir, configfile)) as r:
> + config = decode_byte_list(r.readlines())
> + configs.append(config)
> return configs
>
> def prepare_build(**kwargs):
> @@ -535,10 +512,7 @@ def gen_config(**kwargs):
> log_write(log, "INFO: generate the configuration")
>
> # Select a random toolchain configuration
> - try:
> - configs = get_toolchain_configs(**kwargs)
> - except:
> - return -1
> + configs = get_toolchain_configs(**kwargs)
>
> i = randint(0, len(configs) - 1)
> config = configs[i]
> @@ -928,7 +902,7 @@ def main():
> submitter = args['--submitter'],
> make_opts = (args['--make-opts'] or ''),
> nice = (args['--nice'] or 0),
> - toolchains_url = args['--toolchains-url'],
> + toolchains_path = args['--toolchains-path'],
> upload = upload,
> buildpid = buildpid,
> debug = args['--debug']
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
More information about the buildroot
mailing list