[Buildroot] [PATCHv2] infra: add the transient download mechanism

Arnout Vandecappelle arnout at mind.be
Tue Oct 6 18:43:22 UTC 2020



On 30/09/2020 22:13, Yann E. MORIN wrote:
> Thomas, All,
> 
> On 2020-09-30 21:45 +0200, Thomas Petazzoni spake thusly:
>> On Wed, 30 Sep 2020 19:30:30 +0200
>> "Yann E. MORIN" <yann.morin.1998 at free.fr> wrote:
> [--SNIP--]
>>> +  obvious being the *lack of reproducibility*. Besides, it has its own
>>> +  pitfalls that one must be aware of as well:
>>> +  ** when two builds are running in parallel on the same machine, there
>>> +     is a TOCTOU race for each build to download and extract the archive,
>>> +     so each build may not get what it downloaded, but what the other
>>> +     build did download;
>>
>> This is not a pitfall, it is what we expect from the feature. The
>> feature is "grab" whatever is currently available on the Git repository
>> for that package. It is pretty clear that it is not guarantee that we
>> grab what existed in the Git repository at the moment the user hits
>> "Enter" after typing the make command. There is no way we could
>> atomically snapshot all Git trees of all packages to ensure that what
>> we build is a consistent view of what existed at the moment the build
>> was started.
>>
>> So I think this pitfall is needlessly worrying, and does not actually
>> point to an actual problem. Or more precisely, the problem you're
>> pointing is *exactly* what the transient download mechanism aims at
>> doing.
> 
> I think it is important to stress that, though, because people can be
> very confused but the outcome, one way or the other.
> 
> For example, let's assume that I push my branch, that triggers a
> pipeline in the CI, the pipeline uses my branch as expected. Then I
> repush a modified branch, the new pipeline uses the new HEAD of the
> branch, and again this is as expected..
> 
> But now consider the following scenario:
> 
>   - I push my branch, a pipeline is triggered;

 Here your reasoning is wrong: this feature should *NOT* be used for triggering
a build on push to some repo. For that use case, OVERRIDE_SRCDIR is much more
suitable, because that *will* use the exact sources that correspond to the trigger.

 What this feature is meant for is to do actual continuous integration, i.e.
people are developing code in different repos, and a regularly triggered (or
continuous) build brings all these repos together. It sounds insane to do
something like that, but it is the way that some projects work [1].


 Regards,
 Arnout

[1] https://abseil.io/about/philosophy#we-recommend-that-you-choose-to-live-at-head


>   - before the pipeline has a chance to git-fetch, I do a quick change
>     and repush, a second pipeline is triggered;
>   - the first pipeline uses the code from the second push.
> 
> However, my change was intentional, and I was expecting the CI to test
> the two changes, and I wanted to have the results of both to decide
> which to keep.
> 
> This for example, is very interesting because the CI can do tests I
> can't always do locally, like reproducibility test for performance, for
> example, or test on hardware I don't have at hand (when working from
> home for example), and so on.
> 
> Thus my case is borked. Yeah, I could do two branches, but the fact is
> that if I wait "just enough" between the pushes, it works, while if I
> don't wait "just enough" it does not work.
> 
> Conversely, if I notice that I forgot to commit part of the changes, and
> amend my commit and repush, I don't care what the first pipeline gets;
> I only care about the second.
> 
> So, this is a pitfall: behaviours that may or may not match your
> expectations, depending on what your expectations are, because this
> mechanism caters to both antagonist cases where expectations are
> opposite one from the other.
> 
> [--SNIP--]
>>> +  ** as a consequence, legal-info is not reliable when a package is
>>> +     transient;
>> *That* I believe is the real pitfall.
> 
> Not, it is a consequence of the pitfall.
> 
>>> +  ** if a branch is pushed to an automated build bot (in a CI for
>>> +     example), and then re-pushed to while the build is still in
>>> +     progress, the build may get the wrong version of the branch.
>>
>> This is also *NOT* a pitfall, it is what we ask the transient download
>> mechanism to do.
>>
>> Once this is fixed, I'll be happy to apply the patch!
> 
> Guess we'll have to refer to higher-level hierarchy to take a decision. ;-)
> 
> Thanks for the comments. :-)
> 
> Regards,
> Yann E. MORIN.
> 
>> Best regards,
>>
>> Thomas
>> -- 
>> Thomas Petazzoni, CTO, Bootlin
>> Embedded Linux and Kernel engineering
>> https://bootlin.com
> 


More information about the buildroot mailing list