[Buildroot] Buildroot 2009.02 released
Ulf Samuelsson
ulf.samuelsson at atmel.com
Tue Feb 24 18:15:46 UTC 2009
----- Original Message -----
From: Frank Hoeflich
To: Ulf Samuelsson
Cc: buildroot at busybox.net
Sent: Sunday, February 22, 2009 8:22 AM
Subject: Re: [Buildroot] Buildroot 2009.02 released
Ulf:
>fre 2009-02-20 klockan 09:23 +0100 skrev Peter Korsgaard:
>> >>>>> "Ulf" == Ulf Samuelsson <ulf.samuelsson at atmel.com> writes:
>>
>> Hi,
>>
>> >> Some testing was certainly done, but more is ofcourse always welcome.
>>
>> Ulf> I am not aware that anyone but myself has published any test results.
>>
>> I did for the toolchains, and for everything else I just fixed the
>> problems when I found any.
>>
>> Ulf> There is no recommended way to publish test results.
>> Ulf> No recommended format etc.
>>
>> Sure, like with all other issues - Add an issue to the tracker or mail
>> the list, both preferably with a patch.
>
>You REALLY dont get it , do you?
>
>If you for once stopped seeing Buildroot as a hobby
>and take the view of someone wanting to build
>a root file system without digging deep into
>the internals of Buildroot, then you would realize
>that Bugzilla is NOT the way this shoudl be communicated.
>
OK, enough. Ulf, I am asking you to stop taking this tone with Peter (and for that matter the rest of us).
FWIW I agree with you (and more) that:
1. Buildroot needs a real test plan for the new releases, and some method of
distributing the test suite so that list members can contribute new individual tests as
they see fit in order to fill out the BR test suite.
The traditional way of testing buildroot is to make a "defconfig".
I came to the conclusion that doing this is no good, since this will crash
on the first build error, and you will lose valuable time.
It is also hard to get a nice log for each package.
make <package> from the command line should work (assuming the .config file is OK)
and that is why I generated the build script which goes through a number of packages
and builds them printing out the result.
I think the first decision we need to make, is if we can use defconfigs or not
to fulfil a testplan.
If we decide not to use the defconfigs, then we need to decide
how to implement package specific testing.
Do we continue to use bash scripts, or do we use python or similar.
My build script creates a result file which indicates if the package built OK or not OK.
Not in html/xml format, which is a weakness,at first sight.
We need to decide on the format of the test reports as well as the format of the
documentation.
For each package, it creates a build log which is moved to either
log/OK or log/FAIL depending on success.
I think we should consider making build reports available on a web server.
2. Buildroot should meet its (reasonable) release test plan criteria before a release goes
out the door, like any other software package worth its salt. For now these could be
pretty modest.
So we need to agree on which host distributions we want to support,
and then we need to have a list of reasonable combinations of packages/kernels
Once the list of packages is available, then you could make a number of defconfigs,
or you can forceenable each package by a select statement in the test harness.
By introducing a package specific depend on !BR2_PACKAGE_<PACKAGE>_DISABLE
you will have total control and ensure that you build identically root file systems
for each archtiecture/host
That is also a key decision.
What is then lacking is any real test on H/W.
It is nice if the package, build but that is no good, if the
end result does not boot, or does not behave as expected.
Since some people may lack hardware but still be interested in helping
building on a host, it would be good to publish binaries,
which can be downloaded to targets and tested by others
having access to the board
We need to develop some kind of tests that uses the key functionality.
3. Buildroot test results should be published and visible on the web and should cleanly
hyperlink or otherwise refer back to test plan requirements. Once the list agrees on a
format for results it should be documented in the test plan.
4. Buildroot test vectors should certainly include a reasonable selection of default
configs, a representative set of architectures, host distributions and more.
However, I do *not* agree that:
1. Anyone active on this list is treating BR like a hobby. Nor is it like a commercial
product for which users may expect polished manuals and 24x7 support for their
RTFM questions. It's an open source project; users need to be willing and able to
follow the list and/or query Bugzilla in order to check that something they're seeing is
a known issue.
Your comments above certainly shows that you see the problem with the current approach.
I believe that the goal should be that noone sees any issues at all, unless they are
interested in becoming a developer and contribute.
This goal might not be shared by others, but it is sad, if people are against making
Buildroot easier to use.
For this to work, I think there is a need for configuration of user properties in buildroot.
User properties are distinct from project properties, and should therefore not be
in the defconfigs.
2. Issues posted to Bugzilla are not `in a usable format'. They are searchable, are
capable of being well-documented, may contain descriptive attachments as needed
and have adequate flags for prioritisation and status.
The context for this discussion, is how users perceive Buildroot.
Bugzilla/mailing list is excellent to help resolve a problem, but is fairly
useless/timeconsuming if you want to figure out what will run and what will not.
3. BR development should be targeted toward users `without too much knowledge'.
Such users can always hire an active developer as an interpreter/jungle guide. Linux
and most of the open source world do not lend themselves at all well to users without
a reasonable amount of hands-on knowledge. `Fixing' that would mean grinding a
much larger axe, I think.
When design decisions are made, you often have a several ways to implement things.
Some decisions will result in high maintenance and high learning threshold,
and some will result in low maintenance and a low learning treshold.
I want to see more of the latter, or a good argument why not.
4. All of the testing and documentation issues mentioned should have been addressed
in Buildroot's first release in years. If this release serves 80%+ of the BR public
with workaroundable annoyance only pending a next release of better stability, it will
have done its job. Getting to metricatable quality may take quite some time.
Rome was not built in a day, but I think you can agree that there was *zero* effort
to document anything for the first "release".
I think that if this had gotten *any* priority, we would have now have a list of combinations
host/arch/filesystems where this had been tested without delaying the release.
There would certainly be holes, but since these would be visible, it would allow
people to put in some effort to fill these holes.
It may be that buildbot can nearly accommodate all of this now, or it may be that a good deal of further development is needed.
--Frank
BTW, Thanks for a constructive email!
BR
Ulf Samuelsson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20090224/776084c3/attachment.html>
More information about the buildroot
mailing list