[Buildroot] [RFC v2 00/31] Automatically produce legal compliance info
Thomas De Schampheleire
patrickdepinguin+buildroot at gmail.com
Fri Mar 9 08:47:16 UTC 2012
Hi Luca,
On Wed, Mar 7, 2012 at 9:58 PM, Luca Ceresoli <luca at lucaceresoli.net> wrote:
> Hi,
>
> during the latest two Buildroot Developers Days (in November 2011 and February
> 2012) and in this mailing list there has been some discussion about introducing
> in Buildroot the possibility to automatically derive legally relevant material,
> such as licensing info and source tarballs for open source packages.
>
> I have submitted a first tentative RFC implementation of these features back in
> late January
> (http://lists.busybox.net/pipermail/buildroot/2012-January/049590.html).
>
> This second RFC incorporates many of the additions and modifications that have
> been discussed, as well as many other improvements. Thanks Arnout, ThomasDS and
> others for their review, and the BDD participants for their suggestions.
>
> This code is now fully working, meaning that the core features are there and
> those things that cannot be handled produce big fat warnings.
> I think this code does not need big changes before being merged (unless your
> review is catastrophic).
Thanks for continuing this work, I'm sure it will help many developers
out there.
I have added some suggestions/comments on the first patch. See below
for some other comments.
>
> However I hit a few issues that still need to be sorted out, see the "ISSUE"
> sections below.
>
> My approach is based on two per-package constants in eack .mk file, such as:
> FOOBAR_LICENSE = GPLv3 + LGPLv2.1
> FOOBAR_LICENSE_FILES = COPYING.LGPL demo-app/COPYING.GPL3
> or:
> MYAPP_LICENSE = PROPRIETARY
> # MYAPP_LICENSE_FILES not relevant in this case
> This is the only effort required to the package creator. If <PKG>_LICENSE is
> not specified it defaults to "unknown".
>
> After running 'make legal-info', the following things will be produced in
> $(O)/legal-info/:
> $ find legal-info/ -type f
> legal-info/README # Lists saved stuff, warns about unsaved stuff
> legal-info/licenses.txt # Text of all licenses
> legal-info/buildroot.config # The buildroot config
> legal-info/licenses/buildroot/COPYING # License files, one dir per pkg
> legal-info/licenses/busybox/LICENSE # ...
> legal-info/licenses/...other packages... # ...
> legal-info/manifest.csv # CSV table summarizing all info
> legal-info/sources/busybox-1.19.4.tar.bz2 # tarballs
> legal-info/sources/kmod-5.tar.xz # ...
> legal-info/sources/libtool-2.2.10.tar.gz # ...
> legal-info/sources/...other packages... # ...
I'm a bit confused by licenses.txt: if we already have each individual
license file in the licenses/<package> directory, why do we need an
aggregated set in licenses.txt ?
>
> Given the technical difficulties, the toolchain and the BR sources are not
> saved. Warnings are generated to make sure the user is aware of this.
>
> Following is an explanation of the open issues and future development
> directions. Actually issue 3 is the one where I mostly would like to have
> comments, so you may skip to it if your time is limited. Otherwise seat down
> comfortably and read on.
>
>
> ISSUE 1: the License Repository Feature
> =======================================
>
> The original idea that came out of the last Buildroot Developer Day was to
> maintain in BR a "license repository" holding those licenses that are used
> without change by many projects, such as the GPL family.
> Packages that cannot use this mechanism would need to explicitly define the
> license file in <PKG>_LICENSE_FILES.
>
> The idea of implementation was:
> if (<PKG>_LICENSE_FILES defined):
> copy $(<PKG>_LICENSE_FILES)
> else if (<PKG>_LICENSE = *GPL*, or any other known that's always equal):
> copy file from a "license repo dir" (one copy only per file);
> else:
> copy nothing and warn user
>
> Unfortunately, as Will Moore also pointed out, I discovered that the same
> license (e.g. GPLv2) is not always identical between two packages, although
> the differences might be not substantial.
>
> I do not have precise figures, but the number of variations might be quite
> large. In a config with ~25 packages enabled I got this:
> $ make external-deps|wc -l
> 39
> $ find . -iname "COPYING*" -o -iname "LICENSE*" | \
> xargs grep -l 'Version 2,' | xargs md5sum | \
> awk '{print $1}'|sort|uniq|wc -l
> 20
>
> These are 20 different GPLv2 files! Some only have whitespace changes, some
> have other little differences. An example:
>
> $ diff -u0 -b ./binutils-2.21.1/COPYING ./libtool-2.2.10/COPYING
> --- ./binutils-2.21.1/COPYING 2005-07-14 03:24:56.000000000 +0200
> +++ ./libtool-2.2.10/COPYING 2010-05-20 23:18:41.000000000 +0200
> @@ -4 +4 @@
> - Copyright (C) 1989, 1991 Free Software Foundation, Inc.
> + Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
>
> An added comma.
>
> @@ -18 +18 @@
> -the GNU Library General Public License instead.) You can apply it to
> +the GNU Lesser General Public License instead.) You can apply it to
>
> This updates a reference to another license.
>
> @@ -306,4 +306,3 @@
> - You should have received a copy of the GNU General Public License
> - along with this program; if not, write to the Free Software
> - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> -
> + You should have received a copy of the GNU General Public License along
> + with this program; if not, write to the Free Software Foundation, Inc.,
> + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>
> Just formatting changes here.
>
> @@ -339 +338 @@
> -library. If this is what you want to do, use the GNU Library General
> +library. If this is what you want to do, use the GNU Lesser General
>
> Same as above.
>
> Another example:
>
> $ diff -u0 -b ./binutils-2.21.1/COPYING ./iostat-2.2/LICENSE
> --- ./binutils-2.21.1/COPYING 2005-07-14 03:24:56.000000000 +0200
> +++ ./iostat-2.2/LICENSE 2004-11-25 11:53:11.000000000 +0100
> @@ -5 +5 @@
> - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
>
> Apparently the FSF offices have moved some time in the past.
>
> @@ -308 +308 @@
> - Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
>
> Same as above.
>
> I did not investigate a lot to discover if there are more substantial
> differences. I wanted to stay on the safe side, so I just did not implement the
> "license repository" and the second step of the algorithm. All packages that
> want their license files copied must define <PKG>_LICENSE_FILES. Full stop.
>
> The license repository feature may still be added in the future, but I think
> it should be discussed before. Asking the FSF may be an option, I guess they
> listen to an active community developing free software.
For me treating each license as a different one is OK. As you say,
it's for sure the safest option. It will keep the logic in buildroot
simple.
>
>
> ISSUE 2: Packages with multiple licenses
> ========================================
>
> Some packages (e.g. freetype, qt) allow to choose among different licenses.
> - Should we add an option in menuconfig for choosing the license? This would
> allow to generate "customized" manifest and license files.
> - Or should we save all licenses and and define the package license as, for
> instance:
> QT_LICENSE = GPLv3 or LGPLv2.1 or COMMERCIAL?
>
> I think the first option is more nice and still legally correct, so I
> implemented it for Qt to see how it would look. You can see the implementation
> in a patch near the bottom of this set. It's not currently documented at all,
> I'd first like to know if the approach is good or option 2 (or whatever else)
> is better.
In any case, a commercial license cannot be used without having a
legal agreement with the package author. Since this is a rare case, I
don't think we should try to handle this in buildroot. I'd let the
user handle this case.
Personally, I don't think that many people will be careful in
selecting the appropriate license from the config system. We'd need a
default, and the question becomes what's the default? Some people will
prefer v3 and others v2.
I'd keep it up to the user to fix the csv and related files to reflect
their choice. We can already copy all licenses as you suggest, but I
don't know whether a user should remove the non-appropriate files or
not to be legally ok...
>
>
> ISSUE 3: packages without a license file
> ========================================
>
> Some packages (e.g. tslib) do not have a license file. Instead, the license is
> written in a comment at the top of one or more source files. In such cases we
> planned that the package maintainer would manually copy this text into a file
> in the package directory (where the .mk lives) and the teach the infrastructure
> where it is, so it gets copied from there.
>
> I haven't started working on an implementation of this feature yet, but at
> first sight it's not totally trivial. The problem is the <PKG>_LICENSE_FILES
> definition as it is implemented in the present RFC does not easily allow to
> discriminate between a file in $(O)/build/<pkg>-<ver>/ and a file in
> packages/<pkg>/.
>
> This is how it works now:
> @cp $(addprefix $$($(2)_DIR)/,$$($(3)_LICENSE_FILES)) \
> $(LICENSE_FILES_DIR)/$$($(3)_NAME)/
> It assumes the files are in the package build dir, $(2)_DIR, which allows a
> very simple and clean implementation.
>
> In order to support a copy from the package directory, option 1 is to define
> the full path in <PKG>_LICENSE_FILES, and remove the addprefix from the above
> code snipped. Example:
> FOO_LICENSE_FILES = $(BUILD_DIR)/foo-$(FOO_VERSION)/COPYING
> BAR_LICENSE_FILES = packages/bar/COPYING
> which is not very pleasant to read, and is quite error-prone for the developer
> adding a package.
>
> Option 2 would be to have two distinct constants for the two cases:
> FOO_LICENSE_FILES_IN_BUILD_DIR = COPYING
> BAR_LICENSE_FILES_IN_PACKAGE_DIR = COPYING
> Even with better constant names this would not be extremely beautiful IMHO.
>
> Option 3 is to wait for a smarter idea coming from your reviews! :)
>
>
> TODO in the next patchset before merging into mainline
> ======================================================
>
> - Add to the documentation:
> - some words of advice from Buildroot developers about how to comply to GPL
> and other open source licenses;
> - brief instructions on using this stuff ('make legal-info');
> - instructions in the GENTARGETS section about the _LICENSE and _LICENSE_FILES
> constants.
>
> - Write a few lines of explanation in the log message of the first big commit,
> the one that implements all the logic.
>
> - Save a defconfig instead of the whole .config for the Buildroot configuration?
> Which one would you better like?
A full .config is better I think, as it doesn't depend on the defaults
in buildroot (e.g. version bump).
>
>
> POSSIBLE FUTURE IMPROVEMENTS
> ============================
>
> IMHO these should be planned after the merge of the first core functionality:
> I would like to keep it as simple as possible for a first step.
>
> - How to handle local and override packages?
> - Modify the -source target (_DOWNLOAD macros) to produce tarball?
> But adds a lot of work for normal builds that is used only rarely.
> - Modify the legal-info stuff to re-download them and make a tarball?
> - Assume they are not / almost never used in production and just save
> nothing? Meaning just like the present patchset does, generating a warning.
> My vote here.
>
> - The toolchain is not currently saved (internal, external, ct-NG, no
> discrimination). Actually, only GENTARGETS-based packages are handled, so the
> best approach might be to "simply" migrate the toolchains to GENTARGETS.
>
> - Save the Buildroot sources too. If the sources are not a git clone this might
> be as simple as tar of the current directory and exclude dl and output, but
> this has never been tested. Also, make sure this works for out-of-tree BR
> builds.
>
> - Add a hook for a post-legal-info script.
>
>
> SHOW US THE CODE NOW
> ====================
>
> Ok, the patches are there.
> - The implementation is all in the first commit.
> - A few patches follow to make non-GENTARGETS packages warn about their
> dumbness.
> - Other commits define licenses for some packages.
> - A couple of support commits (for testing only) closes the series.
>
> Changed in v2:
> - squashed together patches 1-4 from RFC v1; now all the legal-info mechanism
> is implmented in a unique patch.
> - rebase on top of current master
> - don't clean $(REDIST_SOURCES_DIR): it is a subdir of $(LEGAL_INFO_DIR), so
> doesn't need to be cleaned twice
> - added legal-info-clean target
> - made legal-info target .PHONY
> - remove the output/legal-info dir before populating it
> - when saving source tarballs, create hardlinks instead of copies if possible
> - add infrastructure to warn the user about info that has not been saved: a
> .warnings file is filled with such info and displayed to the user at the
> end of the legal-info processing
> - ensure manual (non-GENTARGETS-based) packages return error, at least; this
> required to explicitly create a -legal-info target for each of them, or
> they would have been silently skipped.
> - list also Buildroot in the manifest file! :)
> - save the Buildroot .config
> - save license files listed in <PKG>_LICENSE_FILES, both in a separate
> directory for each package and all together in a unique file
> - various cleanups.
>
> Luca
>
> Luca Ceresoli (31):
> legal-info: infrastructure to collect legally-relevant material
> cups: warn that legal-info is not implemented
> fis: warn that legal-info is not implemented
> doom-wad: warn that legal-info is not implemented
> gettext: warn that legal-info is not implemented
> microperl: warn that legal-info is not implemented
> netkitbase: warn that legal-info is not implemented
> netkittelnet: warn that legal-info is not implemented
> newt: warn that legal-info is not implemented
> tinyhttpd: warn that legal-info is not implemented
> ttcp: warn that legal-info is not implemented
> uemacs: warn that legal-info is not implemented
> vpnc: warn that legal-info is not implemented
> xfsprogs: warn that legal-info is not implemented
> mpc: define license
> linux: define license
> m4: define license
> busybox: define license
> bzip2: define license
> directfb: define license
> iostat: define license
> lzo: define license
> lzop: define license
> tslib: define license
> libusb: define license
> pcre: define license
> netsnmp: define license
> berkeleydb: define license
> qt: define license choice
> foobar: create a fake proprietary package (testing only)
> Create test configs (testing only)
>
> Makefile | 52 ++++++++++++++++++++++++-
> configs/legal_info_test2_defconfig | 8 ++++
> configs/legal_info_test_defconfig | 20 ++++++++++
> linux/linux.mk | 2 +
> package/Config.in | 1 +
> package/Makefile.package.in | 59 +++++++++++++++++++++++++++++
> package/berkeleydb/berkeleydb.mk | 2 +
> package/busybox/busybox.mk | 2 +
> package/bzip2/bzip2.mk | 2 +
> package/cups/cups.mk | 4 ++
> package/directfb/directfb.mk | 2 +
> package/fis/fis.mk | 4 ++
> package/foobar/Config.in | 5 ++
> package/foobar/foobar.mk | 15 +++++++
> package/foobar/source/foobar.c | 7 +++
> package/games/doom-wad/doom-wads.mk | 4 ++
> package/gettext/gettext.mk | 4 ++
> package/iostat/iostat.mk | 2 +
> package/libusb/libusb.mk | 2 +
> package/lzo/lzo.mk | 2 +
> package/lzop/lzop.mk | 2 +
> package/m4/m4.mk | 2 +
> package/microperl/microperl.mk | 4 ++
> package/mpc/mpc.mk | 2 +
> package/netkitbase/netkitbase.mk | 4 ++
> package/netkittelnet/netkittelnet.mk | 4 ++
> package/netsnmp/netsnmp.mk | 2 +
> package/newt/newt.mk | 4 ++
> package/pcre/pcre.mk | 2 +
> package/qt/Config.in | 15 +++++++
> package/qt/qt.mk | 9 ++++
> package/tinyhttpd/tinyhttpd.mk | 4 ++
> package/tslib/tslib.mk | 2 +
> package/ttcp/ttcp.mk | 4 ++
> package/uemacs/uemacs.mk | 4 ++
> package/vpnc/vpnc.mk | 4 ++
> package/xfsprogs/xfsprogs.mk | 4 ++
> support/legal-info/README.header | 24 ++++++++++++
> support/legal-info/README.warnings-header | 4 ++
> 39 files changed, 296 insertions(+), 3 deletions(-)
> create mode 100644 configs/legal_info_test2_defconfig
> create mode 100644 configs/legal_info_test_defconfig
> create mode 100644 package/foobar/Config.in
> create mode 100644 package/foobar/foobar.mk
> create mode 100644 package/foobar/source/foobar.c
> create mode 100644 support/legal-info/README.header
> create mode 100644 support/legal-info/README.warnings-header
>
> --
> 1.7.5.4
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
More information about the buildroot
mailing list