[Buildroot] [PATCH 03/38] package: introduce Python package infrastructure

Thomas De Schampheleire patrickdepinguin at gmail.com
Mon Dec 9 10:02:25 UTC 2013


Hi Thomas,

On Sun, Dec 8, 2013 at 11:14 PM, Thomas Petazzoni
<thomas.petazzoni at free-electrons.com> wrote:
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni at free-electrons.com>
> ---
>  docs/manual/adding-packages-python.txt | 136 ++++++++++++++++++++
>  docs/manual/adding-packages.txt        |   2 +
>  package/Makefile.in                    |   1 +
>  package/pkg-python.mk                  | 219 +++++++++++++++++++++++++++++++++
>  4 files changed, 358 insertions(+)
>  create mode 100644 docs/manual/adding-packages-python.txt
>  create mode 100644 package/pkg-python.mk
>
> diff --git a/docs/manual/adding-packages-python.txt b/docs/manual/adding-packages-python.txt
> new file mode 100644
> index 0000000..9b5876f
> --- /dev/null
> +++ b/docs/manual/adding-packages-python.txt
> @@ -0,0 +1,136 @@
> +// -*- mode:doc; -*-
> +// vim: set syntax=asciidoc:
> +
> +Infrastructure for Python packages
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +This infrastructure applies to Python packages that use the standard
> +Python setuptools mechanism as their build system, generally
> +recognizable as the usage of a +setup.py+ script.

I would rather say: 'recognizable by'

> +
> +[[python-package-tutorial]]
> +
> ++python-package+ tutorial
> +^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +First, let's see how to write a +.mk+ file for a Python package,
> +with an example :
> +
> +------------------------
> +01: ################################################################################
> +02: #
> +03: # python-foo
> +04: #
> +05: ################################################################################
> +06:
> +07: PYTHON_FOO_VERSION = 1.0
> +08: PYTHON_FOO_SOURCE = python-foo-$(LIBFOO_VERSION).tar.xz
> +09: PYTHON_FOO_SITE = http://www.foosoftware.org/download
> +10: PYTHON_FOO_LICENSE = BSD-3c
> +11: PYTHON_FOO_LICENSE_FILES = LICENSE
> +12: PYTHON_FOO_ENV = SOME_VAR=1
> +13: PYTHON_FOO_DEPENDENCIES = libmad
> +14: PYTHON_FOO_SETUP_TYPE = distutils
> +15:
> +16: $(eval $(python-package))
> +------------------------
> +
> +On line 7, we declare the version of the package.
> +
> +On line 8 and 9, we declare the name of the tarball (xz-ed tarball
> +recommended) and the location of the tarball on the Web. Buildroot
> +will automatically download the tarball from this location.

Should we clarify here that if the tarball is .gz, it shouldn't be
specified because it's currently the default?

> +
> +On line 10 and 11, we give licensing details about the package (its
> +license on line 10, and the file containing the license text on line
> +11).
> +
> +On line 12, we tell Buildroot to pass custom options to the Python
> ++setup.py+ script when it is configuring the package.
> +
> +On line 13, we declare our dependencies, so that they are built
> +before the build process of our package starts.
> +
> +On line 14, we declare the specific Python build system being used. In
> +this case the +distutils+ Python build system is used. The two
> +supported ones are +distutils+ and +setuptools+.
> +
> +Finally, on line line 16, we invoke the +python-package+ macro that

line line

> +generates all the Makefile rules that actually allows the package to

allow

> +be built.
> +
> +[[python-package-reference]]
> +
> ++python-package+ reference
> +^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The main macro of the Python package infrastructure is
> ++python-package+. It is similar to the +generic-package+ macro. The
> +ability to have Python host packages is also available, with the
> ++host-python-package+ macro.
> +
> +Just like the generic infrastructure, the Python infrastructure works
> +by defining a number of variables before calling the +python-package+
> +or +host-python-package+ macros.
> +
> +First, all the package metadata information variables that exist in

The word First sounds odd to me here. I would rather use 'To start
with,' or 'First of all,'.

> +the generic infrastructure also exist in the Python infrastructure:
> ++PYTHON_FOO_VERSION+, +PYTHON_FOO_SOURCE+, +PYTHON_FOO_PATCH+,
> ++PYTHON_FOO_SITE+, +PYTHON_FOO_SUBDIR+, +PYTHON_FOO_DEPENDENCIES+,
> ++PYTHON_FOO_LICENSE+, +PYTHON_FOO_LICENSE_FILES+, etc.
> +
> +Note however that setting +PYTHON_FOO_INSTALL_STAGING+ to +YES+ has no
> +effect (unless a +PYTHON_FOO_INSTALL_STAGING_CMDS+ variable is
> +defined), since Python modules generally don't need to be installed to
> +the staging directory.
> +
> +For Python packages, one specific variable is mandatory:

Is it needed to say 'For Python packages' here?

> +
> +* +PYTHON_FOO_BUILD_TYPE+, to define which Python build system is used
> +  by the package. The two supported values are +distutils+ and
> +  +setuptools+. If you don't know which one is used in your package,
> +  look at the +setup.py+ file in your package source code, and see
> +  whether it imports things from the +distutils+ module or the
> +  +setuptools+ module.
> +
> +A few additional variables, specific to the Python infrastructure, can
> +also optionaly be defined, depending on the needs. Many of them are

* I would remove 'also' here.
* optionally (two l's)
* The 'depending on the needs' sounds odd to me, suggestions:
"depending on the needs of the package" or "depending on the package's
needs".

> +only useful in very specific cases, typical packages will therefore
> +only use a few of them.
> +
> +* +PYTHON_FOO_ENV+, to specify additional environment variables to
> +  pass to the Python +setup.py+ script (for both the build and install
> +  steps). Note that the infrastructure is automatically passing
> +  several standard variables: defined in +PKG_PYTHON_DISTUTILS_ENV+

Is the colon needed here? After a colon I would have expected a list
of variables that are automatically passed; however what follows is
rather in which containers these variables are put. So I'd replace the
colon with a comma, as you did for INSTALL_OPT.

> +  (for distutils target packages), +HOST_PKG_PYTHON_DISTUTILS_ENV+
> +  (for distutils host packages), +PKG_PYTHON_SETUPTOOLS_ENV+ (for
> +  setuptools target packages) and +HOST_PKG_PYTHON_SETUPTOOLS_ENV+
> +  (for setuptools host packages).
> +
> +* +PYTHON_FOO_BUILD_OPT+, to specify additional options to pass to the
> +  Python +setup.py+ script during the build step. For target distutils
> +  packages, the +PKG_PYTHON_DISTUTILS_BUILD_OPT+ options are already
> +  passed automatically by the infrastructure.
> +
> +* +PYTHON_FOO_INSTALL_OPT+, to specify additional options to pass to
> +  the Python +setup.py+ script during the installation step. Note that
> +  the infrastructure is automatically passing some options, defined in
> +  +PKG_PYTHON_DISTUTILS_INSTALL_OPT+ (for target distutils packages),
> +  +HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPT+ (for host distutils
> +  packages), +PKG_PYTHON_SETUPTOOLS_INSTALL_OPT+ (for target
> +  setuptools packages) and +HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPT+
> +  (for host setuptools packages).
> +
> +With the Python infrastructure, all the steps required to build and
> +install the packages are already defined, and they generally work well
> +for most Python-based packages. However, when required, it is still
> +possible to customize what is done in any particular step:
> +
> +* By adding a post-operation hook (after extract, patch, configure,
> +  build or install). See xref:hooks[] for details.
> +
> +* By overriding one of the steps. For example, even if the Python
> +  infrastructure is used, if the package +.mk+ file defines its own
> +  +PYTHON_FOO_BUILD_CMDS+ variable, it will be used instead of the
> +  default Python one. However, using this method should be restricted
> +  to very specific cases. Do not use it in the general case.
> diff --git a/docs/manual/adding-packages.txt b/docs/manual/adding-packages.txt
> index ae76e74..01277d8 100644
> --- a/docs/manual/adding-packages.txt
> +++ b/docs/manual/adding-packages.txt
> @@ -18,6 +18,8 @@ include::adding-packages-autotools.txt[]
>
>  include::adding-packages-cmake.txt[]
>
> +include::adding-packages-python.txt[]
> +
>  include::adding-packages-hooks.txt[]
>
>  include::adding-packages-gettext.txt[]


What I have not yet seen in the documentation is a naming policy for
python packages. I understood that we expect them all to be named
python-foo, right, so maybe this could be added in the new python
section of the manual?


> diff --git a/package/Makefile.in b/package/Makefile.in
> index 7bc0606..f5d6289 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -371,4 +371,5 @@ include package/pkg-utils.mk
>  include package/pkg-download.mk
>  include package/pkg-autotools.mk
>  include package/pkg-cmake.mk
> +include package/pkg-python.mk
>  include package/pkg-generic.mk
> diff --git a/package/pkg-python.mk b/package/pkg-python.mk
> new file mode 100644
> index 0000000..4556843
> --- /dev/null
> +++ b/package/pkg-python.mk
> @@ -0,0 +1,219 @@
> +################################################################################
> +# Python package infrastructure
> +#
> +# This file implements an infrastructure that eases development of
> +# package .mk files for Python packages. It should be used for all
> +# packages that use Python setup.py/setuptools as their build system.
> +#
> +# See the Buildroot documentation for details on the usage of this
> +# infrastructure
> +#
> +# In terms of implementation, this Python infrastructure requires the
> +# .mk file to only specify metadata informations about the package:
> +# name, version, download URL, etc.
> +#
> +# We still allow the package .mk file to override what the different
> +# steps are doing, if needed. For example, if <PKG>_BUILD_CMDS is
> +# already defined, it is used as the list of commands to perform to
> +# build the package, instead of the default Python behaviour. The
> +# package can also define some post operation hooks.
> +#
> +################################################################################
> +
> +# Passed in the environment of build and install steps of target
> +# distutils based packages.
> +PKG_PYTHON_DISTUTILS_ENV = \
> +       PATH="$(TARGET_PATH)" \
> +       CC="$(TARGET_CC)" \
> +       CFLAGS="$(TARGET_CFLAGS)" \
> +       LDFLAGS="$(TARGET_LDFLAGS)" \
> +       LDSHARED="$(TARGET_CROSS)gcc -shared" \
> +       CROSS_COMPILING=yes \
> +       _python_sysroot=$(STAGING_DIR) \
> +       _python_srcdir=$(PYTHON_DIR) \
> +       _python_prefix=/usr \
> +       _python_exec_prefix=/usr
> +
> +# Passed as options of the build step of target distutils based
> +# packages.

This is probably personal, but I don't feel this comment explains more
than the variable name already does.
Moreover, to repeat the text 'of target distutils based packages' for
each of the ENV, BUILD_OPT, INSTALL_OPT variables seems redundant too.
What about a structure like:

# target distutils-based packages
PKG_PYTHON_DISTUTILS_ENV = ...
PKG_PYTHON_DISTUTILS_BUILD_OPT = ...
PKG_PYTHON_DISTUTIRS_INSTALL_OPT = ...

# host distutils-based packages
....

# target setuptools-based packages
...

# host setuptools-based packages
...



> +PKG_PYTHON_DISTUTILS_BUILD_OPT = \
> +       --executable=/usr/bin/python
> +
> +# Passed as options of the install step of target distutils based
> +# packages.
> +PKG_PYTHON_DISTUTILS_INSTALL_OPT = \
> +       --prefix=$(TARGET_DIR)/usr
> +
> +# Passed in the environment of build and install steps of host
> +# distutils based packages.
> +HOST_PKG_PYTHON_DISTUTILS_ENV = \
> +       PATH="$(HOST_PATH)"
> +
> +# Passed as options of the install step of host distutils based
> +# packages.
> +HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPT = \
> +       --prefix=$(HOST_DIR)/usr
> +
> +# Passed in the environment of the build and install steps of
> +# setuptools based packages.
> +PKG_PYTHON_SETUPTOOLS_ENV = \
> +       PATH="$(TARGET_PATH)" \
> +       PYTHONPATH="$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages" \
> +       PYTHONXCPREFIX="$(STAGING_DIR)/usr/"
> +
> +# Passed in the environment of the build and install steps of
> +# setuptools based packages built for the host.
> +HOST_PKG_PYTHON_SETUPTOOLS_ENV = \
> +       PATH="$(HOST_PATH)" \
> +       PYTHONXCPREFIX="$(HOST_DIR)/usr/"
> +
> +# Passed as options of the install of setuptools based packages.
> +PKG_PYTHON_SETUPTOOLS_INSTALL_OPT = \
> +       --prefix=$(TARGET_DIR)/usr \
> +       --executable=/usr/bin/python \
> +       --single-version-externally-managed \
> +       --root=/
> +
> +# Passed as options of the install of setuptools based packages built
> +# for the host.
> +HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPT = \
> +       --prefix=$(HOST_DIR)/usr
> +
> +################################################################################
> +# inner-python-package -- defines how the configuration, compilation
> +# and installation of a Python package should be done, implements a
> +# few hooks to tune the build process and calls the generic package
> +# infrastructure to generate the necessary make targets
> +#
> +#  argument 1 is the lowercase package name
> +#  argument 2 is the uppercase package name, including an HOST_ prefix
> +#             for host packages
> +#  argument 3 is the uppercase package name, without the HOST_ prefix
> +#             for host packages
> +#  argument 4 is the package directory prefix
> +#  argument 5 is the type (target or host)
> +################################################################################
> +
> +define inner-python-package
> +
> +$(2)_SRCDIR    = $$($(2)_DIR)/$($(2)_SUBDIR)
> +$(2)_BUILDDIR  = $$($(2)_SRCDIR)
> +
> +$(2)_ENV         ?=
> +$(2)_BUILD_OPT   ?=
> +$(2)_INSTALL_OPT ?=
> +
> +ifndef $(2)_SETUP_TYPE
> + ifdef $(3)_SETUP_TYPE
> +  $(2)_SETUP_TYPE = $($(3)_SETUP_TYPE)
> + else
> +  $$(error "$(1): Unknown or undefined <pkg>_SETUP_TYPE")
> + endif
> +endif
> +
> +# Distutils
> +ifeq ($$($(2)_SETUP_TYPE),distutils)
> +ifeq ($(5),target)
> +$(2)_BASE_ENV         = $$(PKG_PYTHON_DISTUTILS_ENV)
> +$(2)_BASE_BUILD_TGT   = build
> +$(2)_BASE_BUILD_OPT   = $$(PKG_PYTHON_DISTUTILS_BUILD_OPT)
> +$(2)_BASE_INSTALL_OPT = $$(PKG_PYTHON_DISTUTILS_INSTALL_OPT)
> +else
> +$(2)_BASE_ENV         = $$(HOST_PKG_PYTHON_DISTUTILS_ENV)
> +$(2)_BASE_BUILD_TGT   = build
> +$(2)_BASE_BUILD_OPT   =
> +$(2)_BASE_INSTALL_OPT = $$(HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPT)
> +endif
> +# Setuptools
> +else ifeq ($$($(2)_SETUP_TYPE),setuptools)
> +ifeq ($(5),target)
> +$(2)_BASE_ENV         = $$(PKG_PYTHON_SETUPTOOLS_ENV)
> +$(2)_BASE_BUILD_TGT   = build -x
> +$(2)_BASE_BUILD_OPT   =
> +$(2)_BASE_INSTALL_OPT = $$(PKG_PYTHON_SETUPTOOLS_INSTALL_OPT)
> +else
> +$(2)_BASE_ENV         = $$(HOST_PKG_PYTHON_SETUPTOOLS_ENV)
> +$(2)_BASE_BUILD_TGT   = build
> +$(2)_BASE_BUILD_OPT   =
> +$(2)_BASE_INSTALL_OPT = $$(HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPT)
> +endif
> +endif
> +
> +# This must be repeated from inner-generic-package, and we need to
> +# exclude the packages added above in various situations, otherwise

Do you mean 'as we need to' ?

added below

and what do you mean with 'in various situations' here?

> +# they get automatically added in the dependencies of the host package
> +# when present in the dependency of the target package, which we do
> +# not necessarily want, especially for host-python-distutilscross.
> +$(2)_DEPENDENCIES ?= $(filter-out host-python host-python-setuptools host-python-distutilscross $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
> +
> +# Target packages need both the python interpreter on the target (for
> +# runtime) and the python interpreter on the host (for
> +# compilation). However, host packages only need the python
> +# interpreter on the host.
> +ifeq ($(5),target)
> +$(2)_DEPENDENCIES += host-python python
> +else
> +$(2)_DEPENDENCIES += host-python
> +endif
> +
> +ifeq ($$($(2)_SETUP_TYPE),setuptools)
> +ifneq ($(2),HOST_PYTHON_SETUPTOOLS)
> +$(2)_DEPENDENCIES += host-python-setuptools
> +ifeq ($(5),target)
> +$(2)_DEPENDENCIES += host-python-distutilscross
> +endif
> +endif
> +endif
> +
> +#
> +# Build step. Only define it if not already defined by the package .mk
> +# file.
> +#
> +ifndef $(2)_BUILD_CMDS
> +define $(2)_BUILD_CMDS
> +       (cd $$($$(PKG)_BUILDDIR)/; \
> +               $$($$(PKG)_BASE_ENV) $$($$(PKG)_ENV) \
> +               $(HOST_DIR)/usr/bin/python setup.py \
> +               $$($$(PKG)_BASE_BUILD_TGT) \
> +               $$($$(PKG)_BASE_BUILD_OPT) $$($$(PKG)_BUILD_OPT))
> +endef
> +endif
> +
> +#
> +# Host installation step. Only define it if not already defined by the
> +# package .mk file.
> +#
> +ifndef $(2)_INSTALL_CMDS
> +define $(2)_INSTALL_CMDS
> +       (cd $$($$(PKG)_BUILDDIR)/; \
> +               $$($$(PKG)_BASE_ENV) $$($$(PKG)_ENV) \
> +               $(HOST_DIR)/usr/bin/python setup.py install \
> +               $$($$(PKG)_BASE_INSTALL_OPT) $$($$(PKG)_INSTALL_OPT))
> +endef
> +endif
> +
> +#
> +# Target installation step. Only define it if not already defined by
> +# the package .mk file.
> +#
> +ifndef $(2)_INSTALL_TARGET_CMDS
> +define $(2)_INSTALL_TARGET_CMDS
> +       (cd $$($$(PKG)_BUILDDIR)/; \
> +               $$($$(PKG)_BASE_ENV) $$($$(PKG)_ENV) \
> +               $(HOST_DIR)/usr/bin/python setup.py install \
> +               $$($$(PKG)_BASE_INSTALL_OPT) $$($$(PKG)_INSTALL_OPT))
> +endef
> +endif
> +
> +# Call the generic package infrastructure to generate the necessary
> +# make targets
> +$(call inner-generic-package,$(1),$(2),$(3),$(4),$(5))
> +
> +endef
> +
> +################################################################################
> +# python-package -- the target generator macro for Python packages
> +################################################################################
> +
> +python-package = $(call inner-python-package,$(call pkgname),$(call UPPERCASE,$(call pkgname)),$(call UPPERCASE,$(call pkgname)),$(call pkgparentdir),target)
> +host-python-package = $(call inner-python-package,host-$(call pkgname),$(call UPPERCASE,host-$(call pkgname)),$(call UPPERCASE,$(call pkgname)),$(call pkgparentdir),host)
> --

Best regards,
Thomas



More information about the buildroot mailing list