The Python build is implemented with GN templates defined in
pw_build/python.gni
. See the .gni file for complete usage documentation.
.. seealso:: :ref:`docs-python-build`
The main Python template is pw_python_package
. Each pw_python_package
target represents a Python package. As described in
:ref:`module-pw_build-python-target`, each pw_python_package
expands to
several subtargets. In summary, these are:
<name>
- Represents the files themselves<name>.lint
- Runs static analysis<name>.tests
- Runs all tests for this package<name>.install
- Installs the package<name>.wheel
- Builds a Python wheel
GN permits using abbreviated labels when the target name matches the directory
name (e.g. //foo
for //foo:foo
). For consistency with this, Python
package subtargets are aliased to the directory when the target name is the
same as the directory. For example, these two labels are equivalent:
//path/to/my_python_package:my_python_package.tests //path/to/my_python_package:tests
The actions in a pw_python_package
(e.g. installing packages and running
Pylint) are done within a single GN toolchain to avoid duplication in
multi-toolchain builds. This toolchain can be set with the
pw_build_PYTHON_TOOLCHAIN
GN arg, which defaults to
$dir_pw_build/python_toolchain:python
.
setup
- List of setup file paths (setup.py or pyproject.toml & setup.cfg), which must all be in the same directory.generate_setup
: As an alternative tosetup
, generate setup files with the keywords in this scope.name
is required. This follows the same structure as asetup.cfg
file's declarative config For example:generate_setup = { metadata = { name = "a_nifty_package" version = "1.2a" } options = { install_requires = [ "a_pip_package" ] } }
sources
- Python sources files in the package.tests
- Test files for this Python package.python_deps
- Dependencies on other pw_python_packages in the GN build.python_test_deps
- Test-only pw_python_package dependencies.other_deps
- Dependencies on GN targets that are not pw_python_packages.inputs
- Other files to track, such as package_data.proto_library
- A pw_proto_library target to embed in this Python package.generate_setup
is required in place of setup if proto_library is used. See :ref:`module-pw_protobuf_compiler-add-to-python-package`.static_analysis
List of static analysis tools to run;"*"
(default) runs all tools. The supported tools are"mypy"
and"pylint"
.pylintrc
- Optional path to a pylintrc configuration file to use. If not provided, Pylint's default rcfile search is used. Pylint is executed from the package's setup directory, so pylintrc files in that directory will take precedence over others.mypy_ini
- Optional path to a mypy configuration file to use. If not provided, mypy's default configuration file search is used. mypy is executed from the package's setup directory, so mypy.ini files in that directory will take precedence over others.
This is an example Python package declaration for a pw_my_module
module.
import("//build_overrides/pigweed.gni") import("$dir_pw_build/python.gni") pw_python_package("py") { setup = [ "pyproject.toml", "setup.cfg", "setup.py", ] sources = [ "pw_my_module/__init__.py", "pw_my_module/alfa.py", "pw_my_module/bravo.py", "pw_my_module/charlie.py", ] tests = [ "alfa_test.py", "charlie_test.py", ] python_deps = [ "$dir_pw_status/py", ":some_protos.python", ] python_test_deps = [ "$dir_pw_build/py" ] pylintrc = "$dir_pigweed/.pylintrc" }
A pw_python_script
represents a set of standalone Python scripts and/or
tests. These files support all of the arguments of pw_python_package
except
those setup
. These targets can be installed, but this only installs their
dependencies.
pw_python_script
allows creating a
:ref:`pw_python_action <module-pw_build-python-action>` associated with the
script. To create an action, pass an action
scope to pw_python_script
.
If there is only a single source file, it serves as the action's script
by
default.
An action in pw_python_script
can always be replaced with a standalone
pw_python_action
, but using the embedded action has some advantages:
- The embedded action target bridges the gap between actions and Python targets. A Python script can be expressed in a single, concise GN target, rather than in two overlapping, dependent targets.
- The action automatically depends on the
pw_python_script
. This ensures that the script's dependencies are installed and the action automatically reruns when the script's sources change, without needing to specify a dependency, a step which is easy to forget. - Using a
pw_python_script
with an embedded action is a simple way to check an existing action's script with Pylint or Mypy or to add tests.
Represents a group of pw_python_package
and pw_python_script
targets.
These targets do not add any files. Their subtargets simply forward to those of
their dependencies.
Represents a set of local and PyPI requirements, with no associated source
files. These targets serve the role of a requirements.txt
file.
When packages are installed by Pigweed, additional version constraints can be
provided using the pw_build_PIP_CONSTRAINTS
GN arg. This option should
contain a list of paths to pass to the --constraint
option of pip
install
. This can be used to synchronize dependency upgrades across a project
which facilitates reproducibility of builds.
Note using multiple pw_python_requirements
that install different versions
of the same package will currently cause unpredictable results, while using
constraints should have correct results (which may be an error indicating a
conflict).
Pigweed also provides some templates to make it easier to bundle Python packages
for deployment. These templates are found in pw_build/python_dist.gni
. See
the .gni file for complete documentation on building distributables.
Collects Python wheels for one or more pw_python_package
targets, plus any
additional pw_python_package
targets they depend on, directly or indirectly.
Note that this does not include Python dependencies that come from outside the
GN build, like packages from PyPI, for example. Those should still be declared
in the package's setup.py
file as usual.
packages
- List ofpw_python_package
targets whose wheels should be included; their dependencies will be pulled in as wheels also.directory
- Output directory for the collected wheels. Defaults to$target_out_dir/$target_name
.
The .wheel
subtarget of every pw_python_package
generates a wheel
(.whl
) for the Python package. The pw_python_wheels
template figures
out which wheels to collect by traversing the pw_python_package_wheels
GN metadata key,
which lists the output directory for each wheel.
Generates a .zip
archive suitable for deployment outside of the project's
developer environment. The generated .zip
contains Python wheels
(.whl
files) for one or more pw_python_package
targets, plus wheels for
any additional pw_python_package
targets in the GN build they depend on,
directly or indirectly. Dependencies from outside the GN build, such as packages
from PyPI, must be listed in packages' setup.py
or setup.cfg
files as
usual.
The .zip
also includes simple setup scripts for Linux,
MacOS, and Windows. The setup scripts automatically create a Python virtual
environment and install the whole collection of wheels into it using pip
.
Optionally, additional files and directories can be included in the archive.
One common example of an additional file to include is a README file with setup
and usage instructions for the distributable. A simple ready-to-use README file
is available at pw_build/py_dist/README.md
.
packages
- A list of pw_python_package targets whose wheels should be included; their dependencies will be pulled in as wheels also.inputs
- An optional list of extra files to include in the generated.zip
, formatted the same way as theinputs
argument topw_zip
targets.dirs
- An optional list of directories to include in the generated.zip
, formatted the same was as thedirs
argument topw_zip
targets.
import("//build_overrides/pigweed.gni") import("$dir_pw_build/python_dist.gni") pw_python_zip_with_setup("my_tools") { packages = [ ":some_python_package" ] inputs = [ "$dir_pw_build/python_dist/README.md > /${target_name}/" ] }
Generates a directory of Python packages from source files suitable for
deployment outside of the project developer environment. The resulting directory
contains only files mentioned in each package's setup.cfg
file. This is
useful for bundling multiple Python packages up into a single package for
distribution to other locations like http://pypi.org.
packages
- A list of :ref:`module-pw_build-pw_python_package` targets to be installed into the build directory. Their dependencies will be pulled in as wheels also.include_tests
- If true, copy Python package tests to atests
subdir.extra_files
- A list of extra files that should be included in the output. The format of each item in this list follows this convention://some/nested/source_file > nested/destination_file
- Source and destination file should be separated by
>
. - The source file should be a GN target label (starting with
//
). - The destination file will be relative to the generated output directory. Parent directories are automatically created for each file. If a file would be overwritten an error is raised.
- Source and destination file should be separated by
generate_setup_cfg
- If included, create a mergedsetup.cfg
for all python Packages using acommon_config_file
as a base. That file should contain the required fields in themetadata
andoptions
sections as shown in Configuring setup() using setup.cfg files.append_git_sha_to_version
andappend_date_to_version
will optionally append the current git SHA or date to the package version string after a+
sign.generate_setup_cfg = { common_config_file = "pypi_common_setup.cfg" append_git_sha_to_version = true append_date_to_version = true }
Using this template will create an additional target for and building a Python
wheel. For example if you define pw_create_python_source_tree("awesome")
the
resulting targets that get created will be:
awesome
- This will create the merged package with all source files in place in the out directory underout/obj/awesome/
.awesome.wheel
- This builds a Python wheel from the above source files underout/obj/awesome._build_wheel/awesome*.whl
.
:octicon:`file;1em` ./pw_env_setup/BUILD.gn
import("//build_overrides/pigweed.gni") import("$dir_pw_build/python_dist.gni") pw_create_python_source_tree("build_python_source_tree") { packages = [ ":some_python_package", ":another_python_package", ] include_tests = true extra_files = [ "//README.md > ./README.md", "//some_python_package/py/BUILD.bazel > some_python_package/BUILD.bazel", "//another_python_package/py/BUILD.bazel > another_python_package/BUILD.bazel", ] generate_setup_cfg = { common_config_file = "pypi_common_setup.cfg" append_git_sha_to_version = true append_date_to_version = true } }
:octicon:`file-directory;1em` ./out/obj/pw_env_setup/build_python_source_tree/
$ tree ./out/obj/pw_env_setup/build_python_source_tree/
├── README.md
├── setup.cfg
├── some_python_package
│ ├── BUILD.bazel
│ ├── __init__.py
│ ├── py.typed
│ ├── some_source_file.py
│ └── tests
│ └── some_source_test.py
└── another_python_package
├── BUILD.bazel
├── __init__.py
├── another_source_file.py
├── py.typed
└── tests
└── another_source_test.py