1 # Python packages folder
3 :warning: **Python 2 will soon be unsupported and removed from the feed - [see below](#python-2-end-of-life)** :warning:
7 1. [Description](#description)
8 2. [Python 2 end-of-life](#python-2-end-of-life)
9 1. [Transition policy / schedule](#transition-policy--schedule)
10 3. [Introduction](#introduction)
11 4. [Build considerations](#build-considerations)
12 5. [General folder structure](#general-folder-structure)
13 6. [Building a Python[3] package](#building-a-python3-package)
14 1. [PKG_BUILD_DIR](#pkg_build_dir)
15 2. [PKG_UNPACK](#pkg_unpack)
16 3. [Include python[3]-package.mk](#include-python3-packagemk)
17 4. [Add Package/<PKG_NAME> OpenWrt definitions](#add-packagepkg_name-openwrt-definitions)
18 5. [Wrapping things up so that they build](#wrapping-things-up-so-that-they-build)
19 6. [Customizing things](#customizing-things)
20 7. [Host-side Python packages for build](#host-side-python-packages-for-build)
24 This section describes specifics for the Python packages that are present in this repo, and how things are structured.
26 In terms of license, contributing guide, etc, all of that information is described in the top [README.md](README.md) file, and it applies here as well. This document attempts to cover only technical aspects of Python/Python3 packages, and maybe some explanations about how things are (and why they are as they are).
28 ## Python 2 end-of-life
30 Python 2 will not be maintained past [1 January 2020](https://pythonclock.org/). As such, we will be transitioning Python 2 programs and libraries to Python 3, and Python 2 packages will be removed in early 2020.
32 (Discussion for how to handle this transition can be found in [#8520](https://github.com/openwrt/packages/issues/8520).)
34 ### Transition policy / schedule
36 A mass removal event ("The Snap") will occur on 31 March 2020, or 2 weeks before the freeze for a 20.x release, whichever is sooner. The exact date will be confirmed when the 20.x release schedule is known, or by 15 March 2020.
38 All Python 2 packages (the Python 2 interpreter, programs that depend on Python 2, and Python 2-only libraries) will be removed during this event.
40 Leading up to "The Snap":
42 * In general, new Python 2 packages are no longer accepted
43 * Exceptions can be made on a case-by-case basis, given extraordinary circumstances or reasons, until 31 May 2019
44 * From 31 May 2019 onward, absolutely no new Python 2 packages will be accepted
46 * The Python 2 interpreter will remain in the feed until "The Snap"
47 * The interpreter will continue to be updated, including the last release in January 2020 (if there is one)
49 * Programs that depend on Python 2 will be transitioned to Python 3 (see [#8893](https://github.com/openwrt/packages/issues/8893))
50 * If a program cannot be transitioned, a suitable replacement will be found
51 * If a replacement cannot be found, the program will be removed during "The Snap"
53 * Python 2 libraries will remain in the feed until "The Snap"
54 * A Python 2-only library will be transitioned to Python 3 (or a suitable replacement found), if its Python 3 version is a dependency of another package in the feed
55 * Python 2 libraries will receive normal updates until 31 October 2019
56 * From 31 October 2019 onward:
57 * Python 2-only libraries will receive security updates only
58 * Python 2 libraries that share the same Makefile as their Python 3 version will continue to receive normal updates
62 This sub-tree came to exist after a number of contributions (Python packages) were made to this repo, and the [lang](lang) subtree grew to a point where a decision was made to move all Python packages under [lang/python](lang/python).
64 It contains the 2 Python interpreters (Python & Python3) and Python packages. Most of the Python packages are downloaded from [pypi.org](https://pypi.org/). Python packages from [pypi.org](https://pypi.org/) are typically preferred when adding new packages.
66 If more packages (than the ones packaged here) are needed, they can be downloaded via [pip or pip3](https://pip.pypa.io). Note that the versions of `pip` & `setuptools` [available in this repo] are the ones that are packaged inside the Python & Python3 packages (yes, Python & Python3 come packaged with `pip` & `setuptools`).
68 ## Build considerations
70 In order to build the Python[3] interpreters, a host Python/Python3 interpreter needs to be built, in order to process some of the build for the target Python/Python3 build. The host Python[3] interpreters are also needed so that Python bytecodes are generated, so the host interpreters need to be the exact versions as on the target. And finally, the host Python[3] interpreters also provide pip & pip3, so that they may be used to install some Python[3] packages that are required to build other Python[3] packages.
71 That's why you'll also see a Python/Python3 build & staging directories.
73 As you're probably thinking, this sounds [and is] somewhat too much complication [just for packaging], but the status of things is-as-it-is, and it's probably much worse than what's currently visible on the surface [with respect to packaging Python[3] & packages].
75 As mentioned earlier, Python[3] packages are shipped with bytecodes, and the reason for this is simply performance & size.
76 The thought/discussion matrix derives a bit like this:
77 * shipping both Python source-code & bytecodes takes too much space on some devices ; Python source code & byte-code take about similar disk-size
78 * shipping only Python source code has a big performance penalty [on some lower end systems] ; something like 500 msecs (Python source-only) -> 70 msecs (Python byte-codes) time reduction for a simple "Hello World" script
79 * shipping only Python byte-codes seems like a good trade-off, and this means that `python-src` & `python3-src` can be provided for people that want the source code
81 By default, automatic Python[3] byte-code generation is disabled when running a Python script, in order to prevent a disk from accidentally filling up. Since some disks reside in RAM, this also means not filling up the RAM. If someone wants to convert Python source to byte-code then he/she is free to compile it [directly on the device] manually via the Python interpreter & library.
83 ## General folder structure
85 The basis of all these packages are:
86 * [lang/python/python](lang/python/python) - The Python 2.7.y interpreter (supposedly, there won't ever by a 2.8.y)
87 * [lang/python/python3](lang/python/python3) - The Python 3.x.y interpreter
89 These 2 are normal OpenWrt packages, which will build the Python interpreters. They also provide `python[3]-pip` & `python[3]-setuptools`. Each Python or Python3 package is actually split into multiple sub-packages [e.g. python-email, python-sqlite3, etc]. This can be viewed inside [lang/python/python/files](lang/python/python/files) & [lang/python/python3/files](lang/python/python3/files).
91 The reason for this splitting, is purely to offer a way for some people to package Python/Python3 in as-minimal-as-possible-and-still-runable way, and also to be somewhat maintainable when packaging. A standard Python[3] installation can take ~20-30 MBs of disk, which can be somewhat big for some people, so there are the `python[3]-base` packages which bring that down to ~5 MBs. This seems to be good enough (and interesting) for a number of people.
93 The Python[3] interpreters are structured like this:
94 * `python-base` (and `python3-base`), which is just the minimal package to startup Python[3] and run basic commands
95 * `python` (and `python3`) are meta-packages, which install almost everything (python[3]-base [plus] Python[3] library [minus] some unit-tests & some windows-y things)
96 * `python-light` (and `python3-light`) are `python` (and `python3`) [minus] packages that are in [lang/python/python/files](lang/python/python/files) or [lang/python/python3/files](lang/python/python3/files) ; the size of these 2 packages may be sensible (and interesting) to another group of people
98 All other Python & Python3 packages (aside from the 2 intepreters) typically use these files:
99 * **python[3]-host.mk** - this file contains paths and build rules for running the Python[3] interpreters on the host-side; they also provide paths to host interprete, host Python lib-dir & so on
100 * **python[3]-package.mk**
101 * includes **python[3]-host.mk**
102 * contains all the default build rules for Python[3] packages; these will be detailed below in the [Building a Python[3] package](#Building a Python[3] package) section
104 **Note** that Python/Python3 packages don't need to use these files (i.e. `python[3]-package.mk` & `python[3]-host.mk`), but they do provide some ease-of-use & reduction of duplicate code, especially when packaging for both Python & Python3. And they do contain some learned-lessons about packaging Python/Python3 packages, so it's a good idea to use them.
106 ## Building a Python[3] package
108 A Python package can be packaged for either Python or Python3 or both.
110 This section will describe both, and then it can be inferred which is for which.
112 Packaging for both Python & Python3 uses the `VARIANT` mechanism for packaging inside OpenWrt. (#### FIXME: find a link for this later if it exists)
116 It's important when packaging for both Python & Python3 to override this variable, so that the build directory differs for each variant.
118 Typically it's just something like:
120 PKG_BUILD_DIR:=$(BUILD_DIR)/$(BUILD_VARIANT)-pyasn1-$(PKG_VERSION)
122 Where `pyasn1` should be some other name, or maybe `PKG_NAME`
124 This should be added before this include:
126 include $(INCLUDE_DIR)/package.mk
131 In many cases, this needs to be overriden. This is usually because the way Python packages are archived, don't follow the convention of other `tar.gz` packages.
135 PKG_UNPACK=$(HOST_TAR) -C $(PKG_BUILD_DIR) --strip-components=1 -xzf $(DL_DIR)/$(PKG_SOURCE)
139 It's not important whether this is after or before `include $(INCLUDE_DIR)/package.mk`
141 ### Include python[3]-package.mk
143 If packaging for Python, add this after `include $(INCLUDE_DIR)/package.mk`
145 include ../python-package.mk
148 If packaging for Python3, add this after `include $(INCLUDE_DIR)/package.mk`
150 include ../python3-package.mk
153 Order doesn't matter between `python-package.mk` & `python3-package.mk`.
155 These will make sure that build rules for Python or Python3 can be specified and picked up for build.
157 ### Add Package/<PKG_NAME> OpenWrt definitions
159 This part is similar to default OpenWrt packages.
160 It's usually recommended to have a `Package/<PKG_NAME>/Default` section that's common for both Python & Python3.
164 define Package/python-lxml/Default
169 DEPENDS:=+libxml2 +libxslt +libexslt
173 Then for each variant do something like:
175 define Package/python-lxml
176 $(call Package/python-lxml/Default)
178 DEPENDS+=+PACKAGE_python-lxml:python-light +PACKAGE_python-lxml:python-codecs
182 define Package/python3-lxml
183 $(call Package/python-lxml/Default)
185 DEPENDS+=+PACKAGE_python3-lxml:python3-light
190 Some considerations here (based on the example above):
191 * be sure to make sure that `DEPENDS` are correct for both variants; as seen in the example above, `python-codecs` is needed only for `python-lxml` (see **[note-encodings](#note-encodings)**)
192 * consider adding conditional DEPENDS for each variant ; so for each Python[3] package add `+PACKAGE_python-lxml:<dep>` as seen in the above example ; the reason for this is build-time reduction ; if you want to build Python3 only packages, this won't build Python & Python packages + dependencies ; this is a known functionality of OpenWrt build deps
193 * there is an exception to the above consideration: if adding `+PACKAGE_python-lxml` conditional deps creates circular dependencies [for some weird reason], then this can be omitted
194 * `VARIANT=python` or `VARIANT=python3` must be added
195 * typically each variant package is named `Package/python3-<something>` & `Package/python3-<something>` ; this convention makes things easier to follow, though it could work without naming things this this
196 * `TITLE` can be something a bit more verbose/neat ; typically the name is short as seen above
198 <a name="note-encodings">**note-encodings**</a>: That's because some character encodings are needed, which are present in `python3-base` but not in `python-light` (but are present in `python-codecs`) ; this is because Python3 is designed to be more Unicode friendly than Python2 (it's one of the fundamental differences between the 2), and Python3 won't start without those encodings being present.
201 Following these, 2 more definitions are required:
203 define Package/python-lxml/description
204 The lxml XML toolkit is a Pythonic binding
205 for the C libraries libxml2 and libxslt.
208 define Package/python3-lxml/description
209 $(call Package/python-lxml/description)
211 (Variant for Python3)
215 Typically, the description is the same for both, so just mentioning that one is a variant of the other is sufficient.
217 ### Wrapping things up so that they build
219 If all the above prerequisites have been met, all that's left is:
222 $(eval $(call PyPackage,python-lxml))
223 $(eval $(call BuildPackage,python-lxml))
225 $(eval $(call Py3Package,python3-lxml))
226 $(eval $(call BuildPackage,python3-lxml))
229 The `$(eval $(call PyPackage,python-lxml))` part will instantiate all the default Python build rules so that the final Python package is packaged into an OpenWrt.
230 And `$(eval $(call BuildPackage,python-lxml))` will bind all the rules generated with `$(eval $(call PyPackage,python-lxml))` into the OpenWrt build system.
232 These packages will contain byte-codes and binaries (shared libs & other stuff).
234 If a user wishes to ship source code, adding 2 more lines creates 2 more packages that ship Python source code:
236 $(eval $(call PyPackage,python-lxml))
237 $(eval $(call PyPackage,python-lxml-src))
238 $(eval $(call BuildPackage,python-lxml))
240 $(eval $(call Py3Package,python3-lxml))
241 $(eval $(call Py3Package,python3-lxml-src))
242 $(eval $(call BuildPackage,python3-lxml))
245 The name `*-src` must be the Python package name; so for `python-lxml-src` a equivalent `python-lxml` name must exist.
247 ### Customizing things
249 Some packages need custom build rules (because they do).
251 The default package build and install processes are defined in `python[3]-package.mk`.
255 The default build process calls `setup.py install` inside the directory where the Python source package is extracted (`PKG_BUILD_DIR`). This "installs" the Python package to an intermediate location (`PKG_INSTALL_DIR`) where it is used by the default install process.
257 There are several Makefile variables that can be used to customize this process (all optional):
259 * `PYTHON_PKG_SETUP_DIR` / `PYTHON3_PKG_SETUP_DIR`: Path where `setup.py` can be found, relative to the package directory (`PKG_BUILD_DIR`).
260 Default: empty value (`setup.py` is in the package directory)
261 * `PYTHON_PKG_SETUP_VARS` / `PYTHON3_PKG_SETUP_VARS`: Additional environment variables to set for the call to `setup.py`. Should be in the form of `VARIABLE1=value VARIABLE2=value ...`.
263 * `PYTHON_PKG_SETUP_GLOBAL_ARGS` / `PYTHON3_PKG_SETUP_GLOBAL_ARGS`: Additional command line arguments to pass to `setup.py`, before / in front of the `install` command.
265 * `PYTHON_PKG_SETUP_ARGS` / `PYTHON3_PKG_SETUP_ARGS`: Additional command line arguments to pass to `setup.py`, after the `install` command.
266 Default: `--single-version-externally-managed`
268 Conceptually, these variables are used in this way (using a Python 2 package as an example):
271 cd $(PKG_BUILD_DIR)/$(PYTHON_PKG_SETUP_DIR)
272 $(PYTHON_PKG_SETUP_VARS) python setup.py $(PYTHON_PKG_SETUP_GLOBAL_ARGS) install $(PYTHON_PKG_SETUP_ARGS)
275 The default build process can be completely overridden by defining custom `PyBuild/Compile` & `Py3Build/Compile` rules in the package Makefile.
279 The default install process copies some/all of the files from `PKG_INSTALL_DIR`, placed there by the build process, to a location passed to the install rule as the first argument (`$(1)`). The OpenWrt build system will then take those files and create the actual .ipk package archives.
281 This default process uses 2 build rules:
282 * `PyPackage/<package>/filespec` & `Py3Package/<package>/filespec` which are Python library files relative to `/usr/lib/pythonX.Y` ; by default this is `/usr/lib/python$(PYTHON[3]_VERSION)/site-packages` (`PYTHON[3]_PKG_DIR`) ; most Python[3] packages generate files that get installed in this sub-folder
283 * `PyPackage/<package>/install` & `Py3Package/<package>/install` is similar to `Package/<package>/install` ; these allow binary (or other files) to be installed on the target
285 Both the 2 above rules generate a `Package/<package>/install` build rule, which gets picked up by the build system. Both can be used together (they are not mutually exclusive), and provide a good enough flexibility for specifying Python[3] packages.
287 The `PyPackage/<package>/filespec` & `Py3Package/<package>/filespec` rules contain one or more lines of the following format (whitespace added for clarity):
290 <one of: +-=> | <file/directory path> | <file permissions>
293 The initial character controls the action that will be taken:
295 * `+`: Install the given path. If the path is a directory, all files and subdirectories inside are installed.
296 * If file permissions is specified (optional), then the file or directory (and all files and subdirectories) are assigned the given permissions; if omitted, then the file or directory retains its original permissions.
297 * `-`: Remove the given path. Useful when most of a directory should be installed except for a few files or subdirectories.
298 * File permissions is not used / ignored in this case.
299 * `=`: Assign the given file permissions to the given path. File permissions is required in this case.
301 As mentioned, the default `PyPackage/<package>/filespec` & `Py3Package/<package>/filespec` install `PYTHON[3]_PKG_DIR`:
304 define PyPackage/python-example/filespec
309 If there is an `examples` directory and `test_*.py` files that can be omitted to save space, this can be specified as:
312 define PyPackage/python-example/filespec
314 -|$(PYTHON_PKG_DIR)/examples
315 -|$(PYTHON_PKG_DIR)/test_*.py
319 ### Host-side Python packages for build
321 These can be installed via pip and ideally they should only be installed like this, because it's a bit simpler than running them through the OpenWrt build system. Build variants on the host-side build are more complicated (and nearly impossible to do sanely) in the current OpenWrt build system.
323 Which is why [for example] if you need python cffi on the host build, it's easier to just add it via:
325 HOST_PYTHON_PACKAGE_BUILD_DEPENDS:="cffi==$(PKG_VERSION)"
326 HOST_PYTHON3_PACKAGE_BUILD_DEPENDS:="cffi==$(PKG_VERSION)"
328 [cffi is one of those packages that needs a host-side package installed for both Python & Python3].
330 This works reasonably well in the current OpenWrt build system, as binaries get built for this package and get installed in the staging-dir `$(STAGING_DIR)/usr/lib/pythonX.Y/site-packages`.