download: improve handling of invalid local files 4e19cbc5533: [download: handle possibly invalid local tarballs] added a FORCE rule to downloaded files, so that they will be always checked by download.pl. As a side-effect, check-compile will fail, forcing unnecessary package rebuilds. The check-compile.txt log shows (for libxml2 for example): Considering target file '.../dl/libxml2-2.9.12.tar.gz'. ... prerequisite 'FORCE' of target '.../dl/libxml2-2.9.12.tar.gz' does not exist. Must remake target '.../dl/libxml2-2.9.12.tar.gz'. ... Giving up on target file '...libxml2-2.9.12/.prepared_...'. Giving up on target file '...libxml2-2.9.12/.configured_...'. Giving up on target file '...libxml2-2.9.12/.built'. Giving up on target file '...stamp/.libxml2_installed'. Giving up on target file '.compile'. Then the package is rebuilt even if it is not otherwise needed. To fix this, instead of always forcing the download target to be remade, check its hash first: if it matches, then the FORCE is not added. Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
build: introduce $(MKHASH) Before this commit, it was assumed that mkhash is in the PATH. While this was fine for the normal build workflow, this led to some issues if make TOPDIR="$(pwd)" -C "$pkgdir" compile was called manually. In most of the cases, I just saw warnings like this: make: Entering directory '/home/.../package/gluon-status-page' bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found [...] While these were only warnings and the package still compiled sucessfully, I also observed that some package even fail to build because of this. After applying this commit, the variable $(MKHASH) is introduced. This variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the correct path. Signed-off-by: Leonardo Mörlein <me@irrelefant.net>
download: add mirror alias for Debian Add an alias for Debian packages and download them from the Debian mirror redirector. Signed-off-by: David Bauer <mail@david-bauer.net>
treewide: unify OpenWrt hosted source via @OPENWRT Multiple sources are hosted on OpenWrts source server only. The source URLs to point to the server vary based on different epochs in OpenWrts history. Replace all by @OPENWRT which is an "empty" mirror, therefore using the fallback servers sources.cdn.openwrt.org and sources.openwrt.org. Signed-off-by: Paul Spooren <mail@aparcar.org>
build: use SPDX license tags The license folder is a core part of OpenWrt and all GPL-2.0 licensed. Use SPDX license tags to allow machines to check licenses. Signed-off-by: Paul Spooren <mail@aparcar.org> [rebase, keep some Copyright lines, sharpen commit message] Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
build: prevent excessive re-evaluation of PKG_VERSION version_abbrev uses $(shell) and the ?= is causing make to run the command over and over again, causing a significant build slowdown Signed-off-by: Felix Fietkau <nbd@nbd.name>
build: add zstd support to pack/unpack functions Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
build: add PKG_SOURCE_URL_FILE support It seems that there is a missing PKG_SOURCE_URL_FILE support. This little fix adds the support for packages to change the name of the downloaded file. Sometimes it is desirable to change the downloaded archive file name, like for mitigating name conflicts for different packages (some files on the server could be named like, e.g. 2018-01-01.tar.gz) or for the cases that there is no name for the file in the URL (e.g. http://someserver.com/download). Signed-off-by: Kamil Wcislo <kamil.wcislo@lpnplant.io> Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Revert faulty tree push Revert "mac80211: add new minstrel_ht patches to improve probing on mt76x2" (9861050b85e5381c93d73ffdbc24c130400e0fb8) Revert "kernel: use bulk free in kfree_skb_list to improve performance" (98b654de2e7502507b31f0fb82befbb48f9c8542) Revert "ramips: add preliminary support for WIO ONE" (085141dc5ba5e763e1bf4c1a2fd6957160810ccc) Revert "ramips: add preliminary support for SGE AP-MTKH7-0006 developer board" (b1db6d0539511f60bf65fbde28b16afa31180e34) Revert "build: use config.site generated by autoconf-lean, drop hardcoded sitefiles" (363ce4329d4f6c00b4a7cb83dd1821fafc0d40ad) Revert "toolchain: add autoconf-lean" (fdb30eed03ae56cd87078b64d9b2fac00799e783) Revert "build: allow overriding the filename on the remote server when downloading" (6fa0e07758524eddf20c9b66dfcb55c490e1961e) Signed-off-by: Felix Fietkau <nbd@nbd.name>
build: allow overriding the filename on the remote server when downloading Github releases usually don't contain the project name in the release filename, which makes them very inconvenient to use from the build system. Add support for naming the local file differently. Signed-off-by: Felix Fietkau <nbd@nbd.name>
build: remove sgid permission from tar Otherwise tar will keep the sgid bit when running from a sgid-set directory, resulting in a different file being generated. Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com> Signed-off-by: Christian Lamparter <chunkeey@gmail.com> [reworked commit message, removed DMARC]
download.mk: enable DownloadMethod/github_archive Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
scripts/dl_github_archive.py: rename from download.py - Make the code more GitHub-specific - Requires mirror hash to work with .gitattributes - Use different API depending on whether PKG_SOURCE_VERSION is a complete commit id or other ref types like tags - Fix removing symbolic link - pre-clean dir_untar for possible leftovers from previous run Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
download.mk: add more comments Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
download.mk: restore the old dl_method implementation Seems like the python download.py dl_method call causes serious performance regression for fresh "make defconfig" as reported in FS#1621. GitHub tarball download will also be disabled with this Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
build: download code from github using archive API A new python script scripts/download.py is added to fetch tarballs using GitHub archive API [1], then repack in a reproducible way same as the current DownloadMethod/git GitHub imposes a 60 reqs/hour rate limit on unauthenticated API access[2]. This affects fetching commit date for feeding tar --mtime= argument. However, observation indicates that archive download is NOT subject to this limit at the moment. In the rare cases where download fails because of this, we will falback to using DownloadMethod/git The missing piece in the GitHub API is that it cannot provide in the tarball dependent submodules's source code. In that case, the implementation will also fallback to using DownloadMethod/git [1] Get archive link, https://developer.github.com/v3/repos/contents/#get-archive-link [2] Rate limiting, https://developer.github.com/v3/#rate-limiting v2 <- v1: - allow passing multiple urls with --urls argument - add commit ts cache. can be helpful on retry Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
downloads.mk: introduce name-agnostic PROJECT_GIT variable Introduce a name-agnostic PROJECT_GIT variable poiting to https://git.openwrt.org/ and declare LEDE_GIT and OPENWRT_GIT as aliases to it. After some transition time we can drop this alias variables. Signed-off-by: Jo-Philipp Wich <jo@mein.io>
build: remove @ as it's causing an error Since $(DownloadMethod/unknown) is being invoked in the expansion of $(call locked ...) anyway, you can't have an @ because the shell doesn't know what to do with it. Signed-off-by: Philip Prindeville <philipp@redfish-solutions.com>
build: let make check warn about use of legacy PKG_MD5SUM variable in feeds The variable rename change has been merged in OpenWrt now, so it's possible to convert the feeds as well. Signed-off-by: Felix Fietkau <nbd@nbd.name>
build: use mkhash to replace various quirky md5sum/openssl calls Signed-off-by: Felix Fietkau <nbd@nbd.name>