staging/chunkeey/scripts, branch v19.07.4

Revert "scripts/download: add sources CDN as first mirror"

2020-08-24T13:56:27Z

This reverts commit c737a9ee6a9c47b6e553ac81bf293b1161e59799. The source CDN has been discontinued in its current form and will take a while to be reestablished. Even then it makes little sense to put a CDN before other CDNs such as kernel.org, apache.org, sourceforge etc. Signed-off-by: Jo-Philipp Wich (cherry picked from commit bf96eb55c82191701030b68810e6f19adbb91eeb)

scripts: Add Buildbot dump-target-info.pl script

2020-08-09T23:43:15Z

The script comes from buildbot.git[0] and is used to print available targets and architectures, which are then build. As the buildbot clones openwrt.git anyway, the script might as well live here to be used for other cases as well, e.g. determining what architectures are available when building Docker containers or show developers an overview which architectures are used by which target. It's called with either the parameter `architectures` or `targets`, showing architectures followed by supported targets or targets, followed by the supported architectures: $ ./scripts/dump-target-info.pl architectures aarch64_cortex-a53 bcm27xx/bcm2710 mediatek/mt7622 mvebu/cortexa53 sunxi/cortexa53 aarch64_cortex-a72 bcm27xx/bcm2711 mvebu/cortexa72 ... $ ./scripts/dump-target-info.pl targets apm821xx/nand powerpc_464fp apm821xx/sata powerpc_464fp ... In the future the the script could be removed from the buildbot repository and maintained only here. Rename `dumpinfo.pl` to `dump-target-info.pl` to improve verbosity of filename. [0]: https://git.openwrt.org/?p=buildbot.git;a=blob;f=scripts/dumpinfo.pl;h=aa97f8d60379076a41b968402e9337cea824ece5;hb=HEAD Signed-off-by: Paul Spooren (cherry picked from commit 656b562aff36d92b0e8586833b59896a55b8a993)

build,json: fix compatibility with Python 3.5

2020-07-13T20:24:30Z

The f-string feature was introduced in Python 3.6. As Buildbots may run on Debian 9, which comes per default with Python 3.5, this would cause an issue. Instead of f-strings use the *legacy* `.format()` function. Signed-off-by: Paul Spooren

build: fix compatibility with python 3.6

2020-07-13T20:24:30Z

On a system python3 is linked to python3.6, fail to perform json_overview_image_info and got `TypeError: __init__() got an unexpected keyword argument 'capture_output'`. This patch emulate the behaviour on python 3.7+. Signed-off-by: Huangbin Zhan (cherry picked from commit 3caad5150c2011a7dac462acaa06d0e69f8ed005)

build,json: fix build failure in case no data is found

2020-07-13T20:24:30Z

Only collect arch_packages if actually generating any output. Fixes: commit f09b9319 ("build,json: store arch_packages in profiles.json"( Signed-off-by: Daniel Golle Signed-off-by: Paul Spooren (cherry picked from commit 3b0f698760ae3a62173a28f18e9e1e3adef9c492)

build,json: store arch_packages in profiles.json

2020-07-13T20:24:30Z

The `arch_packages` contains the supported package architecture. Previously it was necessary to parse the `Packages` index for the line `Architecture:`, requiring both an additional parser and file download. Signed-off-by: Paul Spooren (cherry picked from commit f09b9319c666e343763c7618878a503ad7eb7531)

build: store default/device packages in JSON

2020-07-13T20:24:30Z

With this commit the `profiles.json` contain both the target specific `default_packages` as well as the device specific `device_packages` as a array of strings. This information is required for downstream projects like the various web-based interactive firmware generators. Signed-off-by: Daniel Golle Signed-off-by: Paul Spooren (cherry picked from commit 263f7e5bbd119ebed1f514c16f659a2e2a2b132c)

scripts: JSON merge don't crash if no JSON found

2020-06-30T20:13:05Z

The JSON `WORK_DIR` ($(KDIR)/json_info_files) is only created if the new image generation methods from `image.mk` are used. However some targets like `armvirt` do not use it yet, so the folder is never created. The `json_overview_image_info.py` script used to raise an error if the given `WORK_DIR` isn't a folder, however it should just notify about missing JSON files. This patch removes the Python assert and exists with code 0 even if no JSON files were found, as this is not necessarily an error but simply not yet implemented. Using `glob` on an not existing `Path` results in an empty list, therefore the for loop won't run. Signed-off-by: Paul Spooren CC: Petr Štetiar (cherry picked from commit 14cbd8fb2dd8c81bad06d3c3bb45213685c19c98)

build: refactor JSON info files to `profiles.json`

2020-06-30T20:12:58Z

JSON info files contain machine readable information of built profiles and resulting images. These files were added in commit 881ed09ee6e2 ("build: create JSON files containing image info"). They are useful for firmware wizards and script checking for reproducibility. Currently all JSON files are stored next to the built images, resulting in up to 168 individual files for the ath79/generic target. This patch refactors the JSON creation to store individual per image (not per profile) files in $(BUILD_DIR)/json_info_files and create an single overview file called `profiles.json` in the target directory. Storing per image files and not per profile solves the problem of parallel file writes. If a profiles sysupgrade and factory image are finished at the same time both processes would write to the same JSON file, resulting in randomly broken outputs. Some target like x86/64 do not use the image code yet, resulting in missing JSON files. If no JSON info files were created, no `profiles.json` files is created as it would be empty anyway. As before, this creation is enabled by default only if `BUILDBOT` is set. Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64. Signed-off-by: Paul Spooren [json_info_files dir handling in Make, if case refactoring] Signed-off-by: Petr Štetiar (backported from commit 07449f692ce4c4525e946401f4c3ed0cbbc8c4df) Signed-off-by: Adrian Schmutzler

build: create JSON files containing image info

2020-06-30T20:12:39Z

The JSON info files contain details about the created firmware images per device and are stored next to the created images. The JSON files are stored as "$(IMAGE_PREFIX).json" and contain some device/image meta data as well as a list of created firmware images. An example of openwrt-ramips-rt305x-aztech_hw550-3g.json { "id": "aztech_hw550-3g", "image_prefix": "openwrt-ramips-rt305x-aztech_hw550-3g", "images": [ { "name": "openwrt-ramips-rt305x-aztech_hw550-3g-squashfs-sysupgrade.bin", "sha256": "db2b34b0ec4a83d9bf612cf66fab0dc3722b191cb9bedf111e5627a4298baf20", "type": "sysupgrade" } ], "metadata_version": 1, "supported_devices": [ "aztech,hw550-3g", "hw550-3g" ], "target": "ramips/rt305x", "titles": [ { "model": "HW550-3G", "vendor": "Aztech" }, { "model": "ALL0239-3G", "vendor": "Allnet" } ], "version_commit": "r10920+123-0cc87b3bac", "version_number": "SNAPSHOT" } Signed-off-by: Paul Spooren (backported from commit 881ed09ee6e23f6c224184bb7493253c4624fb9f) Signed-off-by: Adrian Schmutzler