From cdca8bd2953f3c3aa3a1b4cedb89c3b7b9fd4ddb Mon Sep 17 00:00:00 2001 From: Kevin Morris Date: Sat, 6 Nov 2021 16:23:08 -0700 Subject: [PATCH] feat(mkpkglists): added metadata archives Two new archives are available: - packages-meta-v1.json.gz - RPC search formatted data for all packages - ~2.1MB at the time of writing. - packages-meta-ext-v1.json.gz (via --extended) - RPC multiinfo formatted data for all packages. - ~9.8MB at the time of writing. New dependencies are required for this update: - `python-orjson` All archives served out by aur.archlinux.org distribute the Last-Modified header and support the If-Modified-Since header, which should be populated with Last-Modified's value. These should be used by clients to avoid redownloading the archive when unnecessary. Additionally, the new meta archives contain a format suitable for streaming the data as the file is retrieved. It is still in JSON format, however, users can parse package objects line by line after the first '[' found in the file, until the last ']'; both contained on their own lines. Note: This commit is a documentation change and commit body. Signed-off-by: Kevin Morris --- doc/maintenance.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/maintenance.txt b/doc/maintenance.txt index d6094545..2c5c9faf 100644 --- a/doc/maintenance.txt +++ b/doc/maintenance.txt @@ -70,7 +70,8 @@ computations and clean up the database: * aurweb-pkgmaint automatically removes empty repositories that were created within the last 24 hours but never populated. -* aurweb-mkpkglists generates the package list files. +* aurweb-mkpkglists generates the package list files; it takes an optional + --extended flag, which additionally produces multiinfo metadata. * aurweb-usermaint removes the last login IP address of all users that did not login within the past seven days. @@ -79,7 +80,7 @@ These scripts can be installed by running `python3 setup.py install` and are usually scheduled using Cron. The current setup is: ---- -*/5 * * * * aurweb-mkpkglists +*/5 * * * * aurweb-mkpkglists [--extended] 1 */2 * * * aurweb-popupdate 2 */2 * * * aurweb-aurblup 3 */2 * * * aurweb-pkgmaint