When this is unchecked, exceptions cause the resulting stack
trace to be oblivious to the original exception thrown.
This commit changes that behavior so that metrics are created
only when info.response exists.
Signed-off-by: Kevin Morris <kevr@0cost.org>
These comments change every time mkpkglists is run; which
would invalidate the ETag headers disbursed by the gzip
host. This commit removes those changing headers.
Signed-off-by: Kevin Morris <kevr@0cost.org>
We really need caching for this; however, our current caching
method will cause the script to bypass changes to columns
if they have nothing to do with IDs.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Two new archives are available:
- packages-meta-v1.json.gz
- RPC search formatted data for all packages
- ~2.1MB at the time of writing.
- packages-meta-ext-v1.json.gz (via --extended)
- RPC multiinfo formatted data for all packages.
- ~9.8MB at the time of writing.
New dependencies are required for this update:
- `python-orjson`
All archives served out by aur.archlinux.org distribute the Last-Modified
header and support the If-Modified-Since header, which should be
populated with Last-Modified's value. These should be used by clients
to avoid redownloading the archive when unnecessary.
Additionally, the new meta archives contain a format suitable for
streaming the data as the file is retrieved. It is still in JSON
format, however, users can parse package objects line by line after
the first '[' found in the file, until the last ']'; both contained
on their own lines.
Note: This commit is a documentation change and commit body.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
Normally, these scripts are used to update official providers
in the aurweb database along with archives that can be retrieved.
Run both of these scripts in a 5 minute cron job, to both reflect
the live instance database and production load.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Here, we default to using root as the storage directory. Primarily
because it makes sense in Docker; config.dev can always be fixed up
by developers to reflect local system changes.
Signed-off-by: Kevin Morris <kevr@0cost.org>
It seems the ftp mirror from kernel.org cannot be used anymore,
but the https mirror can. So, the default config has been updated
to reflect this; otherwise, aurblup bugs out.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Starlette 0.16.0 has a pretty bad bug in terms of logging which
has been fixed in the 0.17.0 release. That being said, FastAPI has
not yet merged a request at https://github.com/tiangolo/fastapi/pull/4145
which resolves this dependency resolution so we can use the updated
starlette package.
kevr has forked the pull request in question and we are using it
for now in our poetry dependencies to get ahead of the game.
When FastAPI upstream is updated to support 0.17.0, we'll need
to switch this back to using upstream's source.
Signed-off-by: Kevin Morris <kevr@0cost.org>
We don't want to depend on the database to load up data
about the models we define. We now leverage the existing
`aurweb.schema` module for table definitions and set
__table_args__["autoload"] to False.
Signed-off-by: Kevin Morris <kevr@0cost.org>
We really need caching for this; however, our current caching
method will cause the script to bypass changes to columns
if they have nothing to do with IDs.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Two new archives are available:
- packages-meta-v1.json.gz
- RPC search formatted data for all packages
- ~2.1MB at the time of writing.
- packages-meta-ext-v1.json.gz (via --extended)
- RPC multiinfo formatted data for all packages.
- ~9.8MB at the time of writing.
New dependencies are required for this update:
- `python-orjson`
All archives served out by aur.archlinux.org distribute the Last-Modified
header and support the If-Modified-Since header, which should be
populated with Last-Modified's value. These should be used by clients
to avoid redownloading the archive when unnecessary.
Additionally, the new meta archives contain a format suitable for
streaming the data as the file is retrieved. It is still in JSON
format, however, users can parse package objects line by line after
the first '[' found in the file, until the last ']'; both contained
on their own lines.
Note: This commit is a documentation change and commit body.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
This wasn't actually casting to int. We shouldn't be providing
HTTPStatus.CONSTANTS directly anyway, but, in case we do, we now
just convert the status to an int before converting it to a string.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously, we restricted this to gunicorn to get it working on aur-dev.
This change makes it usable through any backend, and also no-op if
PROMETHEUS_MULTIPROC_DIR is not defined.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit provides custom metrics, so we can group requests
into their route paths and not by the arguments given, e.g.
/pkgbase/some-package -> /pkgbase/{name}. We also count RPC
requests as `http_api_requests_total`, split by the RPC
query "type" argument.
- `http_api_requests_total`
- Labels: ["type", "status"]
- `http_requests_total`
- Number of HTTP requests in total.
- Labels: ["method", "path", "status"]
Signed-off-by: Kevin Morris <kevr@0cost.org>
