This also comes with a -w|--workers argument that allows
the caller to set the number of gunicorn workers.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Didn't get this in when the initial request port went down;
here it is.
Auto-accept orphan requests when the package has been out of
date for longer than auto_orphan_age.
Auto-accept deletion requests by the package's maintainer
if the package has been uploaded within auto_deletion_age
seconds ago.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Now, we use an equivalent query to PHP's query, yet we grab
every piece of data we need for all packages asked for in one
database query.
At this time, local benchmarks have shown a slight performance
improvement when compared to PHP.
fastapi 262 requests/sec
php 250 requests/sec
Extras:
- Moved RPCError to the aurweb.exceptions module
Signed-off-by: Kevin Morris <kevr@0cost.org>
Without the increment, we've seen tests failed due to recursion
errors caused by starlette's base middleware. Just make it safe
in case nobody supplies TEST_RECURSION_LIMIT.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change utilizes pytest-xdist to perform a multiproc test
run and reworks aurweb.db's code. We no longer use a global
engine, session or Session, but we now use a memo of engines
and sessions as they are requested, based on the PYTEST_CURRENT_TEST
environment variable, which is available during testing.
Additionally, this change strips several SQLite components
out of the Python code-base.
SQLite is still compatible with PHP and sharness tests, but
not with our FastAPI implementation.
More changes:
------------
- Remove use of aurweb.db.session global in other code.
- Use new aurweb.db.name() dynamic db name function in env.py.
- Added 'addopts' to pytest.ini which utilizes multiprocessing.
- Highly recommended to leave this be or modify `-n auto` to
`-n {cpu_threads}` where cpu_threads is at least 2.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Changes:
-------
- Add aurweb.db.get_session()
- Returns aurweb.db's global `session` instance
- Provides us a way to change the implementation of the session
instance without interrupting user code.
- Use aurweb.db.get_session() in session API methods
- Add docstrings to session API methods
- Refactor aurweb.db.delete
- Normalize aurweb.db.delete to an alias of session.delete
- Refresh instances in places we depend on their non-PK columns
being up to date.
Signed-off-by: Kevin Morris <kevr@0cost.org>
When this is unchecked, exceptions cause the resulting stack
trace to be oblivious to the original exception thrown.
This commit changes that behavior so that metrics are created
only when info.response exists.
Signed-off-by: Kevin Morris <kevr@0cost.org>
These comments change every time mkpkglists is run; which
would invalidate the ETag headers disbursed by the gzip
host. This commit removes those changing headers.
Signed-off-by: Kevin Morris <kevr@0cost.org>
We really need caching for this; however, our current caching
method will cause the script to bypass changes to columns
if they have nothing to do with IDs.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
We don't want to depend on the database to load up data
about the models we define. We now leverage the existing
`aurweb.schema` module for table definitions and set
__table_args__["autoload"] to False.
Signed-off-by: Kevin Morris <kevr@0cost.org>
We really need caching for this; however, our current caching
method will cause the script to bypass changes to columns
if they have nothing to do with IDs.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
This wasn't actually casting to int. We shouldn't be providing
HTTPStatus.CONSTANTS directly anyway, but, in case we do, we now
just convert the status to an int before converting it to a string.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously, we restricted this to gunicorn to get it working on aur-dev.
This change makes it usable through any backend, and also no-op if
PROMETHEUS_MULTIPROC_DIR is not defined.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit provides custom metrics, so we can group requests
into their route paths and not by the arguments given, e.g.
/pkgbase/some-package -> /pkgbase/{name}. We also count RPC
requests as `http_api_requests_total`, split by the RPC
query "type" argument.
- `http_api_requests_total`
- Labels: ["type", "status"]
- `http_requests_total`
- Number of HTTP requests in total.
- Labels: ["method", "path", "status"]
Signed-off-by: Kevin Morris <kevr@0cost.org>
If the If-None-Match header is supplied with a previously
obtained ETag from the same query, a 304 Not Modified is
returned with no content.
This allows clients to completely leverage the ETag header.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Since we're in the hot path, a constant re.compiled
JSONP_EXPR is defined for checks against the callback.
Additionally, reorganized `content_type` and `content`
to avoid performing a DB query when we encounter a
regex mismatch.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change introduces alternate rendering of text/javascript
JSONP-compatible callback content. The `examples/jsonp.html`
HTML document can be used to test this functionality against
a running aurweb server.
Signed-off-by: Kevin Morris <kevr@0cost.org>
When by == 'maintainer', we allow an unspecified keyword,
resulting in a search of orphan packages. Fix our search
check so that when no arg is given, it is set to an empty
str().
We already check for valid args when type is not maintainer,
so there's no need to worry about other args falling through.
Signed-off-by: Kevin Morris <kevr@0cost.org>
