A user that can create databases is now required for tests,
we use the 'root' user in Docker.
Added docker services:
---------------------
- mariadb_test - host localhost:13307
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change utilizes pytest-xdist to perform a multiproc test
run and reworks aurweb.db's code. We no longer use a global
engine, session or Session, but we now use a memo of engines
and sessions as they are requested, based on the PYTEST_CURRENT_TEST
environment variable, which is available during testing.
Additionally, this change strips several SQLite components
out of the Python code-base.
SQLite is still compatible with PHP and sharness tests, but
not with our FastAPI implementation.
More changes:
------------
- Remove use of aurweb.db.session global in other code.
- Use new aurweb.db.name() dynamic db name function in env.py.
- Added 'addopts' to pytest.ini which utilizes multiprocessing.
- Highly recommended to leave this be or modify `-n auto` to
`-n {cpu_threads}` where cpu_threads is at least 2.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Changes:
-------
- Add aurweb.db.get_session()
- Returns aurweb.db's global `session` instance
- Provides us a way to change the implementation of the session
instance without interrupting user code.
- Use aurweb.db.get_session() in session API methods
- Add docstrings to session API methods
- Refactor aurweb.db.delete
- Normalize aurweb.db.delete to an alias of session.delete
- Refresh instances in places we depend on their non-PK columns
being up to date.
Signed-off-by: Kevin Morris <kevr@0cost.org>
When this is unchecked, exceptions cause the resulting stack
trace to be oblivious to the original exception thrown.
This commit changes that behavior so that metrics are created
only when info.response exists.
Signed-off-by: Kevin Morris <kevr@0cost.org>
These comments change every time mkpkglists is run; which
would invalidate the ETag headers disbursed by the gzip
host. This commit removes those changing headers.
Signed-off-by: Kevin Morris <kevr@0cost.org>
We really need caching for this; however, our current caching
method will cause the script to bypass changes to columns
if they have nothing to do with IDs.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Two new archives are available:
- packages-meta-v1.json.gz
- RPC search formatted data for all packages
- ~2.1MB at the time of writing.
- packages-meta-ext-v1.json.gz (via --extended)
- RPC multiinfo formatted data for all packages.
- ~9.8MB at the time of writing.
New dependencies are required for this update:
- `python-orjson`
All archives served out by aur.archlinux.org distribute the Last-Modified
header and support the If-Modified-Since header, which should be
populated with Last-Modified's value. These should be used by clients
to avoid redownloading the archive when unnecessary.
Additionally, the new meta archives contain a format suitable for
streaming the data as the file is retrieved. It is still in JSON
format, however, users can parse package objects line by line after
the first '[' found in the file, until the last ']'; both contained
on their own lines.
Note: This commit is a documentation change and commit body.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
Normally, these scripts are used to update official providers
in the aurweb database along with archives that can be retrieved.
Run both of these scripts in a 5 minute cron job, to both reflect
the live instance database and production load.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Here, we default to using root as the storage directory. Primarily
because it makes sense in Docker; config.dev can always be fixed up
by developers to reflect local system changes.
Signed-off-by: Kevin Morris <kevr@0cost.org>
It seems the ftp mirror from kernel.org cannot be used anymore,
but the https mirror can. So, the default config has been updated
to reflect this; otherwise, aurblup bugs out.
Signed-off-by: Kevin Morris <kevr@0cost.org>
