Provides a convenient way to check for responses on the
mailing list prior to Accepting/Rejecting requests.
We compute the Message-ID hash that can be used to
link back to the article in the mailing list archive.
Signed-off-by: moson-mo <mo-son@mailbox.org>
See doc/git-archive.md for general Git archive specifications
See doc/repos/metadata-repo.md for info and direction related to the new Git metadata archive
A new option has been added for configuration of SMTP timeout:
- notifications.smtp-timeout
During tests, we can change this timeout to be small, so we aren't
depending on hardware-based RNG to pass the timeout.
Without a timeout, users can run into a long-running test for no
particular reason.
Signed-off-by: Kevin Morris <kevr@0cost.org>
As repeats of these traceback notifications were annoying some of
the devops staff, and it took coordination to share tracebacks with
developers, this commit removes that responsibility off of devops
by reporting tracebacks to Gitlab repositories in the form of issues.
- removed ServerErrorNotification
- removed notifications.postmaster configuration option
- added notifications.gitlab-instance option
- added notifications.error-project option
- added notifications.error-token option
- added aurweb.exceptions.handle_form_exceptions, a POST route decorator
Issues are filed confidentially. This change will need updates
in infrastructure's ansible configuration before this can be
applied to aur.archlinux.org.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit does quite a bit:
- Catches unhandled exceptions raised in the route handler and
produces a 500 Internal Server Error Arch-themed response.
- Each unhandled exception causes a notification to be sent to new
`notifications.postmaster` email with a "Traceback ID."
- Traceback ID is logged to the server along with the traceback which
caused the 500: `docker-compose logs fastapi | grep '<traceback_id>'`
- If `options.traceback` is set to `1`, traceback is displayed in
the new 500.html template.
Signed-off-by: Kevin Morris <kevr@0cost.org>
A user that can create databases is now required for tests,
we use the 'root' user in Docker.
Added docker services:
---------------------
- mariadb_test - host localhost:13307
Signed-off-by: Kevin Morris <kevr@0cost.org>
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
It seems the ftp mirror from kernel.org cannot be used anymore,
but the https mirror can. So, the default config has been updated
to reflect this; otherwise, aurblup bugs out.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Two new options have been added:
- [devel] commit_url
- URL including an %s format specifier that can be used to link
to a webpage for the commit.
- [devel] commit_hash
- HEAD's commit hash (produced via `git rev-parse HEAD`)
If a `[devel] commit_hash` is configured, a link to the commit based on
`[devel] commit_url` will be displayed in the aurweb footer in
the form: `HEAD@<commit_hash>`. If no `[devel] commit_url` is
configured, a non-linked hash will be displayed.
Signed-off-by: Kevin Morris <kevr@0cost.org>
New configuration options:
- `[ratelimit] cache`
- A boolean indicating whether we should use configured cache (1)
or database (0) for ratelimiting.
Signed-off-by: Kevin Morris <kevr@0cost.org>
In terms of performance, most queries on this page win over
PHP in query times, with the exception of sorting by Voted or
Notify (https://gitlab.archlinux.org/archlinux/aurweb/-/issues/102).
Otherwise, there are a few modifications: described below.
* Pagination
* The `paginate` Python module has been used in the FastAPI
project
here to implement paging on the packages search page. This
changes how pagination is displayed, however it serves the
same purpose. We'll take advantage of this module in other
places as well.
* Form action
* The form action for actions now use `POST /packages` to
perform. This is currently implemented and will be
addressed in a follow-up commit.
* Input names and values
* Input names and values have been modified to satisfy the
snake_case naming convention we'd like to use as much as
possible.
* Some input names and values were modified to comply with
FastAPI Forms: (IDs[<id>]) -> (IDs, <id>).
Signed-off-by: Kevin Morris <kevr@0cost.org>
As the new-age Python package manager, Poetry brings a lot
of good additions to the table. It allows us to more easily
deal with virtualenvs for the project and resolve dependencies.
As of this commit, `requirements.txt` is replaced by Poetry,
configured at `pyproject.toml`.
In Docker and GitLab, we currently use Poetry in a root fashion.
We should work toward purely using virtualenvs in Docker, but,
for now we'd like to move forward with other things. The project
can still be installed to a virtualenv and used on a user's system
through Poetry; it is just not yet doing so in Docker.
Modifications:
* docker/scripts/install-deps.sh
* Remove python dependencies.
* conf/config.defaults
* Script paths have been updated to use '/usr/bin'.
* docker/git-entrypoint.sh
* Use '/usr/bin/aurweb-git-auth' instead of
'/usr/local/bin/aurweb-git-auth'.
Additions:
* docker/scripts/install-python-deps.sh
* A script used purely to install Python dependencies with Poetry.
This has to be used within the aurweb project directory and
requires system-wide dependencies are installed beforehand.
* Also upgrades system-wide pip.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This includes the addition of the python-fakeredis package,
used for stubbing python-redis when a user does not have a
configured cache.
Signed-off-by: Kevin Morris <kevr@0cost.org>
During development, the lower this value is (must be >= 4)
equals faster User generation. This is particularly useful
for running tests.
In production, a higher value (like 12 which is used by various
popular frameworks) should be used.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This feature was originally introduced by
f961ffd9c7 as a fix for FS#12898
<https://bugs.archlinux.org/task/12898>.
As of today, it is broken because of the `q.SessionID IS NULL` condition
in the WHERE clause, which can’t be true because SessionID is not
nullable. As a consequence, the session limit was not applied.
The fact the absence of the session limit hasn’t caused any issue so
far, and hadn’t even been noticed, suggests the feature is unneeded.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Developers can go to /sso/login to get redirected to the SSO. On
successful login, the ID token is displayed.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
conf/config.dev’s purpose is to provide a lighter configuration template
for developers, and split development-specific options off the default
configuration file.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
aurweb.spawn used to launch only PHP’s built-in server. Now it spawns a
dummy FastAPI application too. Since both stacks spawn their own HTTP
server, aurweb.spawn also spawns nginx as a reverse proxy to mount them
under the same base URL, defined by aur_location in the configuration.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
This program makes it easier for developers to spawn the PHP server
since it fetches automatically what it needs from the configuration
file, rather than having the user explicitly pass arguments to the php
executable.
When the setup gets more complicated as we introduce Python,
aurweb.spawn will keep providing the same interface, while under the
hood it is planned to support running multiple sub-processes.
Its Python interface provides an way for the test suite to spawn the
test server when it needs to perform HTTP requests to the test server.
The current implementation is somewhat weak as it doesn’t detect when a
child process dies, but this is not supposed to happen often, and it is
only meant for aurweb developers.
In the long term, aurweb.spawn will eventually become obsolete, and
replaced by Docker or Flask’s tools.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Add more options to configure the smtplib implementation for sending
notification emails.
The port can be changed using the new smtp-port option.
Encryption can be configured using smtp-use-ssl and smtp-use-starttls.
Keep in mind that you usually also need to change the port when enabling
either of these options.
Authentication can be configured using smtp-user and smtp-password.
Authentication is disabled if either of these values is empty.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Support mail delivery without a local MTA. Instead, an SMTP server can
now be configured using the smtp-server option in the [notifications]
section. In order to use this option, the value of the sendmail option
must be empty.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
The TTL for package details can be much longer than for generic values
since they never change. Note that when an update is pushed via Git, all
packages belonging to that package base are deleted and new packages are
created.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Add a new configuration option to specify the locale directory to use.
This allows the Python scripts to find the translations, even when not
being run from the source code checkout. At the same time, multiple
parallel aurweb setups can still use different sets of translations.
Fixes FS#59278.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
In the process, rename config.proto to config.defaults (because that is
what it is now).
Also use dict.get('key', default_value) when querying os.environ, rather
than an if block, as it is more pythonic/readable/concise, and reduces
the number of dict lookups.
This change allows aurweb configuration to be done via either:
- copying config.defaults to config and modifying values
- creating a new config only containing modified values, next to a
config.defaults containing unmodified values
The motivation for this change is to enable ansible configuration in our
flagship deployment by storing only changed values, and deferring to
config.defaults otherwise.
A side benefit is, it is easier to see what has changed by inspecting
only the site configuration file.
If a config.defaults file does not exist next to $AUR_CONFIG or in
$AUR_CONFIG_DEFAULTS, it is ignored and *all* values are expected to
live in the modified config file.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
