For the `git` service, ./data is always used to provide an
optional overriding of ssh host keys. In aur-dev production
containers, most services which use the data mount use an
internal Docker `data` volume instead.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This only deals with .gz files in the root of the request_uri
and now more. That is: /packages.gz goes through the nginx regex,
but now /cgit/.../snapshot/package.tar.gz is served by the cgit
block.
Signed-off-by: Kevin Morris <kevr@0cost.org>
There is a new ./data bind mount used here. If ssh_host_* keys are
in ./data when the git service starts, they'll override the
container-generated host keys.
Signed-off-by: Kevin Morris <kevr@0cost.org>
A user that can create databases is now required for tests,
we use the 'root' user in Docker.
Added docker services:
---------------------
- mariadb_test - host localhost:13307
Signed-off-by: Kevin Morris <kevr@0cost.org>
Normally, these scripts are used to update official providers
in the aurweb database along with archives that can be retrieved.
Run both of these scripts in a 5 minute cron job, to both reflect
the live instance database and production load.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously, we restricted this to gunicorn to get it working on aur-dev.
This change makes it usable through any backend, and also no-op if
PROMETHEUS_MULTIPROC_DIR is not defined.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The previous method was super bad. Even if a variable was declared,
if it was empty, we would run into a false-positive. Additionally,
the previous method did not allow us to not specify the COMMIT_HASH
variable; which is problematic for development environments.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously CGIT_CLONE_PREFIX_(PHP|FASTAPI), we found that we could
use the same env var in multiple places, including non-cgit-clone-prefix
areas.
So, they were renamed, and one additional prefix was added.
- CGIT_CLONE_PREFIX_PHP -> AURWEB_PHP_PREFIX
- Used for cgit's clone prefix and AUR_CONFIG's aur_location for PHP
- CGIT_CLONE_PREFIX_FASTAPI -> AURWEB_FASTAPI_PREFIX
- Used for cgit's clone prefix and AUR_CONFIG's aur_location for FastAPI
- AURWEB_SSHD_PREFIX
- Used for aurweb's sshd clone prefix shown on package pages
Signed-off-by: Kevin Morris <kevr@0cost.org>
Additionally, clone-prefix is now configurable via environment variables:
- CGIT_CLONE_PREFIX_PHP
- CGIT_CLONE_PREFIX_FASTAPI
These vars can be used by production to customize the clone prefix.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Supply FASTAPI_BACKEND=gunicorn and FASTAPI_WORKERS=<threads_num> to
docker-compose up to use the gunicorn backend.
This is defaulted in production to gunicorn, but FASTAPI_WORKERS
should definitely be configured by any production deployment.
Signed-off-by: Kevin Morris <kevr@0cost.org>
In some cases, when tests fail through Docker, the database
ends up in an invalid state. This causes subsequent runs to
error out with non-sensical DB errors. The `test_initdb.py`
test suite runs tests which setup every modifiable table
in the database, so let's just run it first here to avoid
any invalid test DB state.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit adds a new Arch dependency: `libeatmydata`, which
provides the `eatmydata` executable that stubs out fsync() operations.
We use `eatmydata` to run our sharness and pytests in Docker now.
With `autocommit=True`, required by SQLAlchemy to keep the
session up to date with external DB modifications, many fsync
calls are used in the SQLite backend; especially because we're wiping
and creating records in every DB-bound test.
**Before:**
- mysql: 1m42s (elapsed during pytest run)
- sqlite: 3m06s (elapsed during pytest run)
**After:**
- mysql: 1m40s (elapsed during pytest run)
- sqlite: 1m50s (elapsed during pytest run)
Shout out to @klausenbusk, who suggested this as a possible fix,
and it was. Thanks, Kristian!
Closes#120
Signed-off-by: Kevin Morris <kevr@0cost.org>
Now, when a `./cache/production.{cert,key}.pem` pair is found, it is
used in place of any certificates generated by the `ca` service.
This allows users to customize the certificate that the FastAPI
ASGI server uses as well as the front-end nginx certificates.
Optional:
- ./cache/production.cert.pem
- ./cache/production.key.pem
Fallback:
- ./cache/localhost.cert.pem + ./cache/root.ca.pem (chain)
- ./cache/localhost.key.pem
Signed-off-by: Kevin Morris <kevr@0cost.org>
Additionally, simplify some of the certificate generation
scripts and rename `ca.ext` to `localhost.ext`.
Certificates should be regenerated as of this commit.
Users can run `rm -rf ./cache/*` to clear out any existing
certs, which will cause the `ca` service to regenerate them.
Additionally, since Docker infrastructure has been modified,
a new `aurweb:latest` image will need to be built.
See https://gitlab.archlinux.org/archlinux/aurweb/-/wikis/Docker
Signed-off-by: Kevin Morris <kevr@0cost.org>
Provides a single source of truth for mariadb database
initialization. Previously, php-fpm and fastapi were
racing against each other; while this wasn't an issue,
it was very messy.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The update hook was incorrectly linked to /usr/local/bin/aurweb-git-update,
which was neglected during the original patch regarding dependency
conversion to `poetry`.
Signed-off-by: Kevin Morris <kevr@0cost.org>
PHP was doing this correctly, but FastAPI was doing this
in it's exec script @ docker/scripts/run-fastapi.sh.
Modify the fastapi service so that it does the same thing as
PHP, and the existing "fastapi restart quirk" is no more.
Signed-off-by: Kevin Morris <kevr@0cost.org>
As the new-age Python package manager, Poetry brings a lot
of good additions to the table. It allows us to more easily
deal with virtualenvs for the project and resolve dependencies.
As of this commit, `requirements.txt` is replaced by Poetry,
configured at `pyproject.toml`.
In Docker and GitLab, we currently use Poetry in a root fashion.
We should work toward purely using virtualenvs in Docker, but,
for now we'd like to move forward with other things. The project
can still be installed to a virtualenv and used on a user's system
through Poetry; it is just not yet doing so in Docker.
Modifications:
* docker/scripts/install-deps.sh
* Remove python dependencies.
* conf/config.defaults
* Script paths have been updated to use '/usr/bin'.
* docker/git-entrypoint.sh
* Use '/usr/bin/aurweb-git-auth' instead of
'/usr/local/bin/aurweb-git-auth'.
Additions:
* docker/scripts/install-python-deps.sh
* A script used purely to install Python dependencies with Poetry.
This has to be used within the aurweb project directory and
requires system-wide dependencies are installed beforehand.
* Also upgrades system-wide pip.
Signed-off-by: Kevin Morris <kevr@0cost.org>
python-orjson speeds up a lot of JSON serialization steps,
so we choose to use it over the standard library json module.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This includes the addition of the python-fakeredis package,
used for stubbing python-redis when a user does not have a
configured cache.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This is needed to be able to reach the mysql service from
other hosts or through localhost. Handling both cases here
means that we can support both localhost access and host access.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This script purely removes any existing sqlite and is
used before tests are run. This causes the test flow
to run `aurweb.initdb` again (if ever).
Signed-off-by: Kevin Morris <kevr@0cost.org>
Now, we have `docker/scripts/install-deps.sh`, a script used
by both Docker and .gitlab-ci.yml. We can now focus on changing
deps in this script along as well as documentation going forward.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This makes logging look a little better for development purposes.
Now, `docker-compose logs php-fpm` will only show details about PHP
accesses, while `docker-compose logs nginx` will show accesses
regarding PHP assets.
Signed-off-by: Kevin Morris <kevr@0cost.org>
