This reverts commit 952c24783b.
The issue found was actually:
- If `./aur.git` exists within the aurweb repository locally,
it also ends up in the destination, stopping the aurweb_git_data
volume from being mounted properly.
This was left in when we removed logging.prod.conf in a
previous patch. `test-mysql-entrypoint.sh` takes care of
test logging for us now, so this section is unnecessary.
Closes#261
Signed-off-by: Kevin Morris <kevr@0cost.org>
- On non-localhost communication, this whitelists forwarded headers
on all remote ips
- Add more headers
- Force https X-Forwarded-Proto
- Unset Forwarded header and rely on X-Forwarded-*
Signed-off-by: Kevin Morris <kevr@0cost.org>
When the Docker image is outdated, we need to fetch updated
archlinux-keyring keys to perform an -Syu without problems.
Signed-off-by: Kevin Morris <kevr@0cost.org>
We don't need the https certificates being dealt with in the fastapi
service; we will define our certificates in any frontend nginx
running on top.
Signed-off-by: Kevin Morris <kevr@0cost.org>
`install` includes dependencies present in poetry.lock
and we must stick to them if we wish to pin dependencies.
Signed-off-by: Kevin Morris <kevr@0cost.org>
For the `git` service, ./data is always used to provide an
optional overriding of ssh host keys. In aur-dev production
containers, most services which use the data mount use an
internal Docker `data` volume instead.
Signed-off-by: Kevin Morris <kevr@0cost.org>
A user that can create databases is now required for tests,
we use the 'root' user in Docker.
Added docker services:
---------------------
- mariadb_test - host localhost:13307
Signed-off-by: Kevin Morris <kevr@0cost.org>
Normally, these scripts are used to update official providers
in the aurweb database along with archives that can be retrieved.
Run both of these scripts in a 5 minute cron job, to both reflect
the live instance database and production load.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously, we restricted this to gunicorn to get it working on aur-dev.
This change makes it usable through any backend, and also no-op if
PROMETHEUS_MULTIPROC_DIR is not defined.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Supply FASTAPI_BACKEND=gunicorn and FASTAPI_WORKERS=<threads_num> to
docker-compose up to use the gunicorn backend.
This is defaulted in production to gunicorn, but FASTAPI_WORKERS
should definitely be configured by any production deployment.
Signed-off-by: Kevin Morris <kevr@0cost.org>
In some cases, when tests fail through Docker, the database
ends up in an invalid state. This causes subsequent runs to
error out with non-sensical DB errors. The `test_initdb.py`
test suite runs tests which setup every modifiable table
in the database, so let's just run it first here to avoid
any invalid test DB state.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit adds a new Arch dependency: `libeatmydata`, which
provides the `eatmydata` executable that stubs out fsync() operations.
We use `eatmydata` to run our sharness and pytests in Docker now.
With `autocommit=True`, required by SQLAlchemy to keep the
session up to date with external DB modifications, many fsync
calls are used in the SQLite backend; especially because we're wiping
and creating records in every DB-bound test.
**Before:**
- mysql: 1m42s (elapsed during pytest run)
- sqlite: 3m06s (elapsed during pytest run)
**After:**
- mysql: 1m40s (elapsed during pytest run)
- sqlite: 1m50s (elapsed during pytest run)
Shout out to @klausenbusk, who suggested this as a possible fix,
and it was. Thanks, Kristian!
Closes#120
Signed-off-by: Kevin Morris <kevr@0cost.org>
Now, when a `./cache/production.{cert,key}.pem` pair is found, it is
used in place of any certificates generated by the `ca` service.
This allows users to customize the certificate that the FastAPI
ASGI server uses as well as the front-end nginx certificates.
Optional:
- ./cache/production.cert.pem
- ./cache/production.key.pem
Fallback:
- ./cache/localhost.cert.pem + ./cache/root.ca.pem (chain)
- ./cache/localhost.key.pem
Signed-off-by: Kevin Morris <kevr@0cost.org>
PHP was doing this correctly, but FastAPI was doing this
in it's exec script @ docker/scripts/run-fastapi.sh.
Modify the fastapi service so that it does the same thing as
PHP, and the existing "fastapi restart quirk" is no more.
Signed-off-by: Kevin Morris <kevr@0cost.org>
As the new-age Python package manager, Poetry brings a lot
of good additions to the table. It allows us to more easily
deal with virtualenvs for the project and resolve dependencies.
As of this commit, `requirements.txt` is replaced by Poetry,
configured at `pyproject.toml`.
In Docker and GitLab, we currently use Poetry in a root fashion.
We should work toward purely using virtualenvs in Docker, but,
for now we'd like to move forward with other things. The project
can still be installed to a virtualenv and used on a user's system
through Poetry; it is just not yet doing so in Docker.
Modifications:
* docker/scripts/install-deps.sh
* Remove python dependencies.
* conf/config.defaults
* Script paths have been updated to use '/usr/bin'.
* docker/git-entrypoint.sh
* Use '/usr/bin/aurweb-git-auth' instead of
'/usr/local/bin/aurweb-git-auth'.
Additions:
* docker/scripts/install-python-deps.sh
* A script used purely to install Python dependencies with Poetry.
This has to be used within the aurweb project directory and
requires system-wide dependencies are installed beforehand.
* Also upgrades system-wide pip.
Signed-off-by: Kevin Morris <kevr@0cost.org>
python-orjson speeds up a lot of JSON serialization steps,
so we choose to use it over the standard library json module.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This includes the addition of the python-fakeredis package,
used for stubbing python-redis when a user does not have a
configured cache.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This script purely removes any existing sqlite and is
used before tests are run. This causes the test flow
to run `aurweb.initdb` again (if ever).
Signed-off-by: Kevin Morris <kevr@0cost.org>
Now, we have `docker/scripts/install-deps.sh`, a script used
by both Docker and .gitlab-ci.yml. We can now focus on changing
deps in this script along as well as documentation going forward.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This was completely bugged out. This commit fixes git, provides
two separate cgit servers for the different URL bases and also
supplies a smartgit service for $AURWEB_URL/repo.git interaction.
Docker image needs to be rebuilt with this change:
$ docker build -t aurweb:latest .
Signed-off-by: Kevin Morris <kevr@0cost.org>
By default we now use uvicorn because it has a much
better developer feedback out of the box. We'll work
on hypercorn logging, but for now, hypercorn is usable
via: `docker-compose --env-file docker/hypercorn.env up nginx`.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Now, we have a full collection of services used to run
aurweb over HTTPS using a self-signed CA.
New Docker services:
- `ca` - Certificate authority services
- When the `ca` service is run, it will (if needed) generate
a CA certificate and leaf certificate for localhost AUR
access. This ca is then shared with things like nginx to
use the leaf certificate. Users can import
`./cache/ca.root.pem` into their browser or ca-certificates
as a root CA who issued aurweb's certificate.
- `git` - Start sshd and set it up for aur git access
- `cgit` - Serve cgit with uwsgi on port 3000
- `fastapi` - Serve our FastAPI app with `hypercorn` on port 8000
- `php-fpm` - Serve our PHP-wise aurweb
- `nginx` - Serve FastAPI, PHP and CGit with an HTTPS certificate.
- PHP: https://localhost:8443
- PHP CGit: https://localhost:8443/cgit
- FastAPI: https://localhost:8444
- FastAPI CGit: https://localhost:8444/cgit
Short of it: Run the following in a shell to run PHP and FastAPI
servers on port **8443** and **8444**, respectively.
$ docker-compose up nginx
This will host the PHP, FastAPI, CGit and Git ecosystems.
Git SSH can be knocked at `aur@localhost:2222` as long as you have a
valid public key in the aurweb database.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Instead of using Dockerfile for everything, we've introduced
a docker-compose.yml file and kept the Dockerfile to producing
a pure base image for the services defined.
docker-compose services:
- `mariadb` - Setup mariadb
- `sharness` - Run sharness suites
- `pytest-mysql` - Run pytest suites with MariaDB
- `pytest-sqlite` - Run pytest suites with SQLite
- `test` - Run all tests and produce a collective coverage report
- This target mounts a cache volume and copies any successful
coverage report back to `./cache/.coverage`. Users can run
`./util/fix-coverage ./cache/.coverage` to rewrite source
code paths and move coverage into place to view reports
on your local system.
== Get Started ==
Build `aurweb:latest`.
$ docker build -t aurweb:latest .
Run all tests via `docker-compose`.
$ docker-compose up test
You can also purely run `pytest` in SQLite or MariaDB modes.
$ docker-compose up pytest-sqlite
$ docker-compose up pytest-mysql
Or `sharness` alone, which only uses SQLite internally.
$ docker-compose up sharness
After running tests, coverage reports are stored in `./cache/.coverage`.
This database was most likely created in a different path, and so it
needs to be sanitized with `./util/fix-coverage`.
$ ./util/fix-coverage cache/.coverage
Copied coverage db to /path/to/aurweb/.coverage.
$ coverage report
...
$ coverage html
$ coverage xml
...
Defined components:
**Entrypoints**
- mariadb-entrypoint.sh - setup mariadb and run its daemon
- test-mysql-entrypoint.sh - setup mysql configurations
- test-sqlite-entrypoint.sh - setup sqlite configurations
- tests-entrypoint.sh - setup mysql and sqlite configurations
**Scripts**
- run-mariadb.sh - setup databases
- run-pytests.sh - run pytest suites
- run-sharness.sh - run sharness suites
- run-tests.sh - run both pytests and sharness
**Health**
- mariadb.sh - A healthcheck script for the mariadb service
- pytest.sh - A healthcheck script for the pytest-* services
- sharness.sh - A healthcheck script for the sharness service
This Docker configuration is setup for tests, but should be
extendable for web and git servers.
**Changes to Makefile**
- Remove `.coverage` in the `clean` target
- Add a `coverage` target which prints a report and outputs xml
Signed-off-by: Kevin Morris <kevr@0cost.org>
