This change brings some new additions to our archives:
- SHA-256 .sha256 hexdigests
- We construct our archives in a tmpdir now and move them to the
archive destination when all are completed. This removes some
corrupted downloading when archiving is in-process.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This variable allows users to override the logging.conf used
for Python logging configuration. By default, this is set
to logging.conf, which is a production config. LOG_CONFIG
is treated relative to [options] aurwebdir.
This patch allows us to specify the logging config as opposed
to copying over logging.conf in our test docker and gitlab
test scripts, as well as ease-of-testing as a developer.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Other ports we use are locked to 127.0.0.1. The `git` service,
however, already promotes security in its sshd service and
can't really be abused from an external source. This simplifies
the need to forward to localhost if deploy targets want the sshd
to be available.
Signed-off-by: Kevin Morris <kevr@0cost.org>
For the `git` service, ./data is always used to provide an
optional overriding of ssh host keys. In aur-dev production
containers, most services which use the data mount use an
internal Docker `data` volume instead.
Signed-off-by: Kevin Morris <kevr@0cost.org>
A user that can create databases is now required for tests,
we use the 'root' user in Docker.
Added docker services:
---------------------
- mariadb_test - host localhost:13307
Signed-off-by: Kevin Morris <kevr@0cost.org>
Normally, these scripts are used to update official providers
in the aurweb database along with archives that can be retrieved.
Run both of these scripts in a 5 minute cron job, to both reflect
the live instance database and production load.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously, we restricted this to gunicorn to get it working on aur-dev.
This change makes it usable through any backend, and also no-op if
PROMETHEUS_MULTIPROC_DIR is not defined.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously CGIT_CLONE_PREFIX_(PHP|FASTAPI), we found that we could
use the same env var in multiple places, including non-cgit-clone-prefix
areas.
So, they were renamed, and one additional prefix was added.
- CGIT_CLONE_PREFIX_PHP -> AURWEB_PHP_PREFIX
- Used for cgit's clone prefix and AUR_CONFIG's aur_location for PHP
- CGIT_CLONE_PREFIX_FASTAPI -> AURWEB_FASTAPI_PREFIX
- Used for cgit's clone prefix and AUR_CONFIG's aur_location for FastAPI
- AURWEB_SSHD_PREFIX
- Used for aurweb's sshd clone prefix shown on package pages
Signed-off-by: Kevin Morris <kevr@0cost.org>
Additionally, for now, no-op usage of the MARIADB_SOCKET_DIR
environment variable. This is particularly useful for a serious
production setup, but we don't currently use that.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The default recursion limit used by Docker's archlinux:base-devel
Python package becomes problematic in some cases when running tests
against our FastAPI application using starlette.testclient.TestClient
(aliased to fastapi.testclient.TestClient). starlette ends up with
test failures because it exceeds the recursion limit, but this only
happens when using the `TestClient`. When the ASGI servers are run,
this is not an issue and so in that case, the recursion limit has
not been touched.
This change uses a `TEST_RECURSION_LIMIT` environment variable to
modify the recursion limit of the FastAPI application. This variable
is, by default, only supplied when running pytests in Docker, but
can be force-supplied by the user.
TEST_RECURSION_LIMIT=10000 has been added to `.env` and `.gitlab-ci.yml`.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change exposes the uwsgi daemon we use for cgit on:
- PHP: docker-host:13000
- FastAPI: docker-host:13001
These ports can then be used to take advantage of cgit on
a production server that hosts nginx in front of Docker.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Additionally, clone-prefix is now configurable via environment variables:
- CGIT_CLONE_PREFIX_PHP
- CGIT_CLONE_PREFIX_FASTAPI
These vars can be used by production to customize the clone prefix.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The new `aurweb-image` service does not perform any purpose
other than providing a build definition for 'aurweb:latest'.
With this, `docker-compose build` now just runs once for the
`aurweb-image` service, which builds the image used by all
other services.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Supply FASTAPI_BACKEND=gunicorn and FASTAPI_WORKERS=<threads_num> to
docker-compose up to use the gunicorn backend.
This is defaulted in production to gunicorn, but FASTAPI_WORKERS
should definitely be configured by any production deployment.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Provides a single source of truth for mariadb database
initialization. Previously, php-fpm and fastapi were
racing against each other; while this wasn't an issue,
it was very messy.
Signed-off-by: Kevin Morris <kevr@0cost.org>
These weren't needed at all and provided false negatives in
general. Removed them to let Docker deal with them.
Additionally. 'exit 0' -> 'echo' for ca's command; 'exit 0'
happens to depend on the shell running Docker (it seems).
echo is quite a bit more agnostic.
Moreso, added mariadb deps to php-fpm and fastapi.
Signed-off-by: Kevin Morris <kevr@0cost.org>
In addition, remove some unneeded dependencies on tests.
Though, in the future we _should_ craft tests that use these.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This script purely removes any existing sqlite and is
used before tests are run. This causes the test flow
to run `aurweb.initdb` again (if ever).
Signed-off-by: Kevin Morris <kevr@0cost.org>
This was completely bugged out. This commit fixes git, provides
two separate cgit servers for the different URL bases and also
supplies a smartgit service for $AURWEB_URL/repo.git interaction.
Docker image needs to be rebuilt with this change:
$ docker build -t aurweb:latest .
Signed-off-by: Kevin Morris <kevr@0cost.org>
This caused a bug where generated locale would not be used.
Also, removed appending to /etc/hosts which was bugging out
on Mac OS X. archlinux:base-devel seems to come with a valid
/etc/hosts.
Additionally, remove AUR_CONFIG from Dockerfile. We don't
set it up; just use the defaults during installation.
Signed-off-by: Kevin Morris <kevr@0cost.org>
By default we now use uvicorn because it has a much
better developer feedback out of the box. We'll work
on hypercorn logging, but for now, hypercorn is usable
via: `docker-compose --env-file docker/hypercorn.env up nginx`.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Before, docker build was the only way to transfer new code
over to the docker image. This allows users to execute code
in their working directory.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Now, we have a full collection of services used to run
aurweb over HTTPS using a self-signed CA.
New Docker services:
- `ca` - Certificate authority services
- When the `ca` service is run, it will (if needed) generate
a CA certificate and leaf certificate for localhost AUR
access. This ca is then shared with things like nginx to
use the leaf certificate. Users can import
`./cache/ca.root.pem` into their browser or ca-certificates
as a root CA who issued aurweb's certificate.
- `git` - Start sshd and set it up for aur git access
- `cgit` - Serve cgit with uwsgi on port 3000
- `fastapi` - Serve our FastAPI app with `hypercorn` on port 8000
- `php-fpm` - Serve our PHP-wise aurweb
- `nginx` - Serve FastAPI, PHP and CGit with an HTTPS certificate.
- PHP: https://localhost:8443
- PHP CGit: https://localhost:8443/cgit
- FastAPI: https://localhost:8444
- FastAPI CGit: https://localhost:8444/cgit
Short of it: Run the following in a shell to run PHP and FastAPI
servers on port **8443** and **8444**, respectively.
$ docker-compose up nginx
This will host the PHP, FastAPI, CGit and Git ecosystems.
Git SSH can be knocked at `aur@localhost:2222` as long as you have a
valid public key in the aurweb database.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Instead of using Dockerfile for everything, we've introduced
a docker-compose.yml file and kept the Dockerfile to producing
a pure base image for the services defined.
docker-compose services:
- `mariadb` - Setup mariadb
- `sharness` - Run sharness suites
- `pytest-mysql` - Run pytest suites with MariaDB
- `pytest-sqlite` - Run pytest suites with SQLite
- `test` - Run all tests and produce a collective coverage report
- This target mounts a cache volume and copies any successful
coverage report back to `./cache/.coverage`. Users can run
`./util/fix-coverage ./cache/.coverage` to rewrite source
code paths and move coverage into place to view reports
on your local system.
== Get Started ==
Build `aurweb:latest`.
$ docker build -t aurweb:latest .
Run all tests via `docker-compose`.
$ docker-compose up test
You can also purely run `pytest` in SQLite or MariaDB modes.
$ docker-compose up pytest-sqlite
$ docker-compose up pytest-mysql
Or `sharness` alone, which only uses SQLite internally.
$ docker-compose up sharness
After running tests, coverage reports are stored in `./cache/.coverage`.
This database was most likely created in a different path, and so it
needs to be sanitized with `./util/fix-coverage`.
$ ./util/fix-coverage cache/.coverage
Copied coverage db to /path/to/aurweb/.coverage.
$ coverage report
...
$ coverage html
$ coverage xml
...
Defined components:
**Entrypoints**
- mariadb-entrypoint.sh - setup mariadb and run its daemon
- test-mysql-entrypoint.sh - setup mysql configurations
- test-sqlite-entrypoint.sh - setup sqlite configurations
- tests-entrypoint.sh - setup mysql and sqlite configurations
**Scripts**
- run-mariadb.sh - setup databases
- run-pytests.sh - run pytest suites
- run-sharness.sh - run sharness suites
- run-tests.sh - run both pytests and sharness
**Health**
- mariadb.sh - A healthcheck script for the mariadb service
- pytest.sh - A healthcheck script for the pytest-* services
- sharness.sh - A healthcheck script for the sharness service
This Docker configuration is setup for tests, but should be
extendable for web and git servers.
**Changes to Makefile**
- Remove `.coverage` in the `clean` target
- Add a `coverage` target which prints a report and outputs xml
Signed-off-by: Kevin Morris <kevr@0cost.org>
