external/aurweb
0
0
Fork 0
mirror of https://gitlab.archlinux.org/archlinux/aurweb.git synced 2025-02-03 10:43:03 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update Docker

Kevin Morris 2021-11-24 22:38:41 +00:00
parent 65d6bc297f
commit 6dbe5c26ef
1 changed files with 92 additions and 59 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

151
Docker.md

@ -20,25 +20,87 @@ pipeline/coverage status table below:
## Services
| Service | Port | Profile | Purpose |
|-----------------|---------|---------|---------------------------------|
| `ca` | | | Self-signed CA generation |
| `memcached` | | | Live memcached service |
| `redis` | `16379` | | Live Redis service |
| `mariadb` | `13306` | | Live MariaDB service |
| `mariadb_init` | | | MariaDB initialization |
| `git` | `2222` | | AUR Git repository |
| `smartgit` | | | HTTP Git frontend |
| `cgit-php` | `13000` | | CGit backend (PHP) |
| `cgit-fastapi` | `13001` | | CGit backend (FastAPI) |
| `php-fpm` | `19000` | | Live php-fpm service |
| `fastapi` | `18000` | | Live `$FASTAPI_BACKEND` service |
| `nginx` | `8443` | | Live nginx frontend to php-fpm |
| `nginx` | `8444` | | Live nginx frontend to fastapi |
| `sharness` | | `dev` | Run sharness test suites |
| `pytest-mysql` | | `dev` | Run pytest suites with mariadb |
| `pytest-sqlite` | | `dev` | Run pytest suites with sqlite |
| `test` | | `dev` | Run all tests and linters |
| Service | Host Bind | Purpose |
|-----------------|-------------------|---------------------------------|
| `ca` | | Self-signed CA generation |
| `memcached` | | Live memcached service |
| `redis` | `127.0.0.1:16379` | Live Redis service |
| `mariadb` | `127.0.0.1:13306` | Live MariaDB service |
| `mariadb_test` | `127.0.0.1:13307` | Live test MariaDB service |
| `mariadb_init` | | `mariadb` initialization |
| `git` | `0.0.0.0:2222` | AUR Git SSH interface |
| `smartgit` | | HTTP Git frontend |
| `cgit-php` | `127.0.0.1:13000` | CGit backend (PHP) |
| `cgit-fastapi` | `127.0.0.1:13001` | CGit backend (FastAPI) |
| `php-fpm` | `127.0.0.1:19000` | Live php-fpm service |
| `fastapi` | `127.0.0.1:18000` | Live `$FASTAPI_BACKEND` service |
| `nginx` | `127.0.0.1:8443` | Live nginx frontend to php-fpm |
| `nginx` | `127.0.0.1:8444` | Live nginx frontend to fastapi |
| `sharness` | | Run sharness test suites |
| `pytest-mysql` | | Run pytest suites with mariadb |
| `pytest-sqlite` | | Run pytest suites with sqlite |
| `test` | | Run all tests and linters |
## Profiles
We have two kinds of profiles:
- [default](#profile-default) (Development)
- [aur-dev](#profile-aur-dev) (Production)
#### Profile: default
This profile provides support for the following environment variables:
- `FASTAPI_BACKEND`
- Default: `uvicorn`
- Choices: `uvicorn`, `hypercorn`, `gunicorn`
- Controls the backend used for the `fastapi` service. In production,
`gunicorn` is recommended.
- `FASTAPI_WORKERS`
- Default: `2`
- Only used when `FASTAPI_BACKEND=gunicorn`; controls the number of
workers used.
- `AURWEB_PHP_PREFIX`
- Default: `https://localhost:8443`
- Used to populate `aur_location` in the `php-fpm` service.
- Used to populate `git_clone_uri_anon` in the `php-fpm` service.
- `AURWEB_FASTAPI_PREFIX`
- Default: `https://localhost:8444`
- Used to populate `aur_location` in the `fastapi` service.
- Used to populate `git_clone_uri_anon` in the `fastapi` service.
- `AURWEB_SSHD_PREFIX`
- Default: `ssh://aur@localhost:2222`
- Used to populate `git_clone_uri_priv`.
- `TEST_RECURSION_LIMIT`
- Default: `10000`
- This should be left alone. It is used to configure the recursion
limit used for tests.
- `COMMIT_HASH`
- Default: empty
- When set, used to produce a link to the specified commit hash
in the https://gitlab.archlinux.org/archlinux/aurweb repository.
#### Profile: aur-dev
To use this profile, execute docker-compose with the following:
$ docker-compose -f docker-compose.yml -f docker-compose.aur-dev.yml ...
This profile provides support for the following environment variables,
along with any inherited from [default](#profile-default):
- `MARIADB_SOCKET_DIR`
- Default: `/var/run/mysqld`
- Bind mount a host mariadb instance socket directory.
- `GIT_DATA_DIR`
- Default: `./aur.git`
- Bind mount a Git repository on the host filesystem.
This profile **does not** bind mount aurweb repository components
for live reloading like the [default](#profile-default) profile does.
This means that you'll have to rebuild the Docker image for any
local filesystem changes to take effect.
## Getting Started
@ -56,12 +118,6 @@ Users will notice initially starting up a service can take some time, especially
$ docker-compose run test # First run with no up'd services, takes a while.
$ docker-compose run test # Starts instantly.
## Continued Imaging
After building, users do not need to rebuild the image for code changes in the `aurweb` package. Our `fastapi` Docker service takes advantage of hot reload and has shared volume access to `$aurwebdir/aurweb/`. As a user makes changes to the `aurweb` package, the `fastapi` service automatically reloads with the new code. See the `fastapi` service in `docker-compose.yml` for a complete list of shared local directory volumes.
**NOTE** However, an image rebuild is necessary whenever Docker infrastructure changes. This includes `poetry` dependencies or `$aurwebdir/docker/` modifications.
## Development
During development, users may wish to run instances of FastAPI and PHP side by side. For this, users can depend on the `nginx` service, which hosts PHP at https://localhost:8443 and FastAPI at https://localhost:8444.
@ -75,19 +131,14 @@ For an all in one testing strategy, the `test` service should be preferred, as i
Our test suites also provide coverage data via a shared volume directory at `$aurwebdir/cache/`. After tests are complete, users can copy coverage data output by Docker into their local setup by executing `./util/fix-coverage ./cache/.coverage`. After this is done, users can continue on with standard coverage execution:
$ docker-compose run pytest-mysql
...
$ ./util/fix-coverage ./cache/.coverage
$ coverage report
$ coverage html
$ <your_browser> htmlcov/index.html
Test services which provide coverage data: `pytest-mysql`, `pytest-sqlite`, and `test`.
Test services which provide coverage data: `pytest-mysql` and `test`.
## Production
For production, the [docker-compose.prod.yml](https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu/docker-compose.prod.yml) should be used in unison with the standard `docker-compose.yml`:
For production, the [docker-compose.aur-dev.yml](https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu/docker-compose.aur-dev.yml) profile config should be used in unison with the standard `docker-compose.yml`:
$ docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d <service>
$ docker-compose -f docker-compose.yml -f docker-compose.aur-dev.yml up -d <service>
This provides service overrides which mainly affect volume behavior. In
production, volumes will not be shared with docker from the host.
@ -142,30 +193,12 @@ host to container.
In production, the `cgit-php` and `cgit-fastapi` services should be
configured to use the intended `clone-prefix`.
- `CGIT_CLONE_PREFIX_PHP`
- `AURWEB_PHP_PREFIX`
- Defaulted to `https://localhost:8443`
- `CGIT_CLONE_PREFIX_FASTAPI`
- `AURWEB_FASTAPI_PREFIX`
- Defaulted to `https://localhost:8444`
#### git-bridge (Work In Progress)
In production, hosts should be able to run a instance of sshd in
front of Docker, which allows standard authentication to the host
in addition to customized authentication and shell for a particular
user (`aur` by default).
For this reason, we must provide a persistent endpoint in a Docker
service which bridges to `aurweb-git-auth`. This bridge must customize
output when it is invoking the script via the `git-bridge` RPC API.
This way, the Docker host could configure sshd's `AuthorizedKeysCommand`
to contact this endpoint for auth negotiation, which would tunnel through
to `aurweb-git-serve` when successful.
This feature is being worked on and has had success thus far, however,
it is not yet available and must be simplified as it stands. This section
will be updated when the feature comes to completion and can be used
by production.
- `AURWEB_SSHD_PREFIX`
- Defaulted to `ssh://aur@localhost:2222`
#### fastapi
@ -200,7 +233,7 @@ The `nginx` service should **not be used in production**. When running
aurweb in a production Docker instance, users should configure an
instance in front of Docker to use exposed services as backends:
- `cgit-php` - port `13000`
- `php-fpm` - port `19000`
- `cgit-fastapi` - port `13001`
- `fastapi` - port `18000`
- `cgit-php` @ `127.0.0.1:13000`
- `php-fpm` @ `127.0.0.1:19000`
- `cgit-fastapi` @ `127.0.0.1:13001`
- `fastapi` @ `127.0.0.1:18000`