external/aurweb
0
0
Fork 0
mirror of https://gitlab.archlinux.org/archlinux/aurweb.git synced 2025-02-03 10:43:03 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update Docker

Kevin Morris 2021-10-24 01:11:52 +00:00
parent afb7a4dd04
commit 9fd38d3ed2
1 changed files with 117 additions and 65 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

182
Docker.md

@ -1,37 +1,44 @@
| Branch |
|--------|
| pu |
AURWeb's Docker infrastructure is composed of a [Dockerfile](https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu/Dockerfile) and [docker-compose.yml Services](https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu/docker-compose.yml).
## Contents
- [Status](#status)
- [Services](#services)
- [Getting Started](#getting-started)
- [Development](#development)
- [Production](#production)
## Status
Branches containing Docker features are located in the
pipeline/coverage status table below:
| pu (fastapi) |
|--------------|
| ![pipeline](https://gitlab.archlinux.org/archlinux/aurweb/badges/pu/pipeline.svg?key_text=build) ![coverage](https://gitlab.archlinux.org/archlinux/aurweb/badges/pu/coverage.svg) |
## Services
| Service | Port | Profile | Purpose |
|---------------|-------|---------|--------------------------------|
| ca | | | Self-signed CA generation |
| memcached | | | Live memcached service |
| redis | 16379 | | Live Redis service |
| mariadb | 13306 | | Live MariaDB service |
| mariadb_init | | | MariaDB initialization |
| git | 2222 | | AUR Git repository |
| smartgit | | | HTTP Git frontend |
| cgit-php | | | CGit backend (PHP) |
| cgit-fastapi | | | CGit backend (FastAPI) |
| php-fpm | 19000 | | Live php-fpm service |
| fastapi | 18000 | | Live $FASTAPI_BACKEND service |
| nginx | 8443 | | Live nginx frontend to php-fpm |
| nginx | 8444 | | Live nginx frontend to fastapi |
| sharness | | dev | Run sharness test suites |
| pytest-mysql | | dev | Run pytest suites with mariadb |
| pytest-sqlite | | dev | Run pytest suites with sqlite |
| test | | dev | Run all tests and linters |
| Service | Port | Profile | Purpose |
|-----------------|---------|---------|---------------------------------|
| `ca` | | | Self-signed CA generation |
| `memcached` | | | Live memcached service |
| `redis` | `16379` | | Live Redis service |
| `mariadb` | `13306` | | Live MariaDB service |
| `mariadb_init` | | | MariaDB initialization |
| `git` | `2222` | | AUR Git repository |
| `smartgit` | | | HTTP Git frontend |
| `cgit-php` | `13000` | | CGit backend (PHP) |
| `cgit-fastapi` | `13001` | | CGit backend (FastAPI) |
| `php-fpm` | `19000` | | Live php-fpm service |
| `fastapi` | `18000` | | Live `$FASTAPI_BACKEND` service |
| `nginx` | `8443` | | Live nginx frontend to php-fpm |
| `nginx` | `8444` | | Live nginx frontend to fastapi |
| `sharness` | | `dev` | Run sharness test suites |
| `pytest-mysql` | | `dev` | Run pytest suites with mariadb |
| `pytest-sqlite` | | `dev` | Run pytest suites with sqlite |
| `test` | | `dev` | Run all tests and linters |
## Getting Started
@ -100,55 +107,100 @@ The following services in particular are useful for production:
Note: To see exposed ports on the host, take a look at [Services](#services) at
the beginning of this document.
Deployers will want to use an HTTP front-end like `nginx` which uses
Docker's `php-fpm` and/or `fastapi` services as a backend.
### Production Configuration
Example:
All Docker-configurable environment variables are defaulted within the
[.env](https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu/.env) file.
$ docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d fastapi
$ docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d php-fpm
$ cat /etc/nginx/conf.d/aur-fastapi.conf
server {
listen 443 ssl http2;
server_name aur-fastapi.domain.org;
#### mariadb
ssl_certificate /path/to/aur.cert.pem;
ssl_certificate_key /path/to/aur.key.pem;
The `mariadb` service requires a bind-mounted socket directory from the host,
containing `mysqld.sock` in production:
root /path/to/aurweb_root;
- `MARIADB_SOCKET_DIR`: must contain `mysqld.sock`
- Contains path to mariadb's socket directory on the host.
location / {
try_files $uri @proxy_to_app;
}
This feature means that the database used by all Docker services are
driven by the host. Docker containers will contact the host's `mysqld.sock`
for database usage in general.
# Not yet accessible for production.
location ~ "^/([a-z0-9][a-z0-9.+_-]*?)(\.git)?/(git-(receive|upload)-pack|HEAD|info/refs|objects/(info/(http-)?alternates|packs)|[0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}\.(pack|idx))$" {
include uwsgi_params;
uwsgi_pass smartgit;
uwsgi_modifier1 9;
uwsgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
uwsgi_param PATH_INFO /aur.git/$3;
uwsgi_param GIT_HTTP_EXPORT_ALL "";
uwsgi_param GIT_NAMESPACE $1;
uwsgi_param GIT_PROJECT_ROOT /aurweb;
}
#### git
# Not yet enabled for production.
location ~ ^/cgit {
include uwsgi_params;
rewrite ^/cgit/([^?/]+/[^?]*)?(?:\?(.*))?$ /cgit.cgi?url=$1&$2 last;
uwsgi_modifier1 9;
uwsgi_param CGIT_CONFIG /etc/cgitrc;
uwsgi_pass uwsgi://127.0.0.1:<exposed_cgit_fastapi_port>;
}
The `git` service (and services which depend on it) requires a bind-mounted
repository from the host in production:
location @proxy_to_app {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_buffering off;
proxy_pass https://127.0.0.1:18000;
}
- `GIT_DATA_DIR` (default: `./aur.git/`)
- Contains path to an source repository directory on the Docker
host which is bind mounted in containers.
}
This feature means that all services which depend on the aurweb git repository
will make use of the provided `$GIT_DATA_DIR` source bind mounted from
host to container.
#### cgit-(php|fastapi)
In production, the `cgit-php` and `cgit-fastapi` services should be
configured to use the intended `clone-prefix`.
- `CGIT_CLONE_PREFIX_PHP`
- Defaulted to `https://localhost:8443`
- `CGIT_CLONE_PREFIX_FASTAPI`
- Defaulted to `https://localhost:8444`
#### git-bridge (Work In Progress)
In production, hosts should be able to run a instance of sshd in
front of Docker, which allows standard authentication to the host
in addition to customized authentication and shell for a particular
user (`aur` by default).
For this reason, we must provide a persistent endpoint in a Docker
service which bridges to `aurweb-git-auth`. This bridge must customize
output when it is invoking the script via the `git-bridge` RPC API.
This way, the Docker host could configure sshd's `AuthorizedKeysCommand`
to contact this endpoint for auth negotiation, which would tunnel through
to `aurweb-git-serve` when successful.
This feature is being worked on and has had success thus far, however,
it is not yet available and must be simplified as it stands. This section
will be updated when the feature comes to completion and can be used
by production.
#### fastapi
In production, the `fastapi` service should be configured to use
the `gunicorn` multi-proc, multi-threaded ASGI server frontend.
- `FASTAPI_BACKEND`
- Options: `uvicorn`, `gunicorn`, `hypercorn`
- Defaulted to `uvicorn`
- `FASTAPI_WORKERS`
- Only applicable when `FASTAPI_BACKEND=gunicorn`
- Defaulted to `2`
Tip: Our `php-fpm` service uses 5 workers (php-fpm's default). To
make proper performance comparisons, `fastapi` should be configured
for `gunicorn` with `5` workers.
#### redis/memcached
The `redis` and `memcached` servers are depended on by their respective
users: `fastapi` and `php-fpm`. There is no control over `memcached`,
but `redis` does provide a host-accessible port:
- `redis` - port `16379`
Users can run `redis-cli -p 16379` to gain access to the Docker
`redis` instance.
#### nginx
The `nginx` service should **not be used in production**. When running
aurweb in a production Docker instance, users should configure an
instance in front of Docker to use exposed services as backends:
- `cgit-php` - port `13000`
- `php-fpm` - port `19000`
- `cgit-fastapi` - port `13001`
- `fastapi` - port `18000`