diff --git a/Docker.md b/Docker.md index 562247c..d5231bf 100644 --- a/Docker.md +++ b/Docker.md @@ -1,37 +1,44 @@ -| Branch | -|--------| -| pu | AURWeb's Docker infrastructure is composed of a [Dockerfile](https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu/Dockerfile) and [docker-compose.yml Services](https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu/docker-compose.yml). ## Contents +- [Status](#status) - [Services](#services) - [Getting Started](#getting-started) - [Development](#development) - [Production](#production) +## Status + +Branches containing Docker features are located in the +pipeline/coverage status table below: + +| pu (fastapi) | +|--------------| +| ![pipeline](https://gitlab.archlinux.org/archlinux/aurweb/badges/pu/pipeline.svg?key_text=build) ![coverage](https://gitlab.archlinux.org/archlinux/aurweb/badges/pu/coverage.svg) | + ## Services -| Service | Port | Profile | Purpose | -|---------------|-------|---------|--------------------------------| -| ca | | | Self-signed CA generation | -| memcached | | | Live memcached service | -| redis | 16379 | | Live Redis service | -| mariadb | 13306 | | Live MariaDB service | -| mariadb_init | | | MariaDB initialization | -| git | 2222 | | AUR Git repository | -| smartgit | | | HTTP Git frontend | -| cgit-php | | | CGit backend (PHP) | -| cgit-fastapi | | | CGit backend (FastAPI) | -| php-fpm | 19000 | | Live php-fpm service | -| fastapi | 18000 | | Live $FASTAPI_BACKEND service | -| nginx | 8443 | | Live nginx frontend to php-fpm | -| nginx | 8444 | | Live nginx frontend to fastapi | -| sharness | | dev | Run sharness test suites | -| pytest-mysql | | dev | Run pytest suites with mariadb | -| pytest-sqlite | | dev | Run pytest suites with sqlite | -| test | | dev | Run all tests and linters | +| Service | Port | Profile | Purpose | +|-----------------|---------|---------|---------------------------------| +| `ca` | | | Self-signed CA generation | +| `memcached` | | | Live memcached service | +| `redis` | `16379` | | Live Redis service | +| `mariadb` | `13306` | | Live MariaDB service | +| `mariadb_init` | | | MariaDB initialization | +| `git` | `2222` | | AUR Git repository | +| `smartgit` | | | HTTP Git frontend | +| `cgit-php` | `13000` | | CGit backend (PHP) | +| `cgit-fastapi` | `13001` | | CGit backend (FastAPI) | +| `php-fpm` | `19000` | | Live php-fpm service | +| `fastapi` | `18000` | | Live `$FASTAPI_BACKEND` service | +| `nginx` | `8443` | | Live nginx frontend to php-fpm | +| `nginx` | `8444` | | Live nginx frontend to fastapi | +| `sharness` | | `dev` | Run sharness test suites | +| `pytest-mysql` | | `dev` | Run pytest suites with mariadb | +| `pytest-sqlite` | | `dev` | Run pytest suites with sqlite | +| `test` | | `dev` | Run all tests and linters | ## Getting Started @@ -100,55 +107,100 @@ The following services in particular are useful for production: Note: To see exposed ports on the host, take a look at [Services](#services) at the beginning of this document. -Deployers will want to use an HTTP front-end like `nginx` which uses -Docker's `php-fpm` and/or `fastapi` services as a backend. +### Production Configuration -Example: +All Docker-configurable environment variables are defaulted within the +[.env](https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu/.env) file. - $ docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d fastapi - $ docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d php-fpm - $ cat /etc/nginx/conf.d/aur-fastapi.conf - server { - listen 443 ssl http2; - server_name aur-fastapi.domain.org; +#### mariadb - ssl_certificate /path/to/aur.cert.pem; - ssl_certificate_key /path/to/aur.key.pem; +The `mariadb` service requires a bind-mounted socket directory from the host, +containing `mysqld.sock` in production: - root /path/to/aurweb_root; +- `MARIADB_SOCKET_DIR`: must contain `mysqld.sock` + - Contains path to mariadb's socket directory on the host. - location / { - try_files $uri @proxy_to_app; - } +This feature means that the database used by all Docker services are +driven by the host. Docker containers will contact the host's `mysqld.sock` +for database usage in general. - # Not yet accessible for production. - location ~ "^/([a-z0-9][a-z0-9.+_-]*?)(\.git)?/(git-(receive|upload)-pack|HEAD|info/refs|objects/(info/(http-)?alternates|packs)|[0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}\.(pack|idx))$" { - include uwsgi_params; - uwsgi_pass smartgit; - uwsgi_modifier1 9; - uwsgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend; - uwsgi_param PATH_INFO /aur.git/$3; - uwsgi_param GIT_HTTP_EXPORT_ALL ""; - uwsgi_param GIT_NAMESPACE $1; - uwsgi_param GIT_PROJECT_ROOT /aurweb; - } +#### git - # Not yet enabled for production. - location ~ ^/cgit { - include uwsgi_params; - rewrite ^/cgit/([^?/]+/[^?]*)?(?:\?(.*))?$ /cgit.cgi?url=$1&$2 last; - uwsgi_modifier1 9; - uwsgi_param CGIT_CONFIG /etc/cgitrc; - uwsgi_pass uwsgi://127.0.0.1:; - } +The `git` service (and services which depend on it) requires a bind-mounted +repository from the host in production: - location @proxy_to_app { - proxy_set_header Host $http_host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_redirect off; - proxy_buffering off; - proxy_pass https://127.0.0.1:18000; - } +- `GIT_DATA_DIR` (default: `./aur.git/`) + - Contains path to an source repository directory on the Docker + host which is bind mounted in containers. - } +This feature means that all services which depend on the aurweb git repository +will make use of the provided `$GIT_DATA_DIR` source bind mounted from +host to container. + +#### cgit-(php|fastapi) + +In production, the `cgit-php` and `cgit-fastapi` services should be +configured to use the intended `clone-prefix`. + +- `CGIT_CLONE_PREFIX_PHP` + - Defaulted to `https://localhost:8443` +- `CGIT_CLONE_PREFIX_FASTAPI` + - Defaulted to `https://localhost:8444` + +#### git-bridge (Work In Progress) + +In production, hosts should be able to run a instance of sshd in +front of Docker, which allows standard authentication to the host +in addition to customized authentication and shell for a particular +user (`aur` by default). + +For this reason, we must provide a persistent endpoint in a Docker +service which bridges to `aurweb-git-auth`. This bridge must customize +output when it is invoking the script via the `git-bridge` RPC API. + +This way, the Docker host could configure sshd's `AuthorizedKeysCommand` +to contact this endpoint for auth negotiation, which would tunnel through +to `aurweb-git-serve` when successful. + +This feature is being worked on and has had success thus far, however, +it is not yet available and must be simplified as it stands. This section +will be updated when the feature comes to completion and can be used +by production. + +#### fastapi + +In production, the `fastapi` service should be configured to use +the `gunicorn` multi-proc, multi-threaded ASGI server frontend. + +- `FASTAPI_BACKEND` + - Options: `uvicorn`, `gunicorn`, `hypercorn` + - Defaulted to `uvicorn` +- `FASTAPI_WORKERS` + - Only applicable when `FASTAPI_BACKEND=gunicorn` + - Defaulted to `2` + +Tip: Our `php-fpm` service uses 5 workers (php-fpm's default). To +make proper performance comparisons, `fastapi` should be configured +for `gunicorn` with `5` workers. + +#### redis/memcached + +The `redis` and `memcached` servers are depended on by their respective +users: `fastapi` and `php-fpm`. There is no control over `memcached`, +but `redis` does provide a host-accessible port: + +- `redis` - port `16379` + +Users can run `redis-cli -p 16379` to gain access to the Docker +`redis` instance. + +#### nginx + +The `nginx` service should **not be used in production**. When running +aurweb in a production Docker instance, users should configure an +instance in front of Docker to use exposed services as backends: + +- `cgit-php` - port `13000` +- `php-fpm` - port `19000` +- `cgit-fastapi` - port `13001` +- `fastapi` - port `18000`