external/aurweb
0
0
Fork 0
mirror of https://gitlab.archlinux.org/archlinux/aurweb.git synced 2025-02-03 10:43:03 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
Page: Docker
Pages
Docker Testing Docker Home Python Port Testing Guide _sidebar aur dev.archlinux.org Testing
12 Docker
Kevin Morris edited this page 2021-11-24 22:38:41 +00:00
Table of Contents

AURWeb's Docker infrastructure is composed of a Dockerfile and docker-compose.yml Services.

Contents

Status

Branches containing Docker features are located in the pipeline/coverage status table below:

pu (fastapi)
pipeline coverage

Services

Service Host Bind Purpose
ca Self-signed CA generation
memcached Live memcached service
redis 127.0.0.1:16379 Live Redis service
mariadb 127.0.0.1:13306 Live MariaDB service
mariadb_test 127.0.0.1:13307 Live test MariaDB service
mariadb_init mariadb initialization
git 0.0.0.0:2222 AUR Git SSH interface
smartgit HTTP Git frontend
cgit-php 127.0.0.1:13000 CGit backend (PHP)
cgit-fastapi 127.0.0.1:13001 CGit backend (FastAPI)
php-fpm 127.0.0.1:19000 Live php-fpm service
fastapi 127.0.0.1:18000 Live $FASTAPI_BACKEND service
nginx 127.0.0.1:8443 Live nginx frontend to php-fpm
nginx 127.0.0.1:8444 Live nginx frontend to fastapi
sharness Run sharness test suites
pytest-mysql Run pytest suites with mariadb
pytest-sqlite Run pytest suites with sqlite
test Run all tests and linters

Profiles

We have two kinds of profiles:

Profile: default

This profile provides support for the following environment variables:

  • FASTAPI_BACKEND
    • Default: uvicorn
    • Choices: uvicorn, hypercorn, gunicorn
    • Controls the backend used for the fastapi service. In production, gunicorn is recommended.
  • FASTAPI_WORKERS
    • Default: 2
    • Only used when FASTAPI_BACKEND=gunicorn; controls the number of workers used.
  • AURWEB_PHP_PREFIX
    • Default: https://localhost:8443
    • Used to populate aur_location in the php-fpm service.
    • Used to populate git_clone_uri_anon in the php-fpm service.
  • AURWEB_FASTAPI_PREFIX
    • Default: https://localhost:8444
    • Used to populate aur_location in the fastapi service.
    • Used to populate git_clone_uri_anon in the fastapi service.
  • AURWEB_SSHD_PREFIX
    • Default: ssh://aur@localhost:2222
    • Used to populate git_clone_uri_priv.
  • TEST_RECURSION_LIMIT
    • Default: 10000
    • This should be left alone. It is used to configure the recursion limit used for tests.
  • COMMIT_HASH

Profile: aur-dev

To use this profile, execute docker-compose with the following:

$ docker-compose -f docker-compose.yml -f docker-compose.aur-dev.yml ...

This profile provides support for the following environment variables, along with any inherited from default:

  • MARIADB_SOCKET_DIR
    • Default: /var/run/mysqld
    • Bind mount a host mariadb instance socket directory.
  • GIT_DATA_DIR
    • Default: ./aur.git
    • Bind mount a Git repository on the host filesystem.

This profile does not bind mount aurweb repository components for live reloading like the default profile does. This means that you'll have to rebuild the Docker image for any local filesystem changes to take effect.

Getting Started

To get started, you need to build the aurweb:latest Docker image by issuing the following command:

$ docker-compose build

You can then docker-compose (up|run) any one of the services:

$ docker-compose run test
$ docker-compose up -d nginx

Users will notice initially starting up a service can take some time, especially one with many dependencies. After the initial startup, however, users can run services again without much initialization:

$ docker-compose run test  # First run with no up'd services, takes a while.
$ docker-compose run test  # Starts instantly.

Development

During development, users may wish to run instances of FastAPI and PHP side by side. For this, users can depend on the nginx service, which hosts PHP at https://localhost:8443 and FastAPI at https://localhost:8444.

The ca service will generate a ca.root.pem file in $aurweb_path/cache, which can be imported into your browser as a Certificate Authority. This root is the issuer to the nginx service's certificates mentioned above.

Otherwise, users may also wish to run tests. Users can easily run tests by taking advantage of the pytest-mysql, pytest-sqlite, sharness and test services.

For an all in one testing strategy, the test service should be preferred, as it also runs linter hecks which the aurweb project requires: flake8 and isort. This test is identical to GitLab CI's tests.

Our test suites also provide coverage data via a shared volume directory at $aurwebdir/cache/. After tests are complete, users can copy coverage data output by Docker into their local setup by executing ./util/fix-coverage ./cache/.coverage. After this is done, users can continue on with standard coverage execution:

$ docker-compose run pytest-mysql

Test services which provide coverage data: pytest-mysql and test.

Production

For production, the docker-compose.aur-dev.yml profile config should be used in unison with the standard docker-compose.yml:

$ docker-compose -f docker-compose.yml -f docker-compose.aur-dev.yml up -d <service>

This provides service overrides which mainly affect volume behavior. In production, volumes will not be shared with docker from the host.

The following services in particular are useful for production:

  • mariadb
  • git
  • smartgit
  • cgit-fastapi (fastapi depends on)
  • redis (fastapi depends on)
  • fastapi
  • cgit-php (php-fpm depends on)
  • memcached (php-fpm depends on)
  • php-fpm

Note: To see exposed ports on the host, take a look at Services at the beginning of this document.

Production Configuration

All Docker-configurable environment variables are defaulted within the .env file.

mariadb

The mariadb service requires a bind-mounted socket directory from the host, containing mysqld.sock in production:

  • MARIADB_SOCKET_DIR: must contain mysqld.sock
    • Contains path to mariadb's socket directory on the host.

This feature means that the database used by all Docker services are driven by the host. Docker containers will contact the host's mysqld.sock for database usage in general.

git

The git service (and services which depend on it) requires a bind-mounted repository from the host in production:

  • GIT_DATA_DIR (default: ./aur.git/)
    • Contains path to an source repository directory on the Docker host which is bind mounted in containers.

This feature means that all services which depend on the aurweb git repository will make use of the provided $GIT_DATA_DIR source bind mounted from host to container.

cgit-(php|fastapi)

In production, the cgit-php and cgit-fastapi services should be configured to use the intended clone-prefix.

  • AURWEB_PHP_PREFIX
    • Defaulted to https://localhost:8443
  • AURWEB_FASTAPI_PREFIX
    • Defaulted to https://localhost:8444
  • AURWEB_SSHD_PREFIX
    • Defaulted to ssh://aur@localhost:2222

fastapi

In production, the fastapi service should be configured to use the gunicorn multi-proc, multi-threaded ASGI server frontend.

  • FASTAPI_BACKEND
    • Options: uvicorn, gunicorn, hypercorn
    • Defaulted to uvicorn
  • FASTAPI_WORKERS
    • Only applicable when FASTAPI_BACKEND=gunicorn
    • Defaulted to 2

Tip: Our php-fpm service uses 5 workers (php-fpm's default). To make proper performance comparisons, fastapi should be configured for gunicorn with 5 workers.

redis/memcached

The redis and memcached servers are depended on by their respective users: fastapi and php-fpm. There is no control over memcached, but redis does provide a host-accessible port:

  • redis - port 16379

Users can run redis-cli -p 16379 to gain access to the Docker redis instance.

nginx

The nginx service should not be used in production. When running aurweb in a production Docker instance, users should configure an instance in front of Docker to use exposed services as backends:

  • cgit-php @ 127.0.0.1:13000
  • php-fpm @ 127.0.0.1:19000
  • cgit-fastapi @ 127.0.0.1:13001
  • fastapi @ 127.0.0.1:18000