use Poetry to deal with deps and package install

As the new-age Python package manager, Poetry brings a lot
of good additions to the table. It allows us to more easily
deal with virtualenvs for the project and resolve dependencies.

As of this commit, `requirements.txt` is replaced by Poetry,
configured at `pyproject.toml`.

In Docker and GitLab, we currently use Poetry in a root fashion.
We should work toward purely using virtualenvs in Docker, but,
for now we'd like to move forward with other things. The project
can still be installed to a virtualenv and used on a user's system
through Poetry; it is just not yet doing so in Docker.

Modifications:

* docker/scripts/install-deps.sh
    * Remove python dependencies.
* conf/config.defaults
    * Script paths have been updated to use '/usr/bin'.
* docker/git-entrypoint.sh
    * Use '/usr/bin/aurweb-git-auth' instead of
      '/usr/local/bin/aurweb-git-auth'.

Additions:

* docker/scripts/install-python-deps.sh
    * A script used purely to install Python dependencies with Poetry.
      This has to be used within the aurweb project directory and
      requires system-wide dependencies are installed beforehand.
    * Also upgrades system-wide pip.

Signed-off-by: Kevin Morris <kevr@0cost.org>

.gitlab-ci.yml

@@ -11,8 +11,9 @@ variables:
   DB_HOST: localhost
 
 before_script:
+  - export PATH="$HOME/.poetry/bin:${PATH}"
   - ./docker/scripts/install-deps.sh
-  - pip install -r requirements.txt
+  - ./docker/scripts/install-python-deps.sh
   - useradd -U -d /aurweb -c 'AUR User' aur
   - ./docker/mariadb-entrypoint.sh
   - (cd '/usr' && /usr/bin/mysqld_safe --datadir='/var/lib/mysql') &
@@ -20,7 +21,6 @@ before_script:
   - ./docker/test-mysql-entrypoint.sh # Create mysql AUR_CONFIG.
   - ./docker/test-sqlite-entrypoint.sh # Create sqlite AUR_CONFIG.
   - make -C po all install
-  - python setup.py install --install-scripts=/usr/local/bin
   - python -m aurweb.initdb # Initialize MySQL tables.
   - AUR_CONFIG=conf/config.sqlite python -m aurweb.initdb
   - make -C test clean

Dockerfile

@@ -1,23 +1,25 @@
 FROM archlinux:base-devel
 
+ENV PATH="$HOME/.poetry/bin:${PATH}"
 ENV PYTHONPATH=/aurweb
 ENV AUR_CONFIG=conf/config
 
+# Install system-wide dependencies.
+COPY ./docker/scripts/install-deps.sh /install-deps.sh
+RUN /install-deps.sh
+
 # Copy Docker scripts
 COPY ./docker /docker
 COPY ./docker/scripts/*.sh /usr/local/bin/
 
-# Install system-wide dependencies.
-RUN /docker/scripts/install-deps.sh
-
 # Copy over all aurweb files.
 COPY . /aurweb
 
 # Working directory is aurweb root @ /aurweb.
 WORKDIR /aurweb
 
-# Install pip directories now that we have access to /aurweb.
+# Install Python dependencies.
-RUN pip install -r requirements.txt
+RUN /docker/scripts/install-python-deps.sh
 
 # Add our aur user.
 RUN useradd -U -d /aurweb -c 'AUR User' aur
@@ -27,6 +29,3 @@ RUN ln -sf /usr/share/zoneinfo/UTC /etc/localtime
 
 # Install translations.
 RUN make -C po all install
-
-# Install package and scripts.
-RUN python setup.py install --install-scripts=/usr/local/bin

INSTALL

@@ -45,22 +45,54 @@ read the instructions below.
    if the defaults file does not exist) and adjust the configuration (pay
    attention to disable_http_login, enable_maintenance and aur_location).
 
-4) Install Python modules and dependencies:
+4) Install dependencies.
 
-    # pacman -S python-mysql-connector python-pygit2 python-srcinfo python-sqlalchemy \
+4a) Install system-wide dependencies:
-                python-bleach python-markdown python-alembic hypercorn \
-                python-itsdangerous python-authlib python-httpx \
-                python-jinja python-aiofiles python-python-multipart \
-                python-requests hypercorn python-bcrypt python-email-validator \
-                python-lxml python-feedgen
-    # python3 setup.py install
 
-(FastAPI-Specific)
+    # pacman -S git gpgme cgit pyalpm python-srcinfo curl openssh \
+                uwsgi uwsgi-plugin-cgi php php-fpm
 
-    # pacman -S redis python-redis python-fakeredis python-orjson
+4b) Install Python dependencies via poetry (required):
+
+**NOTE** Users do not need to install pip or poetry dependencies system-wide.
+You may take advantage of Poetry's virtualenv integration to manage
+dependencies. This is merely a demonstration to show users how to without
+a virtualenv. In Docker and CI, we don't yet use a virtualenv.
+
+    ## Install Poetry dependencies system-wide, if not using a virtualenv.
+    # pacman -S python-pip
+
+    ## Ensure pip is upgraded. Poetry depends on it being up to date.
+    # pip install --upgrade pip
+
+    ## Install Poetry.
+    # curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
+    # export PATH="$HOME/.poetry/bin:${PATH}"
+
+    ## Use Poetry to install dependencies and the aurweb package.
+    # poetry lock     # Resolve dependencies
+    # poetry update   # Install/update dependencies
+    # poetry build    # Build the aurweb package
+    # poetry install  # Install the aurweb package and scripts
+
+When installing in a virtualenv, config.defaults must contain the correct
+absolute paths to aurweb scripts, which requires modification.
+
+4c) Setup FastAPI Redis cache (optional).
+
+First, install Redis and start its service.
+
+    # pacman -S redis
     # systemctl enable --now redis
 
-5) Create a new MySQL database and a user and import the aurweb SQL schema:
+Now that Redis is running, ensure that you configure aurweb to use
+the Redis cache by setting `cache = redis` in your AUR config.
+
+In `conf/config.defaults`, the `redis_address` configuration is set
+to `redis://localhost`. This can be set to point to any Redis server
+and will be used as long as `cache = redis`.
+
+5) Create a new database and a user and import the aurweb SQL schema:
 
     $ python -m aurweb.initdb
 

conf/config.defaults

@@ -34,8 +34,8 @@ commit_uri = /cgit/aur.git/commit/?h=%s&id=%s
 snapshot_uri = /cgit/aur.git/snapshot/%s.tar.gz
 enable-maintenance = 1
 maintenance-exceptions = 127.0.0.1
-render-comment-cmd = /usr/local/bin/aurweb-rendercomment
+render-comment-cmd = /usr/bin/aurweb-rendercomment
-localedir = /srv/http/aurweb/aur.git/web/locale/
+localedir = /srv/http/aurweb/web/locale/
 ; memcache, apc, or redis
 ; memcache/apc are supported in PHP, redis is supported in Python.
 cache = none
@@ -49,7 +49,7 @@ request_limit = 4000
 window_length = 86400
 
 [notifications]
-notify-cmd = /usr/local/bin/aurweb-notify
+notify-cmd = /usr/bin/aurweb-notify
 sendmail =
 smtp-server = localhost
 smtp-port = 25
@@ -68,7 +68,7 @@ RSA = SHA256:Ju+yWiMb/2O+gKQ9RJCDqvRg7l+Q95KFAeqM5sr6l2s
 [auth]
 valid-keytypes = ssh-rsa ssh-dss ecdsa-sha2-nistp256 ecdsa-sha2-nistp384 ecdsa-sha2-nistp521 ssh-ed25519 sk-ssh-ecdsa@openssh.com sk-ssh-ed25519@openssh.com
 username-regex = [a-zA-Z0-9]+[.\-_]?[a-zA-Z0-9]+$
-git-serve-cmd = /usr/local/bin/aurweb-git-serve
+git-serve-cmd = /usr/bin/aurweb-git-serve
 ssh-options = restrict
 
 [sso]
@@ -83,7 +83,7 @@ session_secret =
 repo-path = /srv/http/aurweb/aur.git/
 repo-regex = [a-z0-9][a-z0-9.+_-]*$
 git-shell-cmd = /usr/bin/git-shell
-git-update-cmd = /usr/local/bin/aurweb-git-update
+git-update-cmd = /usr/bin/aurweb-git-update
 ssh-cmdline = ssh aur@aur.archlinux.org
 
 [update]

docker/git-entrypoint.sh

@@ -25,7 +25,7 @@ chmod 755 /app
 cat >> $AUTH_SCRIPT << EOF
 #!/usr/bin/env bash
 export AUR_CONFIG="$AUR_CONFIG"
-exec /usr/local/bin/aurweb-git-auth "\$@"
+exec /usr/bin/aurweb-git-auth "\$@"
 EOF
 chmod 755 $AUTH_SCRIPT
 

docker/scripts/install-deps.sh

@@ -5,12 +5,12 @@
 set -eou pipefail
 
 pacman -Syu --noconfirm --noprogressbar \
-    --cachedir .pkg-cache git gpgme \
+    --cachedir .pkg-cache git gpgme nginx redis openssh \
-      nginx redis openssh \
+      mariadb mariadb-libs cgit uwsgi uwsgi-plugin-cgi \
-      mariadb mariadb-libs \
+      php php-fpm memcached php-memcached python-pip pyalpm \
-      cgit uwsgi uwsgi-plugin-cgi \
+      python-srcinfo curl
-      php php-fpm \
+
-      memcached php-memcached \
+# https://python-poetry.org/docs/ Installation section.
-      python-pip pyalpm python-srcinfo
+curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
 
 exec "$@"

docker/scripts/install-python-deps.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+set -eou pipefail
+
+# Upgrade PIP; Arch Linux's version of pip is outdated for Poetry.
+pip install --upgrade pip
+
+# Install the aurweb package and deps system-wide via poetry.
+poetry config virtualenvs.create false
+poetry lock
+poetry update
+poetry build
+poetry install --no-interaction --no-ansi
+
+exec "$@"

pyproject.toml

@@ -0,0 +1,100 @@
+# Poetry build configuration for the aurweb project.
+#
+# Dependencies:
+#   * python >= 3.9
+#   * pip
+#       * poetry
+#       * poetry-dynamic-versioning
+#
+[tool.poetry]
+name = "aurweb"
+version = "5.0.0" # Updated via poetry-dynamic-versioning
+license = "GPL-2.0-only"
+description = "Source code for the Arch User Repository's website"
+homepage = "https://aur.archlinux.org"
+repository = "https://gitlab.archlinux.org/archlinux/aurweb"
+documentation = "https://gitlab.archlinux.org/archlinux/aurweb/-/blob/master/README.md"
+keywords = ["aurweb", "aur", "Arch", "Linux"]
+authors = [
+    "Lucas Fleischer <lfleischer@archlinux.org>",
+    "Eli Schwartz <eschwartz@archlinux.org>",
+    "Kevin Morris <kevr@0cost.org>"
+]
+maintainers = [
+    "Eli Schwartz <eschwartz@archlinux.org>"
+]
+packages = [
+    { include = "aurweb" }
+]
+
+[tool.poetry-dynamic-versioning]
+enable = true
+vcs = "git"
+
+[build-system]
+requires = ["poetry>=1.1.8", "poetry-dynamic-versioning"]
+build-backend = "poetry.masonry.api"
+
+[tool.poetry.urls]
+"Repository" = "https://gitlab.archlinux.org/archlinux/aurweb"
+"Bug Tracker" = "https://gitlab.archlinux.org/archlinux/aurweb/-/issues"
+"Development Mailing List" = "https://lists.archlinux.org/listinfo/aur-dev"
+"General Mailing List" = "https://lists.archlinux.org/listinfo/aur-general"
+"Request Mailing List" = "https://lists.archlinux.org/listinfo/aur-requests"
+
+[tool.poetry.dependencies]
+# poetry-dynamic-versioning is used to produce tool.poetry.version
+# based on git tags.
+poetry-dynamic-versioning = { version = "0.13.1", python = "^3.9" }
+
+# General
+authlib = { version = "0.15.2", python = "^3.9" }
+aiofiles = { version = "0.7.0", python = "^3.9" }
+asgiref = { version = "3.4.1", python = "^3.9" }
+bcrypt = { version = "3.2.0", python = "^3.9" }
+bleach = { version = "3.3.1", python = "^3.9" }
+email-validator = { version = "1.1.3", python = "^3.9" }
+fakeredis = { version = "1.6.0", python = "^3.9" }
+fastapi = { version = "0.66.0", python = "^3.9" }
+feedgen = { version = "0.9.0", python = "^3.9" }
+httpx = { version = "0.18.2", python = "^3.9" }
+hypercorn = { version = "0.11.2", python = "^3.9" }
+itsdangerous = { version = "2.0.1", python = "^3.9" }
+jinja2 = { version = "3.0.1", python = "^3.9" }
+lxml = { version = "4.6.3", python = "^3.9" }
+markdown = { version = "3.3.4", python = "^3.9" }
+orjson = { version = "3.6.3", python = "^3.9" }
+protobuf = { version = "3.17.3", python = "^3.9" }
+pygit2 = { version = "1.6.1", python = "^3.9" }
+python-multipart = { version = "0.0.5", python = "^3.9" }
+redis = { version = "3.5.3", python = "^3.9" }
+requests = { version = "2.26.0", python = "^3.9" }
+werkzeug = { version = "2.0.1", python = "^3.9" }
+
+# SQL
+alembic = { version = "1.6.5", python = "^3.9" }
+sqlalchemy = { version = "1.3.23", python = "^3.9" }
+mysqlclient = { version = "2.0.3", python = "^3.9" }
+
+[tool.poetry.dev-dependencies]
+flake8 = { version = "3.9.2", python = "^3.9" }
+isort = { version = "5.9.3", python = "^3.9" }
+coverage = { version = "5.5", python = "^3.9" }
+pytest = { version = "6.2.4", python = "^3.9" }
+pytest-asyncio = { version = "0.15.1", python = "^3.9" }
+pytest-cov = { version = "2.12.1", python = "^3.9" }
+pytest-tap = { version = "3.2", python = "^3.9" }
+uvicorn = { version = "0.15.0", python = "^3.9" }
+
+[tool.poetry.scripts]
+aurweb-git-auth = "aurweb.git.auth:main"
+aurweb-git-serve = "aurweb.git.serve:main"
+aurweb-git-update = "aurweb.git.update:main"
+aurweb-aurblup = "aurweb.scripts.aurblup:main"
+aurweb-mkpkglists = "aurweb.scripts.mkpkglists:main"
+aurweb-notify = "aurweb.scripts.notify:main"
+aurweb-pkgmaint = "aurweb.scripts.pkgmaint:main"
+aurweb-popupdate = "aurweb.scripts.popupdate:main"
+aurweb-rendercomment = "aurweb.scripts.rendercomment:main"
+aurweb-tuvotereminder = "aurweb.scripts.tuvotereminder:main"
+aurweb-usermaint = "aurweb.scripts.usermaint:main"

requirements.txt

@@ -1,36 +0,0 @@
-# General
-authlib==0.15.2
-aiofiles==0.7.0
-asgiref==3.4.1
-bcrypt==3.2.0
-bleach==3.3.1
-coverage==5.5
-email-validator==1.1.3
-fakeredis==1.6.0
-fastapi==0.66.0
-feedgen==0.9.0
-flake8==3.9.2
-httpx==0.18.2
-hypercorn==0.11.2
-isort==5.9.3
-itsdangerous==2.0.1
-jinja2==3.0.1
-lxml==4.6.3
-markdown==3.3.4
-orjson==3.6.3
-protobuf==3.17.3
-pygit2==1.6.1
-pytest==6.2.4
-pytest-asyncio==0.15.1
-pytest-cov==2.12.1
-pytest-tap==3.2
-python-multipart==0.0.5
-redis==3.5.3
-requests==2.26.0
-uvicorn==0.15.0
-werkzeug==2.0.1
-
-# SQL
-alembic==1.6.5
-sqlalchemy==1.3.23
-mysqlclient==2.0.3

setup.py

@@ -1,36 +0,0 @@
-import re
-import sys
-
-from setuptools import find_packages, setup
-
-version = None
-with open('web/lib/version.inc.php', 'r') as f:
-    for line in f.readlines():
-        match = re.match(r'^define\("AURWEB_VERSION", "v([0-9.]+)"\);$', line)
-        if match:
-            version = match.group(1)
-
-if not version:
-    sys.stderr.write('error: Failed to parse version file!')
-    sys.exit(1)
-
-setup(
-    name="aurweb",
-    version=version,
-    packages=find_packages(),
-    entry_points={
-        'console_scripts': [
-            'aurweb-git-auth = aurweb.git.auth:main',
-            'aurweb-git-serve = aurweb.git.serve:main',
-            'aurweb-git-update = aurweb.git.update:main',
-            'aurweb-aurblup = aurweb.scripts.aurblup:main',
-            'aurweb-mkpkglists = aurweb.scripts.mkpkglists:main',
-            'aurweb-notify = aurweb.scripts.notify:main',
-            'aurweb-pkgmaint = aurweb.scripts.pkgmaint:main',
-            'aurweb-popupdate = aurweb.scripts.popupdate:main',
-            'aurweb-rendercomment = aurweb.scripts.rendercomment:main',
-            'aurweb-tuvotereminder = aurweb.scripts.tuvotereminder:main',
-            'aurweb-usermaint = aurweb.scripts.usermaint:main',
-        ],
-    },
-)