mirror of
https://gitlab.archlinux.org/archlinux/aurweb.git
synced 2025-02-03 10:43:03 +01:00
use Poetry to deal with deps and package install
As the new-age Python package manager, Poetry brings a lot
of good additions to the table. It allows us to more easily
deal with virtualenvs for the project and resolve dependencies.
As of this commit, `requirements.txt` is replaced by Poetry,
configured at `pyproject.toml`.
In Docker and GitLab, we currently use Poetry in a root fashion.
We should work toward purely using virtualenvs in Docker, but,
for now we'd like to move forward with other things. The project
can still be installed to a virtualenv and used on a user's system
through Poetry; it is just not yet doing so in Docker.
Modifications:
* docker/scripts/install-deps.sh
* Remove python dependencies.
* conf/config.defaults
* Script paths have been updated to use '/usr/bin'.
* docker/git-entrypoint.sh
* Use '/usr/bin/aurweb-git-auth' instead of
'/usr/local/bin/aurweb-git-auth'.
Additions:
* docker/scripts/install-python-deps.sh
* A script used purely to install Python dependencies with Poetry.
This has to be used within the aurweb project directory and
requires system-wide dependencies are installed beforehand.
* Also upgrades system-wide pip.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit is contained in:
Kevin Morris
parent
5e6f0cb8d7
commit
2f9994807b
11 changed files with 1756 additions and 106 deletions
|
|
@ -11,8 +11,9 @@ variables:
|
|||
DB_HOST: localhost
|
||||
|
||||
before_script:
|
||||
- export PATH="$HOME/.poetry/bin:${PATH}"
|
||||
- ./docker/scripts/install-deps.sh
|
||||
- pip install -r requirements.txt
|
||||
- ./docker/scripts/install-python-deps.sh
|
||||
- useradd -U -d /aurweb -c 'AUR User' aur
|
||||
- ./docker/mariadb-entrypoint.sh
|
||||
- (cd '/usr' && /usr/bin/mysqld_safe --datadir='/var/lib/mysql') &
|
||||
|
|
@ -20,7 +21,6 @@ before_script:
|
|||
- ./docker/test-mysql-entrypoint.sh # Create mysql AUR_CONFIG.
|
||||
- ./docker/test-sqlite-entrypoint.sh # Create sqlite AUR_CONFIG.
|
||||
- make -C po all install
|
||||
- python setup.py install --install-scripts=/usr/local/bin
|
||||
- python -m aurweb.initdb # Initialize MySQL tables.
|
||||
- AUR_CONFIG=conf/config.sqlite python -m aurweb.initdb
|
||||
- make -C test clean
|
||||
|
|
|
|||
15
Dockerfile
15
Dockerfile
|
|
@ -1,23 +1,25 @@
|
|||
FROM archlinux:base-devel
|
||||
|
||||
ENV PATH="$HOME/.poetry/bin:${PATH}"
|
||||
ENV PYTHONPATH=/aurweb
|
||||
ENV AUR_CONFIG=conf/config
|
||||
|
||||
# Install system-wide dependencies.
|
||||
COPY ./docker/scripts/install-deps.sh /install-deps.sh
|
||||
RUN /install-deps.sh
|
||||
|
||||
# Copy Docker scripts
|
||||
COPY ./docker /docker
|
||||
COPY ./docker/scripts/*.sh /usr/local/bin/
|
||||
|
||||
# Install system-wide dependencies.
|
||||
RUN /docker/scripts/install-deps.sh
|
||||
|
||||
# Copy over all aurweb files.
|
||||
COPY . /aurweb
|
||||
|
||||
# Working directory is aurweb root @ /aurweb.
|
||||
WORKDIR /aurweb
|
||||
|
||||
# Install pip directories now that we have access to /aurweb.
|
||||
RUN pip install -r requirements.txt
|
||||
# Install Python dependencies.
|
||||
RUN /docker/scripts/install-python-deps.sh
|
||||
|
||||
# Add our aur user.
|
||||
RUN useradd -U -d /aurweb -c 'AUR User' aur
|
||||
|
|
@ -27,6 +29,3 @@ RUN ln -sf /usr/share/zoneinfo/UTC /etc/localtime
|
|||
|
||||
# Install translations.
|
||||
RUN make -C po all install
|
||||
|
||||
# Install package and scripts.
|
||||
RUN python setup.py install --install-scripts=/usr/local/bin
|
||||
|
|
|
|||
54
INSTALL
54
INSTALL
|
|
@ -45,22 +45,54 @@ read the instructions below.
|
|||
if the defaults file does not exist) and adjust the configuration (pay
|
||||
attention to disable_http_login, enable_maintenance and aur_location).
|
||||
|
||||
4) Install Python modules and dependencies:
|
||||
4) Install dependencies.
|
||||
|
||||
# pacman -S python-mysql-connector python-pygit2 python-srcinfo python-sqlalchemy \
|
||||
python-bleach python-markdown python-alembic hypercorn \
|
||||
python-itsdangerous python-authlib python-httpx \
|
||||
python-jinja python-aiofiles python-python-multipart \
|
||||
python-requests hypercorn python-bcrypt python-email-validator \
|
||||
python-lxml python-feedgen
|
||||
# python3 setup.py install
|
||||
4a) Install system-wide dependencies:
|
||||
|
||||
(FastAPI-Specific)
|
||||
# pacman -S git gpgme cgit pyalpm python-srcinfo curl openssh \
|
||||
uwsgi uwsgi-plugin-cgi php php-fpm
|
||||
|
||||
# pacman -S redis python-redis python-fakeredis python-orjson
|
||||
4b) Install Python dependencies via poetry (required):
|
||||
|
||||
**NOTE** Users do not need to install pip or poetry dependencies system-wide.
|
||||
You may take advantage of Poetry's virtualenv integration to manage
|
||||
dependencies. This is merely a demonstration to show users how to without
|
||||
a virtualenv. In Docker and CI, we don't yet use a virtualenv.
|
||||
|
||||
## Install Poetry dependencies system-wide, if not using a virtualenv.
|
||||
# pacman -S python-pip
|
||||
|
||||
## Ensure pip is upgraded. Poetry depends on it being up to date.
|
||||
# pip install --upgrade pip
|
||||
|
||||
## Install Poetry.
|
||||
# curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
|
||||
# export PATH="$HOME/.poetry/bin:${PATH}"
|
||||
|
||||
## Use Poetry to install dependencies and the aurweb package.
|
||||
# poetry lock # Resolve dependencies
|
||||
# poetry update # Install/update dependencies
|
||||
# poetry build # Build the aurweb package
|
||||
# poetry install # Install the aurweb package and scripts
|
||||
|
||||
When installing in a virtualenv, config.defaults must contain the correct
|
||||
absolute paths to aurweb scripts, which requires modification.
|
||||
|
||||
4c) Setup FastAPI Redis cache (optional).
|
||||
|
||||
First, install Redis and start its service.
|
||||
|
||||
# pacman -S redis
|
||||
# systemctl enable --now redis
|
||||
|
||||
5) Create a new MySQL database and a user and import the aurweb SQL schema:
|
||||
Now that Redis is running, ensure that you configure aurweb to use
|
||||
the Redis cache by setting `cache = redis` in your AUR config.
|
||||
|
||||
In `conf/config.defaults`, the `redis_address` configuration is set
|
||||
to `redis://localhost`. This can be set to point to any Redis server
|
||||
and will be used as long as `cache = redis`.
|
||||
|
||||
5) Create a new database and a user and import the aurweb SQL schema:
|
||||
|
||||
$ python -m aurweb.initdb
|
||||
|
||||
|
|
|
|||
|
|
@ -34,8 +34,8 @@ commit_uri = /cgit/aur.git/commit/?h=%s&id=%s
|
|||
snapshot_uri = /cgit/aur.git/snapshot/%s.tar.gz
|
||||
enable-maintenance = 1
|
||||
maintenance-exceptions = 127.0.0.1
|
||||
render-comment-cmd = /usr/local/bin/aurweb-rendercomment
|
||||
localedir = /srv/http/aurweb/aur.git/web/locale/
|
||||
render-comment-cmd = /usr/bin/aurweb-rendercomment
|
||||
localedir = /srv/http/aurweb/web/locale/
|
||||
; memcache, apc, or redis
|
||||
; memcache/apc are supported in PHP, redis is supported in Python.
|
||||
cache = none
|
||||
|
|
@ -49,7 +49,7 @@ request_limit = 4000
|
|||
window_length = 86400
|
||||
|
||||
[notifications]
|
||||
notify-cmd = /usr/local/bin/aurweb-notify
|
||||
notify-cmd = /usr/bin/aurweb-notify
|
||||
sendmail =
|
||||
smtp-server = localhost
|
||||
smtp-port = 25
|
||||
|
|
@ -68,7 +68,7 @@ RSA = SHA256:Ju+yWiMb/2O+gKQ9RJCDqvRg7l+Q95KFAeqM5sr6l2s
|
|||
[auth]
|
||||
valid-keytypes = ssh-rsa ssh-dss ecdsa-sha2-nistp256 ecdsa-sha2-nistp384 ecdsa-sha2-nistp521 ssh-ed25519 sk-ssh-ecdsa@openssh.com sk-ssh-ed25519@openssh.com
|
||||
username-regex = [a-zA-Z0-9]+[.\-_]?[a-zA-Z0-9]+$
|
||||
git-serve-cmd = /usr/local/bin/aurweb-git-serve
|
||||
git-serve-cmd = /usr/bin/aurweb-git-serve
|
||||
ssh-options = restrict
|
||||
|
||||
[sso]
|
||||
|
|
@ -83,7 +83,7 @@ session_secret =
|
|||
repo-path = /srv/http/aurweb/aur.git/
|
||||
repo-regex = [a-z0-9][a-z0-9.+_-]*$
|
||||
git-shell-cmd = /usr/bin/git-shell
|
||||
git-update-cmd = /usr/local/bin/aurweb-git-update
|
||||
git-update-cmd = /usr/bin/aurweb-git-update
|
||||
ssh-cmdline = ssh aur@aur.archlinux.org
|
||||
|
||||
[update]
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ chmod 755 /app
|
|||
cat >> $AUTH_SCRIPT << EOF
|
||||
#!/usr/bin/env bash
|
||||
export AUR_CONFIG="$AUR_CONFIG"
|
||||
exec /usr/local/bin/aurweb-git-auth "\$@"
|
||||
exec /usr/bin/aurweb-git-auth "\$@"
|
||||
EOF
|
||||
chmod 755 $AUTH_SCRIPT
|
||||
|
||||
|
|
|
|||
|
|
@ -5,12 +5,12 @@
|
|||
set -eou pipefail
|
||||
|
||||
pacman -Syu --noconfirm --noprogressbar \
|
||||
--cachedir .pkg-cache git gpgme \
|
||||
nginx redis openssh \
|
||||
mariadb mariadb-libs \
|
||||
cgit uwsgi uwsgi-plugin-cgi \
|
||||
php php-fpm \
|
||||
memcached php-memcached \
|
||||
python-pip pyalpm python-srcinfo
|
||||
--cachedir .pkg-cache git gpgme nginx redis openssh \
|
||||
mariadb mariadb-libs cgit uwsgi uwsgi-plugin-cgi \
|
||||
php php-fpm memcached php-memcached python-pip pyalpm \
|
||||
python-srcinfo curl
|
||||
|
||||
# https://python-poetry.org/docs/ Installation section.
|
||||
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
|
||||
|
||||
exec "$@"
|
||||
|
|
|
|||
14
docker/scripts/install-python-deps.sh
Executable file
14
docker/scripts/install-python-deps.sh
Executable file
|
|
@ -0,0 +1,14 @@
|
|||
#!/bin/bash
|
||||
set -eou pipefail
|
||||
|
||||
# Upgrade PIP; Arch Linux's version of pip is outdated for Poetry.
|
||||
pip install --upgrade pip
|
||||
|
||||
# Install the aurweb package and deps system-wide via poetry.
|
||||
poetry config virtualenvs.create false
|
||||
poetry lock
|
||||
poetry update
|
||||
poetry build
|
||||
poetry install --no-interaction --no-ansi
|
||||
|
||||
exec "$@"
|
||||
1577
poetry.lock
generated
Normal file
1577
poetry.lock
generated
Normal file
File diff suppressed because it is too large
Load diff
100
pyproject.toml
Normal file
100
pyproject.toml
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
# Poetry build configuration for the aurweb project.
|
||||
#
|
||||
# Dependencies:
|
||||
# * python >= 3.9
|
||||
# * pip
|
||||
# * poetry
|
||||
# * poetry-dynamic-versioning
|
||||
#
|
||||
[tool.poetry]
|
||||
name = "aurweb"
|
||||
version = "5.0.0" # Updated via poetry-dynamic-versioning
|
||||
license = "GPL-2.0-only"
|
||||
description = "Source code for the Arch User Repository's website"
|
||||
homepage = "https://aur.archlinux.org"
|
||||
repository = "https://gitlab.archlinux.org/archlinux/aurweb"
|
||||
documentation = "https://gitlab.archlinux.org/archlinux/aurweb/-/blob/master/README.md"
|
||||
keywords = ["aurweb", "aur", "Arch", "Linux"]
|
||||
authors = [
|
||||
"Lucas Fleischer <lfleischer@archlinux.org>",
|
||||
"Eli Schwartz <eschwartz@archlinux.org>",
|
||||
"Kevin Morris <kevr@0cost.org>"
|
||||
]
|
||||
maintainers = [
|
||||
"Eli Schwartz <eschwartz@archlinux.org>"
|
||||
]
|
||||
packages = [
|
||||
{ include = "aurweb" }
|
||||
]
|
||||
|
||||
[tool.poetry-dynamic-versioning]
|
||||
enable = true
|
||||
vcs = "git"
|
||||
|
||||
[build-system]
|
||||
requires = ["poetry>=1.1.8", "poetry-dynamic-versioning"]
|
||||
build-backend = "poetry.masonry.api"
|
||||
|
||||
[tool.poetry.urls]
|
||||
"Repository" = "https://gitlab.archlinux.org/archlinux/aurweb"
|
||||
"Bug Tracker" = "https://gitlab.archlinux.org/archlinux/aurweb/-/issues"
|
||||
"Development Mailing List" = "https://lists.archlinux.org/listinfo/aur-dev"
|
||||
"General Mailing List" = "https://lists.archlinux.org/listinfo/aur-general"
|
||||
"Request Mailing List" = "https://lists.archlinux.org/listinfo/aur-requests"
|
||||
|
||||
[tool.poetry.dependencies]
|
||||
# poetry-dynamic-versioning is used to produce tool.poetry.version
|
||||
# based on git tags.
|
||||
poetry-dynamic-versioning = { version = "0.13.1", python = "^3.9" }
|
||||
|
||||
# General
|
||||
authlib = { version = "0.15.2", python = "^3.9" }
|
||||
aiofiles = { version = "0.7.0", python = "^3.9" }
|
||||
asgiref = { version = "3.4.1", python = "^3.9" }
|
||||
bcrypt = { version = "3.2.0", python = "^3.9" }
|
||||
bleach = { version = "3.3.1", python = "^3.9" }
|
||||
email-validator = { version = "1.1.3", python = "^3.9" }
|
||||
fakeredis = { version = "1.6.0", python = "^3.9" }
|
||||
fastapi = { version = "0.66.0", python = "^3.9" }
|
||||
feedgen = { version = "0.9.0", python = "^3.9" }
|
||||
httpx = { version = "0.18.2", python = "^3.9" }
|
||||
hypercorn = { version = "0.11.2", python = "^3.9" }
|
||||
itsdangerous = { version = "2.0.1", python = "^3.9" }
|
||||
jinja2 = { version = "3.0.1", python = "^3.9" }
|
||||
lxml = { version = "4.6.3", python = "^3.9" }
|
||||
markdown = { version = "3.3.4", python = "^3.9" }
|
||||
orjson = { version = "3.6.3", python = "^3.9" }
|
||||
protobuf = { version = "3.17.3", python = "^3.9" }
|
||||
pygit2 = { version = "1.6.1", python = "^3.9" }
|
||||
python-multipart = { version = "0.0.5", python = "^3.9" }
|
||||
redis = { version = "3.5.3", python = "^3.9" }
|
||||
requests = { version = "2.26.0", python = "^3.9" }
|
||||
werkzeug = { version = "2.0.1", python = "^3.9" }
|
||||
|
||||
# SQL
|
||||
alembic = { version = "1.6.5", python = "^3.9" }
|
||||
sqlalchemy = { version = "1.3.23", python = "^3.9" }
|
||||
mysqlclient = { version = "2.0.3", python = "^3.9" }
|
||||
|
||||
[tool.poetry.dev-dependencies]
|
||||
flake8 = { version = "3.9.2", python = "^3.9" }
|
||||
isort = { version = "5.9.3", python = "^3.9" }
|
||||
coverage = { version = "5.5", python = "^3.9" }
|
||||
pytest = { version = "6.2.4", python = "^3.9" }
|
||||
pytest-asyncio = { version = "0.15.1", python = "^3.9" }
|
||||
pytest-cov = { version = "2.12.1", python = "^3.9" }
|
||||
pytest-tap = { version = "3.2", python = "^3.9" }
|
||||
uvicorn = { version = "0.15.0", python = "^3.9" }
|
||||
|
||||
[tool.poetry.scripts]
|
||||
aurweb-git-auth = "aurweb.git.auth:main"
|
||||
aurweb-git-serve = "aurweb.git.serve:main"
|
||||
aurweb-git-update = "aurweb.git.update:main"
|
||||
aurweb-aurblup = "aurweb.scripts.aurblup:main"
|
||||
aurweb-mkpkglists = "aurweb.scripts.mkpkglists:main"
|
||||
aurweb-notify = "aurweb.scripts.notify:main"
|
||||
aurweb-pkgmaint = "aurweb.scripts.pkgmaint:main"
|
||||
aurweb-popupdate = "aurweb.scripts.popupdate:main"
|
||||
aurweb-rendercomment = "aurweb.scripts.rendercomment:main"
|
||||
aurweb-tuvotereminder = "aurweb.scripts.tuvotereminder:main"
|
||||
aurweb-usermaint = "aurweb.scripts.usermaint:main"
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
# General
|
||||
authlib==0.15.2
|
||||
aiofiles==0.7.0
|
||||
asgiref==3.4.1
|
||||
bcrypt==3.2.0
|
||||
bleach==3.3.1
|
||||
coverage==5.5
|
||||
email-validator==1.1.3
|
||||
fakeredis==1.6.0
|
||||
fastapi==0.66.0
|
||||
feedgen==0.9.0
|
||||
flake8==3.9.2
|
||||
httpx==0.18.2
|
||||
hypercorn==0.11.2
|
||||
isort==5.9.3
|
||||
itsdangerous==2.0.1
|
||||
jinja2==3.0.1
|
||||
lxml==4.6.3
|
||||
markdown==3.3.4
|
||||
orjson==3.6.3
|
||||
protobuf==3.17.3
|
||||
pygit2==1.6.1
|
||||
pytest==6.2.4
|
||||
pytest-asyncio==0.15.1
|
||||
pytest-cov==2.12.1
|
||||
pytest-tap==3.2
|
||||
python-multipart==0.0.5
|
||||
redis==3.5.3
|
||||
requests==2.26.0
|
||||
uvicorn==0.15.0
|
||||
werkzeug==2.0.1
|
||||
|
||||
# SQL
|
||||
alembic==1.6.5
|
||||
sqlalchemy==1.3.23
|
||||
mysqlclient==2.0.3
|
||||
36
setup.py
36
setup.py
|
|
@ -1,36 +0,0 @@
|
|||
import re
|
||||
import sys
|
||||
|
||||
from setuptools import find_packages, setup
|
||||
|
||||
version = None
|
||||
with open('web/lib/version.inc.php', 'r') as f:
|
||||
for line in f.readlines():
|
||||
match = re.match(r'^define\("AURWEB_VERSION", "v([0-9.]+)"\);$', line)
|
||||
if match:
|
||||
version = match.group(1)
|
||||
|
||||
if not version:
|
||||
sys.stderr.write('error: Failed to parse version file!')
|
||||
sys.exit(1)
|
||||
|
||||
setup(
|
||||
name="aurweb",
|
||||
version=version,
|
||||
packages=find_packages(),
|
||||
entry_points={
|
||||
'console_scripts': [
|
||||
'aurweb-git-auth = aurweb.git.auth:main',
|
||||
'aurweb-git-serve = aurweb.git.serve:main',
|
||||
'aurweb-git-update = aurweb.git.update:main',
|
||||
'aurweb-aurblup = aurweb.scripts.aurblup:main',
|
||||
'aurweb-mkpkglists = aurweb.scripts.mkpkglists:main',
|
||||
'aurweb-notify = aurweb.scripts.notify:main',
|
||||
'aurweb-pkgmaint = aurweb.scripts.pkgmaint:main',
|
||||
'aurweb-popupdate = aurweb.scripts.popupdate:main',
|
||||
'aurweb-rendercomment = aurweb.scripts.rendercomment:main',
|
||||
'aurweb-tuvotereminder = aurweb.scripts.tuvotereminder:main',
|
||||
'aurweb-usermaint = aurweb.scripts.usermaint:main',
|
||||
],
|
||||
},
|
||||
)
|
||||
Loading…
Add table
Reference in a new issue