aurweb/test/setup.sh
Kevin Morris a467b18474
Merge branch 'pu': pre-v6.0.0
Release v6.0.0 - Python

This documents UX and functional changes for the v6.0.0 aurweb release.
Following this release, we'll be working on a few very nice features
noted at the end of this article in Upcoming Work.

Preface
-------

This v6.0.0 release makes the long-awaited Python port official.

Along with the development of the python port, we have modified a
number of features. There have been some integral changes to how
package requests are dealt with, so _Trusted Users_ should read
the entirety of this document.

Legend
------

There are a few terms which I'd like to define to increase
understanding of these changes as they are listed:

- _self_
    - Refers to a user viewing or doing something regarding their own account
- _/pkgbase/{name}/{action}_
    - Refers to a POST action which can be triggered via the relevent package
      page at `/{pkgbase,packages}/{name}`.

Grouped changes explained in multiple items will always be prefixed with
the same letter surrounded by braces. Example:

- [A] Some feature that does something
- [A] The same feature where another thing has changed

Infrastructure
--------------

- Python packaging is now done with poetry.
- SQLite support has been removed. This was done because even though
  SQLAlchemy is an ORM, SQLite has quite a few SQL-server-like features
  missing both out of the box and integrally which force us to account
  for the different database types. We now only support mysql, and should
  be able to support postgresql without much effort in the future.
  Note: Users wishing to easily spin up a database quickly can use
  `docker-compose up -d mariadb` for a Docker-hosted mariadb service.
- An example systemd service has been included at `examples/aurweb.service`.
- Example wrappers to `aurweb-git-(auth|serve|update)` have been included
  at `examples/aurweb-git-(auth|serve|update).sh` and should be used to
  call these scripts when aurweb is installed into a poetry virtualenv.

HTML
----

- Pagers have all been modified. They still serve the same purpose, but
  they have slightly different display.
- Some markup and methods around the website has been changed for
  post requests, and some forms have been completely reworked.

Package Requests
----------------

- Normal users can now view and close their own requests
- [A] Requests can no longer be accepted through manual closures
- [A] Requests are now closed via their relevent actions
    - Deletion
        - Through `/packages` bulk delete action
        - Through `/pkgbase/{name}/delete`
    - Merge
        - Through `/pkgbase/{name}/merge`
    - Orphan
        - Through `/packages` bulk disown action
        - Through `/pkgbase/{name}/disown`
- Deletion and merge requests (and their closures) are now autogenerated
  if no pre-existing request exists. This was done to increase tracking of
  package modifications performed by those with access to do so (TUs).
- Deletion, merge and orphan request actions now close all (1 or more)
  requests pertaining to the action performed. This comes with the downside
  of multiple notifications sent out about a closure if more than one
  request (or no request) exists for them
- Merge actions now automatically reject other pre-existing merge requests
  with a mismatched `MergeBaseName` column when a merge action is performed
- The last `/requests` page no longer goes nowhere

Package Bulk Actions: /packages
-------------------------------

- The `Merge into` field has been removed. Merges now require being
  performed via the `/pkgbase/{name}/merge` action.

Package View
------------

- Some cached metadata is no longer cached (pkginfo). Previously,
  this was defaulted to a one day cache for some package information.
  If we need to bring this back, we can.

TU Proposals
------------

- A valid username is now required for any addition or removal of a TU.

RPC
---

- `type=get-comment-form` has been removed and is now located at
  `/pkgbase/{name}/comments/{id}/form`.
- Support for versions 1-4 have been removed.
- JSON key ordering is different than PHP's JSON.
- `type=search` performance is overall slightly worse than PHP's. This
  should not heavily affect users, as a 3,000 record query is returned
  in roughly 0.20ms from a local standpoint. We will be working on this
  in aim to push it over PHP.

Archives
--------

- Added metadata archive `packages-meta-v1.json.gz`.
- Added metadata archive `packages-meta-ext-v1.json.gz`.
    - Enable this by passing `--extended` to `aurweb-mkpkglists`.

Performance Changes
-------------------

As is expected from a complete rewrite of the website, performance
has changed across the board. In most places, Python's implementation
now performs better than the pre-existing PHP implementation, with the
exception of a few routes. Notably:

- `/` loads much quicker as it is now persistently cached forcibly
  for five minutes at a time.
- `/packages` search is much quicker.
- `/packages/{name}` view is slightly slower; we are no longer caching
  various pieces of package info for `cache_pkginfo_ttl`, which is
  defaulted to 86400 seconds, or one day.
- Request actions are slower due to the removal of the `via` parameter.
  We now query the database for requests related to the action based on
  the current state of the DB.
- `/rpc?type=info` queries are slightly quicker.
- `/rpc?type=search` queries of low result counts are quicker.
- `/rpc?type=search` queries of large result counts (> 2500) are slower.
    - We are not satisfied with this. We'll be working on pushing this
      over the edge along with the rest of the DB-intensive routes.
      However, the speed degredation is quite negligible for users'
      experience: 0.12ms PHP vs 0.15ms Python on a 3,000 record query
      on my local 4-core 8-thread system.

Upcoming Work
-------------

This release is the first major release of the Python implementation.
We have multiple tasks up for work immediately, which will bring us
a few more minor versions forward as they are completed.

- Update request and tu vote pagers
- Archive differentials
- Archive mimetypes
- (a) Git scripts to ORM conversion
- (a) Sharness removal
- Restriction of number of requests users can submit
2022-01-18 10:39:59 -08:00

233 lines
7.9 KiB
Bash

TOPLEVEL="$(cd "$(dirname "$0")/.." && pwd)"
. "$TOPLEVEL/test/sharness.sh"
# Configure python search path.
PYTHONPATH="$TOPLEVEL"
export PYTHONPATH
# Configure paths to the Git interface scripts.
GIT_AUTH="$TOPLEVEL/aurweb/git/auth.py"
GIT_SERVE="$TOPLEVEL/aurweb/git/serve.py"
GIT_UPDATE="$TOPLEVEL/aurweb/git/update.py"
MKPKGLISTS="$TOPLEVEL/aurweb/scripts/mkpkglists.py"
TUVOTEREMINDER="$TOPLEVEL/aurweb/scripts/tuvotereminder.py"
PKGMAINT="$TOPLEVEL/aurweb/scripts/pkgmaint.py"
USERMAINT="$TOPLEVEL/aurweb/scripts/usermaint.py"
AURBLUP="$TOPLEVEL/aurweb/scripts/aurblup.py"
NOTIFY="$TOPLEVEL/aurweb/scripts/notify.py"
RENDERCOMMENT="$TOPLEVEL/aurweb/scripts/rendercomment.py"
# We reuse some of these scripts when running `env`, so add
# it to PATH; that way, env can pick up the script when loaded.
PATH="${PATH}:${TOPLEVEL}/test/scripts"
export PATH
# Create the configuration file and a dummy notification script.
cat >config <<-EOF
[database]
backend = sqlite
name = aur.db
[options]
aurwebdir = $TOPLEVEL
aur_location = https://aur.archlinux.org
aur_request_ml = aur-requests@lists.archlinux.org
enable-maintenance = 0
maintenance-exceptions = 127.0.0.1
commit_uri = https://aur.archlinux.org/cgit/aur.git/log/?h=%s&id=%s
localedir = $TOPLEVEL/web/locale/
snapshot_uri = /cgit/aur.git/snapshot/%s.tar.gz
[notifications]
notify-cmd = $NOTIFY
sendmail = ./sendmail.sh
sender = notify@aur.archlinux.org
reply-to = noreply@aur.archlinux.org
[auth]
valid-keytypes = ssh-rsa ssh-dss ecdsa-sha2-nistp256 ecdsa-sha2-nistp384 ecdsa-sha2-nistp521 ssh-ed25519
username-regex = [a-zA-Z0-9]+[.\-_]?[a-zA-Z0-9]+$
git-serve-cmd = $GIT_SERVE
ssh-options = restrict
[serve]
repo-path = ./aur.git/
repo-regex = [a-z0-9][a-z0-9.+_-]*$
git-shell-cmd = ./git-shell.sh
git-update-cmd = ./update.sh
ssh-cmdline = ssh aur@aur.archlinux.org
[update]
max-blob-size = 256000
[aurblup]
db-path = $(pwd)/sync/
sync-dbs = test
server = file://$(pwd)/remote/
[mkpkglists]
archivedir = $(pwd)/archive
packagesfile = packages.gz
packagesmetafile = packages-meta-v1.json.gz
packagesmetaextfile = packages-meta-ext-v1.json.gz
pkgbasefile = pkgbase.gz
userfile = users.gz
EOF
cat >sendmail.sh <<-\EOF
#!/bin/sh
cat >>sendmail.out
EOF
chmod +x sendmail.sh
cat >git-shell.sh <<-\EOF
#!/bin/sh
echo $AUR_USER
echo $AUR_PKGBASE
echo $GIT_NAMESPACE
EOF
chmod +x git-shell.sh
cat >update.sh <<-\EOF
#!/bin/sh
echo $AUR_USER
echo $AUR_PKGBASE
EOF
chmod +x update.sh
AUR_CONFIG=config
export AUR_CONFIG
# Create SSH public keys which will be used by the test users later.
AUTH_KEYTYPE_USER=ssh-rsa
AUTH_KEYTEXT_USER=AAAAB3NzaC1yc2EAAAADAQABAAABAQCeUafDK4jqUiRHNQfwHcYjBKLZ4Rc1sNUofHApBP6j91nIvDHZe2VUqeBmFUhBz7kXK4VbXD9nlHMun2HeshL8hXnMzymZ8Wk7+IKefj61pajJkIdttw9Tnayfg7uhg5RbFy9zpEjmGjnIVjSzOXKCwppNT+CNujpKM5FD8gso/Z+l3fD+IwrPwS1SzF1Z99nqI9n2FM/JWZqluvTqnW9WdAvBDfutXxp0R5ZiLI5TAKL2Ssp5rpL70pkLXhv+9sK545zKKlXUFmw6Pi2iVBdqdRsk9ocl49dLiNIh8CYDCO3CRQn+8EnpBhTor2TKQxGJI3mzoBwWJJxoKhD/XlYJ
AUTH_FINGERPRINT_USER=SHA256:F/OFtYAy0JCytAGUi4RUZnOsThhQtFMK7fH1YvFBCpo
AUTH_KEYTYPE_TU=ssh-rsa
AUTH_KEYTEXT_TU=AAAAB3NzaC1yc2EAAAADAQABAAABAQC4Q2Beg6jf2r1LZ4vwT5y10dK8+/c5RaNyTwv77wF2OSLXh32xW0ovhE2lW2gqoakdGsxgM2fTtqMTl29WOsAxlGF7x9XbWhFXFUT88Daq1fAeuihkiRjfBbInSW/WcrFZ+biLBch67addtfkkd4PmAafDeeCtszAXqza+ltBG1oxAGiTXgI3LOhA1/GtLLxsi5sPUO3ZlhvwDn4Sy0aXYx8l9hop/PU4Cjn82hyRa9r+SRxQ3KtjKxcVMnZ8IyXOrBwXTukgSBR/6nSdEmO0JPkYUFuNwh3UGFKuNkrPguL5T+4YDym6czYmZJzQ7NNl2pLKYmYgBwBe5rORlWfN5
AUTH_FINGERPRINT_TU=SHA256:xQGC6j/U1Q3NDXLl04pm+Shr1mjYUXbGMUzlm9vby4k
AUTH_KEYTYPE_MISSING=sha-rsa
AUTH_KEYTEXT_MISSING=AAAAB3NzaC1yc2EAAAADAQABAAABAQC9UTpssBunuTBCT3KFtv+yb+cN0VmI2C9O9U7wHlkEZWxNBK8is6tnDHXBxRuvRk0LHILkTidLLFX22ZF0+TFgSz7uuEvGZVNpa2Fn2+vKJJYMvZEvb/f8VHF5/Jddt21VOyu23royTN/duiT7WIZdCtEmq5C9Y43NPfsB8FbUc+FVSYT2Lq7g1/bzvFF+CZxwCrGjC3qC7p3pshICfFR8bbWgRN33ClxIQ7MvkcDtfNu38dLotJqdfEa7NdQgba5/S586f1A4OWKc/mQJFyTaGhRBxw/cBSjqonvO0442VYLHFxlrTHoUunKyOJ8+BJfKgjWmfENC9ESY3mL/IEn5
AUTH_FINGERPRINT_MISSING=SHA256:uB0B+30r2WA1TDMUmFcaEBjosjnFGzn33XFhiyvTL9w
# Setup fake SSH environment.
SSH_CLIENT='1.2.3.4 1234 22'
SSH_CONNECTION='1.2.3.4 1234 4.3.2.1 22'
SSH_TTY=/dev/pts/0
export SSH_CLIENT SSH_CONNECTION SSH_TTY
# Initialize the test database.
python -m aurweb.initdb --no-alembic
echo "INSERT INTO Users (ID, UserName, Passwd, Email, LangPreference, AccountTypeID) VALUES (1, 'user', '!', 'user@localhost', 'en', 1);" | sqlite3 aur.db
echo "INSERT INTO Users (ID, UserName, Passwd, Email, LangPreference, AccountTypeID) VALUES (2, 'tu', '!', 'tu@localhost', 'en', 2);" | sqlite3 aur.db
echo "INSERT INTO Users (ID, UserName, Passwd, Email, LangPreference, AccountTypeID) VALUES (3, 'dev', '!', 'dev@localhost', 'en', 3);" | sqlite3 aur.db
echo "INSERT INTO Users (ID, UserName, Passwd, Email, LangPreference, AccountTypeID) VALUES (4, 'user2', '!', 'user2@localhost', 'en', 1);" | sqlite3 aur.db
echo "INSERT INTO Users (ID, UserName, Passwd, Email, LangPreference, AccountTypeID) VALUES (5, 'user3', '!', 'user3@localhost', 'en', 1);" | sqlite3 aur.db
echo "INSERT INTO Users (ID, UserName, Passwd, Email, LangPreference, AccountTypeID) VALUES (6, 'user4', '!', 'user4@localhost', 'en', 1);" | sqlite3 aur.db
echo "INSERT INTO Users (ID, UserName, Passwd, Email, LangPreference, AccountTypeID) VALUES (7, 'tu2', '!', 'tu2@localhost', 'en', 2);" | sqlite3 aur.db
echo "INSERT INTO Users (ID, UserName, Passwd, Email, LangPreference, AccountTypeID) VALUES (8, 'tu3', '!', 'tu3@localhost', 'en', 2);" | sqlite3 aur.db
echo "INSERT INTO Users (ID, UserName, Passwd, Email, LangPreference, AccountTypeID) VALUES (9, 'tu4', '!', 'tu4@localhost', 'en', 2);" | sqlite3 aur.db
echo "INSERT INTO SSHPubKeys (UserID, Fingerprint, PubKey) VALUES (1, '$AUTH_FINGERPRINT_USER', '$AUTH_KEYTYPE_USER $AUTH_KEYTEXT_USER');" | sqlite3 aur.db
echo "INSERT INTO SSHPubKeys (UserID, Fingerprint, PubKey) VALUES (2, '$AUTH_FINGERPRINT_TU', '$AUTH_KEYTYPE_TU $AUTH_KEYTEXT_TU');" | sqlite3 aur.db
echo "INSERT INTO Bans (IPAddress, BanTS) VALUES ('1.3.3.7', 0);" | sqlite3 aur.db
echo "INSERT INTO PackageBlacklist (Name) VALUES ('forbidden');" | sqlite3 aur.db
echo "INSERT INTO OfficialProviders (Name, Repo, Provides) VALUES ('official', 'core', 'official');" | sqlite3 aur.db
# Initialize a Git repository and test packages.
GIT_AUTHOR_EMAIL=author@example.com
GIT_AUTHOR_NAME='A U Thor'
GIT_COMMITTER_EMAIL=committer@example.com
GIT_COMMITTER_NAME='C O Mitter'
export GIT_AUTHOR_EMAIL GIT_AUTHOR_NAME
export GIT_COMMITTER_EMAIL GIT_COMMITTER_NAME
(
mkdir aur.git
cd aur.git
git init -q
git config --local commit.gpgsign false
git checkout -q --orphan refs/namespaces/foobar/refs/heads/master
cat >PKGBUILD <<-EOF
pkgname=foobar
pkgver=1
pkgrel=1
pkgdesc='aurweb test package.'
url='https://aur.archlinux.org/'
license=('GPL')
arch=('any')
depends=('python-pygit2')
source=()
md5sums=()
package() {
echo 'Hello world!'
}
EOF
cat >.SRCINFO <<-EOF
pkgbase = foobar
pkgdesc = aurweb test package.
pkgver = 1
pkgrel = 1
url = https://aur.archlinux.org/
arch = any
license = GPL
depends = python-pygit2
pkgname = foobar
EOF
git add PKGBUILD .SRCINFO
git commit -q -m 'Initial import'
sed 's/\(pkgrel.*\)1/\12/' PKGBUILD >PKGBUILD.new
sed 's/\(pkgrel.*\)1/\12/' .SRCINFO >.SRCINFO.new
mv PKGBUILD.new PKGBUILD
mv .SRCINFO.new .SRCINFO
git commit -q -am 'Bump pkgrel'
git checkout -q --orphan refs/namespaces/foobar2/refs/heads/master
cat >PKGBUILD <<-EOF
pkgname=foobar2
pkgver=1
pkgrel=1
pkgdesc='aurweb test package.'
url='https://aur.archlinux.org/'
license=('MIT')
arch=('any')
depends=('python-pygit2')
source=()
md5sums=()
package() {
echo 'Hello world!'
}
EOF
cat >.SRCINFO <<-EOF
pkgbase = foobar2
pkgdesc = aurweb test package.
pkgver = 1
pkgrel = 1
url = https://aur.archlinux.org/
arch = any
license = MIT
depends = python-pygit2
pkgname = foobar2
EOF
git add PKGBUILD .SRCINFO
git commit -q -m 'Initial import'
git checkout -q refs/namespaces/foobar/refs/heads/master
)