Release v6.0.0 - Python
This documents UX and functional changes for the v6.0.0 aurweb release.
Following this release, we'll be working on a few very nice features
noted at the end of this article in Upcoming Work.
Preface
-------
This v6.0.0 release makes the long-awaited Python port official.
Along with the development of the python port, we have modified a
number of features. There have been some integral changes to how
package requests are dealt with, so _Trusted Users_ should read
the entirety of this document.
Legend
------
There are a few terms which I'd like to define to increase
understanding of these changes as they are listed:
- _self_
- Refers to a user viewing or doing something regarding their own account
- _/pkgbase/{name}/{action}_
- Refers to a POST action which can be triggered via the relevent package
page at `/{pkgbase,packages}/{name}`.
Grouped changes explained in multiple items will always be prefixed with
the same letter surrounded by braces. Example:
- [A] Some feature that does something
- [A] The same feature where another thing has changed
Infrastructure
--------------
- Python packaging is now done with poetry.
- SQLite support has been removed. This was done because even though
SQLAlchemy is an ORM, SQLite has quite a few SQL-server-like features
missing both out of the box and integrally which force us to account
for the different database types. We now only support mysql, and should
be able to support postgresql without much effort in the future.
Note: Users wishing to easily spin up a database quickly can use
`docker-compose up -d mariadb` for a Docker-hosted mariadb service.
- An example systemd service has been included at `examples/aurweb.service`.
- Example wrappers to `aurweb-git-(auth|serve|update)` have been included
at `examples/aurweb-git-(auth|serve|update).sh` and should be used to
call these scripts when aurweb is installed into a poetry virtualenv.
HTML
----
- Pagers have all been modified. They still serve the same purpose, but
they have slightly different display.
- Some markup and methods around the website has been changed for
post requests, and some forms have been completely reworked.
Package Requests
----------------
- Normal users can now view and close their own requests
- [A] Requests can no longer be accepted through manual closures
- [A] Requests are now closed via their relevent actions
- Deletion
- Through `/packages` bulk delete action
- Through `/pkgbase/{name}/delete`
- Merge
- Through `/pkgbase/{name}/merge`
- Orphan
- Through `/packages` bulk disown action
- Through `/pkgbase/{name}/disown`
- Deletion and merge requests (and their closures) are now autogenerated
if no pre-existing request exists. This was done to increase tracking of
package modifications performed by those with access to do so (TUs).
- Deletion, merge and orphan request actions now close all (1 or more)
requests pertaining to the action performed. This comes with the downside
of multiple notifications sent out about a closure if more than one
request (or no request) exists for them
- Merge actions now automatically reject other pre-existing merge requests
with a mismatched `MergeBaseName` column when a merge action is performed
- The last `/requests` page no longer goes nowhere
Package Bulk Actions: /packages
-------------------------------
- The `Merge into` field has been removed. Merges now require being
performed via the `/pkgbase/{name}/merge` action.
Package View
------------
- Some cached metadata is no longer cached (pkginfo). Previously,
this was defaulted to a one day cache for some package information.
If we need to bring this back, we can.
TU Proposals
------------
- A valid username is now required for any addition or removal of a TU.
RPC
---
- `type=get-comment-form` has been removed and is now located at
`/pkgbase/{name}/comments/{id}/form`.
- Support for versions 1-4 have been removed.
- JSON key ordering is different than PHP's JSON.
- `type=search` performance is overall slightly worse than PHP's. This
should not heavily affect users, as a 3,000 record query is returned
in roughly 0.20ms from a local standpoint. We will be working on this
in aim to push it over PHP.
Archives
--------
- Added metadata archive `packages-meta-v1.json.gz`.
- Added metadata archive `packages-meta-ext-v1.json.gz`.
- Enable this by passing `--extended` to `aurweb-mkpkglists`.
Performance Changes
-------------------
As is expected from a complete rewrite of the website, performance
has changed across the board. In most places, Python's implementation
now performs better than the pre-existing PHP implementation, with the
exception of a few routes. Notably:
- `/` loads much quicker as it is now persistently cached forcibly
for five minutes at a time.
- `/packages` search is much quicker.
- `/packages/{name}` view is slightly slower; we are no longer caching
various pieces of package info for `cache_pkginfo_ttl`, which is
defaulted to 86400 seconds, or one day.
- Request actions are slower due to the removal of the `via` parameter.
We now query the database for requests related to the action based on
the current state of the DB.
- `/rpc?type=info` queries are slightly quicker.
- `/rpc?type=search` queries of low result counts are quicker.
- `/rpc?type=search` queries of large result counts (> 2500) are slower.
- We are not satisfied with this. We'll be working on pushing this
over the edge along with the rest of the DB-intensive routes.
However, the speed degredation is quite negligible for users'
experience: 0.12ms PHP vs 0.15ms Python on a 3,000 record query
on my local 4-core 8-thread system.
Upcoming Work
-------------
This release is the first major release of the Python implementation.
We have multiple tasks up for work immediately, which will bring us
a few more minor versions forward as they are completed.
- Update request and tu vote pagers
- Archive differentials
- Archive mimetypes
- (a) Git scripts to ORM conversion
- (a) Sharness removal
- Restriction of number of requests users can submit
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
If the person running the tests has a global configuration to sign git
commits, this breaks the testsuite which looks for a key capable of
committing dummy data under a dummy author
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
This resolves logging issues with alembic on aurweb.initdb
in addition to adding more logging utilities for aurweb
and tests in general.
Developers should fetch a logger for their specific module
via `logging.getLogger(__name__)`.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Important note: Python tests will repeatedly clear out tables
that they test against; for this reason, one should always run
the shell tests first. The __init__.py file is necessary for
coverage to collect data from the tests being run.
At this point in FastAPI development, I'd like to encourage a
few things going forward:
1. Any time you contribute to the FastAPI codebase, you **must**
maintain equal or increased coverage on the overall source.
Developers are highly appreciated for adding tests in your
specific domain of addition or modification that may be missing
coverage. Our goal is 100% coverage, and all newly added files
**must** have 100% coverage through tests.
2. All source should be formatted with the autopep8 tool and
kept within an 80 column width, with the exception of HTML
templates.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The new schema was generated with sqlacodegen and then manually adjusted
to fit schema/aur-schema.sql faithfully, both in the organisation of the
code and in the SQL generated by SQLAlchemy.
Initializing the database now requires the new tool aurweb.initdb.
References to aur-schema.sql have been updated and the old schema
dropped.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Since commit a7865ef (Make the locale directory configurable,
2018-07-22), we need to specify the locale directory in the
configuration file.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Add support for translating notification emails and send localized
notifications, based on the user's language preferences. Also, update
the translations Makefile to add strings from the notification script
to the message catalog.
Implements FS#31850.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Add a script to periodically remove old IP addresses from the users
database.
The login IP addresses are stored for spam protection and to prevent
from abuse. It is quite unlikely that we ever need the IP address of a
user whose last login is more than a week old. It makes sense to remove
such IP addresses to protect our users' privacy.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
In addition to the packages list and the package base list, also create
a list of registered user names.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Automatically detect Git commit identifiers, shorten them, and make them
link to the cgit interface.
Implements FS#43290.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Instead of converting package comments from plain text to HTML code when
they are displayed, do the conversion when the comment is posted and
store the rendered result in the database. The conversion itself is done
by a Python script which uses Bleach for sanitizing the text.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Instead of making all tests fail, error out during initialization if the
SQLite schema has not been generated.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Allow for automatically converting the schema into a schema that works
with SQLite by running `make` from the schema/ subdirectory. Use the new
Makefile in the test suite.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
The scripts were moved to aurweb/scripts/ in commit 3718860 (Make
maintenance scripts installable, 2016-10-17). Update the paths in the
test suite accordingly.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Move the Git interface scripts from git-interface/ to aurweb/git/. Use
setuptools to automatically create wrappers which can be installed using
`python3 setup.py install`. Update the configuration files, the test
suite as well as the INSTALL and README files to reflect these changes.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Add a test to make sure that Trusted Users, who already voted on a
proposal, do not receive any reminders.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Instead of only checking whether the notification script is called with
the correct parameters, actually invoke the real notify script and check
whether proper notifications are generated.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Move and rename the existing git-interface tests such that tests for
other scripts can be added easily.
In particular, the following changes are made:
* Move the existing tests from git-interface/test/ to test/.
* Rename t0001-auth.sh to t1100-git-auth.sh.
* Rename t0002-serve.sh to t1200-git-serve.sh.
* Rename t0003-update.sh to t1300-git-update.sh.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-09-29 22:07:24 +02:00
Renamed from git-interface/test/setup.sh (Browse further)
