This causes an issue that should have been obvious from the get-go:
if a package request is up in the AUR, but the package has already
been picked up by an official repository, we would end up returning
a 404 here, leading a TU to not be able to perform an action for
a request's target.
Signed-off-by: Kevin Morris <kevr@0cost.org>
For tests, we only care about emails having a valid syntax.
I don't think we should verify this at all, as aurweb.scripts.notify
will timeout if it cant deliver via sendmail/smtp.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Release v6.0.0 - Python
This documents UX and functional changes for the v6.0.0 aurweb release.
Following this release, we'll be working on a few very nice features
noted at the end of this article in Upcoming Work.
Preface
-------
This v6.0.0 release makes the long-awaited Python port official.
Along with the development of the python port, we have modified a
number of features. There have been some integral changes to how
package requests are dealt with, so _Trusted Users_ should read
the entirety of this document.
Legend
------
There are a few terms which I'd like to define to increase
understanding of these changes as they are listed:
- _self_
- Refers to a user viewing or doing something regarding their own account
- _/pkgbase/{name}/{action}_
- Refers to a POST action which can be triggered via the relevent package
page at `/{pkgbase,packages}/{name}`.
Grouped changes explained in multiple items will always be prefixed with
the same letter surrounded by braces. Example:
- [A] Some feature that does something
- [A] The same feature where another thing has changed
Infrastructure
--------------
- Python packaging is now done with poetry.
- SQLite support has been removed. This was done because even though
SQLAlchemy is an ORM, SQLite has quite a few SQL-server-like features
missing both out of the box and integrally which force us to account
for the different database types. We now only support mysql, and should
be able to support postgresql without much effort in the future.
Note: Users wishing to easily spin up a database quickly can use
`docker-compose up -d mariadb` for a Docker-hosted mariadb service.
- An example systemd service has been included at `examples/aurweb.service`.
- Example wrappers to `aurweb-git-(auth|serve|update)` have been included
at `examples/aurweb-git-(auth|serve|update).sh` and should be used to
call these scripts when aurweb is installed into a poetry virtualenv.
HTML
----
- Pagers have all been modified. They still serve the same purpose, but
they have slightly different display.
- Some markup and methods around the website has been changed for
post requests, and some forms have been completely reworked.
Package Requests
----------------
- Normal users can now view and close their own requests
- [A] Requests can no longer be accepted through manual closures
- [A] Requests are now closed via their relevent actions
- Deletion
- Through `/packages` bulk delete action
- Through `/pkgbase/{name}/delete`
- Merge
- Through `/pkgbase/{name}/merge`
- Orphan
- Through `/packages` bulk disown action
- Through `/pkgbase/{name}/disown`
- Deletion and merge requests (and their closures) are now autogenerated
if no pre-existing request exists. This was done to increase tracking of
package modifications performed by those with access to do so (TUs).
- Deletion, merge and orphan request actions now close all (1 or more)
requests pertaining to the action performed. This comes with the downside
of multiple notifications sent out about a closure if more than one
request (or no request) exists for them
- Merge actions now automatically reject other pre-existing merge requests
with a mismatched `MergeBaseName` column when a merge action is performed
- The last `/requests` page no longer goes nowhere
Package Bulk Actions: /packages
-------------------------------
- The `Merge into` field has been removed. Merges now require being
performed via the `/pkgbase/{name}/merge` action.
Package View
------------
- Some cached metadata is no longer cached (pkginfo). Previously,
this was defaulted to a one day cache for some package information.
If we need to bring this back, we can.
TU Proposals
------------
- A valid username is now required for any addition or removal of a TU.
RPC
---
- `type=get-comment-form` has been removed and is now located at
`/pkgbase/{name}/comments/{id}/form`.
- Support for versions 1-4 have been removed.
- JSON key ordering is different than PHP's JSON.
- `type=search` performance is overall slightly worse than PHP's. This
should not heavily affect users, as a 3,000 record query is returned
in roughly 0.20ms from a local standpoint. We will be working on this
in aim to push it over PHP.
Archives
--------
- Added metadata archive `packages-meta-v1.json.gz`.
- Added metadata archive `packages-meta-ext-v1.json.gz`.
- Enable this by passing `--extended` to `aurweb-mkpkglists`.
Performance Changes
-------------------
As is expected from a complete rewrite of the website, performance
has changed across the board. In most places, Python's implementation
now performs better than the pre-existing PHP implementation, with the
exception of a few routes. Notably:
- `/` loads much quicker as it is now persistently cached forcibly
for five minutes at a time.
- `/packages` search is much quicker.
- `/packages/{name}` view is slightly slower; we are no longer caching
various pieces of package info for `cache_pkginfo_ttl`, which is
defaulted to 86400 seconds, or one day.
- Request actions are slower due to the removal of the `via` parameter.
We now query the database for requests related to the action based on
the current state of the DB.
- `/rpc?type=info` queries are slightly quicker.
- `/rpc?type=search` queries of low result counts are quicker.
- `/rpc?type=search` queries of large result counts (> 2500) are slower.
- We are not satisfied with this. We'll be working on pushing this
over the edge along with the rest of the DB-intensive routes.
However, the speed degredation is quite negligible for users'
experience: 0.12ms PHP vs 0.15ms Python on a 3,000 record query
on my local 4-core 8-thread system.
Upcoming Work
-------------
This release is the first major release of the Python implementation.
We have multiple tasks up for work immediately, which will bring us
a few more minor versions forward as they are completed.
- Update request and tu vote pagers
- Archive differentials
- Archive mimetypes
- (a) Git scripts to ORM conversion
- (a) Sharness removal
- Restriction of number of requests users can submit
This variable allows users to override the logging.conf used
for Python logging configuration. By default, this is set
to logging.conf, which is a production config. LOG_CONFIG
is treated relative to [options] aurwebdir.
This patch allows us to specify the logging config as opposed
to copying over logging.conf in our test docker and gitlab
test scripts, as well as ease-of-testing as a developer.
Signed-off-by: Kevin Morris <kevr@0cost.org>
We were allowing erroneous URLs through, raising exceptions,
from e.g. `http://[localhost:8444/blah`. This patch catches
any ValueErrors raised during the parse process and returns
False, indicating that the validation failed.
This patch also adds testing specifically for `util.valid_homepage`.
We didn't have specific testing for this before; this will allow us
to catch regressions in this area.
Closes#250
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit does quite a bit:
- Catches unhandled exceptions raised in the route handler and
produces a 500 Internal Server Error Arch-themed response.
- Each unhandled exception causes a notification to be sent to new
`notifications.postmaster` email with a "Traceback ID."
- Traceback ID is logged to the server along with the traceback which
caused the 500: `docker-compose logs fastapi | grep '<traceback_id>'`
- If `options.traceback` is set to `1`, traceback is displayed in
the new 500.html template.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Originally left at util/adduser.py, this script allows administrators
to simply add a user to the configured aurweb database.
See --help for options.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This gets raised when a client closes a connection before receiving
a valid response; this is not controllable from our side.
Signed-off-by: Kevin Morris <kevr@0cost.org>
These new decorators are meant to be used without any arguments
and provide aliases to auth_required:
- `auth_required(True) -> requires_auth`
- `auth_required(False) -> requires_guest`
These decorators should be used without arguments, e.g.:
@router.get("/")
@requires_guest
async def my_route(request: Request):
return HTMLResponse()
Signed-off-by: Kevin Morris <kevr@0cost.org>
Move package request routes and related routes to their
respective routers. In addition, move some utility used
for requests over from `aurweb.packages`.
Introduced routers:
- `aurweb.routers.requests`
Introduced package:
- `aurweb.requests`
Introduced module:
- `aurweb.requests.util`
Changes:
- Moved `aurweb.packages.validate` to `aurweb.pkgbase.validate`
- Moved requests listing & request closure routes to
`aurweb.routers.requests`
- Moved pkgbase request creation route to `aurweb.routers.pkgbase`
- Moved `get_pkgreq_by_id` from `aurweb.packages.util` to
`aurweb.requests.util` and fixed its return type hint.
Signed-off-by: Kevin Morris <kevr@0cost.org>
`delete_package` was processing package deletions through `Package`
instances. This doesn't make sense; if we delete a package, we want
to target its package base.
This new function vastly simplifies the previous.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously, when the relationship was None, an <a> would still
wrap the None value erroneously. This addresses that for all
three user fields.
In addition, this commit adds direct testing for the
`templates/partials/packages/details.html` template.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit centralizes comaintainer management with a few new
functions and uses them more appropriately within routes:
- aurweb.packages.util.latest_priority
- aurweb.packages.util.remove_comaintainer
- aurweb.packages.util.remove_comaintainers
- aurweb.packages.util.add_comaintainer
- aurweb.packages.util.add_comaintainers
- aurweb.packages.util.rotate_comaintainers
Closes#117
Signed-off-by: Kevin Morris <kevr@0cost.org>
While this does make more sense to me personally, there is no need
to change how the AUR treats its users; it has been accepted for
ages and not found to be ridden with flaws. Stay with the tried
and true method.
Signed-off-by: Kevin Morris <kevr@0cost.org>
- Use queries more closely aligned to PHP's implementation; removes
the need for separate vote/notification queries.
- Default sort by popularity
Closes#214
Signed-off-by: Kevin Morris <kevr@0cost.org>
Instead of allowing an exception to propogate through the framework
routes, catch it and log out an error about notifications not being
sent.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit also decouples testing regarding this feature
into several test functions.
Signed-off-by: Kevin Morris <kevr@0cost.org>
bump
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit changes several things about how we were handling
package requests.
Modifications (requests):
-------------
- `/requests/{id}/close` no longer provides an Accepted selection.
All manual request closures will cause a rejection.
- Relevent `pkgbase` actions now trigger request closures:
`/pkgbase/{name}/delete` (deletion), `/pkgbase/{name}/merge` (merge)
and `/pkgbase/{name}/disown` (orphan).
- Comment fields have been added to
`/pkgbase/{name}/{delete,merge,disown}`, which is used to set the
`PackageRequest.ClosureComment` on pending requests. If the comment
field is left blank, a closure comment is autogenerated.
- Autogenerated request notifications are only sent out once
as a closure notification.
- Some markup has been fixed.
Modifications (disown/orphan):
-----------------------------
- Orphan requests are now handled through the same path as
deletion/merge.
- We now check for due date when disowning as non-maintainer;
previously, this was only done for display and not functionally.
This check applies to Trusted Users' disowning of a package.
This style of notification flow does reduce our visibility, but
accounting can still be done via the close request; it includes
the action, pkgbase name and the user who accepted it.
Closes#204
Signed-off-by: Kevin Morris <kevr@0cost.org>
