This patch brings in two new features:
- when viewing proposal listings, there is a new Statistics section,
containing the total and active number of Trusted Users found in the
database.
- when viewing a proposal directly, the number of active trusted users
assigned when the proposal was added is now displayed in the details
section.
Closes#323
Signed-off-by: Kevin Morris <kevr@0cost.org>
Not sure where this works, but it doesn't seem to work on my
browser. Achieved the same by forEaching through the array
returned by querySelectorAll instead.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Release v6.0.0 - Python
This documents UX and functional changes for the v6.0.0 aurweb release.
Following this release, we'll be working on a few very nice features
noted at the end of this article in Upcoming Work.
Preface
-------
This v6.0.0 release makes the long-awaited Python port official.
Along with the development of the python port, we have modified a
number of features. There have been some integral changes to how
package requests are dealt with, so _Trusted Users_ should read
the entirety of this document.
Legend
------
There are a few terms which I'd like to define to increase
understanding of these changes as they are listed:
- _self_
- Refers to a user viewing or doing something regarding their own account
- _/pkgbase/{name}/{action}_
- Refers to a POST action which can be triggered via the relevent package
page at `/{pkgbase,packages}/{name}`.
Grouped changes explained in multiple items will always be prefixed with
the same letter surrounded by braces. Example:
- [A] Some feature that does something
- [A] The same feature where another thing has changed
Infrastructure
--------------
- Python packaging is now done with poetry.
- SQLite support has been removed. This was done because even though
SQLAlchemy is an ORM, SQLite has quite a few SQL-server-like features
missing both out of the box and integrally which force us to account
for the different database types. We now only support mysql, and should
be able to support postgresql without much effort in the future.
Note: Users wishing to easily spin up a database quickly can use
`docker-compose up -d mariadb` for a Docker-hosted mariadb service.
- An example systemd service has been included at `examples/aurweb.service`.
- Example wrappers to `aurweb-git-(auth|serve|update)` have been included
at `examples/aurweb-git-(auth|serve|update).sh` and should be used to
call these scripts when aurweb is installed into a poetry virtualenv.
HTML
----
- Pagers have all been modified. They still serve the same purpose, but
they have slightly different display.
- Some markup and methods around the website has been changed for
post requests, and some forms have been completely reworked.
Package Requests
----------------
- Normal users can now view and close their own requests
- [A] Requests can no longer be accepted through manual closures
- [A] Requests are now closed via their relevent actions
- Deletion
- Through `/packages` bulk delete action
- Through `/pkgbase/{name}/delete`
- Merge
- Through `/pkgbase/{name}/merge`
- Orphan
- Through `/packages` bulk disown action
- Through `/pkgbase/{name}/disown`
- Deletion and merge requests (and their closures) are now autogenerated
if no pre-existing request exists. This was done to increase tracking of
package modifications performed by those with access to do so (TUs).
- Deletion, merge and orphan request actions now close all (1 or more)
requests pertaining to the action performed. This comes with the downside
of multiple notifications sent out about a closure if more than one
request (or no request) exists for them
- Merge actions now automatically reject other pre-existing merge requests
with a mismatched `MergeBaseName` column when a merge action is performed
- The last `/requests` page no longer goes nowhere
Package Bulk Actions: /packages
-------------------------------
- The `Merge into` field has been removed. Merges now require being
performed via the `/pkgbase/{name}/merge` action.
Package View
------------
- Some cached metadata is no longer cached (pkginfo). Previously,
this was defaulted to a one day cache for some package information.
If we need to bring this back, we can.
TU Proposals
------------
- A valid username is now required for any addition or removal of a TU.
RPC
---
- `type=get-comment-form` has been removed and is now located at
`/pkgbase/{name}/comments/{id}/form`.
- Support for versions 1-4 have been removed.
- JSON key ordering is different than PHP's JSON.
- `type=search` performance is overall slightly worse than PHP's. This
should not heavily affect users, as a 3,000 record query is returned
in roughly 0.20ms from a local standpoint. We will be working on this
in aim to push it over PHP.
Archives
--------
- Added metadata archive `packages-meta-v1.json.gz`.
- Added metadata archive `packages-meta-ext-v1.json.gz`.
- Enable this by passing `--extended` to `aurweb-mkpkglists`.
Performance Changes
-------------------
As is expected from a complete rewrite of the website, performance
has changed across the board. In most places, Python's implementation
now performs better than the pre-existing PHP implementation, with the
exception of a few routes. Notably:
- `/` loads much quicker as it is now persistently cached forcibly
for five minutes at a time.
- `/packages` search is much quicker.
- `/packages/{name}` view is slightly slower; we are no longer caching
various pieces of package info for `cache_pkginfo_ttl`, which is
defaulted to 86400 seconds, or one day.
- Request actions are slower due to the removal of the `via` parameter.
We now query the database for requests related to the action based on
the current state of the DB.
- `/rpc?type=info` queries are slightly quicker.
- `/rpc?type=search` queries of low result counts are quicker.
- `/rpc?type=search` queries of large result counts (> 2500) are slower.
- We are not satisfied with this. We'll be working on pushing this
over the edge along with the rest of the DB-intensive routes.
However, the speed degredation is quite negligible for users'
experience: 0.12ms PHP vs 0.15ms Python on a 3,000 record query
on my local 4-core 8-thread system.
Upcoming Work
-------------
This release is the first major release of the Python implementation.
We have multiple tasks up for work immediately, which will bring us
a few more minor versions forward as they are completed.
- Update request and tu vote pagers
- Archive differentials
- Archive mimetypes
- (a) Git scripts to ORM conversion
- (a) Sharness removal
- Restriction of number of requests users can submit
This commit does quite a bit:
- Catches unhandled exceptions raised in the route handler and
produces a 500 Internal Server Error Arch-themed response.
- Each unhandled exception causes a notification to be sent to new
`notifications.postmaster` email with a "Traceback ID."
- Traceback ID is logged to the server along with the traceback which
caused the 500: `docker-compose logs fastapi | grep '<traceback_id>'`
- If `options.traceback` is set to `1`, traceback is displayed in
the new 500.html template.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previous conversion was causing a straight up bug when submitting
new proposals. This patch fixes that issue.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This was missed during the initial porting of the /login route.
Modifications:
-------------
- A form is now used for the [Logout] link and some css was
needed to deal with positioning.
Closes#186
Signed-off-by: Kevin Morris <kevr@0cost.org>
These were using the old comment image sources. Slipped in
due to cache and not checking without cache.
Fixed them to use src="/static/images/...".
Signed-off-by: Kevin Morris <kevr@0cost.org>
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
The SQL logic in this file for package metadata now exactly
reflects RPC's search logic, without searching for specific
packages.
Two command line arguments are available:
--extended | Include License, Keywords, Groups, relations
and dependencies.
When --extended is passed, the script will create a
packages-meta-ext-v1.json.gz, configured via packagesmetaextfile.
Archive JSON is in the following format: line-separated package objects
enclosed in a list:
[
{...},
{...},
{...}
]
Signed-off-by: Kevin Morris <kevr@0cost.org>
With this change, we've decoupled some partials shared between
`/pkgbase/{name}` and `/account/{username}/comments`. The comment
actions template now resolves its package base via the `comment`
instance instead of requiring `pkgbase`.
We've also modified the existing package comment routes to
support execution from any location using the `next` parameter.
This allows us to reuse code from package comments for
account comments actions.
Moved the majority of comment editing javascript to its own
.js file.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change now requires that PHP routes do not return HTTP 404
to be considered for the /metrics population. Additionally,
we make a small sanitization here to avoid trailing '/'
characters, unless we're on the homepage route.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Along with this initial requests metric implementation,
we also now serve the `/metrics` route, which grabs request
metrics out of cache and renders them properly for Prometheus.
**NOTE** Metrics are only enabled when the aurweb system admin
has enabled caching by configuring `options.cache` correctly
in `$AUR_CONFIG`. Otherwise, an error is logged about no cache
being configured.
New dependencies have been added which require the use of
`composer`. See `INSTALL` for the dependency section in regards
to composer dependencies and how to install them properly for
aurweb.
Metrics are in the following forms:
aurweb_http_requests_count(method="GET",route="/some_route")
aurweb_api_requests_count(method="GET",route="/rpc",type="search")
This should allow us to search through the requests for specific routes
and queries.
Signed-off-by: Kevin Morris <kevr@0cost.org>
In terms of performance, most queries on this page win over
PHP in query times, with the exception of sorting by Voted or
Notify (https://gitlab.archlinux.org/archlinux/aurweb/-/issues/102).
Otherwise, there are a few modifications: described below.
* Pagination
* The `paginate` Python module has been used in the FastAPI
project
here to implement paging on the packages search page. This
changes how pagination is displayed, however it serves the
same purpose. We'll take advantage of this module in other
places as well.
* Form action
* The form action for actions now use `POST /packages` to
perform. This is currently implemented and will be
addressed in a follow-up commit.
* Input names and values
* Input names and values have been modified to satisfy the
snake_case naming convention we'd like to use as much as
possible.
* Some input names and values were modified to comply with
FastAPI Forms: (IDs[<id>]) -> (IDs, <id>).
Signed-off-by: Kevin Morris <kevr@0cost.org>
With FastAPI starting to come closer to a close, we've got to advertise
this deprecation so that users have some time to adjust before making
the changes. We have not specified a specific time here, but we'd like
this message to reach users of the RPC API for at least a month before
any modifications are made to the interface.
Signed-off-by: Kevin Morris <kevr@0cost.org>
With FastAPI starting to come closer to a close, we've got to advertise
this deprecation so that users have some time to adjust before making
the changes. We have not specified a specific time here, but we'd like
this message to reach users of the RPC API for at least a month before
any modifications are made to the interface.
Signed-off-by: Kevin Morris <kevr@0cost.org>
A few things added with this commit:
- aurweb.packages.util
- A module providing package and pkgbase helpers.
- aurweb.template.register_filter
- A decorator that can be used to register a filter:
@register_filter("some_filter") def f(): pass
Additionally, template partials have been split off a bit
differently. Changes:
- /packages/{name} is defined in packages/show.html.
- partials/packages/package_actions.html is now
partials/packages/actions.html.
- partials/packages/details.html has been added.
- partials/packages/comments.html has been added.
- partials/packages/comment.html has been added.
- models.dependency_type additions: name and id constants.
- models.relation_type additions: name and id constants.
- models.official_provider additions: base official url constant.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Slight markup changes, same style overall and same
form parameters as the PHP implementation.
In addition, we've disabled the "left" and "right"
navigation buttons when we're at the border of the
table.
CSS Changes:
- Added similar styling to submit `<buttons>` that submit `<input>` had.
- Added .results tr td[align="{left,right}"] styling to align
the result table's `More -->` button to the right of the table.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit ports the `/tu/?id={proposal_id}` PHP routes to
FastAPI into two individual GET and POST routes.
With this port of the single proposal view and POST logic,
several things have changed.
- The only parameter used is now `decision`, which
must contain `Yes`, `No`, or `Abstain` as a string.
When an invalid value is given, a BAD_REQUEST response
is returned in plaintext: Invalid 'decision' value.
- The `doVote` parameter has been removed.
- The details section has been rearranged into a set
of divs with specific classes that can be used for
testing. CSS has been added to persist the layout with
the element changes.
- Several errors that can be discovered in the POST path
now trigger their own non-200 HTTPStatus codes.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The Document.execCommand API is deprecated and no longer recommended to
be used. It's replacement is the much simpler navigator.clipboard API
which is supported in all browsers except internet explorer.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
