Derived off of original work done by Leonidas Spyropoulos
at https://gitlab.archlinux.org/archlinux/aurweb/-/merge_requests/503
This revision of that original work finishes off the inconsistencies
mentioned in the original MR and adds a small bit of testing for more
regression checks.
Fixes: #360
Signed-off-by: Kevin Morris <kevr@0cost.org>
Found along with the previous commit to be a security hole in our
implementation. This commit resolves an issue regarding comment editing.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This addresses a severe security issue, which is omitted from this
git message for obscurity purposes.
Otherwise, it allows co-maintainers to see the keyword form when
viewing a package they co-maintain.
Closes#378
Signed-off-by: Kevin Morris <kevr@0cost.org>
This was incorrectly using the particular Package record's name
to format options.snapshot_uri in order to produce URLPath.
It should, instead, use the PackageBase record's name, which
this commit resolves.
Bug reported by thomy2000
Closes#382
Signed-off-by: Kevin Morris <kevr@0cost.org>
Speeds up SSHPubKeys.PubKey searches in a larger database.
Fixed form of the original commit which was reverted,
1a7f6e1fa9
Signed-off-by: Kevin Morris <kevr@0cost.org>
This reverts commit 1a7f6e1fa9.
This commit broke account creation in some way. We'd still like to
do this, but we need to ensure it does not intrude on other facets.
Extra: We should really work out how this even passed tests; it
should not have.
Remove all extra whitespace when parsing Keywords to ensure we don't add
empty keywords in the DB.
Closes: #332
Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
This patch does not include a javascript implementating, but
provides a pure HTML/HTTP method of paging through these lists.
Also fixes erroneous limiting. We now use a hardcoded limit of 20
by default.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This patch brings in two new features:
- when viewing proposal listings, there is a new Statistics section,
containing the total and active number of Trusted Users found in the
database.
- when viewing a proposal directly, the number of active trusted users
assigned when the proposal was added is now displayed in the details
section.
Closes#323
Signed-off-by: Kevin Morris <kevr@0cost.org>
Previously, Python code was looking for suggestions based on
`%<keyword>%`. This was inconsistent with PHP's suggestion
implementation and cause more records to be bundled with a suggestion,
along with supplying misleading suggestions.
Closes#343
Signed-off-by: Kevin Morris <kevr@0cost.org>
A new option has been added for configuration of SMTP timeout:
- notifications.smtp-timeout
During tests, we can change this timeout to be small, so we aren't
depending on hardware-based RNG to pass the timeout.
Without a timeout, users can run into a long-running test for no
particular reason.
Signed-off-by: Kevin Morris <kevr@0cost.org>
People can change comaintainer ownership to suspended users if they
want to.
Suspended users cannot login, so there is no breach of security
here. It does make sense to allow ownership to be changed, imo.
Closes#339
Signed-off-by: Kevin Morris <kevr@0cost.org>
As repeats of these traceback notifications were annoying some of
the devops staff, and it took coordination to share tracebacks with
developers, this commit removes that responsibility off of devops
by reporting tracebacks to Gitlab repositories in the form of issues.
- removed ServerErrorNotification
- removed notifications.postmaster configuration option
- added notifications.gitlab-instance option
- added notifications.error-project option
- added notifications.error-token option
- added aurweb.exceptions.handle_form_exceptions, a POST route decorator
Issues are filed confidentially. This change will need updates
in infrastructure's ansible configuration before this can be
applied to aur.archlinux.org.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change brings some new additions to our archives:
- SHA-256 .sha256 hexdigests
- We construct our archives in a tmpdir now and move them to the
archive destination when all are completed. This removes some
corrupted downloading when archiving is in-process.
Signed-off-by: Kevin Morris <kevr@0cost.org>
There was one blazing issue with the previous implementation regardless
of the multiple records: we were generating fingerprints by storing
the key into a file and reading it with ssh-keygen. This is absolutely
terrible and was not meant to be left around (it was forgotten, my bad).
Took this opportunity to clean up a few things:
- simplify pubkey validation
- centralize things a bit better
Signed-off-by: Kevin Morris <kevr@0cost.org>
By implicitly joining, sqlalchemy joined on
`TUVote.UsersID = TUVoteInfo.SubmitterID`. This should be joining on
`TUVote.VoteID = TUVoteInfo.ID` instead to include all TUVote instances
found in the database.
Closes#266
Signed-off-by: Kevin Morris <kevr@0cost.org>
- the "Flagged Out-of-date on ..." link in the package action panel does
not contain a timezone specifier.
Signed-off-by: Kevin Morris <kevr@0cost.org>
It was found in the aur.al database that some records have
a non-null flagger, but are not flagged. Using the flagger
relationship, we were false redirecting away from the flag
page.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This causes an issue that should have been obvious from the get-go:
if a package request is up in the AUR, but the package has already
been picked up by an official repository, we would end up returning
a 404 here, leading a TU to not be able to perform an action for
a request's target.
Signed-off-by: Kevin Morris <kevr@0cost.org>
For tests, we only care about emails having a valid syntax.
I don't think we should verify this at all, as aurweb.scripts.notify
will timeout if it cant deliver via sendmail/smtp.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Release v6.0.0 - Python
This documents UX and functional changes for the v6.0.0 aurweb release.
Following this release, we'll be working on a few very nice features
noted at the end of this article in Upcoming Work.
Preface
-------
This v6.0.0 release makes the long-awaited Python port official.
Along with the development of the python port, we have modified a
number of features. There have been some integral changes to how
package requests are dealt with, so _Trusted Users_ should read
the entirety of this document.
Legend
------
There are a few terms which I'd like to define to increase
understanding of these changes as they are listed:
- _self_
- Refers to a user viewing or doing something regarding their own account
- _/pkgbase/{name}/{action}_
- Refers to a POST action which can be triggered via the relevent package
page at `/{pkgbase,packages}/{name}`.
Grouped changes explained in multiple items will always be prefixed with
the same letter surrounded by braces. Example:
- [A] Some feature that does something
- [A] The same feature where another thing has changed
Infrastructure
--------------
- Python packaging is now done with poetry.
- SQLite support has been removed. This was done because even though
SQLAlchemy is an ORM, SQLite has quite a few SQL-server-like features
missing both out of the box and integrally which force us to account
for the different database types. We now only support mysql, and should
be able to support postgresql without much effort in the future.
Note: Users wishing to easily spin up a database quickly can use
`docker-compose up -d mariadb` for a Docker-hosted mariadb service.
- An example systemd service has been included at `examples/aurweb.service`.
- Example wrappers to `aurweb-git-(auth|serve|update)` have been included
at `examples/aurweb-git-(auth|serve|update).sh` and should be used to
call these scripts when aurweb is installed into a poetry virtualenv.
HTML
----
- Pagers have all been modified. They still serve the same purpose, but
they have slightly different display.
- Some markup and methods around the website has been changed for
post requests, and some forms have been completely reworked.
Package Requests
----------------
- Normal users can now view and close their own requests
- [A] Requests can no longer be accepted through manual closures
- [A] Requests are now closed via their relevent actions
- Deletion
- Through `/packages` bulk delete action
- Through `/pkgbase/{name}/delete`
- Merge
- Through `/pkgbase/{name}/merge`
- Orphan
- Through `/packages` bulk disown action
- Through `/pkgbase/{name}/disown`
- Deletion and merge requests (and their closures) are now autogenerated
if no pre-existing request exists. This was done to increase tracking of
package modifications performed by those with access to do so (TUs).
- Deletion, merge and orphan request actions now close all (1 or more)
requests pertaining to the action performed. This comes with the downside
of multiple notifications sent out about a closure if more than one
request (or no request) exists for them
- Merge actions now automatically reject other pre-existing merge requests
with a mismatched `MergeBaseName` column when a merge action is performed
- The last `/requests` page no longer goes nowhere
Package Bulk Actions: /packages
-------------------------------
- The `Merge into` field has been removed. Merges now require being
performed via the `/pkgbase/{name}/merge` action.
Package View
------------
- Some cached metadata is no longer cached (pkginfo). Previously,
this was defaulted to a one day cache for some package information.
If we need to bring this back, we can.
TU Proposals
------------
- A valid username is now required for any addition or removal of a TU.
RPC
---
- `type=get-comment-form` has been removed and is now located at
`/pkgbase/{name}/comments/{id}/form`.
- Support for versions 1-4 have been removed.
- JSON key ordering is different than PHP's JSON.
- `type=search` performance is overall slightly worse than PHP's. This
should not heavily affect users, as a 3,000 record query is returned
in roughly 0.20ms from a local standpoint. We will be working on this
in aim to push it over PHP.
Archives
--------
- Added metadata archive `packages-meta-v1.json.gz`.
- Added metadata archive `packages-meta-ext-v1.json.gz`.
- Enable this by passing `--extended` to `aurweb-mkpkglists`.
Performance Changes
-------------------
As is expected from a complete rewrite of the website, performance
has changed across the board. In most places, Python's implementation
now performs better than the pre-existing PHP implementation, with the
exception of a few routes. Notably:
- `/` loads much quicker as it is now persistently cached forcibly
for five minutes at a time.
- `/packages` search is much quicker.
- `/packages/{name}` view is slightly slower; we are no longer caching
various pieces of package info for `cache_pkginfo_ttl`, which is
defaulted to 86400 seconds, or one day.
- Request actions are slower due to the removal of the `via` parameter.
We now query the database for requests related to the action based on
the current state of the DB.
- `/rpc?type=info` queries are slightly quicker.
- `/rpc?type=search` queries of low result counts are quicker.
- `/rpc?type=search` queries of large result counts (> 2500) are slower.
- We are not satisfied with this. We'll be working on pushing this
over the edge along with the rest of the DB-intensive routes.
However, the speed degredation is quite negligible for users'
experience: 0.12ms PHP vs 0.15ms Python on a 3,000 record query
on my local 4-core 8-thread system.
Upcoming Work
-------------
This release is the first major release of the Python implementation.
We have multiple tasks up for work immediately, which will bring us
a few more minor versions forward as they are completed.
- Update request and tu vote pagers
- Archive differentials
- Archive mimetypes
- (a) Git scripts to ORM conversion
- (a) Sharness removal
- Restriction of number of requests users can submit
