we should be suggesting to make a shallow clone to reduce
the amount of data that is being transferred initially
Signed-off-by: moson-mo <mo-son@mailbox.org>
See doc/git-archive.md for general Git archive specifications
See doc/repos/metadata-repo.md for info and direction related to the new Git metadata archive
The POST description was ridiculously confusing. This cleans up the
doc a bit and is hopefully a bit more straight-forward.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This change brings some new additions to our archives:
- SHA-256 .sha256 hexdigests
- We construct our archives in a tmpdir now and move them to the
archive destination when all are completed. This removes some
corrupted downloading when archiving is in-process.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Release v6.0.0 - Python
This documents UX and functional changes for the v6.0.0 aurweb release.
Following this release, we'll be working on a few very nice features
noted at the end of this article in Upcoming Work.
Preface
-------
This v6.0.0 release makes the long-awaited Python port official.
Along with the development of the python port, we have modified a
number of features. There have been some integral changes to how
package requests are dealt with, so _Trusted Users_ should read
the entirety of this document.
Legend
------
There are a few terms which I'd like to define to increase
understanding of these changes as they are listed:
- _self_
- Refers to a user viewing or doing something regarding their own account
- _/pkgbase/{name}/{action}_
- Refers to a POST action which can be triggered via the relevent package
page at `/{pkgbase,packages}/{name}`.
Grouped changes explained in multiple items will always be prefixed with
the same letter surrounded by braces. Example:
- [A] Some feature that does something
- [A] The same feature where another thing has changed
Infrastructure
--------------
- Python packaging is now done with poetry.
- SQLite support has been removed. This was done because even though
SQLAlchemy is an ORM, SQLite has quite a few SQL-server-like features
missing both out of the box and integrally which force us to account
for the different database types. We now only support mysql, and should
be able to support postgresql without much effort in the future.
Note: Users wishing to easily spin up a database quickly can use
`docker-compose up -d mariadb` for a Docker-hosted mariadb service.
- An example systemd service has been included at `examples/aurweb.service`.
- Example wrappers to `aurweb-git-(auth|serve|update)` have been included
at `examples/aurweb-git-(auth|serve|update).sh` and should be used to
call these scripts when aurweb is installed into a poetry virtualenv.
HTML
----
- Pagers have all been modified. They still serve the same purpose, but
they have slightly different display.
- Some markup and methods around the website has been changed for
post requests, and some forms have been completely reworked.
Package Requests
----------------
- Normal users can now view and close their own requests
- [A] Requests can no longer be accepted through manual closures
- [A] Requests are now closed via their relevent actions
- Deletion
- Through `/packages` bulk delete action
- Through `/pkgbase/{name}/delete`
- Merge
- Through `/pkgbase/{name}/merge`
- Orphan
- Through `/packages` bulk disown action
- Through `/pkgbase/{name}/disown`
- Deletion and merge requests (and their closures) are now autogenerated
if no pre-existing request exists. This was done to increase tracking of
package modifications performed by those with access to do so (TUs).
- Deletion, merge and orphan request actions now close all (1 or more)
requests pertaining to the action performed. This comes with the downside
of multiple notifications sent out about a closure if more than one
request (or no request) exists for them
- Merge actions now automatically reject other pre-existing merge requests
with a mismatched `MergeBaseName` column when a merge action is performed
- The last `/requests` page no longer goes nowhere
Package Bulk Actions: /packages
-------------------------------
- The `Merge into` field has been removed. Merges now require being
performed via the `/pkgbase/{name}/merge` action.
Package View
------------
- Some cached metadata is no longer cached (pkginfo). Previously,
this was defaulted to a one day cache for some package information.
If we need to bring this back, we can.
TU Proposals
------------
- A valid username is now required for any addition or removal of a TU.
RPC
---
- `type=get-comment-form` has been removed and is now located at
`/pkgbase/{name}/comments/{id}/form`.
- Support for versions 1-4 have been removed.
- JSON key ordering is different than PHP's JSON.
- `type=search` performance is overall slightly worse than PHP's. This
should not heavily affect users, as a 3,000 record query is returned
in roughly 0.20ms from a local standpoint. We will be working on this
in aim to push it over PHP.
Archives
--------
- Added metadata archive `packages-meta-v1.json.gz`.
- Added metadata archive `packages-meta-ext-v1.json.gz`.
- Enable this by passing `--extended` to `aurweb-mkpkglists`.
Performance Changes
-------------------
As is expected from a complete rewrite of the website, performance
has changed across the board. In most places, Python's implementation
now performs better than the pre-existing PHP implementation, with the
exception of a few routes. Notably:
- `/` loads much quicker as it is now persistently cached forcibly
for five minutes at a time.
- `/packages` search is much quicker.
- `/packages/{name}` view is slightly slower; we are no longer caching
various pieces of package info for `cache_pkginfo_ttl`, which is
defaulted to 86400 seconds, or one day.
- Request actions are slower due to the removal of the `via` parameter.
We now query the database for requests related to the action based on
the current state of the DB.
- `/rpc?type=info` queries are slightly quicker.
- `/rpc?type=search` queries of low result counts are quicker.
- `/rpc?type=search` queries of large result counts (> 2500) are slower.
- We are not satisfied with this. We'll be working on pushing this
over the edge along with the rest of the DB-intensive routes.
However, the speed degredation is quite negligible for users'
experience: 0.12ms PHP vs 0.15ms Python on a 3,000 record query
on my local 4-core 8-thread system.
Upcoming Work
-------------
This release is the first major release of the Python implementation.
We have multiple tasks up for work immediately, which will bring us
a few more minor versions forward as they are completed.
- Update request and tu vote pagers
- Archive differentials
- Archive mimetypes
- (a) Git scripts to ORM conversion
- (a) Sharness removal
- Restriction of number of requests users can submit
Two new archives are available:
- packages-meta-v1.json.gz
- RPC search formatted data for all packages
- ~2.1MB at the time of writing.
- packages-meta-ext-v1.json.gz (via --extended)
- RPC multiinfo formatted data for all packages.
- ~9.8MB at the time of writing.
New dependencies are required for this update:
- `python-orjson`
All archives served out by aur.archlinux.org distribute the Last-Modified
header and support the If-Modified-Since header, which should be
populated with Last-Modified's value. These should be used by clients
to avoid redownloading the archive when unnecessary.
Additionally, the new meta archives contain a format suitable for
streaming the data as the file is retrieved. It is still in JSON
format, however, users can parse package objects line by line after
the first '[' found in the file, until the last ']'; both contained
on their own lines.
Note: This commit is a documentation change and commit body.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Two new archives are available:
- packages-meta-v1.json.gz
- RPC search formatted data for all packages
- ~2.1MB at the time of writing.
- packages-meta-ext-v1.json.gz (via --extended)
- RPC multiinfo formatted data for all packages.
- ~9.8MB at the time of writing.
New dependencies are required for this update:
- `python-orjson`
All archives served out by aur.archlinux.org distribute the Last-Modified
header and support the If-Modified-Since header, which should be
populated with Last-Modified's value. These should be used by clients
to avoid redownloading the archive when unnecessary.
Additionally, the new meta archives contain a format suitable for
streaming the data as the file is retrieved. It is still in JSON
format, however, users can parse package objects line by line after
the first '[' found in the file, until the last ']'; both contained
on their own lines.
Note: This commit is a documentation change and commit body.
Signed-off-by: Kevin Morris <kevr@0cost.org>
* Rename the aur project to aurweb on Transifex.
* Rename aur.pot to aurweb.pot.
* Update documentation and Makefile.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
It is now possible to search for packages that depend on a given package,
for instance:
/rpc/?v=5&type=search&by=depends&arg=ocaml
It is similarly possible to match on "makedepends", "checkdepends" and
"optdepends".
Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
AUR_PRIVILEGED allows people with privileged AUR accounts to evade the
block on non-fast-forward commits. While valid in this case, we should
not do so by default, since in at least one case a TU did this without
realizing there was an existing package.
( https://aur.archlinux.org/packages/rtmidi/ )
Switch to using allow_overwrite to check for destructive actions.
Use .ssh/config "SendEnv" on the TU's side and and sshd_config
"AcceptEnv" in the AUR server to specifically request overwrite access.
TUs should use: `AUR_OVERWRITE=1 git push --force`
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Add a note to the Git/SSH interface documentation that we recommend to
disable automatic garbage collection and use a maintenance script to
cleanup and optimize the Git repository instead.
Also, add a reference to the Git/SSH interface documentation to the Git
repository setup instructions in INSTALL.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
In 002d348 (Describe how to omit "have" lines, 2015-11-14), we added
instructions on how to omit "have" lines originating from other package
repositories. Fix those instructions such that the HEAD ref of the
repository is transferred properly.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
A new feature in Git allows for omitting "have" lines corresponding to
refs outside the current Git namespace. Explain how to enable this
feature in the INSTALL instructions and in the Git interface
documentation.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Convert the RPC interface documentation from web/html/rpc.php to
AsciiDoc and add it to the documentation directory.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Update the link to the project page on Transifex and remove an outdated link
to the Arch Wiki.
Fixes FS#45966.
Signed-off-by: Stefan Auditor <stefan.auditor@erdfisch.de>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Implement a new command that can be used to restore deleted package
bases without having to push a new commit.
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
