Commit graph

66 commits

Author SHA1 Message Date
Lukas Fleischer
dd9c6f3ddc t0003: Add more git-update tests
Add tests for common scenarios that should be detected/handled by the
update hook.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 21:23:20 +02:00
Lukas Fleischer
9a03c7fbdd t0002: Add more git-serve tests
Add tests for common scenarios that should be detected/handled by the
git-serve script.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 21:23:20 +02:00
Lukas Fleischer
008eace8db t0001: Add more git-auth tests
Test the authentication script with an invalid key type and with a key
that does not exist in the database.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 21:02:21 +02:00
Lukas Fleischer
3a41f8d564 git-update: Remove package details before updating
Explicitly remove all package sources, dependencies, relations, licenses
and groups before inserting new ones.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 19:38:42 +02:00
Lukas Fleischer
6e38309c19 git-interface: Add test suite and basic tests
Add basic tests for the Git interface. The test suite is based on
sharness.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 12:12:09 +02:00
Lukas Fleischer
baf8a220ab git-interface: Support SQLite as database backend
In addition to MySQL, add support for SQLite to the database abstraction
layer. Also, add a new configuration option to select the DBMS.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 12:05:22 +02:00
Lukas Fleischer
f2a6bd207d git-interface: Do not use UNIX_TIMESTAMP
Avoid using UNIX_TIMESTAMP which is not part of the SQL standard.
Retrieve the current UNIX time in Python and substitute it into the SQL
queries instead.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 12:05:22 +02:00
Lukas Fleischer
27631f1157 git-interface: Do not use rowcount
Avoid using Cursor.rowcount to obtain the number of rows returned by a
SELECT statement as this is not guaranteed to be supported by every
database engine.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 12:05:22 +02:00
Lukas Fleischer
ecbf32f0cc git-interface: Add AUR_CONFIG environment variable
Introduce a new environment variable that can be used to specify the
path to an aurweb configuration file. If the environment variable is
unset, the default search path is used.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 12:05:22 +02:00
Lukas Fleischer
2f5f5583be git-interface: Factor out configuration file parsing
Add a new module that automatically locates the configuration file and
provides methods to obtain the values of configuration options.

Use the new module instead of ConfigParser everywhere.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 12:05:22 +02:00
Lukas Fleischer
2915abb9d3 git-interface: Add database abstraction layer
Add a new class that connects to the database specified in the
configuration file and provides an interface to execute SQL queries.
Prepared statements with qmark ("?") placeholders are supported.

Replace all direct database accesses with calls to the new abstraction
layer.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 12:05:22 +02:00
Lukas Fleischer
2cd69bf66d git-update: Make maximum blob size configurable
Support setting the maximum blob size in the configuration file.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 11:25:51 +02:00
Lukas Fleischer
87f5f1b407 git-update: Use AUR_PRIVILEGED for forced pushes
Instead of looking up the account type of the current user again, use
the AUR_PRIVILEGED environment variable to check whether the user is
allowed to perform non-fast-forward ref updates.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 01:22:52 +02:00
Lukas Fleischer
7a53ded5fe git-update: Fix some issues reported by pyflakes
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-05 01:22:51 +02:00
Lukas Fleischer
0c1187caa4 git-serve: Deprecate setup-repo
Since 02dd9c5 (git-serve.py: Automatically create repositories,
2015-01-06), one can create new package bases by running `git push`. It
is no longer necessary to run setup-repo manually.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-04 21:37:52 +02:00
Lukas Fleischer
415a2c836d git-update: Notify privileged users of forced uploads
Show a warning when a Trusted User or a developer creates a package that
is blacklisted or already provided by an official package.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-04 13:04:44 +02:00
Lukas Fleischer
573715afd9 git-serve: Refactor environment variable access
Read all environment variables at the beginning of the script and
immediately pre-process their values.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-08-04 13:04:44 +02:00
Lukas Fleischer
d273ee5eb2 Use the official provider list to detect duplicates
Instead of automatically adding packages from the official binary
repositories to the package blacklist, use the official provider list to
prevent users from uploading duplicates.

This does not only result in reduced disk usage but also has a nice
visible side effect. The error messages printed by the update hook now
look like

    error: package already provided by [community]: powerline-fonts

instead of

    error: package is blacklisted: powerline-fonts

which was confusing to most end users.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-05-18 09:22:44 +02:00
Lukas Fleischer
dc41a5afa5 git-serve: Change syntax for setting keywords
Instead of `config <pkgbase> keywords <keyword1> <keyword2>...`, the new
syntax is `set-keywords <pkgbase> <keyword1> <keyword2>...`.

The `config` keyword was rather general and it was not obvious that it
could be used to change package base settings. Instead of replacing it
with an even more verbose expression, remove that unnecessary level of
indirection. Since we do not (and probably never will) support keywords
anywhere else, the chance of name conflicts is small.

Suggested-by: Dave Reisner <dreisner@archlinux.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-02-09 21:03:47 +01:00
Lukas Fleischer
64072461df Add support for package update notifications
Introduce a new notification option to receive notifications when a new
commit is pushed to a package repository.

Implements FS#30109.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-02-07 10:54:20 +01:00
Lukas Fleischer
3412de21d3 Rename the CommentNotify table to PackageNotifications
As a preparatory step to adding support for package notifications on
events other than comments, rename the database table accordingly.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-02-07 10:39:22 +01:00
Johannes Löthberg
7a3a3876a8 git-update: Replace aurinfo.py with python-srcinfo
python-srcinfo is a more transparent and simpler library for parsing
SRCINFO files.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-02-07 09:48:32 +01:00
Lukas Fleischer
f8b3cb97e5 Fix issues reported by pyflakes
Fix several style issues and remove unneeded imports/assignments.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2016-02-06 16:23:01 +01:00
Mark Weiman
39280152ea git-serve: Fix pkgbase_exists()
Fixes a regression introduced in 8c87b1d (git-serve: Add support for
setting keywords, 2015-10-22).

Signed-off-by: Mark Weiman <mark.weiman@markzz.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-11-26 17:22:22 +01:00
Lukas Fleischer
8c87b1df0c git-serve: Add support for setting keywords
This allows for setting keywords using the SSH interface. The syntax is
`config <pkgbase> keywords <keyword1> <keyword2>...`.

Implements FS#45627.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-10-24 16:22:43 +02:00
Lukas Fleischer
62c13e2c0c Do not show ".SRCINFO unchanged" warning on restore
Since 612300b (Show a warning if .SRCINFO is unchanged, 2015-09-29), the
git-update script displays a warning when a ref update does not affect
the content of the package base meta data. We also invoke git-update to
rebuild the package base details in the aurweb database when a package
base is restored via the SSH interface. In that case, fake information
is passed to the update hook: Both the old and the new object IDs refer
to the current HEAD. Check for such "Everything up-to-date" updates and
not display the ".SRCINFO unchanged" in these cases.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-10-03 12:10:25 +02:00
Lukas Fleischer
612300b39c Show a warning if .SRCINFO is unchanged
Warn users when a remote ref update does not change the content of
.SRCINFO such that users are reminded of updating package meta data.

Implements FS#46130.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-09-29 20:54:55 +02:00
Lukas Fleischer
2c20403cc4 git-update: Use proper stop value for slice
Fixes a regression introduced in 4112e57 (Add a restore command to the
SSH interface, 2015-08-14).

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-09-16 22:10:06 +02:00
Lukas Fleischer
4112e572aa Add a restore command to the SSH interface
Implement a new command that can be used to restore deleted package
bases without having to push a new commit.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-08-16 06:49:12 +02:00
Lukas Fleischer
da875276d4 git-update: Add comments
Add some comments to explain the major steps performed in the update
hook.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-08-14 12:52:32 +02:00
Lukas Fleischer
80e06e5fc3 git-update: Remove superfluous assignment
The pkgbase variable already contains the package base name at this
point, no need to reassign it.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-08-14 12:50:35 +02:00
Lukas Fleischer
4bc6c55d98 git-update: Move blacklist reading further down
Since c4870a9 (git-update: Only check HEAD for blacklisted packages,
2015-06-04), only the HEAD commit package name is looked up in the
blacklist. This means that we no longer need to read the blacklist
before running the commit walker. Moving the blacklist reading code
further down makes the code easier to read.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-08-14 12:45:36 +02:00
Johannes Löthberg
950abb0189 aurinfo: Fix parsing with custom file when running script
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-08-04 05:57:02 +02:00
Johannes Löthberg
f19892f7c2 aurinfo: Allow lines starting with spaces
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-08-04 05:57:02 +02:00
Lukas Fleischer
38b1bbe78d git-update: Error out if PKGBUILD is missing
Fixes FS#45646.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-07-16 14:09:55 +02:00
Johannes Löthberg
f4f1921fc6 git-update: Fix error when printing SRCINFO errors
This commit fixes a bug introduced by ae2907a (git: Use .format
everywhere instead of %, 2015-06-27) where passing the error tuple to
.format wasn't prefixed with an asterisk.

Fixes FS#45545.

Reported-by: Marty <vadmium+al@gmail.com>
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-07-04 15:02:04 +02:00
Lukas Fleischer
d0be66015a git-update: Improve error message on invalid pkgbase
When the package base does not match the repository name, we currently
show an error message that only mentions the package base name:

    error: invalid pkgbase: yate-qt4

To the end user this sounds like the package base name is malformed
itself (and we show exactly the same error message when it contains
invalid characters, indeed). Clarify that there is a name mismatch by
also printing the expected value:

    error: invalid pkgbase: yate-qt4, expected: yate4-qt

Inspired by FS#45483.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-27 13:04:03 +02:00
Lukas Fleischer
b036b436aa Add support for multiple SSH public keys
Attaching more than one SSH public key to the same account is useful,
e.g. if one uses different machines to access the AUR SSH interface.
Multiple keys can now be specified by adding multiple lines to the text
area on the account edit form.

Implements FS#45469.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Acked-by: Leonidas Spyropoulos <artafinde@gmail.com>
2015-06-27 13:04:03 +02:00
Johannes Löthberg
4da759e638 git-update: Allow privileged users to bypass blacklist
Fixes FS#45410.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-27 13:04:03 +02:00
Johannes Löthberg
e93f198ede git-serve: Drop direct AccountType checking, use AUR_PRIVILEGED
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-27 13:04:03 +02:00
Johannes Löthberg
ed0e2fd975 git-auth: Set AUR_PRIVILEGED env var for TUs & devs
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-27 13:04:03 +02:00
Johannes Löthberg
e9485531be git: Use AUR_USER env var instead of ForceCommand argument
Also add an utility function for formatting the ForceCommand, using
shlex.quote to quote the value.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-27 13:04:03 +02:00
Johannes Löthberg
ae2907a57e git: Use .format everywhere instead of %
% formatting is deprecated, and .format should be used instead.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-27 13:04:03 +02:00
Johannes Löthberg
61cdf09cd0 Make url and pkgdesc fields optional
The url and pkgdesc PKGBUILD variables are optional, so they should be
in the AUR as well.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-20 14:29:43 +02:00
Johannes Löthberg
5d9a80a546 git-update: Remove regex matching of URL
The regex check that a URL is either http or ftp doesn't really have any
use, and it forces you to leave proper URLs for eg projects hosted on
gopher empty.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-20 14:28:23 +02:00
Lukas Fleischer
3bc4496328 git-update: Fix parsing of versioned optdepends
Fixes FS#43649.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-14 17:58:56 +02:00
Johannes Löthberg
cc6c7a1caa aurinfo: Fix bug with space indents creating split packages
If a .SRCINFO line doesn't start with a tab, check if the key is pkgname
before adding it as a package.

Fixes a bug where if you have accidentally gotten a line indented with
spaces, from that line forward it will think it is a split package,
instead of erroring out.

Reported-by: Raansu <Gero3977@gmail.com>
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-14 17:58:55 +02:00
Johannes Löthberg
3158e038c8 git-serve: Support cloning with simplified paths
This commit adds support for cloning repositories over SSH without a
leading slash or with a trailing .git

Fixes FS#45260

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-12 16:58:00 +02:00
Lukas Fleischer
5225dfef48 git-update: Allow non-fast-forwards for TUs
Although we theoretically never want to lose history, there may be rare
occasions when a forced push is required (e.g. if illegal data is
pushed). Allow Trusted Users and Developers to perform non-fast-forward
pushes.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-11 17:56:51 +02:00
Lukas Fleischer
a0f3060f23 git-update: Deny non-fast-forwards
To make sure we never lose any history, non-fast-forwards are forbidden.
Instead of relying on receive.denyNonFastForwards, add a simple check to
the update hook. This has the added benefit of more flexibility.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-11 17:56:26 +02:00