Commit graph

64 commits

Author SHA1 Message Date
Lukas Fleischer
824744f1d2 Add a field for package base popularity
Create a new field Popularity in the PackageBases table. The field is
updated by the popupdate script and reflects the popularity of a
package. Popularity is the sum of all votes with each vote being
weighted with a factor of 0.98 per day since its creation.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-11 18:59:52 +02:00
Lukas Fleischer
0345914c5c Add popupdate script
This adds a simple script that can be used to recalculate the number of
votes per package. It can be used in a cronjob.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-11 18:59:46 +02:00
Lukas Fleischer
18292e7114 aurblup: Fix path to configuration file
Fixes a regression introduced in commit 10ecd39 (Restructure scripts,
2015-06-01).

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-04 11:25:01 +02:00
Lukas Fleischer
10ecd3982d Restructure scripts
* Move scripts/git-integration/ to git-interface/.
* Move scripts/aurblup/aurblup.py to scripts/aurblup.py.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-02 10:34:34 +02:00
Lukas Fleischer
451e60d91d git-update: Check for missing install and source files
Suggested-by: Marcel Korpel <marcel.korpel@gmail.com>
Suggested-by: carstene1ns <arch@carsten-teibes.de>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-01 23:25:59 +02:00
Lukas Fleischer
b44411fb53 Use gitnamespaces for efficient storage
Instead of using one Git repository per package, use a single large
object storage for space efficiency. The refs of the individual package
bases are divided using gitnamespaces(7) which allows for exposing each
namespace as an independent repository easily. Also, git-serve is
modified to create a branch for each package, allowing to browse the
large repository with cgit.

Helped-by: Florian Pritz <bluewind@xinu.at>
Helped-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-06-01 23:24:52 +02:00
Lukas Fleischer
c3614c4f09 Change default configuration to use SSH port 22
We no longer run a separate SSH daemon on port 2222. Change the default
configuration accordingly. Also remove some configuration files that are
no longer needed.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-05-21 11:20:17 +02:00
Lukas Fleischer
42b9e4dd28 Remove the OpenSSH patch
Extended AuthorizedKeysCommand parameters are now officially supported
by OpenSSH.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-05-21 10:54:20 +02:00
Lukas Fleischer
ef1f3798a0 Update the OpenSSH patch
Use the latest version of Damien Miller's patch to extend the parameters
to the AuthorizedKeysCommand.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2015-04-11 14:08:30 +02:00
Lukas Fleischer
cc1e8aed30 Rename the AUR software to aurweb
Rename the project to help differentiate between the software providing
access to the Arch User Repository and the collection of source packages
itself.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-02-04 09:50:01 +01:00
Lukas Fleischer
3488581046 git-update.py: Check for mandatory fields
Check explicitly whether pkgver, pkgrel, pkgname, pkgdesc and url are
available in each package. If any of these is missing, an exception
might occur when parsing the meta data later.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-15 09:35:01 +01:00
Lukas Fleischer
4dbd562b8e Use custom templates for Git repositories
Do not waste disk space by copying dozens of unneeded sample hooks. Use
a custom template directory that only includes the git-update hook.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-11 18:44:01 +01:00
Lukas Fleischer
cb661cd971 Add to notification list on repository creation
Since commit bf7717a (git-serve.py: Adopt repositories on creation,
2015-01-06), newly created package bases are no longer orphan and
therefore, they are not adopted when pushing the first commit which also
means that the initial submitter is not added to the notification list
automatically. To remedy this, add the owner to the notification list
when setting up a new repository.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-11 01:06:35 +01:00
Lukas Fleischer
3c171d353f Rewrite aurblup in Python
The AUR backend already uses several Python scripts, rewrite the aurblup
helper as well. This has several advantages:

* We can easily use the main configuration file without using any shell
  script wrappers.

* aurblup does not need to be recompiled on libalpm soname bumps.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-09 13:14:02 +01:00
Lukas Fleischer
7dd78de3fd Merge branch 'maint' 2015-01-09 09:38:56 +01:00
Lukas Fleischer
5816cc7faa aurblup: Do not blacklist provides
This partly reverts commit ddc5435 (Add packages' provides and replaces
to the blacklist in aurblup., 2011-02-08). While adding replaces is
fine, blacklisting provides prevents from uploading alternative
implementations of a program which is not what we want.

Fixes FS#43381.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-09 09:38:10 +01:00
Lukas Fleischer
2d96b51cb5 git-update.py: Do not wipe repository descriptions
Only update repository descriptions if there is at least one package in
the package base meta data.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-08 09:18:40 +01:00
Lukas Fleischer
ed7fb91f41 git-update.py: Reject blacklisted packages
Reject commits adding packages which are in the official repositories
already.

Fixes FS#43371.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-08 09:18:40 +01:00
Lukas Fleischer
ebf8f5d16f git-serve.py: Allow TUs to push to any repository
Implements FS#32807.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-07 12:21:22 +01:00
Lukas Fleischer
fc23a9bd50 Add support for package base co-maintainers
This allows for having multiple co-maintainers for AUR packages.
Co-maintainers have push access to the package base Git repository but
are not allowed to change the package base category, disown the package
or modify the list of co-maintainers. The primary maintainer of an AUR
package can edit the list of co-maintainers from the Package Actions
box.

Implements FS#17911.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-07 12:21:21 +01:00
Lukas Fleischer
b32458cb8a git-update.py: Support architecture-specific fields
Fixes FS#43356.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-07 10:41:44 +01:00
Lukas Fleischer
bf7717ac13 git-serve.py: Adopt repositories on creation
Automatically assign ownership when creating a new (empty) repository.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-06 23:33:43 +01:00
Lukas Fleischer
02dd9c518f git-serve.py: Automatically create repositories
Create a fresh Git repository when cloning or pushing using a path that
does not yet exist.

Implements FS#43308.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-06 23:29:35 +01:00
Lukas Fleischer
e8c48e9f4c git-serve.py: Relax path validation
Make the terminating slash in repository URLs optional.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-02 12:20:09 +01:00
Lukas Fleischer
289ff0c700 git-serve.py: Add a command to list repositories
The list-repos command now lists all repositories you maintain. Empty
repositories are prefixed with an asterisk.

Implements FS#43288.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-01 17:29:29 +01:00
Lukas Fleischer
1b627a3f0b git-serve.py: Improve error messages
Also, add a help command that lists available options.

Implements FS#43287.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2015-01-01 17:29:29 +01:00
Lukas Fleischer
965f908e15 git-update.py: Cast epoch to an integer
Convert epoch values before doing integer comparisons. Also, add a
sanity check for the epoch variable.

Reported-by: Ido Rosen <ido@kernel.org>
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-30 12:38:12 +01:00
Lukas Fleischer
9bb93fad75 Add a script to initialize Git repositories
Introduce a script that creates one repository for each package base in
the database.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-28 17:37:05 +01:00
Lukas Fleischer
00e6e4a182 git-serve.py: Use virtual paths
Pretend that all repositories are located in the root directory of the
file system to achieve location transparency.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-28 15:38:52 +01:00
Lukas Fleischer
332d0f61e4 Remove legacy scripts
These are no longer needed. We use Git repositories now.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-27 12:42:12 +01:00
Lukas Fleischer
9ed559eb33 git-update.py: Update repository description
Use the package description of the first package as the Git repository
description.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-27 12:42:12 +01:00
Lukas Fleischer
c3c110c09d Honor the database socket setting in Git scripts
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-27 12:42:12 +01:00
Lukas Fleischer
e1aac98bf5 Add sshd setup instructions
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-27 12:42:12 +01:00
Lukas Fleischer
943b6bc976 Add update hook template
This adds a script that can be used as an update hook to check all
commits for validity and to regenerate the package details page before
updating a named ref.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-27 12:42:12 +01:00
Lukas Fleischer
ad17b9e2b4 Add basic Git authentication/authorization scripts
This adds two scripts to be used together with Git over SSH:

* git-auth.py is supposed to be used as AuthorizedKeysCommand. It checks
  whether the public key belongs to any AUR user and invokes
  git-serve.py, passing the name of the corresponding user as a command
  line argument, if any.

* git-serve.py is a wrapper around git-shell(1) that checks whether the
  user passed as command line argument has access to the Git repository
  that a push operation writes to.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-27 12:42:12 +01:00
Lukas Fleischer
f55e3c21c6 mkpkglists.py: Honor database socket setting
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-12-27 12:42:11 +01:00
Lukas Fleischer
73e7d97277 mkpkglists: Use the main configuration file
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-10-24 10:03:54 +02:00
Lukas Fleischer
76343fb915 Use an INI-style configuration file
Replace web/lib/config.inc.php with an INI-style configuration file.
This allows us to get rid of several globals and makes it easier to use
the same configuration file in external scripts.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-10-24 10:03:54 +02:00
Lukas Fleischer
725a4778db Add a script to generate package lists
This can be used to regularly generate lists of packages and package
bases that are made available under /packages.gz and /pkgbase.gz,
respectively.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-10-10 10:57:47 +02:00
Lukas Fleischer
676595f9bf Prefix package functions with pkg_/pkgbase_
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2014-04-05 15:55:31 +02:00
Lukas Fleischer
6b1ba5e48f aurblup: Update for pacman 4.1 API changes
alpm_db_register_sync() and alpm_option_get_syncdbs() have been renamed.
Update aurblup sources accordingly.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2013-04-28 13:44:32 +02:00
Lukas Fleischer
6dc61e7d9e Remove aurblup configuration file parser
Drop the (very bad) PHP parser and allow for passing all necessary
configuration via command line parameters.

Also, add a convenience wrapper written in PHP that parses the
configuration file and subsequently calls aurblup with correct command
line options.

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2013-02-15 13:25:56 +01:00
Dan McGee
99a2bb81cf aurblup: update for pacman 4 revised libalpm API
Notable changes include the necessary handle object and the splitting
of provides and replaces into alpm_depend_t objects.

Signed-off-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2011-10-24 17:57:52 +02:00
Dan McGee
d20f90ead3 aurblup: remove use of alpm_list_getdata()
This is not strictly necessry as listitem->data is public.

Signed-off-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2011-10-24 17:57:51 +02:00
Dan McGee
156bfbddb9 aurblup: style cleanups
Always use two lines for if statements, use a character constant rather
than the 0 integer when NULL-terminating a string, and remove the
unnecessary NULL check before free(value)- free(NULL) is a no-op and
always safe.

Signed-off-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2011-10-24 17:57:51 +02:00
Dan McGee
f5736ace65 aurblup: remove ';' from queries
The semicolons are not necessary when performing queries via the MySQL
API, so remove them.

Signed-off-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2011-10-24 17:57:51 +02:00
Dan McGee
c69c44050e aurblup: Update config.h.proto
* Switch to x86_64 architecture by default
* Add more repositories (including multilib)
* Update default configuration path to include .php file extension

Signed-off-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2011-08-19 10:21:14 +02:00
Lukas Fleischer
0ac8970b62 Rename package_exists() to pkgid_from_name()
Describe what this function actually does: Return the ID of a package
with a given name and return NULL if such a package doesn't exist.

The function name is chosen in a fashion similar to other functions from
"pkgfuncs.inc.php" (pkgname_from_id(), pkgnotify_from_sid(), ...).

Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2011-08-11 16:17:07 +02:00
Dan McGee
6ae2bc514f Add a upload directory transform script
This goes with the previous patch that moves uploads into segmented
subdirectories. To actually run, follow the DRYRUN instructions.

Signed-off-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2011-08-10 15:23:00 +02:00
Dan McGee
9a79d2105e Segment the upload directory by package name prefix
This implements the following scheme:

* /packages/cower/ --> /packages/co/cower/
* /packages/j/     --> /packages/j/j/
* /packages/zqy/   --> /packages/zq/zqy/

We take up to the first two characters of each package name as a
intermediate subdirectory, and then the full package name lives
underneath that. Shorter named packages live in a single letter
directory.

Why, you ask? Well because earlier today the AUR hit 32,000 entries in
the unsupported/ directory, making new package uploads impossible. While
some might argue we shouldn't have so many damn packages in the repos,
we should be able to handle this case.

Why two characters instead of one? Our two biggest two-char groups, 'pe'
and 'py', both start with 'p', and have nearly 2000 packages each. Go
Python and Perl.

Still needed is a "move the existing data" script, as well as a set of
rewrite rules for those wishing to preserve backward compatible URLs for
any helper programs doing the wrong thing and relying on them.

Signed-off-by: Dan McGee <dan@archlinux.org>
Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de>
2011-08-10 14:34:07 +02:00