Commit graph

2675 commits

Author SHA1 Message Date
Kevin Morris
7c65604dad move off env.py's active code to __name__ == "__main__"
* Moved migrations/env.py's logging initialization and migration execution
  into a `__name__ == "__main__"` stanza so it doesn't immediately happen
  when imported by another module.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:50:51 -07:00
Kevin Morris
2df90ce280 port over base HTML layout from PHP to FastAPI+Jinja2
+ Mounted static files (at web/html) to /static.
+ Added AURWEB_VERSION to aurweb.config (this is used around HTML
  to refer back to aurweb's release on git.archlinux.org), so we
  need it easily accessible in the Python codebase.
+ Implemented basic Jinja2 partials to put together whole aurweb
  pages. This may be missing some things currently and is a WIP
  until this set is ready to be merged.
+ Added config [options] aurwebdir = YOUR_AUR_ROOT; this configuration
  option should specify the root directory of the aurweb project.
  It is used by various parts of the FastAPI codebase to target
  project directories.

Added routes via aurweb.routers.html:
    * POST /language: Set your session language.
    * GET /favicon.ico: Redirect to /static/images/favicon.ico.
        * Some browsers always look for $ROOT/favicon.ico to get an icon
          for the page being loaded, regardless of a specified "shortcut
          icon" given in a <link> directive.
    * GET /: Home page; WIP.

* Updated aurweb.routers.html.language passes query parameters to
  its next redirection.

When calling aurweb.templates.render_template, the context passed should
be formed via the aurweb.templates.make_context. See
aurweb.routers.html.index for an example of this.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:50:51 -07:00
Jelle van der Waa
1ff822bb14 Use the clipboard API for copy paste
The Document.execCommand API is deprecated and no longer recommended to
be used. It's replacement is the much simpler navigator.clipboard API
which is supported in all browsers except internet explorer.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-06-05 19:50:51 -07:00
Marcus Andersson
bda9256ab1 Add error color when package is orphaned
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-06-05 19:49:55 -07:00
Kevin Morris
c1e29e90ca aurweb: Globalize a Translator instance, add more utility
+ Added SUPPORTED_LANGUAGES, a global constant dictionary of
  language => display pairs for languages we support.
+ Add Translator.get_translator, a function used to retrieve a
  translator after initializing it (if needed). Use `fallback=True`
  while creating languages, in case we setup a language that we
  don't have a translation for, it will noop the translation.
  This is particularly useful for "en," since we do not translate
  it, but doing this will allow us to go through our normal translation
  flow in any case.
+ Added typing.
+ Added get_request_language, a function that grabs the language for
  a request session, defaulting to aurweb.config [options] default_lang.
+ Added get_raw_translator_for_request, a function that retrieves
  the concrete translation object for a given language.
+ Added tr, a jinja2 contextfilter that can be used to inline translate
  strings in jinja2 templates.
+ Added `python-jinja` dep to .gitlab-ci.yml. This needs to be
  included in documentation before this set is merged in.
+ Introduce pytest units (test_l10n.py) in `test` along with
  __init__.py, which marks `test` as a test package.
+ Additionally, fix up notify.py to use the global translator. Also
  reduce its source width to <= 80 by newlining some code.
+ Additionally, prepare locale in .gitlab-ci.yml and add
  aurweb.config [options] localedir to config.dev with YOUR_AUR_ROOT
  like others.

Signed-off-by: Kevin Morris <kevr@0cost.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-06-05 19:49:42 -07:00
Leonidas Spyropoulos
21140e28a8 Filter out current username from co-maintainers list.
Closes: #8

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-06-05 19:49:42 -07:00
Kevin Morris
cd3e880264 add Dockerfile
This docker file downloads deps, sets up some things beforehand and
finishes with running our entire collection of tests.

Signed-off-by: Kevin Morris <kevr@0cost.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-06-05 19:49:19 -07:00
Kevin Morris
52ab056e18 update documentation for FastAPI tests and deps.
Additionally, we now ask for two more favors from contributors:

1. All source modified or added within a patchset **must** maintain
   equivalent or increased coverage by providing tests that use the
   functionality.
2. Please keep your source within an 80 column width.

PS: Sneak a few test Makefile and gitlab fixes.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:49:19 -07:00
Kevin Morris
57c11ae13f install aurweb package & init db on GitLab CI
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:49:19 -07:00
Kevin Morris
6d08789ac1 add test_popupdate.py
We had no coverage over aurweb.scripts.popupdate. This test covers
all of its functionality.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:49:19 -07:00
Kevin Morris
4b7609681d add test_exceptions.py
This helps gain coverage over aurweb.exceptions regardless
of their actual use in the testing base.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:49:19 -07:00
Kevin Morris
e800cefe95 Makefile: run pytest units
Important note: Python tests will repeatedly clear out tables
that they test against; for this reason, one should always run
the shell tests first. The __init__.py file is necessary for
coverage to collect data from the tests being run.

At this point in FastAPI development, I'd like to encourage a
few things going forward:

1. Any time you contribute to the FastAPI codebase, you **must**
   maintain equal or increased coverage on the overall source.
   Developers are highly appreciated for adding tests in your
   specific domain of addition or modification that may be missing
   coverage. Our goal is 100% coverage, and all newly added files
   **must** have 100% coverage through tests.
2. All source should be formatted with the autopep8 tool and
   kept within an 80 column width, with the exception of HTML
   templates.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:49:19 -07:00
Kevin Morris
4230772e3b add .coveragerc, update .gitignore
Now, .coveragerc enforces a minimum overall 82% coverage, as is
the current standing. Providing less than 100% coverage for added
code should reduce the overall coverage and eventually reach 81%
or below, causing coverage report to fail on execution.

Developers should increase the failure minimum as they increase
coverage across the uncovered code.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:49:19 -07:00
Kevin Morris
bac38edd48 [db] fix schema and migration for case insensitivity
Some of the columns that were changed still want to be
case insensitive. Good thing our tables have nice
separation.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 15:08:18 -07:00
Leonidas Spyropoulos
b1121dc6ca Adds Alembic migration for DB/Tables conversion to utf8mb4
MySql defaults to `utf8` and case insensitive collation so migrate these to case sensitive and `utf8mb4`

Closes #21

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
2021-05-18 05:42:36 -07:00
Leonidas Spyropoulos
0d68b914bf Conditionally apply SSOAccountId migration to support misaligned databases
Closes: #34

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
2021-05-18 05:42:36 -07:00
Kevin Morris
82f6d2ce75 alembic: fix ef39fcd6e1cd downgrade
op.drop_constraint requires a valid field to drop the constraint on.
Without this, downgrade cannot occur.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-05-18 05:42:36 -07:00
Kevin Morris
b41422450a aurweb.db: only pass check_same_thread with sqlite
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-05-18 05:42:36 -07:00
Kevin Morris
25393dc326 Fix incorrect construction of MySQL SQLAlchemy URL
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-05-18 05:42:36 -07:00
Marcus Andersson
4fa220850f
Add error color when package is orphaned
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-05-13 16:50:51 -04:00
Eli Schwartz
e454a0200c
git update hook: gracefully error on completely broken .SRCINFO
I've seen this happen a bunch of times now. Someone cannot push to the
AUR, and the error report is some traceback with a KeyError which is
difficult to understand without context:

remote: Traceback (most recent call last):
remote:   File "/srv/http/aurweb/aur.git/hooks/update", line 33, in <module>
remote:     sys.exit(load_entry_point('aurweb==5.0.0', 'console_scripts', 'aurweb-git-update')())
remote:   File "/usr/lib/python3.9/site-packages/aurweb-5.0.0-py3.9.egg/aurweb/git/update.py", line 306, in main
remote: KeyError: 'pkgbase'

Eventually it turns out that their .SRCINFO file is... badly corrupted.
Generally, they managed to accidentally commit an *empty* file instead
of a .SRCINFO, and in all cases, the problem was on the very first
lookup for 'pkgbase'.

Point people to the actual failing commit, and have a nicely formatted
message indicating that the .SRCINFO is completely invalid.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-05-10 23:22:00 -04:00
Kevin Morris
19652d6cbe
swap uvicorn out for hypercorn
uvicorn is subjectively nicer to play with for local dev work, but
hypercorn is required in order to do HTTP/2 which is fairly
performance-important.

Signed-off-by: Kevin Morris <kevr@0cost.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
Co-authored-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-05-10 23:22:00 -04:00
Kristian Klausen
bab74dd307
Update wiki links to the new short URL
Done with: find -type f -exec sed -Ee ':wiki.archlinux.org: s:(wiki.archlinux.org)/index.php/:\1/title/:g' -i {} \;

Fixes #16

[1] https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/335

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-05-09 18:06:32 -04:00
Eli Schwartz
8ec170b3e0
dos2unix a file with Windows linebreaks that editors and human reviewers hate
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-05-02 21:46:35 -04:00
Eli Schwartz
c3035a9039
add https://EditorConfig.org setup to ensure consistent style
Mostly here to make sure people continue to use tabbed indents for php
and the TAP tests, since that is what they are currently using.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
2021-04-28 18:06:06 -04:00
Jelle van der Waa
d668ef0bcd Resolve SQL Error when deleting an account
The account deletion code tries to remove user from PackageNotifications
using the wrong column UsersID to identify the user by id. In the
PackagePackageNotifications table the foreign key is called UserID. In
the future ideally this would be unified into UserID for all tables.

Closes: #12
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-03-29 18:39:02 -04:00
Felix Yan
21c457817f Use jsDelivr instead of Google CDN for jquery
jsdelivr is another free CDN service for open source projects.

The main motivation for this change is that it is the only one that works fairly
well across the globe. The Google CDN service is known to be hardly
accessible in mainland China, unfortunately.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:26:32 -05:00
Lukas Fleischer
933d2705f9 Fetch Transifex image from https://www.transifex.com
Fixes GitLab issue #3.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Lukas Fleischer
62b413f6b7 .gitignore: add test/trash directory*
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Justin Kromlinger
d5d333005e RSS: Decrease cache time and increase item count
I think after 10-15 years we might want to adjust those values. With a
30min cache and 20 items per creation I would bet some new AUR packages
might be swept under the carpet.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Justin Kromlinger
eb11943fed RSS: Always provide a GUID
https://validator.w3.org/feed/docs/warning/MissingGuid.html
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Justin Kromlinger
1d0c6ffe24 RSS: Make sure image title matches channel title
https://validator.w3.org/feed/docs/warning/ImageTitleDoesntMatch.html
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Justin Kromlinger
78dbbd3dfa RSS: Set proper content type header
https://validator.w3.org/feed/docs/warning/UnexpectedContentType.html
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Justin Kromlinger
568e0d2fa3 RSS: Add atom self link
https://validator.w3.org/feed/docs/warning/MissingAtomSelfLink.html
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Jakub Klinkovský
bc972089a1 Fix WHERE clause for keyword search queries with empty keywords
When the keyword parameter is empty, the AND clause has to be omitted,
otherwise we get an SQL syntax error:

... WHERE PackageBases.PackagerUID IS NOT NULL AND () ...

This got broken in commit 9e30013aa4fc6ce3a3c9f6f83a6fe789c1fc2456
Author: Kevin Morris <kevr.gtalk@gmail.com>
Date:   Sun Jul 5 18:19:06 2020 -0700

Support conjunctive keyword search in RPC interface

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Jakub Klinkovský
3062a78a92 gendummydata.py: optimize iteration for big numbers of pkgs
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Jakub Klinkovský
51a3535820 gendummydata.py: set MAX_USERS and MAX_PKGS to more realistic values
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Jakub Klinkovský
879c0622d6 gendummydata.py: set exit code to 1 when there is an error
Of course the default exit code is 0...

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Jakub Klinkovský
92e315465b gendummydata.py: remove unused database connection variables
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Filipe Laíns
db75a5528e doc: simplify database setup instructions in TESTING
Signed-off-by: Filipe Laíns <lains@archlinux.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Filipe Laíns
e62d472708 doc: add missing gendummydata.py dependencies in TESTING
Signed-off-by: Filipe Laíns <lains@archlinux.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Filipe Laíns
4e4f5855f1 doc: fix AUR_CONFIG in TESTING
Signed-off-by: Filipe Laíns <lains@archlinux.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Filipe Laíns
83d228d9e8 spawn: expand AUR_CONFIG to the full path
This allows using a relative path for the config. PHP didn't play well
with it.

Signed-off-by: Filipe Laíns <lains@archlinux.org>
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Frédéric Mangano-Tarumi
8c28ba6e7f Redirect to referer after SSO login
Introduce a `redirect` query argument to SSO login endpoints so that
users are redirected to the page they were originally on when they
clicked the Login link.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:21 -05:00
Frédéric Mangano-Tarumi
87815d37c0 Remove the per-user session limit
This feature was originally introduced by
f961ffd9c7 as a fix for FS#12898
<https://bugs.archlinux.org/task/12898>.

As of today, it is broken because of the `q.SessionID IS NULL` condition
in the WHERE clause, which can’t be true because SessionID is not
nullable. As a consequence, the session limit was not applied.

The fact the absence of the session limit hasn’t caused any issue so
far, and hadn’t even been noticed, suggests the feature is unneeded.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:25:19 -05:00
Frédéric Mangano-Tarumi
be31675b65 Guard OAuth exceptions to provide better messages
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:24:30 -05:00
Frédéric Mangano-Tarumi
5fb4fc12de HTML error pages for FastAPI
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:24:30 -05:00
Frédéric Mangano-Tarumi
202ffd8923 Update last login information on SSO login
Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:24:30 -05:00
Frédéric Mangano-Tarumi
9290eee138 Stop redirecting stderr with proc_open
Error outputs were piped to a temporary buffer that wasn’t read by
anyone, making debugging hard because errors were completely silenced.
By not explicitly redirecting stderr on proc_open, the subprocess
inherits its parent stderr.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:24:30 -05:00
Frédéric Mangano-Tarumi
a1a742b518 aurweb.spawn: Support stdout redirections to non-tty
Only ttys have a terminal size. If we can’t obtain it, we’ll just use 80
as a sane default.

Signed-off-by: Lukas Fleischer <lfleischer@archlinux.org>
2021-02-20 11:24:30 -05:00