Previously, we were running a single ORM query for every single package
to check for its voted or notified states. Now, we perform a single
ORM query for each of the set of voted or notified packages in
relation with the request user.
This improves performance drastically at the expense of some
manual code additions and set-dependency; i.e. we add a bit
more complexity and roundabout way of getting our data.
Closes: https://gitlab.archlinux.org/archlinux/aurweb/-/issues/102
Signed-off-by: Kevin Morris <kevr@0cost.org>
For SQLAlchemy to automatically understand updates from the
external world, it must use an `autocommit=True` in its session.
This change breaks how we were using commit previously, as
`autocommit=True` causes SQLAlchemy to commit when a
SessionTransaction context hits __exit__.
So, a refactoring was required of our tests: All usage of
any `db.{create,delete}` must be called **within** a
SessionTransaction context, created via new `db.begin()`.
From this point forward, we're going to require:
```
with db.begin():
db.create(...)
db.delete(...)
db.session.delete(object)
```
With this, we now get external DB modifications automatically
without reloading or restarting the FastAPI server, which we
absolutely need for production.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The new `extend_query` and `urlencode` filters are way cleaner ways
to achieve what we did with `dedupe_qs`.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The first addition, extend_query, can be used to take an existing
query parameter dictionary and inject an *additions as replacement
key/value pairs.
The second, to_qs, converts a query parameter dictionary to
a query string in the form "a=b&c=d".
These two functions simplify and make dedupe_qs and quote_plus more
efficient in terms of constructing custom query string overrides.
Signed-off-by: Kevin Morris <kevr@0cost.org>
The possibly plural version of `tr`, `tn` provides a way to translate
strings into singular or plural form based on a given integer
being 1 or not 1.
Example use:
```
{{ 1 | tn("%d package found.", "%d packages found.") | format(1) }}
```
Signed-off-by: Kevin Morris <kevr@0cost.org>
Added:
- User.voted_for(package)
- Has a user voted for a particular package?
- User.notified(package)
- Is a user being notified about a particular package?
- User.packages()
- Entire collection of Package objects related to User.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Just like some of the other tables, we have some constant
records that we use to denote types of things. This commit
adds constants which correlate with these record constants.
Signed-off-by: Kevin Morris <kevr@0cost.org>
A helper function which provides a textual string conversion
of a particular Status column.
In a PackageRequest, Status is split up into four different types:
- PENDING : "Pending", PENDING_ID: 0
- CLOSED : "Closed", CLOSED_ID: 1
- ACCEPTED : "Accepted", ACCEPTED_ID: 2
- REJECTED : "Rejected", REJECTED_ID: 3
This commit adds constants for the textual strings and the
IDs. It also adds a PackageRequest.status_display() function which
grabs the proper display string for a particular Status ID.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This puts one more toward completion of the homepage
overall; we'll need to still implement the authenticated
user dashboard after this.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This includes the addition of the python-fakeredis package,
used for stubbing python-redis when a user does not have a
configured cache.
Signed-off-by: Kevin Morris <kevr@0cost.org>
These were being displayed regardless of the dep type
and state of DepDesc. This is fixed with this commit.
Signed-off-by: Kevin Morris <kevr@0cost.org>
A few things added with this commit:
- aurweb.packages.util
- A module providing package and pkgbase helpers.
- aurweb.template.register_filter
- A decorator that can be used to register a filter:
@register_filter("some_filter") def f(): pass
Additionally, template partials have been split off a bit
differently. Changes:
- /packages/{name} is defined in packages/show.html.
- partials/packages/package_actions.html is now
partials/packages/actions.html.
- partials/packages/details.html has been added.
- partials/packages/comments.html has been added.
- partials/packages/comment.html has been added.
- models.dependency_type additions: name and id constants.
- models.relation_type additions: name and id constants.
- models.official_provider additions: base official url constant.
Signed-off-by: Kevin Morris <kevr@0cost.org>
With a recent curl update, it now rejects URLs with spaces in it.
We should probably fix this so that we can sanitize urls with spaces
to be used properly, but for now, just remove spaces in the directory.
Signed-off-by: Kevin Morris <kevr@0cost.org>
There are slight differences in that, with `python-feedgen`,
an empty description field completely omits the description,
but includes the description when there is one.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Slight markup changes, same style overall and same
form parameters as the PHP implementation.
In addition, we've disabled the "left" and "right"
navigation buttons when we're at the border of the
table.
CSS Changes:
- Added similar styling to submit `<buttons>` that submit `<input>` had.
- Added .results tr td[align="{left,right}"] styling to align
the result table's `More -->` button to the right of the table.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Another part of the "Trusted User" collection of routes.
This allows a Trusted User to create a proposal.
New Routes:
- get `/addvote/`
- post `/addvote/`
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit ports the `/tu/?id={proposal_id}` PHP routes to
FastAPI into two individual GET and POST routes.
With this port of the single proposal view and POST logic,
several things have changed.
- The only parameter used is now `decision`, which
must contain `Yes`, `No`, or `Abstain` as a string.
When an invalid value is given, a BAD_REQUEST response
is returned in plaintext: Invalid 'decision' value.
- The `doVote` parameter has been removed.
- The details section has been rearranged into a set
of divs with specific classes that can be used for
testing. CSS has been added to persist the layout with
the element changes.
- Several errors that can be discovered in the POST path
now trigger their own non-200 HTTPStatus codes.
Signed-off-by: Kevin Morris <kevr@0cost.org>
A new middleware which redirects requests going to '/route?id=some_id'
to '/route/some_id'. In the FastAPI application, we'll prefer using
restful layouts where possible where resource-based ids are
parameters of the request uri: '/route/{resource_id}'.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit implements the '/tu' Trusted User index page.
In addition to this functionality, this commit introduces
the following jinja2 filters:
- dt: util.timestamp_to_datetime
- as_timezone: util.as_timezone
- dedupe_qs: util.dedupe_qs
- urlencode: urllib.parse.quote_plus
There's also a new decorator that can be used to enforce
permissions: `account_type_required`. If a user does not
meet account type requirements, they are redirected to '/'.
```
@auth_required(True)
@account_type_required({"Trusted User"})
async def some_route(request: fastapi.Request):
return Response("You are a Trusted User!")
```
Routes added:
- `GET /tu`: aurweb.routers.trusted_user.trusted_user
Signed-off-by: Kevin Morris <kevr@0cost.org>
This clones the end goal behavior of PHP, but it does not
concern itself with the revision form array at all.
Since this page on PHP renders out the entire list of
terms that a user needs to accept, we can treat a
POST request with the "accept" checkbox enabled as a
request to accept all unaccepted (or outdated revision)
terms.
This commit also adds in a new http middleware used to
redirect authenticated users to '/tos' if they have not
yet accepted all terms.
Signed-off-by: Kevin Morris <kevr@0cost.org>
Instead of using Dockerfile for everything, we've introduced
a docker-compose.yml file and kept the Dockerfile to producing
a pure base image for the services defined.
docker-compose services:
- `mariadb` - Setup mariadb
- `sharness` - Run sharness suites
- `pytest-mysql` - Run pytest suites with MariaDB
- `pytest-sqlite` - Run pytest suites with SQLite
- `test` - Run all tests and produce a collective coverage report
- This target mounts a cache volume and copies any successful
coverage report back to `./cache/.coverage`. Users can run
`./util/fix-coverage ./cache/.coverage` to rewrite source
code paths and move coverage into place to view reports
on your local system.
== Get Started ==
Build `aurweb:latest`.
$ docker build -t aurweb:latest .
Run all tests via `docker-compose`.
$ docker-compose up test
You can also purely run `pytest` in SQLite or MariaDB modes.
$ docker-compose up pytest-sqlite
$ docker-compose up pytest-mysql
Or `sharness` alone, which only uses SQLite internally.
$ docker-compose up sharness
After running tests, coverage reports are stored in `./cache/.coverage`.
This database was most likely created in a different path, and so it
needs to be sanitized with `./util/fix-coverage`.
$ ./util/fix-coverage cache/.coverage
Copied coverage db to /path/to/aurweb/.coverage.
$ coverage report
...
$ coverage html
$ coverage xml
...
Defined components:
**Entrypoints**
- mariadb-entrypoint.sh - setup mariadb and run its daemon
- test-mysql-entrypoint.sh - setup mysql configurations
- test-sqlite-entrypoint.sh - setup sqlite configurations
- tests-entrypoint.sh - setup mysql and sqlite configurations
**Scripts**
- run-mariadb.sh - setup databases
- run-pytests.sh - run pytest suites
- run-sharness.sh - run sharness suites
- run-tests.sh - run both pytests and sharness
**Health**
- mariadb.sh - A healthcheck script for the mariadb service
- pytest.sh - A healthcheck script for the pytest-* services
- sharness.sh - A healthcheck script for the sharness service
This Docker configuration is setup for tests, but should be
extendable for web and git servers.
**Changes to Makefile**
- Remove `.coverage` in the `clean` target
- Add a `coverage` target which prints a report and outputs xml
Signed-off-by: Kevin Morris <kevr@0cost.org>
This also updates `test/README.md` to be a bit more specific
and precise with our current state of testing.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This resolves logging issues with alembic on aurweb.initdb
in addition to adding more logging utilities for aurweb
and tests in general.
Developers should fetch a logger for their specific module
via `logging.getLogger(__name__)`.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This commit introduces a middleware function which adds
the following security headers to each response:
- Content-Security-Policy
- This includes a new `nonce`, which is tied to a user
via authentication middleware. Both an anonymous user
and an authenticated user recieve their own random nonces.
- X-Content-Type-Options
- Referrer-Policy
- X-Frame-Options
They are then tested for existence in test/test_routes.py.
Note: The overcomplicated-looking asyncio behavior in the
middleware function is used to avoid a warning about the old
coroutine awaits being deprecated. See
https://docs.python.org/3/library/asyncio-task.html#asyncio.wait
for more detail.
Signed-off-by: Kevin Morris <kevr@0cost.org>
This function adds f"SameSite={value}" to each cookie's header
stored in response.
This is needed because starlette does not currently support
the `samesite` argument in Response.set_cookie. It is merged,
however, and waiting for next release.
Signed-off-by: Kevin Morris <kevr@0cost.org>
