Commit graph

2527 commits

Author SHA1 Message Date
Kevin Morris
38dc2bb99d Sanitize and modernize pytests
Some of these tests were written before some of our convenient
tooling existed. Additionally, some of the tests were not
cooperating with PEP-8 guidelines or isorted.

This commit does the following:
    - Replaces all calls to make_(user|session) with
      aurweb.db.create(Model, ...).
    - Replace calls to session.add(...) + session.commit() with
      aurweb.db.create.
    - Removes the majority of calls to (session|aurweb.db).delete(...).
    - Replaces session.query calls with aurweb.db.query.
    - Initializes all mutable globals in pytest fixture setup().
    - Makes mutable global declarations more concise:
      `var1, var2 = None, None` -> `var1 = var2 = None`
    - Defines a warning exclusion for test/test_ssh_pub_key.py.
    - Removes the aurweb.testing.models module.
    - Removes some useless pytest.fixture yielding.

As of this commit, developers should use the following guidelines
when writing tests:
    - Always use aurweb.db.(create|delete|query) for database
      operations, where possible.
    - Always define mutable globals in the style: `var1 = var2 = None`.
    - `yield` the most dependent model in pytest setup fixture **iff**
      you must delete records after test runs to maintain database
      integrity. Example: test/test_account_type.py.

This all makes the test code look and behave much cleaner.
Previously, aurweb.testing.setup_test_db was buggy and leaving
objects around in SQLAlchemy's IdentityMap.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
f2121fb833 simplify test_package_keyword.py
We no longer need to delete records like this; in fact, it causes
errors now. Fix this by removing the deletions and allow
setup_test_db to do it's job.

We'll need to do this for other tests as well.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
15b1332656 add Package SQLAlchemy ORM model
Additionally, add an optional **kwargs passing via make_relationship.
This allows us to use things like `uselist=False`, which was needed
for test/test_package.py.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
621e459dfb aurweb.models.user: Remove session.commit() from construction
We don't want to do this on construction. We only want to do this
when we want to actually add the user to the database (or modify it).

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
f8a6049de2 aurweb.db.session: Use autoflush=True for Sessions
We'd like SQLAlchemy to automatically maintain flushes for us.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
794868b20f aurweb.testing.setup_test_db: Expunge objects
This is needed to avoid redundant objects in SQLAlchemy's
IdentityMap, since we pass a direct .execute to delete
the tables passed in. Additionally, remove our engine.connect()
call in favor of relying on the already-established Session.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
b692b11f62 add Group SQLAlchemy ORM model
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
e1ab02c2bf Fix database initialization in test_term.py
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
718fa48a5c add AcceptedTerm SQLAlchemy ORM model
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
29db2ee513 add Term SQLAlchemy ORM model
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
fb21015811 add PackageKeyword SQLAlchemy ORM model
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
a7e5498197 add PackageBase SQLAlchemy ORM model
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
822905be7d bugfix: relax next verification
AUR renders its own 404 Not Found page when a bad route
is encountered. Introducing the previous verification
caused an error in this case when setting a language
while viewing the Not Found page. So, instead of checking
through routes, just make sure that the next parameter
starts with a '/' character, which removes the possibility
of any cross attacks.

+ Removed aurweb.asgi.routes; no longer needed.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Leonidas Spyropoulos
32abdbafae fastapi: Jinja contextfilter renamed to pass_context
Closes: #23

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
2021-06-05 20:11:17 -07:00
Kevin Morris
4f928b4577 add account (view) route
+ Added get /account/{username} route.
+ Added account/show.html template which shows a single use

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
4e9ef6fb00 add account edit (settings) routes
* Added account_url filter to jinja2 environment. This produces a path
  to the user's account url (/account/{username}).
* Updated archdev-navbar to link to new edit route.
+ Added migrate_cookies(request, response) to aurweb.util, a function
  that simply migrates the request cookies to response and returns it.
+ Added account_edit tests to test_accounts_routes.py.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
d323c1f95b add python-lxml to dependencies
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
c94793b0b1 add user registration routes
* Added /register get and post routes.
+ Added default attributes to AnonymousUser, including a new
  AnonymousList which behaves like an sqlalchemy relationship
  list.
+ aurweb.util: Added validation functions for various user fields
  used throughout registration.
+ test_accounts_routes: Added get|post register route tests.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
19b4a896f1 add openssh to test dependencies
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
df0a637d2b add aurweb.captcha, a CAPTCHA utility module
This CAPTCHA workflow is the same workflow used by our current
PHP implementation of account registration.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
7a6a38592e add python-email-validator dependency
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
a5be6fc9be aurweb.templates: add make_variable_context
A new make_context wrapper which additionally includes either
query parameters (get) or form data (post) in the context.

Use this to simplify setting context variables for form data
in particular.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
9052688ed2 add aurweb.time module
This module includes timezone-based utilities for a FastAPI request.
This commit introduces use of the AURTZ cookie within get_request_timezone.
This cookie should be set to the user or session's timezone.

* `make_context` has been modified to parse the request's timezone
  and include the "timezone" and "timezones" variables, along with
  a timezone specified "now" date.
+ Added `Timezone` attribute to aurweb.testing.requests.Request.user.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
07d5907ecd aurweb.auth: add user credentials and matcher functions
This clones the behavior already present in the PHP implementation,
but it uses a global dict with credential constant keys to
validation functions to determine if a given user has a credential.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
670f711b59 add SSHPubKey ORM model
Includes `aurweb.models.ssh_pub_key.get_fingerprint(pubkey)` helper.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
9fdbe3f775 add authenticated User LangPreference tracking
+ Use User.LangPreference when there is no set AURSID
  if request.user.is_authenticated is true.
+ Updated post /language to update LangPreference when
  request.user.is_authenticated.
+ Restore language during test where we change it.
+ Added the user attribute to aurweb.testing.requests.Request.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
a33d076d8b add passreset routes
Introduced `get|post` `/passreset` routes. These routes mimic the
behavior of the existing PHP implementation, with the exception of
HTTP status code returns.

Routes added:
    GET /passreset
    POST /passreset

Routers added:
    aurweb.routers.accounts

* On an unknown user or mismatched resetkey (where resetkey must ==
  user.resetkey), return HTTP status NOT_FOUND (404).
* On another error in the request, return HTTP status BAD_REQUEST (400).

Both `get|post` routes requires that the current user is **not**
authenticated, hence `@auth_required(False, redirect="/")`.

+ Added auth_required decorator to aurweb.auth.
+ Added some more utility to aurweb.models.user.User.
+ Added `partials/error.html` template.
+ Added `passreset.html` template.
+ Added aurweb.db.ConnectionExecutor functor for paramstyle logic.
  Decoupling the executor logic from the database connection logic
  is needed for us to easily use the same logic with a fastapi
  database session, when we need to use aurweb.scripts modules.

At this point, notification configuration is now required to complete
tests involved with notifications properly, like passreset.
`conf/config.dev` has been modified to include [notifications] sendmail,
sender and reply-to overrides. Dockerfile and .gitlab-ci.yml have been
updated to setup /etc/hosts and start postfix before running tests.

* setup.cfg: ignore E741, C901 in aurweb.routers.accounts

These two warnings (shown in the commit) are not dangerous and a bi-product
of maintaining compatibility with our current code flow.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
4423326cec add the request parameter to render_template
This allows us to inspect things about the request we're rendering from.

* Use render_template(request, ...) in aurweb.routers.auth

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
5d4a5deddf implement login + logout routes and templates
+ Added route: GET `/login` via `aurweb.routers.auth.login_get`
+ Added route: POST `/login` via `aurweb.routers.auth.login_post`
+ Added route: GET `/logout` via `aurweb.routers.auth.logout`
+ Added route: POST `/logout` via `aurweb.routers.auth.logout_post`
* Modify archdev-navbar.html template to toggle displays on auth state
+ Added login.html template

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
56f2798279 add aurweb.auth and authentication to User
+ Added aurweb.auth.AnonymousUser
    * An instance of this model is returned as the request user
      when the request is not authenticated
+ Added aurweb.auth.BasicAuthBackend
+ Add starlette's AuthenticationMiddleware to app middleware,
  which uses our BasicAuthBackend facility
+ Added User.is_authenticated()
+ Added User.authenticate(password)
+ Added User.login(request, password)
+ Added User.logout(request)
+ Added repr(User(...)) representation
+ Added aurweb.auth.auth_required decorator.

This change uses the same AURSID logic in the PHP implementation.

Additionally, introduce a few helpers for authentication,
one of which being `User.update_password(password, rounds = 12)`
where `rounds` is a configurable number of salt rounds.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
137c050f99 add python-bcrypt dependency
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
1922e5380d add aurweb.models.session.Session ORM database object
+ Added aurweb.util module.
    - Added make_random_string function.
+ Added aurweb.db.make_random_value function.
    - Takes a model and a column and introspects them to figure out the
      proper column length to create a random string for; then creates
      a unique string for that column.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
adc9fccb7d add aurweb.models.ban.Ban ORM mapping
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
a836892cde aurweb.db: add query, create, delete helpers
Takes sqlalchemy kwargs or stanzas:

query(Model, Model.Column == value)
query(Model, and_(Model.Column == value, Model.Column != "BAD!"))

Updated tests to reflect the new utility and a comment about upcoming
function deprecation is added to get_account_type().

From here on, phase out the use of get_account_type().

+ aurweb.db: Added create utility function
+ aurweb.db: Added delete utility function

The `delete` function can be used to delete a record by search
kwargs directly.

Example:
    delete(User, User.ID == 6)

All three functions added in this commit are typically useful to
perform these operations without having to import aurweb.db.session.
Removes a bit of redundancy overall.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Kevin Morris
5185df629e move aurweb.testing to its own package
+ Added aurweb.testing.setup_test_db(*tables)
+ Added aurweb.testing.models.make_user(**kwargs)
+ Added aurweb.testing.models.make_session(**kwargs)
+ Added aurweb.testing.requests.Client
+ Added aurweb.testing.requests.Request
* Updated test_l10n.py to use our new Request

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 20:11:17 -07:00
Leonidas Spyropoulos
64bc93926f Add support for configuring database with port instead of socket
Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
2021-06-05 20:11:17 -07:00
Kristian Klausen
ac31f520ea Add coverage report for "Test Coverage Visualization"[1]
[1] https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html
2021-06-05 20:11:17 -07:00
Leonidas Spyropoulos
7b7c3abbe2 Conditionally apply SSOAccountId migration to support misaligned databases
Closes: #34

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
2021-06-05 20:11:17 -07:00
Leonidas Spyropoulos
72f755817c Adds Alembic migration for DB/Tables conversion to utf8mb4
MySql defaults to `utf8` and case insensitive collation so migrate these to case sensitive and `utf8mb4`

Closes #21

Signed-off-by: Leonidas Spyropoulos <artafinde@gmail.com>
2021-06-05 20:11:04 -07:00
Kevin Morris
66189c4460 alembic: restore logging, fix pytest conflicts
In this case, when running pytests, we do not allow alembic to
configure loggers.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:52:56 -07:00
Kevin Morris
3f1f03e03c aurweb.db: only pass check_same_thread with sqlite
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:52:56 -07:00
Kevin Morris
e0eb6b0e76 test_db: remove use of mkdtemp and os.removedirs
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:52:56 -07:00
Marcus Andersson
1d5827007f Adding route tests
Removing status code from 404 title

Removing status code from 503 title

Adding id to 503 error box

Indatation fix
2021-06-05 19:52:56 -07:00
Marcus Andersson
f6744d3e39 Adding error 503 catcher 2021-06-05 19:52:56 -07:00
Marcus Andersson
cdf75ced9a Adding error 404 catcher 2021-06-05 19:52:56 -07:00
Kevin Morris
82f3871a83 Support SQLAlchemy 1.4 URL.create recommendation
This fixes a deprecating warning when using SQLAlchemy 1.4.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:52:56 -07:00
Kevin Morris
81856f3b64 Fix incorrect construction of MySQL SQLAlchemy URL
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:52:49 -07:00
Kevin Morris
02311eab76 add test_initdb.py
IMPORTANT: This test completely wipes out the database it's using.
Make sure you've got AUR_CONFIG set to a test database configuration!

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:52:42 -07:00
Kevin Morris
8a47afd2ea add aurweb.models.user.User
+ Added aurweb.models.user.User class. This is the first example
  of an sqlalchemy ORM model. We can search for users via for example:
  `session.query(User).filter(User.ID==1).first()`, where `session` is
  a configured `aurweb.db.session` object.
+ Along with the User class, defined the AccountType class.
  Each User maintains a relationship to its AccountType via User.AccountType.
+ Added AccountType.users backref.

Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:52:42 -07:00
Kevin Morris
e860d828b6 add aurweb.testing, a module with testing utilities
Signed-off-by: Kevin Morris <kevr@0cost.org>
2021-06-05 19:52:42 -07:00