aurweb/test

aurweb Test Collection

To run all tests, you may run make -C test sh and pytest within the project root:

$ make -C test sh    # Run Sharness tests.
$ poetry run pytest  # Run Pytest suites.

For more control, you may use the prove or pytest command, which receives a directory or a list of files to run, and produces a report.

Each test script is standalone, so you may run them individually. Some tests may receive command-line options to help debugging.

Dependencies

For all tests to run dependencies provided via poetry are required:

$ poetry install

Logging

Tests also require the logging.test.conf logging configuration file to be used. You can specify the LOG_CONFIG environment variable to override:

$ export LOG_CONFIG=logging.test.conf

logging.test.conf enables debug logging for the aurweb package, for which we run tests against.

Test Configuration

To perform any tests, we need to supply aurweb with a valid configuration. For development (and testing) purposes, an example conf/config.dev can be slightly modified.

Start off by copying config.dev to a new configuration.

$ cp -v conf/config.dev conf/config

First, we must tell aurweb where the root of our project lives by replacing YOUR_AUR_ROOT with the path to the aurweb repository.

$ sed -i "s;YOUR_AUR_ROOT;/path/to/aurweb;g" conf/config

Test Databases

Python tests create and drop hashed database names based on PYTEST_CURRENT_TEST. To run tests with a database, the database user must have privileges to create and drop their own databases. Typically, this is the root user, but can be configured for any other user:

GRANT ALL ON *.* TO 'user'@'localhost' WITH GRANT OPTION

The aurweb platform is intended to use the mysql backend, but the sqlite backend is still used for sharness tests. These tests will soon be replaced with pytest suites and sqlite removed.

After ensuring you've configured a test database, users can continue on to Running Tests.

Running tests

Makefile test targets: sh, clean.

Recommended method of running tests: pytest.

Legacy sharness tests: make -C test sh.

aurweb is currently going through a refactor where the majority of sharness tests have been replaced with pytest units. There are still a few sharness tests around, and they are required to gain as much coverage as possible over an entire test run. Users should be writing pytest units for any new features.

Run tests from the project root.

$ cd /path/to/aurweb

Ensure you have the proper AUR_CONFIG and LOG_CONFIG exported:

$ export AUR_CONFIG=conf/config
$ export LOG_CONFIG=logging.test.conf

To run sharness shell test suites (requires Arch Linux):

$ make -C test sh

To run pytest Python test suites:

# With poetry-installed aurweb
$ poetry run pytest

# With globally-installed aurweb
$ pytest

After tests are run, one can produce coverage reports.

# Print out a CLI coverage report.
$ coverage report

# Produce an HTML-based coverage report.
$ coverage html

Writing Python tests (current)

Almost all of our pytest suites use the database in some way. There are a few particular testing utilities in aurweb that one should keep aware of to aid testing code:

• db_test pytest fixture
  • Prepares test databases for the module and cleans out database tables for each test function requiring this fixture.
• aurweb.testing.requests.Request
  • A fake stripped down version of fastapi.Request that can be passed to any functions in our codebase which use fastapi.Request parameters.

Example code:

import pytest

from aurweb import db
from aurweb.models.account_type import USER_ID
from aurweb.models.user import User
from aurweb.testing.requests import Request

# We need to use the `db_test` fixture at some point
# during our test functions.
@pytest.fixture(autouse=True)
def setup(db_test: None) -> None:
    return

# Or... specify it in a dependency fixture.
@pytest.fixture
def user(db_test: None) -> User:
    with db.begin():
        user = db.create(User, Username="test",
                         Email="test@example.org",
                         Passwd="testPassword",
                         AccountTypeID=USER_ID)
    yield user

def test_user_login(user: User):
    assert isinstance(user, User) is True

    fake_request = Request()
    sid = user.login(fake_request, "testPassword")
    assert sid is not None

Writing Sharness tests (legacy)

Shell test scripts must follow the Test Anything Protocol specification: http://testanything.org/tap-specification.html

Python tests must be compatible with pytest and included in pytest test/ execution after setting up a configuration.

Tests must support being run from any directory. They may use $0 to determine their location. Python scripts should expect aurweb to be installed and importable without toying with os.path or PYTHONPATH.

Tests written in shell should use sharness. In general, new tests should be consistent with existing tests unless they have a good reason not to.

Debugging Sharness tests

By default, make -C test is quiet and does not print out verbose information about tests being run. If a test is failing, one can look into verbose details of sharness tests by executing them with the --verbose flag. Example: ./t1100_git_auth.t --verbose. This is particularly useful when tests happen to fail in a remote continuous integration environment, where the reader does not have complete access to the runner.