10 Python Port

Kevin Morris edited this page 2021-09-10 00:47:27 +00:00

Table of Contents

• Overview
• State of Development
• Docker
• Getting Started
• Getting Familiar
• Contribution Guidelines
• Who, Where?
• Filing Bugs

Overview

At some point, a porting of aurweb to Python was proposed on the aur-dev mailing list. In May 2020 it was accepted as a milestone (reference: https://lists.archlinux.org/pipermail/aur-dev/2020-May/004848.html). The HTTP framework of choice in the conversation settled on FastAPI along with the popular database management library, SQLAlchemy and Alembic to manage its migrations.

This effort has been started and its beginnings now exist in the pu branch of aurweb, located at https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu.

As we port the PHP implementation of aurweb over to Python, we should keep a few things in mind:

1. Existing API must persist in Python; there can be no difference told between FastAPI and PHP. Browseable pages are another story, however -- as long as they contain similar content, the same CSS style and functionality, implementations can be changed. With that being said; we should strive to pay attention to any bugs with the existing PHP implementation and proactively solve them.
2. As tasks in this port are approached, developers should strive to increase test coverage wherever they can. Do not allow this to overwhelmingly consume your time, all help is highly valued and appreciated. Our goal is 100% coverage across the Python codebase (see Contribution Guidelines).

All contributions, advice or other assistance is greatly appreciated and welcome. We encourage you to speak with us if you need any help or wish to discuss any topics regarding the project; our contact information can be found down in the Who, Where? section.

State of Development

We use a GitLab issue, Path to Python/FastAPI, to track the state of development of the PHP to Python porting efforts. Refer there for development task details.

Docker

See the Docker wiki page for an in-depth look at our services and images.

Getting Started

First, see INSTALL and test/README.md for lists of python packages required to run aurweb on Arch Linux.

This Python port of aurweb uses a number of dependencies to simplify life. For specifics, see the Getting Familiar section.

When running aurweb for development, we can host it manually with an ASGI server, like uvicorn.

Before one can do so, they must prepare international translations and a configuration for development.

# Prepare a development configuration.
aurweb$ sed -r "s;YOUR_AUR_ROOT;$(pwd);g" conf/config.dev > conf/config

# Export some variables.
aurweb$ export AUR_CONFIG=conf/config
aurweb$ export PYTHONPATH="$(pwd)"

# Install international translations to web/locale.
aurweb$ make -C po all install

That being done, we've also got to initialize our development database.

aurweb$ python -m aurweb.initdb

If you want to run all tests and collect coverage, run the following statements.

aurweb$ make -C test
aurweb$ coverage report --include='aurweb/*'

With all that being done, you can now launch an ASGI server daemon and browse FastAPI aurweb.

# uvicorn's --reload will hot reload the application on source changes.
aurweb$ uvicorn --reload aurweb.asgi:app

Now, an HTTP server should be running at http://127.0.0.1:8000, where you can browse and explore the project.

NOTE: uvicorn is a bit more friendly for development, while hypercorn has massive transport improvements for production. They can be used interchangeably, but hypercorn is recommended for production.

Getting Familiar

The Python 3 port of this project uses a few frameworks and tools to make life easier for developers and provide modern tooling. Some may be unfamiliar with these packages and modules, and so, here, we attempt to provide pointers to all of them. Below, you will find a listing of our chosen technologies, each with a small description and some links to getting started with them and references where possible.

All of these libraries are used in the Getting Started section above and must be installed first.

• FastAPI - https://fastapi.tiangolo.com
  • A Flask-like, more lightweight HTTP framework.
• SQLAlchemy - https://www.sqlalchemy.org
  • A database connector library for Python.
• SQLAlchemy ORM - https://docs.sqlalchemy.org/en/13/orm
  • [O]bject [R]elational [M]apping tools which allow a developer to represent database tables with Python objects.
• Alembic - https://alembic.sqlalchemy.org/en/latest
  • An SQLAlchemy instrumentation tool used to initialize databases and apply migrations to them.
• Jinja2 - https://jinja.palletsprojects.com/en/2.11.x
  • A template framework which we use to generate HTML files. Jinja2 templates allow a developer to use "Python tags" defined which he/she can define in addition to helpful built-ins.
• Requests - https://docs.python-requests.org/en/master
  • An HTTP client library used to perform requests in a simple way.
• Bcrypt - https://github.com/pyca/bcrypt
  • A crypto library used for aurweb internal passwords and password salts.
• LXML - https://lxml.de
  • An XML-interfacing library, which also supports HTML and allows us to perform xpath searches. This library is typically used to scrape HTML webpages.
• Pytest - https://docs.pytest.org/en/6.2.x
  • A test framework for Python.
• Pytest-cov - https://github.com/pytest-dev/pytest-cov
  • A module used to collect and report coverage from Python execution.

Some modules required by back-end code include the following.

• Pygit2 - https://github.com/libgit2/pygit2
  • A git interface for Python.
• Srcinfo - https://github.com/kyrias/python-srcinfo
  • .SRCINFO parsing library for Arch Linux packages.
• Markdown - https://python-markdown.github.io
  • Used to display markdown.
• Bleach - https://github.com/mozilla/bleach
  • HTML sanitization.

Additionally, this project uses the following programs to orchestrate development and/or production.

• hypercorn - https://gitlab.com/pgjones/hypercorn
  • An HTTP/2 compliant ASGI server application. This can be upstreamed in an nginx configuration to host the application publicly.

Contribution Guidelines

1. All contributions must maintain 100% test coverage over any newly added or modified source code
   • Test coverage can be maintained by writing pytest units that exercise all paths in the code that your commits affect
2. Do not exceed 80 columns in any Python source file
3. If not otherwise specified, code should maintain PEP-8 compliance in terms of code style and spacing. autopep8 can be used to automatically format your code
4. All new Python code should be isorted
5. SQLAlchemy ORM model attributes should be Capitalized if they are literal table columns or forward references, otherwise snake_cased for backref references.
6. Use typed arguments as often as possible. We'd like to show users the types of arguments without relying on documentation, and this helps a ton with that.

A few small clues about some of our local aurweb package utilities that should be used in tests:

• Always use aurweb.db.(create|delete|query) for database operations, where possible.
• Always define mutable globals in the style: var1 = var2 = None.
• yield the most dependent model in pytest setup fixture iff you must delete records after test runs to maintain database integrity. Example: test/test_account_type.py.

Who, Where?

Looking for the code?

• Development Branch: pu
  • https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu
  • This branch contains all merged code from the Python/FastAPI efforts.

Want to chat?

• Come talk with us in IRC on Libera in the #archlinux-aurweb channel
  • Note: You must be registered with NickServ to join

Just to clarify here: elibrokeit and lfleischer are long time maintainers of aurweb and are still maintaining the project aside from this port; they are still the authority on this project and should always be informed of any absolutely critical aurweb issues that we may not be aware of. kevr shall be maintaining the progress of the Python port, especially because he has the free time to do so.

A little anti-social?

• You can email kevr directly at kevr@0cost.org

Filing Bugs

We are slowly relocating over to GitLab, as the original Arch Linux bugtracker, bugs.archlinux.org, is moving toward deprecation.

If you found a bug, have a feature request or general issue with the port (and/or project in general), please create an issue in the new GitLab repository. This location will be the most active as far as the Python port is concerned.

Of course, we will always observe the aur-dev mailing list while it's in service for contributors who do not own an Arch Linux GitLab account.