From ae6442d44a74e8f2cb94f49f505903437203eb83 Mon Sep 17 00:00:00 2001 From: kevr Date: Tue, 11 May 2021 02:42:29 +0000 Subject: [PATCH] Create Python Port --- Python-Port.md | 140 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 Python-Port.md diff --git a/Python-Port.md b/Python-Port.md new file mode 100644 index 0000000..eb9dcc7 --- /dev/null +++ b/Python-Port.md @@ -0,0 +1,140 @@ +## Overview + +At some point, a porting of aurweb to Python was proposed on the `aur-dev` mailing list. In May 2020 it was accepted as a milestone (reference: https://lists.archlinux.org/pipermail/aur-dev/2020-May/004848.html). The HTTP framework of choice in the conversation settled on [FastAPI](https://fastapi.tiangolo.com) along with the popular database management library, [SQLAlchemy](https://www.sqlalchemy.org) and [Alembic](https://alembic.sqlalchemy.org/en/latest) to manage its migrations. + +This effort has been started and its beginnings now exist in the `pu` branch of aurweb, located at https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu. + +As we port the PHP implementation of aurweb over to Python, we should keep a few things in mind: +1. Existing API **must** persist in Python; there can be no difference told between FastAPI and PHP. Browseable pages are another story, however -- as long as they contain similar content, the same CSS style and functionality, implementations can be changed. With that being said; we should strive to pay attention to any bugs with the existing PHP implementation and proactively solve them. +2. As tasks in this port are approached, developers should strive to increase test coverage wherever they can. Do not allow this to overwhelmingly consume your time, all help is highly valued and appreciated. Our goal is **100%** coverage across the Python codebase (see [Contribution Guidelines](#contribution-guidelines)). + +All contributions, advice or other assistance is greatly appreciated and welcome. We encourage you to speak with us if you need any help or wish to discuss any topics regarding the project; our contact information can be found down in the [Who, Where?](#who-where) section. + +## State of Development + +We use a GitLab issue, [Path to Python](https://gitlab.archlinux.org/archlinux/aurweb/-/issues/18), to track the state of development of the PHP to Python porting efforts. Refer there for development task details. + +## Docker + +If you want to build the project and see its test run with docker, you can run `docker build .` which should result in a full test run of both legacy *sharness* and our new *pytest* suites. At the end, it'll produce a text coverage report of the executed tests. Keep in mind, this is expected to be run at the root of the `aurweb` repository; it requires relative directories to populate Docker. + +## Getting Started +
+ +First, see `INSTALL` and `test/README.md` for lists of python packages required to run aurweb on Arch Linux. + +This Python port of aurweb uses a number of dependencies to simplify life. For specifics, see the [Getting Familiar](#getting-familiar) section. + +When running `aurweb` for development, we can host it manually with an ASGI server, like `uvicorn`. + +Before one can do so, they must prepare international translations and a configuration for development. + + # Prepare a development configuration. + aurweb$ sed -r "s;YOUR_AUR_ROOT;$(pwd);g" conf/config.dev > conf/config + + # Export some variables. + aurweb$ export AUR_CONFIG=conf/config + aurweb$ export PYTHONPATH="$(pwd)" + + # Install international translations to web/locale. + aurweb$ make -C po all install + +That being done, we've also got to initialize our development database. + + aurweb$ python -m aurweb.initdb + +If you want to run all tests and collect coverage, run the following statements. + + aurweb$ make -C test + aurweb$ coverage report --include='aurweb/*' + +With all that being done, you can now launch an ASGI server daemon and browse FastAPI aurweb. + + # uvicorn's --reload will hot reload the application on source changes. + aurweb$ uvicorn --reload aurweb.asgi:app + +Now, an HTTP server should be running at http://127.0.0.1:8000, where you can browse and explore the project. + +**NOTE:** `hypercorn` can be used in place of `uvicorn` here if you wish. + +## Getting Familiar +
+ +The Python 3 port of this project uses a few frameworks and tools to make life easier for developers and provide modern tooling. Some may be unfamiliar with these packages and modules, and so, here, we attempt to provide pointers to all of them. Below, you will find a listing of our chosen technologies, each with a small description and some links to getting started with them and references where possible. + +All of these libraries are used in the [Getting Started](#getting-started) section above and must be installed first. + +* FastAPI - [https://fastapi.tiangolo.com](https://fastapi.tiangolo.com) + * A Flask-like, more lightweight HTTP framework. +* SQLAlchemy - [https://www.sqlalchemy.org](https://www.sqlalchemy.org) + * A database connector library for Python. +* SQLAlchemy ORM - [https://docs.sqlalchemy.org/en/13/orm](https://docs.sqlalchemy.org/en/13/orm) + * [**O**]bject [**R**]elational [**M**]apping tools which allow a developer to represent database tables with Python objects. +* Alembic - [https://alembic.sqlalchemy.org/en/latest](https://alembic.sqlalchemy.org/en/latest) + * An SQLAlchemy instrumentation tool used to initialize databases and apply [migrations](https://cloud.google.com/architecture/database-migration-concepts-principles-part-1) to them. +* Jinja2 - [https://jinja.palletsprojects.com/en/2.11.x](https://jinja.palletsprojects.com/en/2.11.x) + * A template framework which we use to generate HTML files. Jinja2 templates allow a developer to use "Python tags" defined which he/she can define in addition to helpful built-ins. +* Requests - [https://docs.python-requests.org/en/master](https://docs.python-requests.org/en/master) + * An HTTP client library used to perform requests in a simple way. +* Bcrypt - [https://github.com/pyca/bcrypt](https://github.com/pyca/bcrypt) + * A crypto library used for aurweb internal passwords and password salts. +* LXML - [https://lxml.de](https://lxml.de) + * An XML-interfacing library, which also supports HTML and allows us to perform xpath searches. This library is typically used to scrape HTML webpages. +* Pytest - [https://docs.pytest.org/en/6.2.x](https://docs.pytest.org/en/6.2.x) + * A test framework for Python. +* Pytest-cov - [https://github.com/pytest-dev/pytest-cov](https://github.com/pytest-dev/pytest-cov) + * A module used to collect and report coverage from Python execution. + +Some modules required by legacy code include the following. + +* Pygit2 - [https://github.com/libgit2/pygit2](https://github.com/libgit2/pygit2) + * A git interface for Python. +* Srcinfo - [https://github.com/kyrias/python-srcinfo](https://github.com/kyrias/python-srcinfo) + * .SRCINFO parsing library for Arch Linux packages. +* Markdown - [https://python-markdown.github.io](https://python-markdown.github.io) + * Used to display markdown. +* Bleach - [https://github.com/mozilla/bleach](https://github.com/mozilla/bleach) + * HTML sanitization. + +Additionally, this project uses the following programs to orchestrate development and/or production. + +* hypercorn - [https://gitlab.com/pgjones/hypercorn](https://gitlab.com/pgjones/hypercorn) + * An HTTP/2 compliant ASGI server application. This can be upstreamed in an `nginx` configuration to host the application publicly. + +## Contribution Guidelines +
+ +1. All contributions **must** maintain 100% test coverage over any newly added or modified +source code + * Test coverage can be maintained by writing `pytest` units that exercise all paths in + the code that your commits affect +2. Do not exceed 80 columns in any Python source file +3. If not otherwise specified, code should maintain PEP-8 compliance in terms of code style +and spacing. `autopep8` can be used to automatically format your code +4. All new Python code should be `isort`ed +5. SQLAlchemy ORM model attributes should be Capitalized if they are literal table columns, otherwise snake_cased in accordance with PEP-8 guidelines + +## Who, Where? +
+ +**Looking for the code?** +- Development Branch: **pu** + - https://gitlab.archlinux.org/archlinux/aurweb/-/tree/pu + - This branch contains all merged code from the Python/FastAPI efforts. + +**Want to chat?** +- Come talk with us in IRC on **Freenode** in the **#archlinux-aurweb** channel + - Note: You must be registered with NickServ to join + +Just to clarify here: **elibrokeit** and **lfleischer** are long time maintainers of aurweb and are still maintaining the project aside from this port; they are still the authority on this project and should always be informed of any absolutely critical aurweb issues that we may not be aware of. **kevr** shall be maintaining the progress of the Python port, especially because he has the free time to do so. + +**A little anti-social?** +- You can email [kevr](https://gitlab.archlinux.org/kevr) directly at [kevr@0cost.org](mailto:kevr@0cost.org) + +## Filing Bugs + +We are slowly relocating over to [GitLab](https://gitlab.archlinux.org/archlinux/aurweb), as the original Arch Linux bugtracker, bugs.archlinux.org, is moving toward deprecation. + +If you found a bug, have a feature request or general issue with the port (and/or project in general), please create an issue in the new [GitLab repository](https://gitlab.archlinux.org/archlinux/aurweb). This location will be the most active as far as the Python port is concerned. + +Of course, we will always observe the `aur-dev` mailing list while it's in service for contributors who do not own an Arch Linux GitLab account. \ No newline at end of file