Compare commits

..

26 commits

Author SHA1 Message Date
Leonidas Spyropoulos
8ca61eded2
chore(release): prepare for 6.2.16 2025-01-13 15:52:13 +00:00
Leonidas Spyropoulos
a9bf714dae
fix: bump deps for python 3.13 and vulnerability
pygit2 and watchfiles for precompiled wheels
greenlet for python 3.13 compatibility
python-multipart for security vulnerability

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
2025-01-12 20:39:02 +00:00
Leonidas Spyropoulos
3e3173b5c9
chore: avoid cache for new pacman 7
Pacman 7 introduced sandboxing which breaks cache in containers due to permissions on containers

Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
2025-01-12 20:39:02 +00:00
Leonidas Spyropoulos
eca8bbf515
chore(release): prepare for 6.2.15
Signed-off-by: Leonidas Spyropoulos <artafinde@archlinux.org>
2024-09-15 12:03:17 +03:00
Jelle van der Waa
edc1ab949a perf(captcha): simplify count() query for user ids
Using .count() isn't great as it runs a count query on a subquery which
selects all fields in the Users table. This rewrites it into a simple
SELECT count(ID) from USers query.
2024-09-12 12:29:46 +00:00
Muflone
97cc6196eb fix: reduce the number of subqueries against Packages by preloading the existing dependencies names from AUR 2024-08-21 01:36:15 +02:00
Muflone
77ef87c882 housekeep: code re-formatted by black for lint pipeline 2024-08-20 21:00:46 +00:00
Muflone
a40283cdb2 fix: reduce the number of subqueries against User by loading eagerly the Users from PackageComaintainer 2024-08-20 21:00:46 +00:00
Levente Polyak
4f68532ee2
chore(mariadb): fix mysql deprecation warnings by using mariadb commands
Mariadb has scheduled to remove the deprecated mysql drop-in interface.
Let's adapt which also removes a lot of warnings while spinning up the
service.
2024-08-19 15:26:36 +02:00
Levente Polyak
439ccd4aa3
feat(docker): add full grafana, prometheus, tempo setup for local dev
This is a very useful stack for local development as well, by allowing
to easily access a local grafana instance and look at the accessed
endpoints, query usage and durations etc.
As a nice side effect this also makes sure we have an easy way to
actually test any changes to the opentelemetry integration in an actual
environment instead of just listening to a raw socket.
2024-08-19 15:26:29 +02:00
Levente Polyak
8dcf0b2d97
fix(docker): fix compose race conditions on mariadb_init
We want the dependent services to wait until the initialization service
of mariadb finishes, but also properly accept if it already exited
before a leaf service gets picked up and put into created state. By
using the service_completed_successfully signal, we can ensure precisely
this, without being racy and leading to none booted services.

While at it, remove the compose version identifiers as docker-compose
deprecated them and always warned about when running docker-compose.
2024-08-19 15:26:21 +02:00
Leonidas Spyropoulos
88e8db4404
chore(release): prepare version 6.2.14 2024-08-17 17:28:26 +01:00
Sven-Hendrik Haase
b730f6447d
feat: Add opentelemtry-based tracing
This adds tracing to fastapi, redis, and sqlalchemy. It uses the
recommended OLTP exporter to send the tracing data.
2024-08-17 11:27:26 +01:00
Leonidas Spyropoulos
92f5bbd37f
housekeep: reformat asgi.py 2024-08-17 01:31:43 +01:00
Jelle van der Waa
6c6ecd3971
perf(aurweb): create a context with what is required
The pkgbase/util.py `make_context` helper does a lot of unrelated
expensive queries which are not required for any of the templates. Only
the 404 template shows git_clone_uri_* and pkgbase.
2024-08-16 21:32:22 +02:00
Leonidas Spyropoulos
9b12eaf2b9
chore(release): prepare version 6.2.13 2024-08-16 16:03:40 +01:00
Jelle van der Waa
d1a66a743e
perf(aurweb/pkgbase): use exists() to avoid fetching a row
The previous approach fetched the matching row, by using `exists()`
SQLAlchemy changes the query to a `SELECT 1`.
2024-08-09 16:07:17 +02:00
Jelle van der Waa
b65d6c5e3a
perf(aurweb/pkgbase): only relevant queries when logged in
Don't query for notify, requests and vote information when the user is
not logged in as this information is not shown.
2024-08-09 16:07:17 +02:00
Jelle van der Waa
d393ed2352
fix(templates): hide non-actionable links when not logged in
A non-logged in user cannot vote/enable notifications or submit a
request so hide these links.
2024-08-09 16:07:17 +02:00
Leonidas Spyropoulos
a16fac9b95
fix: revert mysqlclient to 2.2.3 2024-08-09 11:02:13 +01:00
renovate
5dd65846d1
chore(deps): update dependency coverage to v7.6.1 2024-08-05 11:25:17 +00:00
renovate
a1b2d231c3
fix(deps): update dependency aiofiles to v24 2024-08-04 20:25:21 +00:00
renovate
f306b6df7a
fix(deps): update dependency fastapi to ^0.112.0 2024-08-04 12:25:03 +00:00
renovate
0d17895647
fix(deps): update dependency gunicorn to v22 2024-08-04 10:24:33 +00:00
renovate
36a56e9d3c
fix(deps): update all non-major dependencies 2024-08-04 09:24:29 +00:00
Diego Viola
80d3e5f7b6 housekeep: update .editorconfig url
Signed-off-by: Diego Viola <diego.viola@gmail.com>
2024-08-03 11:58:58 +00:00
27 changed files with 1582 additions and 975 deletions

View file

@ -1,5 +1,5 @@
# EditorConfig configuration for aurweb # EditorConfig configuration for aurweb
# https://EditorConfig.org # https://editorconfig.org
# Top-most EditorConfig file # Top-most EditorConfig file
root = true root = true

View file

@ -19,9 +19,9 @@ variables:
lint: lint:
stage: .pre stage: .pre
before_script: before_script:
- pacman -Sy --noconfirm --noprogressbar --cachedir .pkg-cache - pacman -Sy --noconfirm --noprogressbar
archlinux-keyring archlinux-keyring
- pacman -Syu --noconfirm --noprogressbar --cachedir .pkg-cache - pacman -Syu --noconfirm --noprogressbar
git python python-pre-commit git python python-pre-commit
script: script:
- export XDG_CACHE_HOME=.pre-commit - export XDG_CACHE_HOME=.pre-commit
@ -60,7 +60,7 @@ test:
path: coverage.xml path: coverage.xml
.init_tf: &init_tf .init_tf: &init_tf
- pacman -Syu --needed --noconfirm --cachedir .pkg-cache terraform - pacman -Syu --needed --noconfirm terraform
- export TF_VAR_name="aurweb-${CI_COMMIT_REF_SLUG}" - export TF_VAR_name="aurweb-${CI_COMMIT_REF_SLUG}"
- TF_ADDRESS="${CI_API_V4_URL}/projects/${TF_STATE_PROJECT}/terraform/state/${CI_COMMIT_REF_SLUG}" - TF_ADDRESS="${CI_API_V4_URL}/projects/${TF_STATE_PROJECT}/terraform/state/${CI_COMMIT_REF_SLUG}"
- cd ci/tf - cd ci/tf
@ -97,7 +97,7 @@ provision_review:
- deploy_review - deploy_review
script: script:
- *init_tf - *init_tf
- pacman -Syu --noconfirm --needed --cachedir .pkg-cache ansible git openssh jq - pacman -Syu --noconfirm --needed ansible git openssh jq
# Get ssh key from terraform state file # Get ssh key from terraform state file
- mkdir -p ~/.ssh - mkdir -p ~/.ssh
- chmod 700 ~/.ssh - chmod 700 ~/.ssh

View file

@ -14,6 +14,12 @@ from fastapi import FastAPI, HTTPException, Request, Response
from fastapi.responses import RedirectResponse from fastapi.responses import RedirectResponse
from fastapi.staticfiles import StaticFiles from fastapi.staticfiles import StaticFiles
from jinja2 import TemplateNotFound from jinja2 import TemplateNotFound
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from sqlalchemy import and_ from sqlalchemy import and_
from starlette.exceptions import HTTPException as StarletteHTTPException from starlette.exceptions import HTTPException as StarletteHTTPException
from starlette.middleware.authentication import AuthenticationMiddleware from starlette.middleware.authentication import AuthenticationMiddleware
@ -22,7 +28,6 @@ from starlette.middleware.sessions import SessionMiddleware
import aurweb.captcha # noqa: F401 import aurweb.captcha # noqa: F401
import aurweb.config import aurweb.config
import aurweb.filters # noqa: F401 import aurweb.filters # noqa: F401
import aurweb.pkgbase.util as pkgbaseutil
from aurweb import aur_logging, prometheus, util from aurweb import aur_logging, prometheus, util
from aurweb.aur_redis import redis_connection from aurweb.aur_redis import redis_connection
from aurweb.auth import BasicAuthBackend from aurweb.auth import BasicAuthBackend
@ -54,6 +59,17 @@ instrumentator().add(prometheus.http_requests_total())
instrumentator().instrument(app) instrumentator().instrument(app)
# Instrument FastAPI for tracing
FastAPIInstrumentor.instrument_app(app)
resource = Resource(attributes={"service.name": "aurweb"})
otlp_endpoint = aurweb.config.get("tracing", "otlp_endpoint")
otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
span_processor = BatchSpanProcessor(otlp_exporter)
trace.set_tracer_provider(TracerProvider(resource=resource))
trace.get_tracer_provider().add_span_processor(span_processor)
async def app_startup(): async def app_startup():
# https://stackoverflow.com/questions/67054759/about-the-maximum-recursion-error-in-fastapi # https://stackoverflow.com/questions/67054759/about-the-maximum-recursion-error-in-fastapi
# Test failures have been observed by internal starlette code when # Test failures have been observed by internal starlette code when
@ -215,7 +231,13 @@ async def http_exception_handler(request: Request, exc: HTTPException) -> Respon
if matches and len(tokens) == 2: if matches and len(tokens) == 2:
try: try:
pkgbase = get_pkg_or_base(matches.group(1)) pkgbase = get_pkg_or_base(matches.group(1))
context = pkgbaseutil.make_context(request, pkgbase) context["pkgbase"] = pkgbase
context["git_clone_uri_anon"] = aurweb.config.get(
"options", "git_clone_uri_anon"
)
context["git_clone_uri_priv"] = aurweb.config.get(
"options", "git_clone_uri_priv"
)
except HTTPException: except HTTPException:
pass pass

View file

@ -1,4 +1,5 @@
import fakeredis import fakeredis
from opentelemetry.instrumentation.redis import RedisInstrumentor
from redis import ConnectionPool, Redis from redis import ConnectionPool, Redis
import aurweb.config import aurweb.config
@ -7,6 +8,8 @@ from aurweb import aur_logging
logger = aur_logging.get_logger(__name__) logger = aur_logging.get_logger(__name__)
pool = None pool = None
RedisInstrumentor().instrument()
class FakeConnectionPool: class FakeConnectionPool:
"""A fake ConnectionPool class which holds an internal reference """A fake ConnectionPool class which holds an internal reference

View file

@ -3,6 +3,7 @@
import hashlib import hashlib
from jinja2 import pass_context from jinja2 import pass_context
from sqlalchemy import func
from aurweb.db import query from aurweb.db import query
from aurweb.models import User from aurweb.models import User
@ -11,7 +12,8 @@ from aurweb.templates import register_filter
def get_captcha_salts(): def get_captcha_salts():
"""Produce salts based on the current user count.""" """Produce salts based on the current user count."""
count = query(User).count() count = query(func.count(User.ID)).scalar()
salts = [] salts = []
for i in range(0, 6): for i in range(0, 6):
salts.append(f"aurweb-{count - i}") salts.append(f"aurweb-{count - i}")

View file

@ -298,9 +298,12 @@ def get_engine(dbname: str = None, echo: bool = False):
connect_args["check_same_thread"] = False connect_args["check_same_thread"] = False
kwargs = {"echo": echo, "connect_args": connect_args} kwargs = {"echo": echo, "connect_args": connect_args}
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from sqlalchemy import create_engine from sqlalchemy import create_engine
_engines[dbname] = create_engine(get_sqlalchemy_url(), **kwargs) engine = create_engine(get_sqlalchemy_url(), **kwargs)
SQLAlchemyInstrumentor().instrument(engine=engine)
_engines[dbname] = engine
if is_sqlite: # pragma: no cover if is_sqlite: # pragma: no cover
setup_sqlite(_engines.get(dbname)) setup_sqlite(_engines.get(dbname))

View file

@ -2,6 +2,7 @@ from typing import Any
from fastapi import Request from fastapi import Request
from sqlalchemy import and_ from sqlalchemy import and_
from sqlalchemy.orm import joinedload
from aurweb import config, db, defaults, l10n, time, util from aurweb import config, db, defaults, l10n, time, util
from aurweb.models import PackageBase, User from aurweb.models import PackageBase, User
@ -26,6 +27,8 @@ def make_context(
if not context: if not context:
context = _make_context(request, pkgbase.Name) context = _make_context(request, pkgbase.Name)
is_authenticated = request.user.is_authenticated()
# Per page and offset. # Per page and offset.
offset, per_page = util.sanitize_params( offset, per_page = util.sanitize_params(
request.query_params.get("O", defaults.O), request.query_params.get("O", defaults.O),
@ -38,12 +41,15 @@ def make_context(
context["pkgbase"] = pkgbase context["pkgbase"] = pkgbase
context["comaintainers"] = [ context["comaintainers"] = [
c.User c.User
for c in pkgbase.comaintainers.order_by( for c in pkgbase.comaintainers.options(joinedload(PackageComaintainer.User))
PackageComaintainer.Priority.asc() .order_by(PackageComaintainer.Priority.asc())
).all() .all()
] ]
context["unflaggers"] = context["comaintainers"].copy() if is_authenticated:
context["unflaggers"].extend([pkgbase.Maintainer, pkgbase.Flagger]) context["unflaggers"] = context["comaintainers"].copy()
context["unflaggers"].extend([pkgbase.Maintainer, pkgbase.Flagger])
else:
context["unflaggers"] = []
context["packages_count"] = pkgbase.packages.count() context["packages_count"] = pkgbase.packages.count()
context["keywords"] = pkgbase.keywords context["keywords"] = pkgbase.keywords
@ -60,17 +66,28 @@ def make_context(
).order_by(PackageComment.CommentTS.desc()) ).order_by(PackageComment.CommentTS.desc())
context["is_maintainer"] = bool(request.user == pkgbase.Maintainer) context["is_maintainer"] = bool(request.user == pkgbase.Maintainer)
context["notified"] = request.user.notified(pkgbase) if is_authenticated:
context["notified"] = request.user.notified(pkgbase)
else:
context["notified"] = False
context["out_of_date"] = bool(pkgbase.OutOfDateTS) context["out_of_date"] = bool(pkgbase.OutOfDateTS)
context["voted"] = request.user.package_votes.filter( if is_authenticated:
PackageVote.PackageBaseID == pkgbase.ID context["voted"] = db.query(
).scalar() request.user.package_votes.filter(
PackageVote.PackageBaseID == pkgbase.ID
).exists()
).scalar()
else:
context["voted"] = False
context["requests"] = pkgbase.requests.filter( if is_authenticated:
and_(PackageRequest.Status == PENDING_ID, PackageRequest.ClosedTS.is_(None)) context["requests"] = pkgbase.requests.filter(
).count() and_(PackageRequest.Status == PENDING_ID, PackageRequest.ClosedTS.is_(None))
).count()
else:
context["requests"] = []
context["popularity"] = popularity(pkgbase, time.utcnow()) context["popularity"] = popularity(pkgbase, time.utcnow())

View file

@ -190,6 +190,17 @@ async def package(
if not all_deps: if not all_deps:
deps = deps.limit(max_listing) deps = deps.limit(max_listing)
context["dependencies"] = deps.all() context["dependencies"] = deps.all()
# Existing dependencies to avoid multiple lookups
context["dependencies_names_from_aur"] = [
item.Name
for item in db.query(models.Package)
.filter(
models.Package.Name.in_(
pkg.package_dependencies.with_entities(models.PackageDependency.DepName)
)
)
.all()
]
# Package requirements (other packages depend on this one). # Package requirements (other packages depend on this one).
reqs = pkgutil.pkg_required(pkg.Name, [p.RelName for p in rels_data.get("p", [])]) reqs = pkgutil.pkg_required(pkg.Name, [p.RelName for p in rels_data.get("p", [])])

View file

@ -177,3 +177,6 @@ expiry_time_search = 600
expiry_time_statistics = 300 expiry_time_statistics = 300
; number of seconds after a cache entry for rss queries expires, default is 5 minutes ; number of seconds after a cache entry for rss queries expires, default is 5 minutes
expiry_time_rss = 300 expiry_time_rss = 300
[tracing]
otlp_endpoint = http://localhost:4318/v1/traces

View file

@ -73,3 +73,6 @@ pkgnames-repo = pkgnames.git
[aurblup] [aurblup]
db-path = YOUR_AUR_ROOT/aurblup/ db-path = YOUR_AUR_ROOT/aurblup/
[tracing]
otlp_endpoint = http://tempo:4318/v1/traces

View file

@ -1,5 +1,4 @@
version: "3.8" ---
services: services:
ca: ca:
volumes: volumes:

View file

@ -1,16 +1,10 @@
version: "3.8" ---
services: services:
ca: ca:
volumes: volumes:
- ./data:/data - ./data:/data
- step:/root/.step - step:/root/.step
mariadb_init:
depends_on:
mariadb:
condition: service_healthy
git: git:
volumes: volumes:
- git_data:/aurweb/aur.git - git_data:/aurweb/aur.git
@ -21,9 +15,6 @@ services:
- git_data:/aurweb/aur.git - git_data:/aurweb/aur.git
- ./data:/data - ./data:/data
- smartgit_run:/var/run/smartgit - smartgit_run:/var/run/smartgit
depends_on:
mariadb:
condition: service_healthy
fastapi: fastapi:
volumes: volumes:

View file

@ -1,3 +1,4 @@
---
# #
# Docker service definitions for the aurweb project. # Docker service definitions for the aurweb project.
# #
@ -16,8 +17,6 @@
# #
# Copyright (C) 2021 aurweb Development # Copyright (C) 2021 aurweb Development
# All Rights Reserved. # All Rights Reserved.
version: "3.8"
services: services:
aurweb-image: aurweb-image:
build: . build: .
@ -49,7 +48,7 @@ services:
image: aurweb:latest image: aurweb:latest
init: true init: true
entrypoint: /docker/mariadb-entrypoint.sh entrypoint: /docker/mariadb-entrypoint.sh
command: /usr/bin/mysqld_safe --datadir=/var/lib/mysql command: /usr/bin/mariadbd-safe --datadir=/var/lib/mysql
ports: ports:
# This will expose mariadbd on 127.0.0.1:13306 in the host. # This will expose mariadbd on 127.0.0.1:13306 in the host.
# Ex: `mysql -uaur -paur -h 127.0.0.1 -P 13306 aurweb` # Ex: `mysql -uaur -paur -h 127.0.0.1 -P 13306 aurweb`
@ -81,7 +80,7 @@ services:
environment: environment:
- MARIADB_PRIVILEGED=1 - MARIADB_PRIVILEGED=1
entrypoint: /docker/mariadb-entrypoint.sh entrypoint: /docker/mariadb-entrypoint.sh
command: /usr/bin/mysqld_safe --datadir=/var/lib/mysql command: /usr/bin/mariadbd-safe --datadir=/var/lib/mysql
ports: ports:
# This will expose mariadbd on 127.0.0.1:13307 in the host. # This will expose mariadbd on 127.0.0.1:13307 in the host.
# Ex: `mysql -uaur -paur -h 127.0.0.1 -P 13306 aurweb` # Ex: `mysql -uaur -paur -h 127.0.0.1 -P 13306 aurweb`
@ -107,8 +106,10 @@ services:
test: "bash /docker/health/sshd.sh" test: "bash /docker/health/sshd.sh"
interval: 3s interval: 3s
depends_on: depends_on:
mariadb:
condition: service_healthy
mariadb_init: mariadb_init:
condition: service_started condition: service_completed_successfully
volumes: volumes:
- mariadb_run:/var/run/mysqld - mariadb_run:/var/run/mysqld
@ -122,6 +123,9 @@ services:
healthcheck: healthcheck:
test: "bash /docker/health/smartgit.sh" test: "bash /docker/health/smartgit.sh"
interval: 3s interval: 3s
depends_on:
mariadb:
condition: service_healthy
cgit-fastapi: cgit-fastapi:
image: aurweb:latest image: aurweb:latest
@ -152,8 +156,10 @@ services:
entrypoint: /docker/cron-entrypoint.sh entrypoint: /docker/cron-entrypoint.sh
command: /docker/scripts/run-cron.sh command: /docker/scripts/run-cron.sh
depends_on: depends_on:
mariadb:
condition: service_healthy
mariadb_init: mariadb_init:
condition: service_started condition: service_completed_successfully
volumes: volumes:
- ./aurweb:/aurweb/aurweb - ./aurweb:/aurweb/aurweb
- mariadb_run:/var/run/mysqld - mariadb_run:/var/run/mysqld
@ -182,6 +188,12 @@ services:
condition: service_healthy condition: service_healthy
cron: cron:
condition: service_started condition: service_started
mariadb:
condition: service_healthy
mariadb_init:
condition: service_completed_successfully
tempo:
condition: service_healthy
volumes: volumes:
- archives:/var/lib/aurweb/archives - archives:/var/lib/aurweb/archives
- mariadb_run:/var/run/mysqld - mariadb_run:/var/run/mysqld
@ -281,6 +293,56 @@ services:
- ./test:/aurweb/test - ./test:/aurweb/test
- ./templates:/aurweb/templates - ./templates:/aurweb/templates
grafana:
# TODO: check if we need init: true
image: grafana/grafana:11.1.3
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_DISABLE_LOGIN_FORM=true
- GF_LOG_LEVEL=warn
# check if depends ar ecorrect, does stopping or restarting a child exit grafana?
depends_on:
prometheus:
condition: service_healthy
tempo:
condition: service_healthy
ports:
- "127.0.0.1:3000:3000"
volumes:
- ./docker/config/grafana/datasources:/etc/grafana/provisioning/datasources
prometheus:
image: prom/prometheus:latest
command:
- --config.file=/etc/prometheus/prometheus.yml
- --web.enable-remote-write-receiver
- --web.listen-address=prometheus:9090
healthcheck:
# TODO: check if there is a status route
test: "sh /docker/health/prometheus.sh"
interval: 3s
ports:
- "127.0.0.1:9090:9090"
volumes:
- ./docker/config/prometheus.yml:/etc/prometheus/prometheus.yml
- ./docker/health/prometheus.sh:/docker/health/prometheus.sh
tempo:
image: grafana/tempo:2.5.0
command:
- -config.file=/etc/tempo/config.yml
healthcheck:
# TODO: check if there is a status route
test: "sh /docker/health/tempo.sh"
interval: 3s
ports:
- "127.0.0.1:3200:3200"
- "127.0.0.1:4318:4318"
volumes:
- ./docker/config/tempo.yml:/etc/tempo/config.yml
- ./docker/health/tempo.sh:/docker/health/tempo.sh
volumes: volumes:
mariadb_test_run: {} mariadb_test_run: {}
mariadb_run: {} # Share /var/run/mysqld/mysqld.sock mariadb_run: {} # Share /var/run/mysqld/mysqld.sock

View file

@ -47,7 +47,7 @@ Luckily such data can be generated.
docker compose exec fastapi /bin/bash docker compose exec fastapi /bin/bash
pacman -S words fortune-mod pacman -S words fortune-mod
./schema/gendummydata.py dummy.sql ./schema/gendummydata.py dummy.sql
mysql aurweb < dummy.sql mariadb aurweb < dummy.sql
``` ```
The generation script may prompt you to install other Arch packages before it The generation script may prompt you to install other Arch packages before it

View file

@ -0,0 +1,42 @@
---
apiVersion: 1
deleteDatasources:
- name: Prometheus
- name: Tempo
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
access: proxy
url: http://prometheus:9090
orgId: 1
editable: false
jsonData:
timeInterval: 1m
- name: Tempo
type: tempo
uid: tempo
access: proxy
url: http://tempo:3200
orgId: 1
editable: false
jsonData:
tracesToMetrics:
datasourceUid: 'prometheus'
spanStartTimeShift: '1h'
spanEndTimeShift: '-1h'
serviceMap:
datasourceUid: 'prometheus'
nodeGraph:
enabled: true
search:
hide: false
traceQuery:
timeShiftEnabled: true
spanStartTimeShift: '1h'
spanEndTimeShift: '-1h'
spanBar:
type: 'Tag'
tag: 'http.path'

View file

@ -0,0 +1,15 @@
---
global:
scrape_interval: 60s
scrape_configs:
- job_name: tempo
static_configs:
- targets: ['tempo:3200']
labels:
instance: tempo
- job_name: aurweb
static_configs:
- targets: ['fastapi:8000']
labels:
instance: aurweb

54
docker/config/tempo.yml Normal file
View file

@ -0,0 +1,54 @@
---
stream_over_http_enabled: true
server:
http_listen_address: tempo
http_listen_port: 3200
log_level: info
query_frontend:
search:
duration_slo: 5s
throughput_bytes_slo: 1.073741824e+09
trace_by_id:
duration_slo: 5s
distributor:
receivers:
otlp:
protocols:
http:
endpoint: tempo:4318
log_received_spans:
enabled: false
metric_received_spans:
enabled: false
ingester:
max_block_duration: 5m
compactor:
compaction:
block_retention: 1h
metrics_generator:
registry:
external_labels:
source: tempo
storage:
path: /tmp/tempo/generator/wal
remote_write:
- url: http://prometheus:9090/api/v1/write
send_exemplars: true
traces_storage:
path: /tmp/tempo/generator/traces
storage:
trace:
backend: local
wal:
path: /tmp/tempo/wal
local:
path: /tmp/tempo/blocks
overrides:
metrics_generator_processors: [service-graphs, span-metrics, local-blocks]

View file

@ -1,2 +1,2 @@
#!/bin/bash #!/bin/bash
exec mysqladmin ping --silent exec mariadb-admin ping --silent

2
docker/health/prometheus.sh Executable file
View file

@ -0,0 +1,2 @@
#!/bin/sh
exec wget -q http://prometheus:9090/status -O /dev/null

2
docker/health/tempo.sh Executable file
View file

@ -0,0 +1,2 @@
#!/bin/sh
exec wget -q http://tempo:3200/status -O /dev/null

View file

@ -6,8 +6,8 @@ MYSQL_DATA=/var/lib/mysql
mariadb-install-db --user=mysql --basedir=/usr --datadir=$MYSQL_DATA mariadb-install-db --user=mysql --basedir=/usr --datadir=$MYSQL_DATA
# Start it up. # Start it up.
mysqld_safe --datadir=$MYSQL_DATA --skip-networking & mariadbd-safe --datadir=$MYSQL_DATA --skip-networking &
while ! mysqladmin ping 2>/dev/null; do while ! mariadb-admin ping 2>/dev/null; do
sleep 1s sleep 1s
done done
@ -15,17 +15,17 @@ done
DATABASE="aurweb" # Persistent database for fastapi. DATABASE="aurweb" # Persistent database for fastapi.
echo "Taking care of primary database '${DATABASE}'..." echo "Taking care of primary database '${DATABASE}'..."
mysql -u root -e "CREATE USER IF NOT EXISTS 'aur'@'localhost' IDENTIFIED BY 'aur';" mariadb -u root -e "CREATE USER IF NOT EXISTS 'aur'@'localhost' IDENTIFIED BY 'aur';"
mysql -u root -e "CREATE USER IF NOT EXISTS 'aur'@'%' IDENTIFIED BY 'aur';" mariadb -u root -e "CREATE USER IF NOT EXISTS 'aur'@'%' IDENTIFIED BY 'aur';"
mysql -u root -e "CREATE DATABASE IF NOT EXISTS $DATABASE;" mariadb -u root -e "CREATE DATABASE IF NOT EXISTS $DATABASE;"
mysql -u root -e "CREATE USER IF NOT EXISTS 'aur'@'%' IDENTIFIED BY 'aur';" mariadb -u root -e "CREATE USER IF NOT EXISTS 'aur'@'%' IDENTIFIED BY 'aur';"
mysql -u root -e "GRANT ALL ON aurweb.* TO 'aur'@'localhost';" mariadb -u root -e "GRANT ALL ON aurweb.* TO 'aur'@'localhost';"
mysql -u root -e "GRANT ALL ON aurweb.* TO 'aur'@'%';" mariadb -u root -e "GRANT ALL ON aurweb.* TO 'aur'@'%';"
mysql -u root -e "CREATE USER IF NOT EXISTS 'root'@'%' IDENTIFIED BY 'aur';" mariadb -u root -e "CREATE USER IF NOT EXISTS 'root'@'%' IDENTIFIED BY 'aur';"
mysql -u root -e "GRANT ALL ON *.* TO 'root'@'%' WITH GRANT OPTION;" mariadb -u root -e "GRANT ALL ON *.* TO 'root'@'%' WITH GRANT OPTION;"
mysqladmin -uroot shutdown mariadb-admin -uroot shutdown
exec "$@" exec "$@"

View file

@ -13,7 +13,7 @@ pacman -Sy --noconfirm --noprogressbar archlinux-keyring
# Install other OS dependencies. # Install other OS dependencies.
pacman -Syu --noconfirm --noprogressbar \ pacman -Syu --noconfirm --noprogressbar \
--cachedir .pkg-cache git gpgme nginx redis openssh \ git gpgme nginx redis openssh \
mariadb mariadb-libs cgit-aurweb uwsgi uwsgi-plugin-cgi \ mariadb mariadb-libs cgit-aurweb uwsgi uwsgi-plugin-cgi \
python-pip pyalpm python-srcinfo curl libeatmydata cronie \ python-pip pyalpm python-srcinfo curl libeatmydata cronie \
python-poetry python-poetry-core step-cli step-ca asciidoc \ python-poetry python-poetry-core step-cli step-ca asciidoc \

2043
poetry.lock generated

File diff suppressed because it is too large Load diff

View file

@ -16,7 +16,7 @@ combine_as_imports = true
# #
[tool.poetry] [tool.poetry]
name = "aurweb" name = "aurweb"
version = "v6.2.12" version = "v6.2.16"
license = "GPL-2.0-only" license = "GPL-2.0-only"
description = "Source code for the Arch User Repository's website" description = "Source code for the Arch User Repository's website"
homepage = "https://aur.archlinux.org" homepage = "https://aur.archlinux.org"
@ -52,13 +52,13 @@ build-backend = "poetry.masonry.api"
"Request Mailing List" = "https://lists.archlinux.org/listinfo/aur-requests" "Request Mailing List" = "https://lists.archlinux.org/listinfo/aur-requests"
[tool.poetry.dependencies] [tool.poetry.dependencies]
python = ">=3.9,<3.13" python = ">=3.10,<3.14"
# poetry-dynamic-versioning is used to produce tool.poetry.version # poetry-dynamic-versioning is used to produce tool.poetry.version
# based on git tags. # based on git tags.
# General # General
aiofiles = "^23.2.1" aiofiles = "^24.0.0"
asgiref = "^3.8.1" asgiref = "^3.8.1"
bcrypt = "^4.1.2" bcrypt = "^4.1.2"
bleach = "^6.1.0" bleach = "^6.1.0"
@ -69,34 +69,42 @@ httpx = "^0.27.0"
itsdangerous = "^2.1.2" itsdangerous = "^2.1.2"
lxml = "^5.2.1" lxml = "^5.2.1"
orjson = "^3.10.0" orjson = "^3.10.0"
protobuf = "^5.26.1" pygit2 = "^1.17.0"
pygit2 = "^1.14.1" python-multipart = "0.0.19"
python-multipart = "^0.0.9"
redis = "^5.0.3" redis = "^5.0.3"
requests = "^2.31.0" requests = "^2.31.0"
paginate = "^0.5.6" paginate = "^0.5.6"
# SQL # SQL
alembic = "^1.13.1" alembic = "^1.13.1"
mysqlclient = "^2.2.4" mysqlclient = "^2.2.3"
Authlib = "^1.3.0" Authlib = "^1.3.0"
Jinja2 = "^3.1.3" Jinja2 = "^3.1.3"
Markdown = "^3.6" Markdown = "^3.6"
Werkzeug = "^3.0.2" Werkzeug = "^3.0.2"
SQLAlchemy = "^1.4.52" SQLAlchemy = "^1.4.52"
greenlet = "3.1.1" # Explicitly add greenlet (dependency of SQLAlchemy) for python 3.13 support
# ASGI # ASGI
uvicorn = "^0.29.0" uvicorn = "^0.30.0"
gunicorn = "^21.2.0" gunicorn = "^22.0.0"
Hypercorn = "^0.16.0" Hypercorn = "^0.17.0"
prometheus-fastapi-instrumentator = "^7.0.0"
pytest-xdist = "^3.5.0" pytest-xdist = "^3.5.0"
filelock = "^3.13.3" filelock = "^3.13.3"
posix-ipc = "^1.1.1" posix-ipc = "^1.1.1"
pyalpm = "^0.10.6" pyalpm = "^0.10.6"
fastapi = "^0.110.0" fastapi = "^0.112.0"
srcinfo = "^0.1.2" srcinfo = "^0.1.2"
tomlkit = "^0.12.0" tomlkit = "^0.13.0"
# Tracing
prometheus-fastapi-instrumentator = "^7.0.0"
opentelemetry-api = "^1.26.0"
opentelemetry-sdk = "^1.26.0"
opentelemetry-exporter-otlp-proto-http = "^1.26.0"
opentelemetry-instrumentation-fastapi = "^0.47b0"
opentelemetry-instrumentation-redis = "^0.47b0"
opentelemetry-instrumentation-sqlalchemy = "^0.47b0"
[tool.poetry.dev-dependencies] [tool.poetry.dev-dependencies]
coverage = "^7.4.4" coverage = "^7.4.4"
@ -104,7 +112,7 @@ pytest = "^8.1.1"
pytest-asyncio = "^0.23.0" pytest-asyncio = "^0.23.0"
pytest-cov = "^5.0.0" pytest-cov = "^5.0.0"
pytest-tap = "^3.4" pytest-tap = "^3.4"
watchfiles = "^0.21.0" watchfiles = "^1.0.4"
[tool.poetry.scripts] [tool.poetry.scripts]
aurweb-git-auth = "aurweb.git.auth:main" aurweb-git-auth = "aurweb.git.auth:main"

View file

@ -24,71 +24,73 @@
{{ "Search wiki" | tr }} {{ "Search wiki" | tr }}
</a> </a>
</li> </li>
{% if not out_of_date %} {% if request.user.is_authenticated() %}
{% if not out_of_date %}
<li>
<a href="/pkgbase/{{ pkgbase.Name }}/flag">
{{ "Flag package out-of-date" | tr }}
</a>
</li>
{% else %}
<li>
<span class="flagged">
<a href="/pkgbase/{{ pkgbase.Name }}/flag-comment">
{{
"Flagged out-of-date (%s)"
| tr | format(date_strftime(pkgbase.OutOfDateTS, "%Y-%m-%d"))
}}
</a>
</span>
</li>
{% if request.user.has_credential(creds.PKGBASE_UNFLAG, approved=unflaggers) %}
<li>
<form action="/pkgbase/{{ pkgbase.Name }}/unflag" method="post">
<input class="button text-button"
type="submit"
name="do_UnFlag"
value="{{ 'Unflag package' | tr }}"
/>
</form>
</li>
{% endif %}
{% endif %}
<li> <li>
<a href="/pkgbase/{{ pkgbase.Name }}/flag"> {% if not voted %}
{{ "Flag package out-of-date" | tr }} <form action="/pkgbase/{{ pkgbase.Name }}/vote" method="post">
</a> <input type="submit"
</li> class="button text-button"
{% else %} name="do_Vote"
<li> value="{{ 'Vote for this package' | tr }}" />
<span class="flagged">
<a href="/pkgbase/{{ pkgbase.Name }}/flag-comment">
{{
"Flagged out-of-date (%s)"
| tr | format(date_strftime(pkgbase.OutOfDateTS, "%Y-%m-%d"))
}}
</a>
</span>
</li>
{% if request.user.has_credential(creds.PKGBASE_UNFLAG, approved=unflaggers) %}
<li>
<form action="/pkgbase/{{ pkgbase.Name }}/unflag" method="post">
<input class="button text-button"
type="submit"
name="do_UnFlag"
value="{{ 'Unflag package' | tr }}"
/>
</form> </form>
{% else %}
<form action="/pkgbase/{{ pkgbase.Name }}/unvote" method="post">
<input type="submit"
class="button text-button"
name="do_UnVote"
value="{{ 'Remove vote' | tr }}" />
</form>
{% endif %}
</li>
<li>
{% if notified %}
<form action="/pkgbase/{{ pkgbase.Name }}/unnotify" method="post">
<input type="submit"
class="button text-button"
name="do_UnNotify"
value="{{ 'Disable notifications' | tr }}"
/>
</form>
{% else %}
<form action="/pkgbase/{{ pkgbase.Name }}/notify" method="post">
<input type="submit"
class="button text-button"
name="do_Notify"
value="{{ 'Enable notifications' | tr }}"
/>
</form>
{% endif %}
</li> </li>
{% endif %}
{% endif %} {% endif %}
<li>
{% if not voted %}
<form action="/pkgbase/{{ pkgbase.Name }}/vote" method="post">
<input type="submit"
class="button text-button"
name="do_Vote"
value="{{ 'Vote for this package' | tr }}" />
</form>
{% else %}
<form action="/pkgbase/{{ pkgbase.Name }}/unvote" method="post">
<input type="submit"
class="button text-button"
name="do_UnVote"
value="{{ 'Remove vote' | tr }}" />
</form>
{% endif %}
</li>
<li>
{% if notified %}
<form action="/pkgbase/{{ pkgbase.Name }}/unnotify" method="post">
<input type="submit"
class="button text-button"
name="do_UnNotify"
value="{{ 'Disable notifications' | tr }}"
/>
</form>
{% else %}
<form action="/pkgbase/{{ pkgbase.Name }}/notify" method="post">
<input type="submit"
class="button text-button"
name="do_Notify"
value="{{ 'Enable notifications' | tr }}"
/>
</form>
{% endif %}
</li>
{% if request.user.has_credential(creds.PKGBASE_EDIT_COMAINTAINERS, approved=[pkgbase.Maintainer]) %} {% if request.user.has_credential(creds.PKGBASE_EDIT_COMAINTAINERS, approved=[pkgbase.Maintainer]) %}
<li> <li>
<a href="/pkgbase/{{ pkgbase.Name }}/comaintainers"> <a href="/pkgbase/{{ pkgbase.Name }}/comaintainers">
@ -111,11 +113,13 @@
</li> </li>
{% endif %} {% endif %}
{% endif %} {% endif %}
<li> {% if request.user.is_authenticated() %}
<a href="/pkgbase/{{ pkgbase.Name }}/request?{{ {'next': '/pkgbase/%s' | format(pkgbase.Name)} | urlencode }}"> <li>
{{ "Submit Request" | tr }} <a href="/pkgbase/{{ pkgbase.Name }}/request?{{ {'next': '/pkgbase/%s' | format(pkgbase.Name)} | urlencode }}">
</a> {{ "Submit Request" | tr }}
</li> </a>
</li>
{% endif %}
{% if request.user.has_credential(creds.PKGBASE_DELETE) %} {% if request.user.has_credential(creds.PKGBASE_DELETE) %}
<li> <li>
<a href="/pkgbase/{{ pkgbase.Name }}/delete?next=/packages"> <a href="/pkgbase/{{ pkgbase.Name }}/delete?next=/packages">

View file

@ -20,7 +20,7 @@
{% endif %} {% endif %}
{% else -%} {% else -%}
</a> </a>
{%- if dep.is_aur_package() -%} {%- if dep.DepName in dependencies_names_from_aur -%}
<sup><small>AUR</small></sup> <sup><small>AUR</small></sup>
{% endif %} {% endif %}
{% endif %} {% endif %}

View file

@ -507,7 +507,9 @@ def test_package_requests_display(
client: TestClient, user: User, package: Package, pkgreq: PackageRequest client: TestClient, user: User, package: Package, pkgreq: PackageRequest
): ):
# Test that a single request displays "1 pending request". # Test that a single request displays "1 pending request".
cookies = {"AURSID": user.login(Request(), "testPassword")}
with client as request: with client as request:
request.cookies = cookies
resp = request.get(package_endpoint(package)) resp = request.get(package_endpoint(package))
assert resp.status_code == int(HTTPStatus.OK) assert resp.status_code == int(HTTPStatus.OK)
@ -530,6 +532,7 @@ def test_package_requests_display(
# Test that a two requests display "2 pending requests". # Test that a two requests display "2 pending requests".
with client as request: with client as request:
request.cookies = cookies
resp = request.get(package_endpoint(package)) resp = request.get(package_endpoint(package))
assert resp.status_code == int(HTTPStatus.OK) assert resp.status_code == int(HTTPStatus.OK)