You've already forked .autoupdate
Replace per-package push scripts with a self-healing AUR reconciler
The six nightly workflows each cloned a package repo and ran that repo's own
push.sh. Two problems compounded into months of silent breakage.
push.sh pushed to origin before the AUR. When the AUR push failed, origin
already carried the new version, so every later run hit "same (old) version
specified", bare-exit 0'd, and the workflow went green while the AUR rotted.
reflex-appimage sat at 1.0.11 locally against 1.0.10 published; nethlink-appimage
was never submitted at all.
The AUR pushes were failing because .SRCINFO was empty -- 0 bytes in
reflex-appimage, 1 byte in nethlink-appimage and pman-helper, against 500-2000
in the healthy ones. The AUR rejects an invalid .SRCINFO. Both PKGBUILDs verify
clean under a real Arch makepkg, so this was the CI environment, and with no
set -e the script committed and pushed the truncated file regardless.
pman-helper failed independently: the workflows queried the Gitea endpoint
/repos/{owner}/{repo}/repo/tags, which 404s -- the correct path is /tags. The
empty result fell through to an interactive read that got EOF, and it committed
pkgver="" plus the sha256 of a zero-byte download.
The replacement is a reconciler rather than a version-bump script. A new
upstream release, a manual push to origin, and a push that failed last night
all take the same path, and the AUR reconcile runs on every pass -- so a failed
push simply retries. Sync is decided by comparing tree content against the AUR,
not by whether this run happened to find something new.
pkgrel is now derived from the AUR rather than from local history, which is
what makes it idempotent: a version differing from the published one resets it
to 1, a packaging change increments it, and once published the next run sees no
difference and does nothing. Editing a PKGBUILD no longer needs a manual bump.
Checksums are computed directly instead of via makepkg -g / updpkgsums. Those
generate every arch array in one pass into a single directory, so when two
arches rename to the same target the second finds the first one's file and
emits an identical, silently wrong sum -- verified against freetube-appimage.
Downloads go to content-addressed cache paths instead. Multi-arch works for
when it is needed; single-arch is the degenerate case of the same loop.
Guards for each observed failure: a real Arch makepkg in an archlinux container
as non-root, set -euo pipefail throughout, .SRCINFO validated against the
PKGBUILD before any commit, and a failed version lookup exiting 1 so it can
never again be mistaken for a quiet "no new version" night.
Package-specific detail moves out of this repo entirely, into an .autoupdate
manifest committed to each package repo. sha256 only; sha1sums and md5sums are
removed. deskflow-bin and freetube-appimage are dropped from automation, having
been deleted from the AUR.
Covered by test.sh: 37 assertions against local bare repos and a local HTTP
server, no network and never touching the real AUR. The retry, pkgrel
idempotence and multi-arch collision tests were each verified by mutation.
This commit is contained in:
200
common.sh
Executable file
200
common.sh
Executable file
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared helpers for the AUR autoupdater.
|
||||
# Sourced by autoupdate.sh and the standalone scripts; never executed directly.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Endpoints. Overridable so the test suite can point at local bare repos
|
||||
# instead of the real AUR; defaults are the production values.
|
||||
AUR_SSH_BASE="${AUR_SSH_BASE:-ssh://aur@aur.archlinux.org}"
|
||||
AUR_HTTPS_BASE="${AUR_HTTPS_BASE:-https://aur.archlinux.org}"
|
||||
|
||||
log() { printf '==> %s\n' "$*"; }
|
||||
info() { printf ' -> %s\n' "$*"; }
|
||||
warn() { printf 'WARNING: %s\n' "$*" >&2; }
|
||||
die() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
# The old push.sh fell back to an interactive `read` when no version was passed.
|
||||
# In CI that reads EOF and yields an empty version, which is how pman-helper
|
||||
# committed pkgver="" and the sha256 of a zero-byte download.
|
||||
require_noninteractive() {
|
||||
[ -t 0 ] && die "refusing to prompt: this script must never depend on interactive input"
|
||||
return 0
|
||||
}
|
||||
|
||||
# A version we are willing to write into a PKGBUILD.
|
||||
VERSION_RE='^[0-9][0-9a-zA-Z._+]*$'
|
||||
|
||||
validate_version() {
|
||||
local ver="${1-}" what="${2-version}"
|
||||
[ -n "$ver" ] || die "${what} is empty"
|
||||
[[ "$ver" =~ $VERSION_RE ]] || die "${what} '${ver}' does not look like a version"
|
||||
return 0
|
||||
}
|
||||
|
||||
# Read a scalar assignment out of a PKGBUILD without sourcing it.
|
||||
# Handles pkgver="1.2.3", pkgver='1.2.3' and bare pkgver=1.2.3.
|
||||
pkgbuild_get() {
|
||||
local var="$1" file="${2:-PKGBUILD}"
|
||||
sed -n -E "s/^${var}=[\"']?([^\"']*)[\"']?[[:space:]]*(#.*)?$/\1/p" "$file" | head -n1
|
||||
}
|
||||
|
||||
# Rewrite a scalar assignment in place, preserving double-quoted style.
|
||||
pkgbuild_set() {
|
||||
local var="$1" val="$2" file="${3:-PKGBUILD}"
|
||||
grep -qE "^${var}=" "$file" || die "no ${var}= assignment in ${file}"
|
||||
sed -i -E "s|^${var}=.*|${var}=\"${val}\"|" "$file"
|
||||
}
|
||||
|
||||
# Expand a PKGBUILD's arrays/vars by sourcing it in a subshell and printing the
|
||||
# result. Safe enough: these are our own package definitions. Used instead of
|
||||
# grep/sed so that ${pkgver} / ${_srcname} substitutions actually resolve.
|
||||
#
|
||||
# Usage: pkgbuild_expand <file> <bash snippet printing what you want>
|
||||
pkgbuild_expand() {
|
||||
local file="$1" snippet="$2"
|
||||
(
|
||||
set +euo pipefail
|
||||
# Neutralise anything that would run during a real build.
|
||||
# shellcheck disable=SC1090
|
||||
source "$file" >/dev/null 2>&1 || true
|
||||
eval "$snippet"
|
||||
)
|
||||
}
|
||||
|
||||
# Print the elements of an array variable, one per line, NUL-safe enough for
|
||||
# source entries (which never contain newlines).
|
||||
pkgbuild_array() {
|
||||
local file="$1" name="$2"
|
||||
pkgbuild_expand "$file" "printf '%s\n' \"\${${name}[@]}\" 2>/dev/null"
|
||||
}
|
||||
|
||||
# True if the named array exists in the PKGBUILD (even if empty).
|
||||
pkgbuild_has_array() {
|
||||
local file="$1" name="$2"
|
||||
[ "$(pkgbuild_expand "$file" "declare -p ${name} >/dev/null 2>&1 && echo yes")" = "yes" ]
|
||||
}
|
||||
|
||||
# Replace `name=( ... )` in a PKGBUILD with the given values.
|
||||
#
|
||||
# This exists to replace the line-counting sed in the old freetube push.sh
|
||||
# (`sednewline="n;"` accumulating `n;n;n;`), which silently corrupts the file
|
||||
# whenever a source array's length changes. Matches `^name=(` through its
|
||||
# closing `)`, so it handles both one-line and multi-line blocks, and appends
|
||||
# the array if it is missing entirely.
|
||||
replace_array() {
|
||||
local file="$1" name="$2"; shift 2
|
||||
local block v
|
||||
if [ "$#" -eq 1 ]; then
|
||||
block="${name}=(\"$1\")"
|
||||
else
|
||||
block="${name}=("
|
||||
for v in "$@"; do block+=$'\n'" \"${v}\""; done
|
||||
block+=$'\n'")"
|
||||
fi
|
||||
|
||||
if ! grep -qE "^${name}=\(" "$file"; then
|
||||
printf '%s\n' "$block" >> "$file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
awk -v name="$name" -v block="$block" '
|
||||
BEGIN { skip = 0 }
|
||||
skip {
|
||||
# Consume until the line that closes the array.
|
||||
if ($0 ~ /\)[[:space:]]*$/) skip = 0
|
||||
next
|
||||
}
|
||||
$0 ~ "^" name "=\\(" {
|
||||
print block
|
||||
# A one-line `name=("x")` closes on the same line; a block does not.
|
||||
if ($0 !~ /\)[[:space:]]*$/) skip = 1
|
||||
next
|
||||
}
|
||||
{ print }
|
||||
' "$file" > "${file}.tmp"
|
||||
|
||||
mv "${file}.tmp" "$file"
|
||||
}
|
||||
|
||||
# Delete a whole `name=( ... )` block (used to retire sha1sums/md5sums).
|
||||
remove_array() {
|
||||
local file="$1" name="$2"
|
||||
grep -qE "^${name}=\(" "$file" || return 0
|
||||
awk -v name="$name" '
|
||||
BEGIN { skip = 0 }
|
||||
skip { if ($0 ~ /\)[[:space:]]*$/) skip = 0; next }
|
||||
$0 ~ "^" name "=\\(" { if ($0 !~ /\)[[:space:]]*$/) skip = 1; next }
|
||||
{ print }
|
||||
' "$file" > "${file}.tmp"
|
||||
mv "${file}.tmp" "$file"
|
||||
}
|
||||
|
||||
# Split a source entry into its rename target and its URL/path.
|
||||
# "foo.AppImage::https://x/y.AppImage" -> name=foo.AppImage loc=https://x/y.AppImage
|
||||
# "https://x/y.tar.gz" -> name=y.tar.gz loc=https://x/y.tar.gz
|
||||
# "reflex.desktop" -> name=reflex.desktop loc=reflex.desktop
|
||||
source_name() {
|
||||
local e="$1"
|
||||
if [[ "$e" == *"::"* ]]; then printf '%s' "${e%%::*}"; else printf '%s' "${e##*/}"; fi
|
||||
}
|
||||
source_loc() { local e="$1"; printf '%s' "${e##*::}"; }
|
||||
|
||||
is_remote() { [[ "$(source_loc "$1")" =~ ^(https?|ftp):// ]]; }
|
||||
|
||||
# VCS sources (git+https://..., svn+..., etc) are not fixed artefacts and
|
||||
# makepkg expects a literal SKIP for them rather than a digest.
|
||||
is_vcs() { [[ "$(source_loc "$1")" =~ ^(git|svn|hg|bzr)(\+|:) ]]; }
|
||||
|
||||
# Download a URL once per run and hand back the cached path.
|
||||
#
|
||||
# Downloads go to a content-addressed cache path, never to the source entry's
|
||||
# `::` rename target. That matters for multi-arch: freetube's x86_64 and
|
||||
# aarch64 entries both rename to freetube.AppImage, so writing to the rename
|
||||
# target makes the second arch find the first arch's file and silently hash
|
||||
# the wrong binary. This is the same collision that makes `makepkg -g`
|
||||
# unusable for multi-arch PKGBUILDs.
|
||||
download_cached() {
|
||||
local url="$1" key path
|
||||
[ -n "${DL_CACHE:-}" ] || die "download_cached: DL_CACHE is not set"
|
||||
mkdir -p "$DL_CACHE"
|
||||
key="$(printf '%s' "$url" | sha256sum | cut -c1-32)"
|
||||
path="${DL_CACHE}/${key}"
|
||||
|
||||
if [ ! -s "$path" ]; then
|
||||
info "fetching ${url}" >&2
|
||||
curl --silent --show-error --location --fail --retry 3 --retry-delay 2 \
|
||||
--max-time 900 -o "${path}.part" "$url" \
|
||||
|| die "download failed: ${url}"
|
||||
mv "${path}.part" "$path"
|
||||
fi
|
||||
printf '%s' "$path"
|
||||
}
|
||||
|
||||
# Regenerate .SRCINFO and refuse to accept a broken one.
|
||||
#
|
||||
# This is the guard that would have prevented every observed breakage: an empty
|
||||
# or inconsistent .SRCINFO is exactly what the AUR rejects, and the old script
|
||||
# committed and pushed it anyway because it had no `set -e`.
|
||||
regen_srcinfo() {
|
||||
local dir="${1:-.}"
|
||||
local out rc=0
|
||||
out="$(cd "$dir" && makepkg --printsrcinfo 2>&1)" || rc=$?
|
||||
|
||||
if [ "$rc" -ne 0 ] || [ -z "${out//[[:space:]]/}" ]; then
|
||||
printf '%s\n' "$out" >&2
|
||||
die "makepkg --printsrcinfo produced no usable output (exit ${rc}).
|
||||
Is this running under a real Arch makepkg? The CI job must use the
|
||||
archlinux container; Ubuntu's build environment silently yields an
|
||||
empty .SRCINFO, which the AUR then rejects."
|
||||
fi
|
||||
|
||||
printf '%s\n' "$out" > "${dir}/.SRCINFO"
|
||||
|
||||
local pb_ver sr_ver
|
||||
pb_ver="$(pkgbuild_get pkgver "${dir}/PKGBUILD")"
|
||||
sr_ver="$(sed -n -E 's/^[[:space:]]*pkgver = (.*)$/\1/p' "${dir}/.SRCINFO" | head -n1)"
|
||||
[ -n "$sr_ver" ] || die ".SRCINFO has no pkgver"
|
||||
[ "$pb_ver" = "$sr_ver" ] \
|
||||
|| die ".SRCINFO pkgver '${sr_ver}' disagrees with PKGBUILD pkgver '${pb_ver}'"
|
||||
}
|
||||
Reference in New Issue
Block a user