Freepod · product upgrades · 15 September 2026

Can a local 27B model keep our app catalog up to date?

Freepod curates self-hosted apps (Immich, Nextcloud, Vaultwarden, …) and maintains its own Helm charts for them. We wrote a skill that lets an unattended agent notice a new upstream release, work out everything that has to change for Freepod to ship it, and open a pull request. Then we ran it 24 times against rewound scenarios, in dry-run mode, on a single local GPU.

Verdict: good enough to hand to the nightly service. Every outcome the service can hit in a dry run came out correct and machine-readable. The remaining misses are judgment calls that a human reviews anyway, in draft pull requests. It's a probabilistic process, and perfect is the enemy of good.

dry runs (runs 01–24)

5 / 6

result statuses exercised, all schema-valid

20–42 min

per product at the chosen settings

5 / 6

hardest judgment call correct since the last rule change

Setup

The agent

Model: Qwen 3.8 27B (Q4) on llama.cpp, one GPU
Harness: the pi coding agent, one fresh session per product
Skill: products/UPGRADING/SKILL.md, about 480 lines, plus a per-product "Upstream references" section in each README
Output: a PR body, a patch, and result.json, which the service reads

The test bench

Scenarios: real master with one product rewound to an older release, in a local fixture repository
Dry runs only: read-only gh, nothing pushed to GitHub
Graded against a known expected answer per scenario; result.json validated against its JSON Schema
One inference session at a time, even when another agent shares the GPU

Every reachable outcome, correct and schema-valid

Runs 18–23 used the final skill. Each wrote a result.json that validates against the schema the nightly service consumes.

would_open

Ready tag-only PR: Vaultwarden 1.35.0 → 1.37.3 (run 18). Draft for a major: Nextcloud 32 → 33.0.9 (run 22).

would_skip

A human declined this exact upgrade before, so a real run stops; the dry run still does all the work: Immich v2.7.5 → v3.2.1 (runs 21, 23, 24).

skipped

Upstream image doesn't exist. It didn't guess a replacement and asked a human (run 19).

up_to_date

Already on the newest tag: nothing written but result.json (run 20).

opened

Real PRs, in the early manual phase (#111–#114). Not a dry-run status.

failed

Not hit by any run yet.

Thinking effort: medium wins

Same skill, same Immich scenario, three effort levels. All three got the right answer, so this is purely about cost.

mediumserver default

34 min

low

54 min

xhigh

111 min

Wall time for one product, all three graded as passing.

Effort	Run	Wall time	Turns	Peak context	Thinking (chars)	Result
medium	15	34 min	57	79k	56k	pass
low	17	54 min	73	94k	87k	pass
xhigh	16	111 min	98	154k	150k	pass

Low effort thinks a little less per turn but takes more turns, so it ends up slower. Medium is also the server's default, so production needs no extra configuration. Earlier turns' thinking is fed back to the model automatically: the agent returns it, and the chat template keeps it. We verified this in the request captures, so no setting is needed there either.

Full clones beat blobless clones

The agent diffs upstream repositories between two release tags. We started with blobless clones to save download time. In practice every file read became a separate network request, and a few careless git greps stalled the GPU for many minutes.

Blobless dropped

About 27 idle GPU minutes in stalls across runs 10, 16 and 21
Even a path-limited grep fetched about 2 files a second (548 files: about 4.5 minutes)
Needed a git guard, which a bare -- slipped past

Full clone adopted

Immich, the biggest repo we diff: 22 s, 325 MB
After that, a whole-tree grep plus history search: 0.9 s
Immich run 51 → 33 min, with no stalls (runs 21 → 23)
A 1 GB size check skips giants like nextcloud/server (6.9 GB)

A shallow clone wasn't an option. The skill backs up "this upstream change is already handled" claims with git log -S between the two tags, and that needs the history.

Where it still misses: judgment

The hardest call in our scenarios: Immich v3 changed its Valkey healthcheck from redis-cli ping || exit 1 to redis-cli ping | grep -q PONG || exit 1. Our chart's probes run a plain valkey-cli ping. Is that a cosmetic change, or a fix we should mirror?

It's a fix. Against a server that answers with an error, a plain valkey-cli ping prints NOAUTH Authentication required. and still exits 0, so our probe passes on a broken server; the PONG form exits 1. We verified this against valkey:9.0-alpine.

04 ✗ before rules05 ✗ before rules 06 ✓13 ✓14 ✗ 15 ✓16 ✓17 ✓21 ✓23 ✗24 ✓

Two skill rules turned this around. First, the review table must show our current value next to upstream's old and new values. Second, calling a change cosmetic needs evidence; without it, mirror the change. Since then it's 5 of 6. The one miss (run 23) found the upstream commit, read the diff, and argued that "the trailing || exit 1 makes it always pass", which is backwards. Run 24, with the identical skill and scenario, mirrored the fix.

We accept this rather than chase it. The rule exists and the evidence was found; the miss is model variance. Chart-changing upgrades always arrive as draft PRs with a Needs human review section, so a miss costs one un-mirrored fix in a reviewed PR, not a broken deployment.

A softer weak spot is evidence quality. In run 22 the agent decided correctly that Nextcloud 33's new Snowflake server ID needs no setting for our single-replica chart, but it argued from the changelog. The configuration reference it should have read was one link away.

All runs

Skill length is in lines. Times from run 20 on are on a GPU shared with another agent's tests, so they're less comparable.

Run	Scenario	Skill	Time	Peak ctx	Grade	Main finding
01	Vaultwarden 1.35.0 → 1.37.3	416	19m	59k	very good	Missed the invite fix's link to our bootstrap Job
02	Nextcloud 32.0.6 → 33.x	418	killed	43k	n/a	Docker Hub paging; reverse-engineered docs from source
03	Nextcloud 32.0.6 → 33.0.9	424	37m	75k	very good	A wrong "one step" claim; 12 minutes trimming the changelog
04	Immich v2.7.5 → v3.2.1	425	37m	92k	failed	Grepped for `healthcheck`, found nothing, never compared the probes
05	same as 04	440	40m	95k	failed	Opened the file, called the PONG fix "cosmetic"
06	same as 04	446	33m	69k	pass	Mirrored the probe fix, bumped the chart, escalated x86-64-v2
07	same as 01 (regression)	446	20m	57k	pass+	Also fixed run 01's invite miss
08	same as 03, trimmed skill	429	29m	61k	good	Fixed run 03's issues; one unverified claim
09	full catalog, real master	431	29m	72k	good	Two rules conflicted on a skip; placeholder product skipped
10	same as 08, thinking not preserved	429	49m	81k	good	+50% turns, +70% thinking; 8.5-minute grep stall
11	full catalog, real master	433	21m	56k	mixed	A trim dropped a rule; one product produced no output
12	full catalog, real master	436	47m	102k	pass	All outputs complete; spotted leftover remote branches
13	Immich (latest skill)	436	33m	73k	pass	The chart-edit path survives the trim
14	same as 13, larger thinking budget	436	33m	80k	failed	PONG called "equivalent"; the budget wasn't the limit
15	same as 13 + "Ours" column + evidence rule	439	34m	79k	pass	Our value next to upstream's made the mismatch explicit
16	same as 15, xhigh effort	439	111m	154k	pass	Same quality at 2× context, 3× thinking; a 12-minute stall
17	same as 15, low effort	439	54m	94k	pass	Same answer, slower: low isn't cheaper
18	same as 01, with `result.json`	481	23m	57k	very good	Schema-valid `would_open`; ready tag-only PR
19	upstream image doesn't exist	481	7m	17k	pass	`skipped`; didn't guess a replacement image
20	already on the newest tag	481	8m	—	pass	`up_to_date`, nothing else written
21	Immich, final skill, blobless	481	51m	81k	pass	`would_skip` + draft; a 7-minute grep stall
22	Nextcloud, full clones	481	42m	76k	good	No stalls; Snowflake right, but argued from the changelog
23	Immich, full clones	481	33m	72k	failed	Fast and clean, but misread the PONG fix
24	same as 23, second sample	481	41m	93k	pass	Mirrored the PONG fix, bumped the chart: run 23's miss was variance

Harness lessons

Make the test bench match production

No container runtime: one run started Docker containers on the dev host to test a theory. Production has no Docker, so the harness now refuses it too.
Guards need negative tests: the "give git a path" guard accepted a bare --, and the agent took it literally.
Read-only GitHub through a gh shim, with a local fixture as origin, keeps dry runs away from the real repo.

Sharing one GPU between agents

One inference session at a time, across all agents, enforced with a shared flock. A second session thrashes the KV cache.
Private fixtures per harness: two agents sharing one clone reset each other's scenario mid-run.
The lock is the signal, not an agent going idle: the runs are detached processes.

Open proposals

Not applied yet. Skill changes go through the maintainer.

Placeholders in the result.json template. It shows a literal "dry_run": true and "pr_url": "<opened only>". A model copying it verbatim would get dry_run wrong in a real run, or write "", which fails the schema. Use placeholders, or have the runner set dry_run itself.
Demote pasted upstream headings. Release notes' ## What's Changed ends up under our ### release headings and reads as a top-level PR section (runs 01, 07, 18, 22).
A Nextcloud README pitfall: with a single replica the default Snowflake serverid fallback is fine; revisit if the chart ever scales out.
Pending README additions: Nextcloud's major-version doc URLs and the "latest patch first" pitfall, and Immich's migration-guide pointer. They're ready locally but not on master yet.

Settled

Decision	Setting	Evidence
Thinking effort	medium (server default)	Runs 15–17
Preserved thinking	on, no setting needed	Request captures + template render
Session scope	one product per fresh session	Runs 09–12
Upstream clones	full, skip repos over 1 GB	Runs 21–23 and a clone benchmark
Machine-readable outcome	`result.json`, schema-validated	Runs 18–24
Nightly cost	about 1.5–2 hours for the whole catalog	20–42 min per product

Freepod · github.com/erikvanzijst/freepod · Evaluation of products/UPGRADING/SKILL.md, 14–15 September 2026. All runs were dry runs, and nothing was pushed to GitHub.