thoughts (05)

Context

It started with a simple need: relaying notifications from n8n on Coolify to ntfy¹. The problem - I couldn’t figure out a sane way to save an n8n workflow to git (and didn’t feel like spending more time on it), so instead of fighting it any further, I wrote the thing myself, from scratch, in Zig².

That’s how ntfy.zig came to be - a small relay: it takes in webhooks (Coolify and GitHub Actions/Deployments, for now), turns them into a title/message/priority, and forwards them to ntfy.

Months of vibe-coding can basically be summed up as the standard disclaimer under any LLM client - “use it, but verify” - which is why it seemed like a good excuse to put some QA experience to use and run stress tests with Locust.

Setup

One stress/ directory in the repo: a fake ntfy endpoint simulating latency/failure rate/timeouts, Locust mixing traffic from both webhook channels (Coolify + GitHub, including a deliberately wrong HMAC signature - the secret GitHub uses to sign the webhook payload so the recipient can verify the request actually came from GitHub - to test 401 rejection), and a script tying it all together, all inside a single uv environment on free-threaded Python 3.14t. Full technical details, including why free-threaded actually makes sense here, are in PR #14.

First run - something’s off

At 20 users everything looked great: median 5-6ms. I cranked it up to 300 users to find the ceiling and got:

OSError(49, "Can't assign requested address")

Uh oh. OSError 49 means the ephemeral port pool is exhausted - and that happened before the CPU even broke a sweat.

Root cause and fix

The cause: every single forward to ntfy built a brand new HTTP client from scratch and closed it immediately after. Which means a fresh TCP connection (and for https, a fresh TLS handshake) for every single webhook. Zero connection pooling³ - hence the rapid port exhaustion.

The fix: one http.Client built once at startup and shared across all requests⁴, instead of creating a new one for every forward. TCP connections start getting reused instead of multiplying endlessly.

verification

150 users, ~550 req/s. lsof on the ntfy.zig process showed 11 persistent connections to the fake ntfy instead of a new one per request. Zero port-exhaustion errors.

Metric	Before	After
TCP connections at 150 VU / ~550 req/s	~1 per request	~11 persistent
Throughput (local)	~70 req/s (wait_time-bound)	~550 req/s

Conclusion

The stress test didn’t need to be clever to catch a real bug - it was enough to push the user count high enough to hit the port limit instead of the CPU limit. It’s a strange world, honestly, where you can write and load-test an application in a language you don’t actually know⁵.

By the way: the whole app sits at around 2MB of RAM at rest - so on top of working correctly, it’s also pretty easy on resources.

Footnotes

1. A nice, fairly minimalist self-hosted push notification server. ↩

2. That’s basically my approach to life, in a nutshell. In more ways than one. ↩

3. A mechanism where a connection, once opened, is kept alive and reused for subsequent requests instead of being closed and reopened every time. ↩

4. context7 often has stale docs for niche libraries like Zig’s fresh-off-the-press stdlib - so Sonnet 5, working on the new std.http, happily navigated straight through the library’s source code instead of relying on the (outdated) documentation. ↩

5. And apparently so does the model - except it, unlike me, has considerably more patience for reading library source code. ↩

I’ve recently returned to self-hosting — one of those periodic relapses where you get to re-solve old problems by testing different approaches to things you already “solved” once before¹. This time it was monitoring’s turn.

I was surprised that right after setting up OIDC I couldn’t add a single dashboard in Grafana — login worked, yet permissions were missing.

Setup

Grafana already sat behind Authelia (self-hosted SSO, single user, in the admins group) via native OIDC² (auth.generic_oauth). The task seemed trivial: everyone in the admins group should automatically get the GrafanaAdmin role in Grafana on login, instead of assigning it manually in the UI — which would get overwritten on every login anyway, since allow_sign_up: true re-provisions the user each time.

Grafana

The standard, documented path: groups scope on the OIDC client + role_attribute_path (a JMESPath³ expression) on Grafana’s side.

I provision Grafana via Ansible, using the grafana.grafana.grafana role. The key piece of config is the grafana_ini_oauth variable:

grafana_ini_oauth:
  "auth.generic_oauth":
    enabled: true
    name: Authelia
    client_id: grafana
    client_secret: "{{ grafana_authelia_oidc_secret | default('') }}"
    scopes: "openid profile email groups"
    auth_url: "https://authelia.example.com/api/oidc/authorization"
    token_url: "https://authelia.example.com/api/oidc/token"
    api_url: "https://authelia.example.com/api/oidc/userinfo"
    use_pkce: true
    allow_sign_up: true
    role_attribute_path: "contains(groups[*], 'admins') && 'GrafanaAdmin' || 'Viewer'"

The admin user is the only member of the admins group in Authelia — hence contains(groups[*], 'admins') in the expression above. A necessary condition for this to work: the groups scope must be added to Grafana’s OIDC client, just like in the config above.

What was supposed to happen

Grafana’s docs for role_attribute_path describe a three-tier fallback: first id_token, if nothing’s found there — userinfo (api_url), if still nothing — access_token. Sounds like solid redundancy.

What happened in practice

In Grafana 13.2.1 the fallback to userinfo is broken — confirmed by grafana/grafana#106686. If the claim isn’t in id_token, Grafana doesn’t check userinfo the way the docs claim it does.

Meanwhile, since version 4.39 Authelia stopped putting scope claims (like groups) into id_token by default — they only land in userinfo. That’s a sensible security tightening on its own (less PII in a token that ends up on various sites), but combined with Grafana’s broken fallback it produces a silent blackout: exactly the claim that matters for the role never arrives anywhere.⁴

The fix on Authelia’s side

identity_providers.oidc.claims_policies on Authelia’s side — explicitly forces specific claims back into id_token per-client, referenced from the client via claims_policy: <name>.

identity_providers:
  oidc:
    # ...
    claims_policies:
      grafana: # policy name, arbitrary, referenced below
        id_token:
          - 'groups' # force this claim into the id_token

    clients:
      - client_id: 'other'
        # no claims_policy — doesn't get this treatment

      - client_id: 'grafana'
        client_secret: '...'
        claims_policy: 'grafana' # attach the policy to the client
        redirect_uris:
          - 'https://grafana.example.com/login/generic_oauth'
        scopes:
          - 'openid'
          - 'profile'
          - 'email'
          - 'groups' # must be requested, otherwise there's nothing to force

With these settings, logging into Grafana as admin works again — another old problem checked off, solved anew and differently than last time.

And, having gained the ability to add a dashboard, did I add one?

Footnotes

1. RPi-Monitor, never forget ↩

2. OIDC (OpenID Connect) — a login (SSO) protocol built on top of OAuth2. ↩

3. JMESPath — a query language for extracting and transforming data from JSON documents, used here to pull groups out of the token claims. ↩

4. Who’s spent ages figuring out how to add a new dashboard, only to eventually realize they’re stuck with a Viewer role? Author of this post of course. ↩

Lately I’ve been working on my own vibe-designed todo app rewrite in Rust¹. Along the way I started looking for a sensible path to optimizing Docker image builds on GitHub Actions.

One of the solutions you can easily stumble upon is cargo-chef — it separates dependency fetching from building the actual code to better cache layers. The problem is it’s designed for simpler applications with a single lib.rs. In a workspace with more crates it doesn’t cache what it should, and every push ends up recompiling everything from scratch.

Two approaches I tried: buildkit cache dance² — speeds things up a bit, but doesn’t cache brilliantly and adds its own time overhead³ — and swatinem/rust-cache, which works nicely on GHA for binary caching, but deploying Kartoteka as a Docker image on Coolify rules that action out.

Research turned up sccache as worth trying — I made an attempt, but Hetzner S3 is a no-go⁴. So Cloudflare R2 on the free tier as a potential alternative⁵.

For now I’m parking at cache dance. Not ruling out going toward my own runner with local cache — just on a slightly bigger server.

Footnotes

1. Kartoteka — somewhere between Todoist, Jira, Amazing Marvin, and Toggl. Classic scope creep. ↩

2. Example in the Kartoteka repo. ↩

3. In Kartoteka’s case there’s also the exotic stack — Leptos — which means deploying the server binary together with the built frontend/wasm. ↩

4. Hetzner S3 bills storage in 1 TB blocks. A bit astronomical for one innocent backup and build artifacts. ↩

5. Yes, I know — suggesting I go back to Cloudflare when I’ve spent two weeks rewriting an MCP from a Cloudflare Worker to axum/rmcp specifically to get off the Cloudflare ecosystem, sure. ↩

It’s an old note, just for archival purposes. There might be newer alternatives. Also, old blog template used another method of referencing urls, don’t have it yet here, so leaving it as-is: link

I have a need to share how much I love Zotero. It was a tool that was a significant part of my academic journey and I had fallen in love while I was writing my B.Sc. thesis¹, and all coexisting add-ons were indirectly responsible for about 50 references mentioned. I’ve recently also incorporated Zotero into my workflow alongside another side-project on Notion.

Other remarkable features that Zotero offers:

• Awesome notes on existing PDFs. A bit better than hypothes.is² what was my main driver in B.Sc. era for taking notes from PDFs.³
• Collection mechanism - it’s a bit specific approach for zotero and needs time to be used to.
• Group collections that you can share your findings in teams.⁴

What add-ons were helpful?

Firefox extension

Zotero Connector for Firefox simplifies the process of adding new database entries. A single click can trigger metadata extraction, and in some cases, even full-text PDF retrieval.⁵

Better Bibtex

Better Bibtex allows exporting yours database in format compatible with Latex.

Only Latex?

If you need a place to take notes and mention/link your entries, Obsidian might be a good place to start. There is a great tutorial that was a great inspiration, available in this Zotero–Obsidian academic workflow guide.

There is a newer approach, however, that might work a bit better - Obsidian Zotero Plugin. Haven’t tested yet.

But what if I want to share my zotero-based entries with friends?

Sometime tool just doesn’t suit well, and forcing others to use obsidian might be imposing useless cognitive overload, so then Notion comes into play with notero extension. Works surprisingly smooth.

To wrap up, there are more efficient workflows than relying solely on a Word database or gathering references in a single designated file. And they may be integrated with your favorite applications

Footnotes

1. That’s the one tag which supposedly is a river and doesn’t match to other apps listed. ↩

2. Zotero has much more colors to highlight text. ↩

3. While PDF support in Zotero is superior, I was mainly taking notes from websites where hypothesis were better. Look in further sources section for great review of note-taking tools. ↩

4. Sharing is caring, or something like that. ↩

5. I was thinking if there may be an extension using other, less legal sources. Yep, they exist. ↩

Be aware that it is a n old post. Just reference or an inspiration if someone is lost.

Years ago I had a thought to seperate repo with hugo theme and the one with rendered page due to possible use of utterances¹. There are my experiences with both approaches:

deploying page to the same repo

• Astro docs provides a good starting point

deploying site elsewhere

• There is a a convinient github-action
• Can you reuse withastro/action@v0 to build it? You might, but it is designed for scenario above, this action creates an artifact that is consumed by actions/deploy-pages@v1 action.
  • tl;dr: there are better ways

Example of deploying astro blog to github pages

There is my example of working github action.

Things worth to note:

• By using actions to install node and pnpm there is huge time savings due to caching.
• There should be a .nojekyll file in blog root - otherwise css is messed up²

Footnotes

1. Is it a good idea? One of tutorials on integrating utterances with astro doesn’t have it either. I hope not to see it as a opinion whether this is good option, but someday I will give it a try. ↩

2. Source ↩