The problem: update fatigue
Every NixOS machine you manage needs its flake inputs updated, its configuration rebuilt, and the new generation activated. Do it manually and you either fall behind on security patches or spend your weekends SSH-ing into servers. Script it naively and you ship untested updates straight to production.
The ideal workflow:
- CI updates
flake.lockon a schedule and shows you exactly what changed. - You review and merge a pull request with per-host package diffs.
- Hosts self-upgrade from the merged commit — no manual intervention, no surprises.
This guide builds exactly that with a Forgejo Actions workflow and a small NixOS module.
Architecture
The system has two independent timers, staggered one hour apart:
| Component | Runs at | What it does |
|---|---|---|
| Forgejo Actions workflow | 04:00 UTC | Updates flake.lock, builds all hosts before and after, opens a PR with diffs |
| NixOS auto-upgrade timer | 05:00 (host-local) | Fetches main, builds own configuration, runs nixos-rebuild switch |
Hosts never modify flake.lock themselves. They always use --no-update-lock-file and build whatever version of nixpkgs (and other inputs) the CI committed to main. This keeps every machine on the same, reviewed set of inputs.
Step 1: The CI workflow
Create .forgejo/workflows/update.yaml in your NixOS flake repository. This workflow runs daily, updates flake inputs, and opens a pull request with per-host package diffs so you can review before merging.
name: Update Flake Inputs
on:
schedule:
- cron: "0 4 * * *" # Run daily at 04:00 UTC — one hour before hosts auto-upgrade.
workflow_dispatch: # Allow triggering manually from the Forgejo UI.
jobs:
update:
name: Update flake.lock and diff all hosts
runs-on: nixos-builder # A native NixOS runner — leverages the host Nix store as cache.
steps:
# --- 1. SSH setup ---
# Write the deploy key to a temp file so git can push branches
# and sign commits over SSH. The key is cleaned up in the final step.
- name: Configure SSH key for git push and commit signing
run: |
SSH_DIR="$RUNNER_TEMP/.ssh"
mkdir -p "$SSH_DIR"
echo "${{ secrets.GIT_PRIVATE_KEY }}" > "$SSH_DIR/forgejo_key"
chmod 600 "$SSH_DIR/forgejo_key"
# Point git at the deploy key for all SSH operations.
SSH_BIN="$(command -v ssh)"
export GIT_SSH_COMMAND="$SSH_BIN -i $SSH_DIR/forgejo_key -o StrictHostKeyChecking=no"
# Persist into subsequent steps via Forgejo environment file.
echo "GIT_SSH_COMMAND=$GIT_SSH_COMMAND" >> "$FORGEJO_ENV"
echo "SSH_KEY_PATH=$SSH_DIR/forgejo_key" >> "$FORGEJO_ENV"
- name: Checkout repository
uses: https://data.forgejo.org/actions/checkout@v6
# --- 2. Git identity ---
# Configure the CI bot identity and enable SSH commit signing.
# ⚠️ Replace remote URL, user.name, and user.email with your values.
- name: Configure git identity and SSH remote
run: |
git config user.name "ci"
git config user.email "ci@example.com"
git config gpg.format ssh
git config user.signingkey "$SSH_KEY_PATH"
git config commit.gpgsign true
git remote set-url origin git@your-forgejo:nix/nixos.git
# --- 3. Discover hosts ---
# Evaluate the flake to list all nixosConfigurations, filtering out
# any ending in "-minimal" (installer images, etc.).
# Adjust the regex to match your naming convention.
- name: Identify host configurations
id: hosts
run: |
HOSTS=$(nix eval .#nixosConfigurations \
--apply 'cs: builtins.filter
(n: builtins.match ".*-minimal" n == null)
(builtins.attrNames cs)' \
--json | nix run nixpkgs#jq -- -r '.[]' \
| tr '\n' ' ')
echo "Found hosts: $HOSTS"
echo "list=$HOSTS" >> "$FORGEJO_OUTPUT"
# --- 4. Snapshot: build BEFORE the update ---
# Build every host's system closure from the current flake.lock.
# The result symlinks (result-before-<host>) are kept for diffing later.
# `|| true` ensures a single failing host doesn't abort the whole run.
- name: Build current configurations (before update)
run: |
for host in ${{ steps.hosts.outputs.list }}; do
echo "::group::Building current $host"
nix build \
".#nixosConfigurations.$host.config.system.build.toplevel" \
-o "result-before-$host" || true
echo "::endgroup::"
done
# --- 5. Update flake.lock ---
# Pull the latest revisions of all flake inputs (nixpkgs, home-manager, etc.).
- name: Update flake inputs
run: nix flake update
# --- 6. Early exit if nothing changed ---
# If all inputs resolved to the same revisions, skip the rest.
- name: Check for changes
id: check
run: |
if git diff --quiet flake.lock; then
echo "changed=false" >> "$FORGEJO_OUTPUT"
echo "No flake.lock changes — nothing to do."
else
echo "changed=true" >> "$FORGEJO_OUTPUT"
fi
# --- 7. Snapshot: build AFTER the update & diff ---
# Rebuild every host with the updated lock file, then compare the
# before/after store paths using nvd. The diff shows exactly which
# packages were added, removed, or changed version — the equivalent
# of comparing two NixOS generations before either is deployed.
- name: Build updated configurations and generate diffs
id: diffs
if: steps.check.outputs.changed == 'true'
run: |
DIFF_REPORT=""
for host in ${{ steps.hosts.outputs.list }}; do
echo "::group::Building updated $host"
nix build \
".#nixosConfigurations.$host.config.system.build.toplevel" \
-o "result-after-$host" || true
echo "::endgroup::"
# Only diff if both builds succeeded.
if [ -e "result-before-$host" ] && \
[ -e "result-after-$host" ]; then
HOST_DIFF=$(nix run nixpkgs#nvd -- diff \
"result-before-$host" "result-after-$host" 2>&1 || true)
DIFF_REPORT="${DIFF_REPORT}### ${host}"$'\n'
DIFF_REPORT="${DIFF_REPORT}\`\`\`"$'\n'
DIFF_REPORT="${DIFF_REPORT}${HOST_DIFF}"$'\n'
DIFF_REPORT="${DIFF_REPORT}\`\`\`"$'\n\n'
fi
done
# Export the full report as a multi-line output variable.
{
echo "report<<DIFF_EOF"
printf '%s' "$DIFF_REPORT"
echo "DIFF_EOF"
} >> "$FORGEJO_OUTPUT"
# --- 8. Open a pull request ---
# Commit the updated flake.lock to a dated branch, then call the
# Forgejo API to create a PR whose body contains the per-host diffs.
# ⚠️ Replace the API URL with your Forgejo instance and repo path.
- name: Create branch, commit, and open pull request
if: steps.check.outputs.changed == 'true'
env:
DIFF_REPORT: ${{ steps.diffs.outputs.report }}
FORGEJO_TOKEN: ${{ forgejo.token }} # Provided automatically by Forgejo.
run: |
DATE=$(date +%Y-%m-%d)
BRANCH="auto/flake-update-$DATE"
# Remove a stale branch from a previous run on the same day.
git push origin --delete "$BRANCH" 2>/dev/null || true
git checkout -b "$BRANCH"
git add flake.lock
git commit -m "chore: update flake inputs $DATE"
git push origin "$BRANCH"
# Compose the PR body with the generation diff report.
BODY=$(printf '%s\n\n%s\n\n%s\n%s\n%s' \
"## Automated flake.lock update ($DATE)" \
"Package changes per host (via nvd):" \
"$DIFF_REPORT" \
"---" \
"*Auto-generated by the update workflow.*")
# Create the pull request via the Forgejo REST API.
nix run nixpkgs#curl -- -sf -X POST \
-H "Authorization: token $FORGEJO_TOKEN" \
-H "Content-Type: application/json" \
"https://your-forgejo/api/v1/repos/nix/nixos/pulls" \
-d "$(nix run nixpkgs#jq -- -n \
--arg title "chore: update flake inputs $DATE" \
--arg body "$BODY" \
--arg head "$BRANCH" \
--arg base "main" \
'{title: $title, body: $body, head: $head, base: $base}')"
# --- 9. Cleanup ---
# Always remove the SSH key and build symlinks, even on failure.
- name: Cleanup
if: always()
run: |
rm -f "$SSH_KEY_PATH"
rm -f result-before-* result-after-*Where it runs
The workflow runs on a NixOS host (runner label nixos-builder), not inside a container. This is important for two reasons:
- It needs a working Nix installation with direct access to
/nix/store. The host’s Nix store acts as a persistent build cache — derivations that haven’t changed since the last run are already in the store and don’t need to be rebuilt or downloaded again. - It builds full NixOS system configurations (
system.build.toplevel), which are large closures that benefit greatly from an existing store.
You can run this workflow in a container (e.g. a Docker-based Forgejo runner with Nix installed), but each run would start with a cold Nix store. That means every derivation is fetched or built from scratch, turning a job that takes minutes on a NixOS host into one that can take significantly longer. If you go the container route, mounting a persistent volume for /nix/store and /nix/var/nix/db helps, but a native NixOS runner remains the most efficient option.
If you already run a Forgejo runner on a NixOS machine, point this workflow at it. Otherwise, register a new runner on any NixOS host with the label nixos-builder.
Secrets and configuration
The workflow needs one secret and a few hardcoded values you must adapt to your setup.
Required secrets
| Secret | Purpose |
|---|---|
GIT_PRIVATE_KEY | SSH private key used for two things: pushing the update branch (git push) and signing the commit (gpg.format ssh). The key must have write access to the repository. Store it in Forgejo → Repository Settings → Secrets. |
The FORGEJO_TOKEN used to create the pull request via the API is provided automatically by Forgejo (${{ forgejo.token }}). No manual setup is required.
Values to customize
These are hardcoded in the YAML and must match your environment:
| Value | Line in workflow | What to set |
|---|---|---|
| Git remote URL | git remote set-url origin ... | Your Forgejo SSH URL, e.g. git@git.example.com:nix/nixos.git |
| API endpoint | https://your-forgejo/api/v1/repos/... | Your Forgejo instance URL and repository path |
| Git identity | user.name / user.email | The name and email for CI commits |
| Runner label | runs-on: nixos-builder | Must match the label of your registered NixOS runner |
Optional / tunable
| Field | Default | Notes |
|---|---|---|
cron: schedule | 0 4 * * * (04:00 UTC) | Adjust to any cron expression that suits your timezone or review habits |
workflow_dispatch | enabled | Allows manual triggering from the Forgejo UI — remove the key if not wanted |
| Host filter | excludes *-minimal hosts | The builtins.filter in the “Identify host configurations” step skips hosts matching .*-minimal — adjust the regex to your naming convention |
| PR base branch | main | Change if your default branch has a different name |
What happens on each run
- Build before — every host configuration is built from the current
flake.lockvianix build .#nixosConfigurations.<host>.config.system.build.topleveland stored as a symlinkresult-before-<host>pointing into the Nix store. - Update —
nix flake updatepulls the latest nixpkgs, home-manager, disko, and any other inputs. - Build after — the same hosts are rebuilt with the updated lock file and stored as
result-after-<host>. - Diff —
nvdcompares the two store paths per host and reports added, removed, and version-changed packages. This is the equivalent of comparing two NixOS generations before either of them is deployed. - PR — the updated
flake.lockis committed to anauto/flake-update-<date>branch and a pull request is opened with the full diff report embedded in the body.
The resulting pull request body contains a per-host section like this:
## Automated flake.lock update (2025-04-12)
Package changes per host (via nvd):
### webserver
[nvd output showing package upgrades, additions, and removals]
### devbox
[nvd output for this host]You review the PR, see exactly which packages changed on which host, and merge when ready. Nothing reaches your machines until you merge to main.
Tip: fully hands-off with auto-merge. If you trust the CI builds and don’t want to review every update manually, most Git forges (Forgejo, GitHub, GitLab) support auto-merging PRs once all required checks pass. Enable branch protection with a required status check for the build job, then configure auto-merge on the PR. The PR will merge itself as soon as CI is green — turning the entire pipeline into a zero-touch flow where hosts upgrade daily without any human interaction. You can still review the merged diff after the fact and roll back if needed.
Step 2: The auto-upgrade NixOS module
Create a module that wraps system.autoUpgrade so hosts pull from your Git repository on a schedule:
# modules/auto-upgrade.nix
{
config,
lib,
...
}:
let
hostname = config.networking.hostName;
in
{
options.services.auto-upgrade.enable =
lib.mkEnableOption "automatic daily flake-based NixOS upgrade";
config = lib.mkIf config.services.auto-upgrade.enable {
system.autoUpgrade = {
enable = true;
# Fetch the latest main branch from your Git server.
# Each host selects its own nixosConfiguration by hostname.
flake = "git+https://your-forgejo/nix/nixos.git#${hostname}";
# Never update flake.lock on the host — CI handles that.
flags = [
"--no-update-lock-file"
];
# Run daily at 05:00 (host-local time), one hour after CI.
dates = "05:00";
# Do not reboot automatically.
# Most services restart on nixos-rebuild switch.
allowReboot = false;
# "switch" activates immediately.
# Use "boot" to defer activation until next reboot.
operation = "switch";
};
};
}Key design decisions:
--no-update-lock-file— the most important flag. Hosts consume whateverflake.lockis onmain. They never resolve inputs independently, so every machine converges on the same package set.flake = "git+https://...#${hostname}"— each host selects its ownnixosConfigurationoutput by hostname. One repository, many machines.dates = "05:00"— staggered one hour after the CI runs at 04:00 UTC. This gives you a window to review the PR. If it is not merged yet, hosts simply rebuild from the currentmain(a no-op if nothing changed).allowReboot = false— most NixOS services restart onswitch. Set totrueif you need kernel or initrd updates to take effect immediately.
Step 3: Enable on each host
Include the module in your flake and enable it per host:
# flake.nix (simplified)
{
outputs = { nixpkgs, ... }: {
nixosConfigurations = {
webserver = nixpkgs.lib.nixosSystem {
modules = [
./modules/auto-upgrade.nix
./hosts/webserver
];
};
devbox = nixpkgs.lib.nixosSystem {
modules = [
./modules/auto-upgrade.nix
./hosts/devbox
];
};
};
};
}Then in each host’s configuration:
# hosts/webserver/default.nix
{
services.auto-upgrade.enable = true;
}For machines you deploy manually (like the CI builder itself, or test machines), simply omit the option or set it to false.
Step 4: Verify the setup
After deploying the configuration to your hosts, check that the systemd timer and service are in place:
# Check the timer schedule and when it last fired
$ systemctl status nixos-upgrade.timer
● nixos-upgrade.timer
Loaded: loaded
Active: active (waiting)
Trigger: tomorrow at 05:00
# Check the last upgrade run
$ journalctl -u nixos-upgrade.service -n 30The nixos-upgrade.service logs show the full nixos-rebuild switch output — which generation was activated, which services restarted, and any errors.
The full flow
Here is what happens every day without any manual intervention:
04:00 UTC Forgejo Actions runs on CI builder
├─ nix flake update
├─ Build all hosts before & after
├─ Generate per-host nvd diffs
└─ Open PR with diff report
You Review PR, check package changes, merge to main
05:00 Each NixOS host (systemd timer)
├─ git fetch main (via flake URL)
├─ nix build own configuration
└─ nixos-rebuild switchIf you don’t merge the PR before 05:00, hosts simply rebuild from the current main — effectively a no-op. The update waits until you merge.
Rollback
If a bad update slips through, NixOS makes rollback trivial:
# Roll back to the previous generation
$ sudo nixos-rebuild switch --rollback
# Or boot into a previous generation from the bootloaderEvery generation is kept in the Nix store until garbage-collected, so you can always go back.
Adapting the schedule
| What to change | Where | Default |
|---|---|---|
| CI update time | .forgejo/workflows/update.yaml → cron: | 0 4 * * * (04:00 UTC) |
| Host upgrade time | modules/auto-upgrade.nix → dates | 05:00 |
| Auto-reboot | modules/auto-upgrade.nix → allowReboot | false |
| Upgrade operation | modules/auto-upgrade.nix → operation | switch |
For desktops you might prefer operation = "boot" so the new configuration only activates on the next reboot, avoiding disruption during work hours. For servers, switch is usually the right choice since services restart gracefully.
Why this works well
- No unreviewed changes reach production. The PR gate means you always see what changed before it deploys.
- Hosts never drift from each other. Every machine builds from the same
flake.lockonmain. - Zero manual SSH sessions. Once the module is enabled, upgrades are fully automatic.
- Safe by default. If CI fails to build a host, the PR shows the error. If a host fails to build, it stays on its current generation. If you merge something bad,
nixos-rebuild switch --rollbackfixes it in seconds. - Works for any fleet size. Whether you run two machines or twenty, the same workflow scales — one PR, one merge, all hosts converge.