RunBSD

← Minecraft Backups

Daily Pipeline-Health Digest

8 August 2026 (updated 18 August 2026)

What this does

Closes the last open item from the Minecraft Backups page's "Needed improvements" list: confirming a clean night used to mean manually checking logs across four separate machines (Mac mini, gmktec, seneca, and batcave). daily-digest-fabric.sh runs once a night, pulls today's relevant log lines from all four, and emails one combined summary via Mail.app/AppleScript — the same alerting mechanism backup-fabric.sh already uses.

Scheduled via a LaunchAgent (com.david.daily-digest.plist) at 5:00 AM — late enough that Shockbyte's download (2AM), check-version.sh → backup-fabric.sh (immediately after), seneca's tape backup (3AM), and gmktec's tarsnap job have all had time to finish for the night.

As of 9 August 2026, this lives in its own dedicated project — the digest script, its LaunchAgent, and its logs are no longer tracked as a small addition alongside the Mac backup script's own files. It's grown enough since the 8 August build (see "Summary line" below) to stand on its own, the same way each other branch on the Minecraft Backups page already does.

Added 18 August 2026: the Fabric/Vanilla split. The LaunchAgent now points at daily-digest-fabric.sh (renamed from daily-digest.sh), which checks the fabric-specific log paths across all four hosts — tarsnap-create-fabric.log, tarsnap-rotate-fabric.log, forward-fabric-to-batcave.log, zfs-snapshot-prune-fabric.log, zfs-tape-backup-fabric.log, and backup-fabric.log. There's no Vanilla-side digest coverage yet — this script only reports on the Fabric branch of the pipeline.

Summary line (added 9 August 2026)

Before the per-host detail below, the digest opens with a compact === Summary === block: one [PASS]/[FAIL] line per branch — gmktec's tarsnap archive creation, gmktec's tarsnap-rotate, gmktec's forward-to-batcave, batcave's zfs-snapshot-prune, seneca's tape backup, and the Mac's own backup run. Each verdict is grounded in that branch's own logging conventions (a known "reached-completion" marker, ERROR:/WARNING: line counts, and so on) rather than just checking whether a log line exists for today — see each check's own inline comment in daily-digest-fabric.sh for the exact marker it looks for. This turns "read six logs and eyeball them" into "check one line per branch, and only dig into the detail section below when something says [FAIL]."

As of 18 August 2026, the gmktec tarsnap-create check is new — that job has no custom logging of its own, so it's verified by looking for today's date in tarsnap-create-fabric.log (the dedicated script's own log — see the Tarsnap Backup page) rather than by parsing tarsnap's CLI output directly, since dhw can't read tarsnap's root-only key file to run tarsnap --list-archives itself.

What it pulls, and from where

The detail section below the summary pulls the fabric-specific logs across all four hosts:

Each remote log is checked for existence before being grepped (ssh host "[ -f /path/to/log ]", with output silenced via >/dev/null 2>&1 so a routine "file doesn't exist yet" check doesn't print SSH connection noise) — a missing log just gets logged as "... log is missing" and the script moves on to the next one, instead of dying on the first host that's ever offline or hasn't produced a log file yet.

Authentication

Every SSH call uses the same dedicated, passphrase-less key backup-fabric.sh uses for gmktec and seneca as of 8 August 2026 (~/.ssh/backup_automation) — not my general-purpose personal key, and no ssh-agent/Keychain involved at all. See the backup.sh & DVD Archiving page's "SSH authentication" section for the full reasoning (narrower blast radius on this particular Mac, and no reboot-loses-the-loaded-key failure mode to self-heal around, since there's no agent in the loop to lose anything).

Resolved 18 August 2026 (was an open item): this same key is now also authorized directly on batcave, added and verified 8 August 2026 per the script's own comment — so batcave's log is read with a direct SSH call, not the nested gmktec-hop this page previously described. That nested-hop architecture (and the batcave_key name it referenced) is no longer accurate and has been removed from this page.

Alerting

mail_digest() mirrors backup-fabric.sh's mail_alert() pattern — AppleScript driving the already-signed-in Mail.app account, since this Mac's LaunchAgent runs inside the GUI session and there's no MTA installed here. A few bugs were caught and fixed while building this:

As of 9 August 2026: the working log file is dated (daily-digest-fabric-YYYY-MM-DD.log) and kept permanently as a running history in the logs folder, rather than being deleted after a successful send. A prune step at the end of the script keeps the most recent 14 dated logs (DIGEST_KEEP=14) and removes older ones, running after mailing so it never touches today's own file.

Code added 18 August 2026: two independent failure-notification channels, on top of Mail.app. If mail_digest() itself fails (Mail.app signed out, a revoked AppleScript/Automation permission, a macOS update, etc.), relying solely on Mail.app to report that failure is circular — so two more channels were written into the script, each a completely different delivery path so whatever breaks Mail.app is unlikely to break these too.

Both were briefly missing from daily-digest-fabric.sh (the fabric branch was cut from an older baseline before these existed) and were restored during the 18 August 2026 drift check — confirmed unintentional on my part, not a deliberate removal.

LaunchAgent

LaunchAgent gotcha worth remembering (found 18 August 2026): editing a plist's ProgramArguments doesn't take effect just because the file on disk changed — launchd keeps running whatever was loaded into memory at bootstrap time until you explicitly reload it (launchctl bootout + bootstrap, or kickstart -k after a reload) or reboot. I hit this directly: after pointing the plist at daily-digest-fabric.sh, manually running the script produced a correct digest, but kicking the LaunchAgent kept producing the old digest — because launchd was still running the old, in-memory daily-digest.sh reference from before the edit. A reboot picked up the change. Worth remembering for any future plist edit on this Mac, not just this script.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.david.daily-digest</string>

    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/caffeinate</string>
        <string>-i</string>
        <string>/Users/david/scripts/shockbyte_dl/daily-digest-fabric.sh</string>
    </array>

    <key>WorkingDirectory</key>
    <string>/Users/david/scripts/shockbyte_dl</string>

    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>5</integer>
        <key>Minute</key>
        <integer>0</integer>
    </dict>

    <key>RunAtLoad</key>
    <false/>

    <key>StandardOutPath</key>
    <string>/Users/david/scripts/shockbyte_dl/logs/stdout.log</string>

    <key>StandardErrorPath</key>
    <string>/Users/david/scripts/shockbyte_dl/logs/stderr.log</string>

    <key>ProcessType</key>
    <string>Interactive</string>
</dict>
</plist>

caffeinate -i keeps the Mac from idle-sleeping mid-run, same reasoning as backup-fabric.sh's own LaunchAgent — this script needs network reachability to four hosts. ProcessType: Interactive matches the GUI-session requirement for the Mail.app-driven alerting. The script needs its executable bit set (chmod +x daily-digest-fabric.sh), since caffeinate execs it directly rather than through an interpreter.

daily-digest-fabric.sh

#!/bin/zsh

# /Users/david/scripts/shockbyte_dl/daily-digest-fabric.sh

# Daily digest notification email script. This script attempts to
# aggregate various log files from the different machines involved
# in the separate branches of the backup scheme, then email them
# as a daily digest of the Minecraft backup process.
#
# As of 2026-08-09: leads with a compact PASS/FAIL summary line per
# branch, followed by the same full raw log detail as before for anyone
# who wants to dig in -- the summary augments the email, it doesn't
# replace anything. Each check below is grounded in that script's own
# actual logging conventions (see the comment above each one), except
# gmktec's tarsnap archive-creation cron job, which has no custom
# logging of its own at all -- that one is verified by checking
# tarsnap-rotate.sh's own archive listing (it already runs `tarsnap
# --list-archives` as root every night) for a matching archive, rather
# than by parsing tarsnap's own CLI output or trying to run tarsnap
# directly as dhw (confirmed 2026-08-09 that dhw can't read tarsnap's
# root-only key file).
#
# Also as of 2026-08-09: the working log file is named per-day
# (daily-digest-YYYY-MM-DD.log) and kept permanently in the logs folder
# as a running history, rather than being deleted after a successful
# send (and kept only on failure, as a scratch file for that day's
# investigation) the way it worked before.

# Dated so every day's digest is kept as a permanent record rather than
# being overwritten or deleted -- see the module comment above.

TODAY_ISO=$(date -I)
TODAY_YMD=$(date +%Y%m%d)
LOGFILE="/Users/david/scripts/shockbyte_dl/logs/daily-digest-fabric-$TODAY_ISO.log"

# Somehow the first $1 prints "The date in latest-version.log matches today's date. Continuing with backup."
log() { echo "$1" >> "$LOGFILE"; }

DIGEST_EMAIL="david@gromzen.com"

# Independent notification channel for when mail_digest() itself fails
# (Mail.app signed out, a revoked AppleScript/Automation permission, a
# macOS update, etc.) -- a completely different app/account/delivery path
# so whatever breaks Mail.app is very unlikely to also break this.
# PLACEHOLDER: "minecraft-backup-alerts" is a stand-in topic name. ntfy.sh
# topics are unauthenticated and global, so David needs to pick/create
# his own actual topic (and subscribe to it in the ntfy app/site) before
# this does anything useful -- see mail_digest()'s failure branch below.
TOPIC="minecraft-backup-alerts"

# External dead-man's-switch heartbeat (Healthchecks.io): pinged at
# the very end of the script, gated on mail_digest succeeding, so a missed
# ping means either the digest didn't run at all or Mail.app failed --
# and the alert comes from Healthchecks.io's own infrastructure, not
# this Mac. LIVE as of 2026-08-23: real check created at healthchecks.io,
# ping URL below is no longer a placeholder.
HEALTHCHECK_URL="https://hc-ping.com/61471105-fc84-4c2c-9713-3d0770ec0906"

# Local (site-internal) dead-man's-switch heartbeat, complementing the
# Healthchecks.io one above rather than replacing it: overwrites a
# timestamp file on batcave over the same SSH_KEY already used for the
# log-pull calls below. Covers the same "digest never ran / Mail.app
# broke" failures via a second, independent delivery path -- one that
# still works even if this Mac's internet is down but the LAN (and
# batcave) is still up, which Healthchecks.io can't reach in that case.
# Watched by mac-heartbeat-alert.sh on batcave (Minecraft Backup
# Instance Processing project) -- see that project's
# improvements-to-be-implemented.md for the deploy steps.
HEARTBEAT_HOST="batcave"
HEARTBEAT_PATH="/home/dhw/mac-heartbeat/last-success"

# Same dedicated, passphrase-less key backup.sh uses for gmktec/seneca as
# of 2026-08-08 -- not David's general-purpose personal key, and no
# ssh-agent/Keychain involved at all. See backup.sh's own comment above
# its SSH_KEY for the full reasoning (narrower blast radius on this Mac
# specifically, and sidesteps the reboot-loses-the-loaded-key failure
# mode entirely rather than self-healing around it). Also authorized
# directly on batcave (added and verified 2026-08-08), not just gmktec
# and seneca.
SSH_KEY="/Users/david/.ssh/backup_automation"

SUMMARY=""
add_summary() { SUMMARY="${SUMMARY}${1}"$'\n'; }

# This Mac mini's LaunchAgent runs in the GUI session (per the
# SSH_AUTH_SOCK note in the README), so we can drive the already-signed-in
# Mail.app account via AppleScript.
mail_digest() {
    local subject
    subject="Daily Backup Digest"
    local full_body
    full_body=$(cat "$LOGFILE")

    local as_subject="${subject//\\/\\\\}"
    as_subject="${as_subject//\"/\\\"}"
    local as_body="${full_body//\\/\\\\}"
    as_body="${as_body//\"/\\\"}"

    local osascript_output
    osascript_output=$(osascript <<APPLESCRIPT 2>&1
tell application "Mail"
    set newMessage to make new outgoing message with properties {subject:"backup-digest: ${as_subject}", content:"${as_body}", visible:false}
    tell newMessage
        make new to recipient at end of to recipients with properties {address:"${DIGEST_EMAIL}"}
        send
    end tell
end tell
APPLESCRIPT
)
    if [ $? -ne 0 ]; then
        log "WARNING: mail_digest failed to send via Mail.app (subject: $subject) -- osascript output: $osascript_output"
        # Second, independent notification channel -- doesn't depend on
        # Mail.app/AppleScript at all, so it still gets through when that's
        # the thing that's broken. See TOPIC placeholder note near the top.
        curl -s -d "Minecraft backup digest FAILED to send via Mail.app on $(date -I): $osascript_output" "ntfy.sh/${TOPIC}" >/dev/null 2>&1
        return 1
    fi
}

# ---------------------------------------------------------------------
# Fetch everything first (each into a variable), then derive PASS/FAIL
# for the summary, then log summary + full detail in that order at the
# very end. Fetching once and reusing the result avoids a second round
# of SSH calls just to re-fetch the same lines for the detail section.
# ---------------------------------------------------------------------

# --- gmktec: tarsnap archive creation ---
gmktec_create_log=""
if ssh -i "$SSH_KEY" dhw@gmktec "[ -f /var/log/tarsnap-create-fabric.log ]" >/dev/null 2>&1; then
    gmktec_create_log=$(ssh -i "$SSH_KEY" dhw@gmktec sed -n "/$TODAY_ISO/,/$TODAY_ISO/p" /var/log/tarsnap-create-fabric.log)
fi
if [ -z "$gmktec_create_log" ]; then
    add_summary "[FAIL] gmktec tarsnap-create-fabric -- no log entries found for today"
#else echo "$gmktec_create_log" | grep "$TODAY_ISO"
else
    add_summary "[PASS] gmktec tarsnap-create-fabric"
fi

# --- gmktec: tarsnap-rotate.log ---
gmktec_rotate_log=""
if ssh -i "$SSH_KEY" dhw@gmktec "[ -f /var/log/tarsnap-rotate-fabric.log ]" >/dev/null 2>&1; then
    gmktec_rotate_log=$(ssh -i "$SSH_KEY" dhw@gmktec grep "$TODAY_ISO" /var/log/tarsnap-rotate-fabric.log)
fi
if [ -z "$gmktec_rotate_log" ]; then
    add_summary "[FAIL] gmktec tarsnap-rotate-fabric -- no log entries found for today"
elif echo "$gmktec_rotate_log" | grep -q "finished (mode=commit)\|nothing to do"; then
    err_count=$(echo "$gmktec_rotate_log" | grep -c "ERROR:")
    if [ "$err_count" -gt 0 ]; then
        add_summary "[FAIL] gmktec tarsnap-rotate-fabric -- completed but $err_count archive deletion(s) failed, see detail below"
    else
        add_summary "[PASS] gmktec tarsnap-rotate-fabric"
    fi
else
    add_summary "[FAIL] gmktec tarsnap-rotate-fabric -- did not reach completion, see detail below"
fi

# --- gmktec: forward-to-batcave.log ---
# Success marker: "forward-to-batcave complete." Any real failure logs
# an "ERROR:" line and exits before reaching that.
gmktec_forward_log=""
if ssh -i "$SSH_KEY" dhw@gmktec "[ -f /home/dhw/forward-to-batcave/forward-fabric-to-batcave.log ]" >/dev/null 2>&1; then
    gmktec_forward_log=$(ssh -i "$SSH_KEY" dhw@gmktec grep "$TODAY_ISO" /home/dhw/forward-to-batcave/forward-fabric-to-batcave.log)
fi
if [ -z "$gmktec_forward_log" ]; then
    add_summary "[FAIL] gmktec forward-fabric-to-batcave -- no log entries found for today"
elif echo "$gmktec_forward_log" | grep -q "forward-fabric-to-batcave complete"; then
    add_summary "[PASS] gmktec forward-fabric-to-batcave"
else
    add_summary "[FAIL] gmktec forward-fabric-to-batcave -- did not complete, see detail below"
fi

# --- batcave: zfs-snapshot-prune-fabric.log ---
# Success marker: "Snapshot + prune complete for ..." (or "already
# exists" if triggered twice in one day -- also a pass). Snapshots that
# could not be destroyed during the prune are logged as "WARNING:" lines
# and don't fail the run, but are surfaced on the summary line.
batcave_log=""
if ssh -i "$SSH_KEY" dhw@batcave "[ -f /home/dhw/zfs-snapshot-prune-fabric/zfs-snapshot-prune-fabric.log ]" >/dev/null 2>&1; then
    # NOT a plain grep for $TODAY_ISO: zfs writes its own errors to stderr
    # WITHOUT a timestamp (e.g. "cannot destroy snapshots: permission denied"),
    # so a date-match filter silently drops exactly the line that explains a
    # WARNING. This awk instead tracks which day the most recent *timestamped*
    # line belongs to, and carries that state onto the untimestamped lines that
    # follow it -- so today's raw zfs errors come through and other days' don't.
    batcave_log=$(ssh -i "$SSH_KEY" dhw@batcave "awk -v d='$TODAY_ISO' '/^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] /{cur=(\$1==d)} cur' /home/dhw/zfs-snapshot-prune-fabric/zfs-snapshot-prune-fabric.log")
fi
if [ -z "$batcave_log" ]; then
    add_summary "[FAIL] batcave zfs-snapshot-prune-fabric -- no log entries found for today"
elif echo "$batcave_log" | grep -q "Snapshot + prune complete\|already exists"; then
    warn_count=$(echo "$batcave_log" | grep -c "WARNING:")
    if [ "$warn_count" -gt 0 ]; then
        add_summary "[PASS] batcave zfs-snapshot-prune-fabric (with $warn_count warning(s), see detail below)"
    else
        add_summary "[PASS] batcave zfs-snapshot-prune-fabric"
    fi
else
    add_summary "[FAIL] batcave zfs-snapshot-prune-fabric -- did not complete, see detail below"
fi

# --- seneca: zfs-tape-backup.log ---
# Success marker: "Backup complete. State updated: ..." (or "already
# backed up today" if this ran twice in one day -- also a pass). Hard
# failures are all prefixed "ERROR:"; capacity warnings and prune
# failures are prefixed "WARNING:" but don't fail the run.
seneca_log=""
if ssh -i "$SSH_KEY" dhw@seneca "[ -f /var/log/zfs-tape-backup-fabric.log ]" >/dev/null 2>&1; then
    seneca_log=$(ssh -i "$SSH_KEY" dhw@seneca grep "$TODAY_ISO" /var/log/zfs-tape-backup-fabric.log)
fi
if [ -z "$seneca_log" ]; then
    add_summary "[FAIL] seneca tape backup fabric -- no log entries found for today"
elif echo "$seneca_log" | grep -q "Backup complete\|already backed up today"; then
    warn_count=$(echo "$seneca_log" | grep -c "WARNING:")
    if [ "$warn_count" -gt 0 ]; then
        add_summary "[PASS] seneca tape backup fabric (with $warn_count warning(s), see detail below)"
    else
        add_summary "[PASS] seneca tape backup fabric"
    fi
else
    add_summary "[FAIL] seneca tape backup fabric -- did not reach completion, see detail below"
fi

# --- Mac: backup-fabric.log ---
# Success marker: "all finished!" at the very end of a run. ERROR: lines
# can appear even on a run that still reaches "all finished!" (most
# failure branches in backup-fabric.sh log and continue rather than aborting),
# so those are checked independently and treated as a FAIL regardless. A
# skipped DVD burn ("WARNING: Burn step skipped due to failed pre-check(s)")
# is also treated as a FAIL rather than a warning-only PASS, even though
# backup-fabric.sh logs it with a WARNING: prefix like other non-fatal
# warnings.
mac_log=""
if [ -f "/Users/david/scripts/shockbyte_dl/logs/backup-fabric.log" ]; then
    mac_log=$(grep "$TODAY_ISO" /Users/david/scripts/shockbyte_dl/logs/backup-fabric.log)
fi
if [ -z "$mac_log" ]; then
    add_summary "[FAIL] Mac backup-fabric.sh -- no log entries found for today"
elif ! echo "$mac_log" | grep -q "all finished!"; then
    add_summary "[FAIL] Mac backup-fabric.sh -- did not reach completion, see detail below"
else
    err_count=$(echo "$mac_log" | grep -c "ERROR:")
    burn_fail_count=$(echo "$mac_log" | grep -c "WARNING: Burn step skipped due to failed pre-check(s)")
    warn_count=$(echo "$mac_log" | grep -c "WARNING:")
    if [ "$err_count" -gt 0 ]; then
        add_summary "[FAIL] Mac backup-fabric.sh -- completed with $err_count ERROR line(s), see detail below"
    elif [ "$burn_fail_count" -gt 0 ]; then
        add_summary "[FAIL] Mac backup-fabric.sh -- DVD burn failed (skipped due to failed pre-check(s)), see detail below"
    elif [ "$warn_count" -gt 0 ]; then
        add_summary "[PASS] Mac backup-fabric.sh (with $warn_count warning(s), see detail below)"
    else
        add_summary "[PASS] Mac backup-fabric.sh"
    fi
fi

# ---------------------------------------------------------------------
# Log summary first, then the same full detail as before.
# ---------------------------------------------------------------------
log "=== Summary ==="
log ""
log "$SUMMARY"
log "=== Detail ==="
log ""

log "gmktec's tarsnap create fabric log:"
log ""
if [ -n "$gmktec_create_log" ]; then
    log "$gmktec_create_log"
else
    log "gmktec's tarsnap create fabric log is missing (/var/log/tarsnap-create-fabric.log not found on gmktec)"
fi
log ""

log "gmktec's tarsnap rotate fabric log:"
log ""
if [ -n "$gmktec_rotate_log" ]; then
    log "$gmktec_rotate_log"
else
    log "gmktec's tarsnap rotate log is missing (/var/log/tarsnap-rotate-fabric.log not found on gmktec)"
fi
log ""

log "gmktec's forward fabric to batcave log:"
log ""
if [ -n "$gmktec_forward_log" ]; then
    log "$gmktec_forward_log"
else
    log "gmktec's forward-to-batcave log is missing (/home/dhw/forward-to-batcave/forward-fabric-to-batcave.log not found on gmktec)"
fi
log ""

log "batcave's zfs snapshot and prune fabric log:"
log ""
if [ -n "$batcave_log" ]; then
    log "$batcave_log"
else
    log "batcave's zfs snapshot and prune fabric log is missing (/home/dhw/zfs-snapshot-prune-fabric/zfs-snapshot-prune-fabric.log not found on batcave)"
fi
log ""

log "seneca's tape backup log:"
log ""
if [ -n "$seneca_log" ]; then
    log "$seneca_log"
else
    log "seneca's tape backup log is missing (/var/log/zfs-tape-backup-fabric.log not found on seneca)"
fi
log ""

log "Mac's backup fabric log:"
log ""
if [ -n "$mac_log" ]; then
    log "$mac_log"
else
    log "Mac's backup fabric log is missing (/Users/david/scripts/shockbyte_dl/logs/backup-fabric.log not found on David's MacBook Air)"
fi
log ""

# Mail the daily-digest. The dated log file is always left in place
# afterward now (see LOGFILE above) -- a send failure still gets its own
# WARNING line logged by mail_digest() itself above, so that's preserved
# in the permanent record too rather than only being visible on the one
# day it happened.
if mail_digest; then
    # External dead-man's-switch heartbeat -- only pinged when
    # mail_digest actually succeeded, not unconditionally. A missed ping
    # means either the digest never ran at all or Mail.app failed, and
    # Healthchecks.io's own infrastructure (not this Mac) is what alerts
    # David when the expected check-in doesn't arrive.
    curl -fsS -m 10 --retry 3 "$HEALTHCHECK_URL" >/dev/null 2>&1

    # Local heartbeat, same gating as the Healthchecks.io ping above --
    # see HEARTBEAT_HOST/HEARTBEAT_PATH note near the top. A failure here
    # is logged but deliberately doesn't fail the whole run: batcave
    # being briefly unreachable shouldn't affect whether tonight's
    # digest counts as sent (the Healthchecks.io ping above already
    # covers that independently).
    if ! ssh -i "$SSH_KEY" -o ConnectTimeout=10 "dhw@${HEARTBEAT_HOST}" \
        "mkdir -p \$(dirname ${HEARTBEAT_PATH}) && date -u +%Y-%m-%dT%H:%M:%SZ > ${HEARTBEAT_PATH}" >/dev/null 2>&1; then
        log "WARNING: local heartbeat push to ${HEARTBEAT_HOST} failed"
    fi
fi

# ---------------------------------------------------------------------
# Prune old digest logs, keeping the most recent DIGEST_KEEP (currently
# 14, ~2 weeks). Runs after mailing today's digest, so it never touches
# today's own file (always the newest) and can't affect what got sent.
# Filenames sort correctly as plain strings since they're YYYY-MM-DD, so
# a reverse sort + skip-the-first-N is enough -- no date parsing needed.
# ---------------------------------------------------------------------
DIGEST_LOG_DIR="/Users/david/scripts/shockbyte_dl/logs"
DIGEST_KEEP=14

old_digests=$(ls -1 "$DIGEST_LOG_DIR"/daily-digest-fabric-*.log 2>/dev/null | sort -r | tail -n +$((DIGEST_KEEP + 1)))
if [ -n "$old_digests" ]; then
    echo "$old_digests" | while IFS= read -r old_log; do
        rm -f "$old_log"
    done
fi

Status

Added and debugged 8 August 2026. The [PASS]/[FAIL] summary line (see "Summary line" above) was added 9 August 2026. As of 18 August 2026, the script was renamed to daily-digest-fabric.sh as part of the pipeline-wide Fabric/Vanilla split, and picked up code for two new failure-notification channels (ntfy.sh, Healthchecks.io — see "Alerting" above) plus a new tarsnap-create-fabric check. Healthchecks.io went live 23 August 2026 (real check + ping URL in place); ntfy.sh is still not live — it still needs a real TOPIC value in place of its placeholder (see "Open items" below) before it actually notifies anyone. Local copies of both the script and its plist also have the DVD-reminder-equivalent fixes for this branch (ntfy/healthcheck code restoration, caffeinate -i wrapper) applied; confirm both are actually deployed to the Mac before counting on any of this. Once proven stable across more nights post-split, this still closes item 4 ("One daily pipeline-health digest") on the Minecraft Backups page's "Needed improvements" list.

Open items / possible follow-ups