RunBSD

Shockbyte Backup Automation

15 July 2026

What this does

Every night at 2AM: logs into the Shockbyte panel, downloads the latest JJAC_Survival_World backup, unzips it, prunes never-visited chunks, rsyncs the trimmed world to gmktec, and archives copies of the raw zip to a few dated locations — all unattended, even with the screen locked. No manual login needed under normal circumstances.

The problem: Cloudflare

panel.shockbyte.com and auth.shockbyte.com sit behind Cloudflare bot protection. AppleScript driving Safari (JS clicks, keystrokes, accessibility-tree clicks, coordinate clicks) was reliably blocked — every approach, every time.

Playwright driving a real, visible Chrome window (headless=False) gets through fine. Headless mode does not. Don't switch this back to headless — it will just start failing again.

Files

shockbyte_download_backup.py    nightly downloader
shockbyte_auth.py                shared Keychain + login-flow logic
shockbyte_login_setup_auto.py   standalone forced session refresh
shockbyte_login_setup.py        manual fallback login (by hand)
backup.sh                        unzip, dechunk, rsync, archive
com.david.shockbyte-backup.plist    launchd schedule

One-time setup

python3 -m venv myvenv
source myvenv/bin/activate
pip3 install playwright psutil
playwright install chromium

psutil is used by shockbyte_download_backup.py to force-kill a stalled Chrome instance if a download hangs and the browser's own close() doesn't respond (see kill_marked_chrome() below). Optional but strongly recommended — without it, a stalled download just logs a warning and leaves the hung Chrome process running.

Store credentials in Keychain — not Passwords.app!

Keychain Access.app → File → New Password Item:

Name:     Shockbyte
Account:  <login email>
Password: <login password>

Note to self: Keychain Access.app and the newer Passwords.app store different item types. Only Keychain Access's "generic password" is readable via security find-generic-password — a Safari-saved login under the same name won't be found by the script.

The plaintext SHOCKBYTE_USERNAME constant (not secret, only the password comes from Keychain) now lives in shockbyte_auth.py, not shockbyte_login_setup_auto.py — that's the shared module both login scripts import, so it's the one to edit.

Get an initial session

python3 shockbyte_login_setup_auto.py

Approve the Keychain permission prompt with "Always Allow" so future unattended runs don't hang waiting on it.

Login form quirks (reverse-engineered by hand)

Set up the LaunchAgent

mkdir -p ~/scripts/shockbyte_dl/logs
cp com.david.shockbyte-backup.plist ~/Library/LaunchAgents/
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.david.shockbyte-backup.plist
launchctl enable gui/$(id -u)/com.david.shockbyte-backup
launchctl kickstart -k gui/$(id -u)/com.david.shockbyte-backup

Key detail: gui/<uid>, not system — runs inside the actual logged-in GUI session, which keeps working even with the screen locked (confirmed by testing). Sleep still kills it though — disable sleep in Energy settings.

Note to self: a bootstrap failure ("5: Input/output error") usually just means the service is already loaded from an earlier attempt — check with launchctl print gui/$(id -u)/com.david.shockbyte-backup before assuming something's actually broken.

The schedule

<key>StartCalendarInterval</key>
<dict>
    <key>Hour</key><integer>2</integer>
    <key>Minute</key><integer>0</integer>
</dict>

Keeping the session alive

Fully automatic now, no calendar reminder needed. Every run of shockbyte_download_backup.py:

  1. Checks the real expiry of the saved session (the KEYCLOAK_SESSION cookie) before doing anything else.
  2. If there's no saved session at all, or it's within 48 hours of expiring, proactively logs back in via shockbyte_auth.py's login_and_refresh_state() (Keychain-based) before attempting the download — reusing its own already-launched browser for that login rather than spawning a second browser process.
  3. If it still gets bounced to the login page during the actual download (session expired despite the above, or expired between the proactive check and the download itself), it runs the same refresh once more and retries the download a single time before giving up.
  4. As a safety net independent of all that, it also logs a loud warning once you're within 3 days of expiry, in case the automatic refresh itself ever silently stops working.

If the automated login itself ever breaks (e.g. Cloudflare starts blocking it again, or the Shockbyte login form changes), the script exits with code 1 and tells you to run shockbyte_login_setup.py by hand as a fallback.

Handling stalled downloads

Failure mode this closes off: Playwright's Download.save_as() (and .path()/.failure()) has no timeout of its own — per the docs, it "will wait for the download to finish if necessary," i.e. forever. A download that starts fine but then stalls mid-transfer (flaky network, Chrome deciding to pause it) used to hang the script indefinitely, leaving a stalled Chrome window sitting open and "waiting to resume" for hours, unnoticed.

Two layers now guard against this in shockbyte_download_backup.py:

  1. Per-download stall timeout. Once a download is confirmed, save_as() runs on a side thread while the main thread waits up to DOWNLOAD_STALL_TIMEOUT_SECONDS (currently 600s / 10 min). If it's still running after that, the download is cancelled and treated as "stalled" rather than waited on forever. Because a stall may mean Chrome's own networking stack is wedged (not just the file), a stalled attempt retries with a brand-new browser instance rather than reusing the same one — up to MAX_DOWNLOAD_ATTEMPTS (currently 2) full attempts.
  2. Outer browser-flow timeout. All of the above (session refresh, navigation, and every download attempt/retry) runs in a background daemon thread with its own deadline, BROWSER_FLOW_TIMEOUT_SECONDS (currently 1800s / 30 min). This is the backstop for cases even the stall timeout can't catch — e.g. browser.close()'s known hang with real Chrome. If the outer deadline fires, the script force-kills the specific Chrome process that was still in flight via kill_marked_chrome(), using a unique --shockbyte-marker flag passed at launch (harmless to Chrome, which ignores unknown flags) plus psutil to find and kill anything tagged with it. This is what prevents a stalled run from leaving an orphaned Chrome process running unattended overnight — the original failure mode this was built to fix.

A run that exhausts every retry (or times out at the outer level) exits with code 2, same as a page-structure-change error — check the log for which one it was.

backup.sh dependencies

If backup.sh fails, shockbyte_download_backup.py exits with code 4 (rather than the 0 it'd otherwise return) so a failure here is distinguishable in logs/monitoring from a successful full run.

Two safety guards were added to the original version of backup.sh: it now hard-stops before running any cleanup if unzip fails or if cd ~/Downloads/instance doesn't actually succeed (to avoid rm -rf running against the wrong directory), and the final rm -rf instance retries a couple of times with a short pause to absorb an intermittent "directory not empty" race with Spotlight/Finder regenerating .DS_Store files. The rest of the script (including the NAS-mount section's "log the error and keep going" behavior) is untouched deliberately: a blanket set -e would mean a single unreachable NAS some night aborts archiving to the local and synced locations too, which is worse than the current behavior of degrading gracefully.

Exit codes (shockbyte_download_backup.py)

Troubleshooting

Bonus gotcha: tarsnap on gmktec wouldn't fire from /etc/crontab

Looked like it ran fine — showed up in the cron log every night — but no archive ever appeared. Turned out nothing was actually wrong with the job: /etc/crontab entries mail their output locally by default, and with no MTA configured on that box, any real error just vanished into the void, silently, forever.

Fix: redirect output explicitly so failures are actually visible.

0   3   *   *   *   root   tarsnap -c -f shockbyte-minecraft-`date +\%Y\%m\%d` /home/dhw/tarsnap-backups/shockbyte/minecraft/JJAC_Survival_World-vanilla >> /var/log/tarsnap-cron.log 2>&1

Known Playwright quirk: browser.close() can hang forever

Observed with real Chrome (channel="chrome") — happens after the important work (download, or a Keychain login) already succeeded, so no data is lost, but the script/terminal never gets control back without a Ctrl-C.

Fix: run the whole browser flow in a background daemon thread with a generous, bounded timeout. If the close hangs, the main thread just stops waiting and exits anyway — a stuck daemon thread can't block that.

ZFS-to-Tape Backup System — Setup Summary

Host: gmktec (FreeBSD 15.0-RELEASE). Tape drive: HP DAT72 USB (/dev/sa0 / /dev/nsa0, SCSI-over-USB via umass). Purpose: daily incremental backup of a Minecraft world to tape, on top of the existing tarsnap cloud backup of the same directory.

What's backed up

Scripts

Two scripts, deployed to /usr/local/sbin/:

zfs-tape-backup.sh

#!/bin/sh
#
# zfs-tape-backup.sh
#
# Daily snapshot + incremental zfs send to a no-rewind tape device.
# Writes a FULL send on the first run (or when the full-backup interval
# has elapsed) and an INCREMENTAL send (relative to the previous
# snapshot) on every other run. Each send is appended to the tape as a
# new tape file, since /dev/nsa0 does not rewind after writing.
#
# Edit the CONFIG section below for your setup, then run this from
# cron (see the example at the bottom of this file).

set -eu -o pipefail

# Block size for writes hitting the tape device. Must not exceed the
# controller's si_iosize_max (commonly 65536 / 64k on USB SCSI bridges -
# check dmesg for the actual value if you hit "cannot split request").
# zfs send's own output buffer size can exceed this, so we always pipe
# through dd to re-chunk writes to a safe size.
TAPE_BS="64k"

# Capacity tracking. DAT72 is rated at 36GB native (uncompressed) capacity
# (decimal GB, i.e. 36,000,000,000 bytes) - this is the manufacturer spec,
# not a measured value, so treat it as approximate. We track cumulative
# bytes actually written (post-compression, since that's what dd reports)
# and warn once you cross CAPACITY_WARN_PERCENT of that figure.
TAPE_NATIVE_CAPACITY_BYTES=36000000000
CAPACITY_WARN_PERCENT=80

# Email alerting. Uses the system mail(1) command, which routes through
# whatever MTA is configured in /etc/mail/mailer.conf (DMA by default on
# FreeBSD 14+). Set to "" to disable email alerts entirely.
ALERT_EMAIL="david@gromzen.com"

# ---------------------------------------------------------------------
# CONFIG - edit these for your environment
# ---------------------------------------------------------------------
DATASET="zroot/home/dhw/jjac-survival-world"   # the ZFS dataset holding the world dir
TAPE="/dev/nsa0"                  # no-rewind tape device
STATE_DIR="/var/db/zfs-tape-backup"
STATE_FILE="$STATE_DIR/state"     # tracks last snapshot + tape position
LOG_FILE="/var/log/zfs-tape-backup.log"
FULL_INTERVAL_DAYS=30             # take a fresh full backup this often
LOCK_FILE="/var/run/zfs-tape-backup.lock"

# ---------------------------------------------------------------------
# Setup
# ---------------------------------------------------------------------
mkdir -p "$STATE_DIR"
touch "$LOG_FILE"

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') $*" >> "$LOG_FILE"
}

mail_alert() {
    # $1 = subject, $2 = body
    if [ -z "$ALERT_EMAIL" ]; then
        return 0
    fi
    if echo "$2" | mail -s "$1" "$ALERT_EMAIL"; then
        log "Sent alert email: $1"
    else
        log "WARNING: failed to send alert email: $1"
    fi
}

# Prevent overlapping runs
if [ -e "$LOCK_FILE" ]; then
    log "ERROR: lock file $LOCK_FILE exists, another run may be in progress. Aborting."
    mail_alert "ZFS tape backup did not run on $(hostname)" \
        "Lock file $LOCK_FILE already exists - a previous run may be stuck or crashed. Backup was skipped. Check $LOG_FILE."
    exit 1
fi
trap 'rm -f "$LOCK_FILE"' EXIT
touch "$LOCK_FILE"

TODAY=$(date '+%Y-%m-%d')
SNAP_NAME="${DATASET}@${TODAY}"

# ---------------------------------------------------------------------
# Read previous state, if any
# ---------------------------------------------------------------------
LAST_SNAP=""
LAST_FULL_DATE=""
TAPE_FILE_POS=0
TOTAL_BYTES_WRITTEN=0

if [ -f "$STATE_FILE" ]; then
    # shellcheck disable=SC1090
    . "$STATE_FILE"
fi

# Guard against running twice in one day
if [ "$SNAP_NAME" = "$LAST_SNAP" ]; then
    log "Snapshot $SNAP_NAME already exists / already backed up today. Nothing to do."
    exit 0
fi

# ---------------------------------------------------------------------
# Decide full vs incremental
# ---------------------------------------------------------------------
DO_FULL=0
if [ -z "$LAST_SNAP" ]; then
    DO_FULL=1
    log "No previous state found - performing initial FULL backup."
elif [ -z "$LAST_FULL_DATE" ]; then
    DO_FULL=1
else
    DAYS_SINCE_FULL=$(( ( $(date -j -f '%Y-%m-%d' "$TODAY" '+%s') - $(date -j -f '%Y-%m-%d' "$LAST_FULL_DATE" '+%s') ) / 86400 ))
    if [ "$DAYS_SINCE_FULL" -ge "$FULL_INTERVAL_DAYS" ]; then
        DO_FULL=1
        log "Full-backup interval reached ($DAYS_SINCE_FULL days since last full) - performing FULL backup."
    fi
fi

# ---------------------------------------------------------------------
# Reposition the tape to the correct append point
# ---------------------------------------------------------------------
# We never trust that the tape is already sitting wherever the last
# successful run left it. Instead, every run explicitly rewinds to the
# beginning and forward-spaces past exactly the number of known-good
# files ($TAPE_FILE_POS). This is deliberate, not just belt-and-suspenders:
#
#   - A rewind is one of the documented ways to clear the sa(4) driver's
#     "frozen" state (set when a previous write failed to close out its
#     terminating filemark(s) - see /var/log/messages for
#     "tape is now frozen" if this has happened before). So this step
#     doubles as automatic recovery from that state, without needing a
#     manual `mt rewind` or reboot.
#   - It makes positioning self-correcting regardless of *why* the tape
#     might be in the wrong place - not just this one known failure mode.
#
# Runs BEFORE any ZFS snapshot is taken, and if it fails, we abort
# loudly rather than risk writing (and overwriting existing backups)
# at the wrong tape position.
log "Repositioning tape: rewind, then forward-space $TAPE_FILE_POS file(s)"
if ! mt -f "$TAPE" rewind >>"$LOG_FILE" 2>&1; then
    log "ERROR: mt rewind failed while repositioning tape. Aborting before touching ZFS."
    mail_alert "ZFS tape backup FAILED on $(hostname)" \
        "mt rewind failed while repositioning the tape before today's backup. No snapshot was taken and nothing was written. Check $LOG_FILE and consider checking the drive/cabling in person."
    exit 1
fi
if [ "$TAPE_FILE_POS" -gt 0 ]; then
    if ! mt -f "$TAPE" fsf "$TAPE_FILE_POS" >>"$LOG_FILE" 2>&1; then
        log "ERROR: mt fsf $TAPE_FILE_POS failed while repositioning tape. Aborting before touching ZFS."
        mail_alert "ZFS tape backup FAILED on $(hostname)" \
            "mt fsf $TAPE_FILE_POS failed while repositioning the tape before today's backup. No snapshot was taken and nothing was written. This could mean the tape doesn't contain as much data as expected (wrong/blank tape loaded?) - check $LOG_FILE and verify the correct tape is in the drive before retrying."
        exit 1
    fi
fi
log "Tape repositioned successfully to file #$TAPE_FILE_POS"

# ---------------------------------------------------------------------
# Take today's snapshot
# ---------------------------------------------------------------------
if ! zfs snapshot "$SNAP_NAME"; then
    log "ERROR: zfs snapshot $SNAP_NAME failed. Aborting."
    mail_alert "ZFS tape backup FAILED on $(hostname)" \
        "zfs snapshot $SNAP_NAME failed - backup did not run. Check $LOG_FILE."
    exit 1
fi
log "Created snapshot $SNAP_NAME"

# ---------------------------------------------------------------------
# Send to tape
# ---------------------------------------------------------------------
DD_STATS=$(mktemp)

if [ "$DO_FULL" -eq 1 ]; then
    log "Sending FULL stream of $SNAP_NAME to $TAPE (tape file #$TAPE_FILE_POS, bs=$TAPE_BS)"
    if ! zfs send "$SNAP_NAME" | dd of="$TAPE" bs="$TAPE_BS" 2>"$DD_STATS"; then
        cat "$DD_STATS" >> "$LOG_FILE"
        log "ERROR: zfs send (full) to $TAPE failed."
        mail_alert "ZFS tape backup FAILED on $(hostname)" \
            "FULL backup of $SNAP_NAME to $TAPE failed. See $LOG_FILE for details.

$(cat "$DD_STATS")"
        rm -f "$DD_STATS"
        zfs destroy "$SNAP_NAME" 2>/dev/null || true
        exit 1
    fi
    SEND_TYPE="FULL"
    LAST_FULL_DATE="$TODAY"
else
    log "Sending INCREMENTAL stream ${LAST_SNAP} -> ${SNAP_NAME} to $TAPE (tape file #$TAPE_FILE_POS, bs=$TAPE_BS)"
    if ! zfs send -i "$LAST_SNAP" "$SNAP_NAME" | dd of="$TAPE" bs="$TAPE_BS" 2>"$DD_STATS"; then
        cat "$DD_STATS" >> "$LOG_FILE"
        log "ERROR: zfs send -i $LAST_SNAP $SNAP_NAME to $TAPE failed."
        mail_alert "ZFS tape backup FAILED on $(hostname)" \
            "INCREMENTAL backup ($LAST_SNAP -> $SNAP_NAME) to $TAPE failed. See $LOG_FILE for details.

$(cat "$DD_STATS")"
        rm -f "$DD_STATS"
        zfs destroy "$SNAP_NAME" 2>/dev/null || true
        exit 1
    fi
    SEND_TYPE="INCREMENTAL"
fi

cat "$DD_STATS" >> "$LOG_FILE"

# Parse the byte count out of dd's summary line, e.g.:
#   "1140916072 bytes transferred in 335.187011 secs (3403819 bytes/sec)"
# Fall back to 0 if the format ever changes, rather than failing the backup.
BYTES_THIS_RUN=$(grep -oE '^[0-9]+ bytes transferred' "$DD_STATS" | awk '{print $1}')
rm -f "$DD_STATS"
if [ -z "${BYTES_THIS_RUN:-}" ]; then
    log "WARNING: could not parse bytes-written from dd output; capacity tracking for this run skipped."
    BYTES_THIS_RUN=0
fi

TOTAL_BYTES_WRITTEN=$((TOTAL_BYTES_WRITTEN + BYTES_THIS_RUN))

log "Wrote $SEND_TYPE stream for $SNAP_NAME as tape file #$TAPE_FILE_POS ($BYTES_THIS_RUN bytes)"

# ---------------------------------------------------------------------
# Capacity check
# ---------------------------------------------------------------------
CAPACITY_PERCENT=$((TOTAL_BYTES_WRITTEN * 100 / TAPE_NATIVE_CAPACITY_BYTES))
log "Cumulative bytes written to this tape: $TOTAL_BYTES_WRITTEN / $TAPE_NATIVE_CAPACITY_BYTES (${CAPACITY_PERCENT}% of rated native capacity)"
if [ "$CAPACITY_PERCENT" -ge "$CAPACITY_WARN_PERCENT" ]; then
    log "WARNING: tape is at ${CAPACITY_PERCENT}% of its rated native capacity (threshold: ${CAPACITY_WARN_PERCENT}%). Consider rotating in a fresh tape soon."
    mail_alert "ZFS tape backup: tape nearing capacity on $(hostname)" \
        "Tape is at ${CAPACITY_PERCENT}% of its rated native capacity (${TOTAL_BYTES_WRITTEN} / ${TAPE_NATIVE_CAPACITY_BYTES} bytes).
Warning threshold: ${CAPACITY_WARN_PERCENT}%.
Consider rotating in a fresh tape soon. See $STATE_DIR/tape-index.log for the full history on this tape."
fi

# ---------------------------------------------------------------------
# Update state
# ---------------------------------------------------------------------
NEW_TAPE_FILE_POS=$((TAPE_FILE_POS + 1))
cat > "$STATE_FILE" <<EOF
LAST_SNAP="$SNAP_NAME"
LAST_FULL_DATE="$LAST_FULL_DATE"
TAPE_FILE_POS=$NEW_TAPE_FILE_POS
TOTAL_BYTES_WRITTEN=$TOTAL_BYTES_WRITTEN
EOF

# Also append a permanent record to the log of what's on the tape and
# where, since $STATE_FILE only tracks the *current* pointer.
echo "$(date '+%Y-%m-%d %H:%M:%S') TAPE_FILE=$TAPE_FILE_POS TYPE=$SEND_TYPE SNAP=$SNAP_NAME BYTES=$BYTES_THIS_RUN CUMULATIVE_BYTES=$TOTAL_BYTES_WRITTEN" >> "$STATE_DIR/tape-index.log"

log "Backup complete. State updated: LAST_SNAP=$SNAP_NAME TAPE_FILE_POS=$NEW_TAPE_FILE_POS"

exit 0

# ---------------------------------------------------------------------
# Example cron entry (run as root, daily at 2am):
#   0 2 * * * /usr/local/sbin/zfs-tape-backup.sh
#
# When you swap in a fresh/blank tape, delete $STATE_FILE (but keep
# tape-index.log for your records, or archive it) so the next run
# starts over with a FULL backup at tape file #0 and resets the
# cumulative capacity counter (TOTAL_BYTES_WRITTEN) to 0.
# ---------------------------------------------------------------------

zfs-tape-restore.sh

#!/bin/sh
#
# zfs-tape-restore.sh
#
# Restores a dataset from a tape written by zfs-tape-backup.sh.
# Reads tape files in order starting from the beginning, receiving
# the full stream first and then each incremental in sequence.
#
# Usage: zfs-tape-restore.sh <target-dataset> [num-files]
#
#   target-dataset   e.g. zroot/home/dhw/jjac-survival-world-restore
#                    (must not already exist)
#   num-files        how many tape files to restore (default: all files
#                     recorded in the tape-index.log)
#
# Check /var/db/zfs-tape-backup/tape-index.log first to see how many
# files are on the tape and what each one is (FULL or INCREMENTAL).
#
# Example:
#   zfs-tape-restore.sh zroot/home/dhw/jjac-survival-world-restore

set -eu -o pipefail

TAPE="/dev/nsa0"
TAPE_BS="64k"   # must match the block size used when writing (see zfs-tape-backup.sh)
INDEX_LOG="/var/db/zfs-tape-backup/tape-index.log"

TARGET="${1:-}"
if [ -z "$TARGET" ]; then
    echo "Usage: $0 <target-dataset> [num-files]" >&2
    exit 1
fi

if [ -n "${2:-}" ]; then
    NUM_FILES="$2"
elif [ -f "$INDEX_LOG" ]; then
    NUM_FILES=$(wc -l < "$INDEX_LOG" | tr -d ' ')
else
    echo "No tape-index.log found and no file count given - specify num-files explicitly." >&2
    exit 1
fi

echo "Restoring $NUM_FILES tape file(s) into dataset: $TARGET"
echo "Rewinding tape..."
mt -f "$TAPE" rewind

i=0
while [ "$i" -lt "$NUM_FILES" ]; do
    if [ "$i" -eq 0 ]; then
        echo "Receiving tape file #0 (expected FULL stream)..."
        dd if="$TAPE" bs="$TAPE_BS" | zfs receive "$TARGET"
    else
        echo "Receiving tape file #$i (expected INCREMENTAL stream)..."
        dd if="$TAPE" bs="$TAPE_BS" | zfs receive -F "$TARGET"
        # -F rolls the target back to match the incoming incremental's
        # origin snapshot if needed; safe here since we're restoring
        # in strict order into a dedicated restore target.
    fi

    i=$((i + 1))
    # No explicit fsf needed here: per sa(4), once a read consumes the
    # terminating zero-length read at a filemark, the no-rewind device
    # is already positioned at the start of the next tape file.
done

echo "Restore complete. Snapshots now present on $TARGET:"
zfs list -t snapshot -r "$TARGET"

Key technical issues hit and fixed

  1. Block size mismatch (si_iosize_max) — the USB SCSI bridge in the DAT72 adapter caps single writes at 65536 bytes (64KB). zfs send's own write buffer exceeds this, causing cannot split request errors. Fix: always pipe zfs send through dd of=/dev/nsa0 bs=64k to re-chunk writes to a safe size, rather than writing directly.
  2. Frozen tape state — an aborted write (triggered by the block-size issue) left the sa(4) driver in a "frozen" state (visible in dmesg, not in mt status), refusing further I/O until explicitly cleared, since a failed write can leave the terminating filemark(s) unclosed. Fix: mt -f /dev/nsa0 rewind clears it — and this is no longer just a manual recovery step. The backup script now rewinds and re-forward-spaces to the correct tape file (mt rewind, then mt fsf $TAPE_FILE_POS) at the start of every run, before touching ZFS. This both self-corrects tape position each time (rather than trusting wherever the last run left off) and doubles as automatic recovery from the frozen state, so a prior failed write no longer requires manual intervention. If the rewind/reposition step itself fails, the script aborts before taking a snapshot or writing anything, and sends an email alert.
  3. Restore positioning bug — the original restore script called mt fsf 1 between reads to advance to the next tape file. This was wrong: per sa(4), once a sequential read consumes the terminating zero-length read at a filemark, the no-rewind device is already positioned at the start of the next file. The extra fsf skipped past the real next file onto nonexistent data, causing an I/O error. Fix: removed the explicit skip; reads now walk file-to-file on their own.
  4. Validating restores against a moving target — diffing a restored copy against the live source directory is unreliable if rsync runs in between, since it compares two different points in time. Fix: diff against the ZFS snapshot's frozen state instead, via the hidden .zfs/snapshot/<name> path, e.g.: 
    diff -rq /home/dhw/.../JJAC_Survival_World-vanilla/.zfs/snapshot/2026-07-16 \
         "$(zfs get -H -o value mountpoint <restored-dataset>)"
    Both full and full+incremental restore chains have been validated this way with a clean (no-diff) result.

Capacity tracking

Email alerting

Cron

0 2 * * * /usr/local/sbin/zfs-tape-backup.sh

Add via /etc/crontab (needs a username field: 0 2 * * * root /usr/local/sbin/zfs-tape-backup.sh) or crontab -e — whichever's more reliable in your environment.

Open items / possible follow-ups