Backups stopped for 31 hours
The NAS ran out of space on Thursday night and restic failed at 04:00. The alert went to an address I stopped reading in spring.
- Status
- Resolved
- Severity
- Major
- Started
- Resolved
- Duration
Backups stopped for 31 hours. The NAS ran out of space on Thursday night and restic failed at 04:00. The alert went to an address I stopped reading in spring, so I only noticed on Saturday, when the weekly check showed Thursday’s snapshot as the newest.
No data was lost. Nothing needed a restore in those 31 hours, and that was luck.
Timeline
- 2026-11-06 04:00 CET restic fails on all three nodes with
no space left on device. The failure mail goes to root, which forwards it to an address I no longer read. - 2026-11-07 09:30 Investigating: The weekly look at
restic snapshotsshows Thursday’s snapshot from 04:00 as the newest. - 09:41 Identified:
/srvon the NAS is full, 7.3T of 7.3T. The storagenas-dumphas kept every weekly Proxmox dump since June. - 10:05 Old dumps deleted, the last three per guest kept. 1.7 TiB free.
- 10:12 Monitoring: restic started by hand on all three nodes.
- 11:00 Resolved: Three new snapshots, and
restic checkfinds no errors.
$ journalctl -u restic-backup --since 2026-11-06 -n 3 --output short-iso
2026-11-06T04:00:01+01:00 pve-1 systemd[1]: Starting restic-backup.service - restic backup to the NAS...
2026-11-06T04:00:14+01:00 pve-1 backup.sh[48211]: Fatal: unable to save snapshot: no space left on device
2026-11-06T04:00:14+01:00 pve-1 systemd[1]: restic-backup.service: Failed with result 'exit-code'.What went wrong
Proxmox writes a dump of every guest to nas-dump on Thursdays at 22:00. When I added that storage in chapter 3 of the series, I left its retention at the default, which keeps everything. Twenty-one Thursdays of dumps came to 2 TiB, and this Thursday’s run filled /srv at 23:40.
restic did the right thing. Its unit mailed root, and root’s mail is forwarded to the address in /etc/aliases, which I stopped reading when I moved my mail in spring. Nothing had failed before, so nothing had gone there yet.
What changed
- The storage keeps the last three dumps per guest:
pvesm set nas-dump --prune-backups keep-last=3. - Failed units send a push to my phone through ntfy and a mail to the address I read, set in one variable of the next loft-ansible release.
- The NAS warns at 80 % full, by the same route.
- The weekly check on the NAS reads one twentieth of the stored data back every Saturday.
subscribe
New entries and the next chapter by mail, nothing else.