> ## Content Index
> Fetch the complete content index at: https://logbuch-homelab.nichedthemes.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Backups stopped for 31 hours
- URL: https://logbuch-homelab.nichedthemes.com/backups-stopped-for-31-hours/
- Published: 2026-11-08T10:00:00.000Z
- Updated: 2026-11-08T10:00:00.000Z
- Description: The NAS ran out of space on Thursday night and restic failed at 04:00. The alert went to an address I stopped reading in spring.
- Author: Tomasz Wieland
- Tags: Homelab, Backups, Proxmox, #incident, #resolved, #major, #Import 2026-11-19 07:37

Backups stopped for 31 hours. The NAS ran out of space on Thursday night and restic failed at 04:00\. The alert went to an address I stopped reading in spring, so I only noticed on Saturday, when the weekly check showed Thursday’s snapshot as the newest.

No data was lost. Nothing needed a restore in those 31 hours, and that was luck.

## Timeline

- 2026-11-06 04:00 CET restic fails on all three nodes with `no space left on device`. The failure mail goes to root, which forwards it to an address I no longer read.
- 2026-11-07 09:30 Investigating: The weekly look at `restic snapshots` shows Thursday’s snapshot from 04:00 as the newest.
- 09:41 Identified: `/srv` on the NAS is full, 7.3T of 7.3T. The storage `nas-dump` has kept every weekly Proxmox dump since June.
- 10:05 Old dumps deleted, the last three per guest kept. 1.7 TiB free.
- 10:12 Monitoring: restic started by hand on all three nodes.
- 11:00 Resolved: Three new snapshots, and `restic check` finds no errors.

```console
$ journalctl -u restic-backup --since 2026-11-06 -n 3 --output short-iso
2026-11-06T04:00:01+01:00 pve-1 systemd[1]: Starting restic-backup.service - restic backup to the NAS...
2026-11-06T04:00:14+01:00 pve-1 backup.sh[48211]: Fatal: unable to save snapshot: no space left on device
2026-11-06T04:00:14+01:00 pve-1 systemd[1]: restic-backup.service: Failed with result 'exit-code'.
```

## What went wrong

Proxmox writes a dump of every guest to `nas-dump` on Thursdays at 22:00\. When I added that storage in [chapter 3 of the series](https://logbuch-homelab.nichedthemes.com/the-nas-as-storage-for-the-cluster/), I left its retention at the default, which keeps everything. Twenty-one Thursdays of dumps came to 2 TiB, and this Thursday’s run filled `/srv` at 23:40.

restic did the right thing. Its unit mailed root, and root’s mail is forwarded to the address in `/etc/aliases`, which I stopped reading when I moved my mail in spring. Nothing had failed before, so nothing had gone there yet.

## What changed

- The storage keeps the last three dumps per guest: `pvesm set nas-dump --prune-backups keep-last=3`.
- Failed units send a push to my phone through ntfy and a mail to the address I read, set in one variable of the next loft-ansible release.
- The NAS warns at 80 % full, by the same route.
- The weekly check on the NAS reads one twentieth of the stored data back every Saturday.