pve-2 keeps losing its network link
Three times in ten hours pve-2 dropped off the network for a few seconds and left the cluster. Power saving on its network card is the likeliest cause.
- Status
- Monitoring
- Severity
- Minor
- Started
- Resolved
- Duration
pve-2 lost its network link three times between Thursday evening and Friday morning, for 9 to 14 seconds each time. The cluster stayed quorate on the other two nodes, and the guests on pve-2 kept running, though nothing could reach them while the link was down. PCIe power saving on the network card is the likeliest cause, and it has been off since Friday 18:20.
Timeline
- 2026-11-12 21:47 CET pve-2 loses its link for 9 seconds and leaves the cluster. It rejoins on its own.
- 2026-11-13 03:15 The same again, for 14 seconds.
- 07:58 Investigating: A third drop, 11 seconds. All three are on pve-2, and the kernel log says the card lost its link.
- 12:30 Identified: The card goes into a PCIe power-saving state and does not always come back in time. Turning that off is the usual fix for this card.
- 18:20 Monitoring: pve-2 rebooted with
pcie_aspm=off. No drop since.
What the log shows
$ journalctl -k --since "2026-11-12 21:40" --grep "Link is" --output short-iso
2026-11-12T21:47:03+01:00 pve-2 kernel: igc 0000:01:00.0 enp1s0: NIC Link is Down
2026-11-12T21:47:12+01:00 pve-2 kernel: igc 0000:01:00.0 enp1s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TXThe change
--- a/etc/default/grub
+++ b/etc/default/grub
@@ -9 +9 @@
-GRUB_CMDLINE_LINUX_DEFAULT="quiet"
+GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_aspm=off"Then update-grub and a reboot, after its guests had moved to pve-1 and pve-3.
Why it is not resolved yet
The other two nodes have the same card and have not dropped once, so the cause may sit in pve-2 itself. This stays at monitoring until pve-2 has gone a week without a drop. If it drops again, the cable and the switch port are next.
subscribe
New entries and the next chapter by mail, nothing else.