ZFS on Linux: snapshot directories are empty
Due to a known bug in ZFS for Linux, all file system snapshots created by ZFS thru its userspace utilities cannot be further accessed after their creation. What happens is that ZFS correctly creates the snapshots upon request, but is then unable to mount them on the file system to access their content.
On Debian I use this snapshot utility:
$ cat /etc/cron.d/zfs-auto-snapshot
PATH="/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"
*/15 * * * * root which zfs-auto-snapshot > /dev/null || exit 0 ; zfs-auto-snapshot --quiet --syslog --label=frequent --keep=4 //
which produces periodic snapshots of the ZFS pool mounted on /media/raid1/:
$ zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
pool_mirror@zfs-auto-snap_weekly-2024-07-28-0447 136G - 427G -
pool_mirror@zfs-auto-snap_daily-2024-08-03-0428 21.3G - 428G -
pool_mirror@zfs-auto-snap_daily-2024-08-04-0446 4.99M - 436G -
pool_mirror@zfs-auto-snap_weekly-2024-08-04-0447 4.72M - 436G -
pool_mirror@zfs-auto-snap_hourly-2024-08-04-2317 27.6M - 443G -
[...]
pool_mirror@zfs-auto-snap_frequent-2024-08-05-1045 161M - 448G -
pool_mirror@zfs-auto-snap_frequent-2024-08-05-1100 237M - 448G -
and I can confirm that the snapshots directories are also created:
root@debianBunker:/media/raid1/.zfs/snapshot# ls -lh
total 0
drwxrwxrwx 1 root root 0 Aug 3 06:28 zfs-auto-snap_daily-2024-08-03-0428
drwxrwxrwx 1 root root 0 Aug 4 06:46 zfs-auto-snap_daily-2024-08-04-0446
drwxrwxrwx 1 root root 0 Aug 5 06:28 zfs-auto-snap_daily-2024-08-05-0428
[...]
drwxrwxrwx 1 root root 0 Jul 28 06:47 zfs-auto-snap_weekly-2024-07-28-0447
drwxrwxrwx 1 root root 0 Aug 4 06:47 zfs-auto-snap_weekly-2024-08-04-0447
that's fine so far. However, if I cd into these snapshot directories, they are all empty.. my data is not here!
root@debianBunker:/media/raid1/.zfs/snapshot# cd zfs-auto-snap_daily-2024-08-03-0428/
root@debianBunker:/media/raid1/.zfs/snapshot/zfs-auto-snap_daily-2024-08-03-0428# ls -alh
total 0
drwxrwxrwx 1 root root 0 Aug 3 06:28 .
drwxrwxrwx 23 root root 2 Aug 5 13:17 ..
root@debianBunker:/media/raid1/.zfs/snapshot/zfs-auto-snap_daily-2024-08-03-0428#
and this is the corresponding log of the failure to automount:
# tail -n1 /proc/spl/kstat/zfs/dbgmsg
1722857003 ffff88818d57f000 zfs_ctldir.c:1161:zfsctl_snapshot_mount(): Unable to automount /.zfs/snapshot/zfs-auto-snap_daily-2024-08-03-0428 error=512
Doing an ls inside these directories is supposed to trigger ZFS to automount them. But it doesn't happen and so far no one has any clue why. This bug is known to occur frequently on a root ZFS, and seldom on non-root mount points. Mine is not on root.
There are solutions for both cases:
with ZFS on root, binding root to
/sysrootis known to work:
# mkdir /sysroot
# mount -o bind / /sysroot
this makes the snapshots accessible at/sysroot/.zfs/, and they are correctly auto-mounted uponls.with ZFS not on root (my case) manually mounting the snapshot without bind works:
# mount -t zfs pool_mirror@zfs-auto-snap_weekly-2024-07-28-0447 /media/snapshot/
# cd /media/snapshot
# ls
gateway.conf lost+found main_backup ve-103 ve-106 ve-109 ve-111
gateway.conf~ lxc-monitord.log ve-102 ve-104.ovz ve-107 ve-110 ve-200
Comments
davide
Someone pranked me! Who changed the post title!?
Edit: fixed. I must have pranked myself, between my impossibly slow 50Kbs LTE modem, my broken browser cache and my buggy KDE3 clipboard it's hard to keep up.
Due to a known bug in LowEndSpirit, all topics created by davide thru his browser cannot be further accessed by their title after their creation.
Doing an F5 on the front page is supposed to trigger LES to display them. But it doesn't happen and so far no one has any clue why. This bug is known to occur frequently on a root of LES, and seldom on non-root forum categories.
I read pranked as spanked.
HostBrr aff best VPS; VirmAche aff worst VPS.
Unable to push-up due to shoulder injury 😣
Papa Oles would love to spank me until my butt is all nice and red, cooked and Crunchy. They love red in family. Nothing binds a family more than American dollars.
The Olympic Movement has reached LowEndSpirit:
Please keep it civil and avoid insults (or allegations - at least without hard evidence).
Relja of House Novović, the First of His Name, King of the Plains, the Breaker of Chains, WirMach Wolves pack member
BikeGremlin's web-hosting reviews