ZFS on Linux: snapshot directories are empty

davide

Due to a known bug in ZFS for Linux, all file system snapshots created by ZFS thru its userspace utilities cannot be further accessed after their creation. What happens is that ZFS correctly creates the snapshots upon request, but is then unable to mount them on the file system to access their content.

On Debian I use this snapshot utility:

$ cat /etc/cron.d/zfs-auto-snapshot
PATH="/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"

*/15 * * * * root which zfs-auto-snapshot > /dev/null || exit 0 ; zfs-auto-snapshot --quiet --syslog --label=frequent --keep=4 //

which produces periodic snapshots of the ZFS pool mounted on /media/raid1/:

$ zfs list -t snapshot
NAME                                                 USED  AVAIL     REFER  MOUNTPOINT
pool_mirror@zfs-auto-snap_weekly-2024-07-28-0447     136G      -      427G  -
pool_mirror@zfs-auto-snap_daily-2024-08-03-0428     21.3G      -      428G  -
pool_mirror@zfs-auto-snap_daily-2024-08-04-0446     4.99M      -      436G  -
pool_mirror@zfs-auto-snap_weekly-2024-08-04-0447    4.72M      -      436G  -
pool_mirror@zfs-auto-snap_hourly-2024-08-04-2317    27.6M      -      443G  -
[...]
pool_mirror@zfs-auto-snap_frequent-2024-08-05-1045   161M      -      448G  -
pool_mirror@zfs-auto-snap_frequent-2024-08-05-1100   237M      -      448G  -

and I can confirm that the snapshots directories are also created:

root@debianBunker:/media/raid1/.zfs/snapshot# ls -lh
total 0
drwxrwxrwx 1 root root 0 Aug  3 06:28 zfs-auto-snap_daily-2024-08-03-0428
drwxrwxrwx 1 root root 0 Aug  4 06:46 zfs-auto-snap_daily-2024-08-04-0446
drwxrwxrwx 1 root root 0 Aug  5 06:28 zfs-auto-snap_daily-2024-08-05-0428
[...]
drwxrwxrwx 1 root root 0 Jul 28 06:47 zfs-auto-snap_weekly-2024-07-28-0447
drwxrwxrwx 1 root root 0 Aug  4 06:47 zfs-auto-snap_weekly-2024-08-04-0447

that's fine so far. However, if I cd into these snapshot directories, they are all empty.. my data is not here!

root@debianBunker:/media/raid1/.zfs/snapshot# cd zfs-auto-snap_daily-2024-08-03-0428/
root@debianBunker:/media/raid1/.zfs/snapshot/zfs-auto-snap_daily-2024-08-03-0428# ls -alh
total 0
drwxrwxrwx  1 root root 0 Aug  3 06:28 .
drwxrwxrwx 23 root root 2 Aug  5 13:17 ..
root@debianBunker:/media/raid1/.zfs/snapshot/zfs-auto-snap_daily-2024-08-03-0428# 

and this is the corresponding log of the failure to automount:

# tail -n1 /proc/spl/kstat/zfs/dbgmsg
1722857003   ffff88818d57f000 zfs_ctldir.c:1161:zfsctl_snapshot_mount(): Unable to automount /.zfs/snapshot/zfs-auto-snap_daily-2024-08-03-0428 error=512

Doing an ls inside these directories is supposed to trigger ZFS to automount them. But it doesn't happen and so far no one has any clue why. This bug is known to occur frequently on a root ZFS, and seldom on non-root mount points. Mine is not on root.

There are solutions for both cases:

• with ZFS on root, binding root to /sysroot is known to work:
  # mkdir /sysroot
# mount -o bind / /sysroot
  this makes the snapshots accessible at /sysroot/.zfs/, and they are correctly auto-mounted upon ls.

• with ZFS not on root (my case) manually mounting the snapshot without bind works:
  # mount -t zfs pool_mirror@zfs-auto-snap_weekly-2024-07-28-0447 /media/snapshot/
# cd /media/snapshot
# ls
gateway.conf lost+found main_backup ve-103 ve-106 ve-109 ve-111
gateway.conf~ lxc-monitord.log ve-102 ve-104.ovz ve-107 ve-110 ve-200

dosai

davide

davide

@dosai said:
davide

Someone pranked me! Who changed the post title!?
Edit: fixed. I must have pranked myself, between my impossibly slow 50Kbs LTE modem, my broken browser cache and my buggy KDE3 clipboard it's hard to keep up.

DataRecovery

@davide said:
Someone pranked me! Who changed the post title!?

Due to a known bug in LowEndSpirit, all topics created by davide thru his browser cannot be further accessed by their title after their creation.

Doing an F5 on the front page is supposed to trigger LES to display them. But it doesn't happen and so far no one has any clue why. This bug is known to occur frequently on a root of LES, and seldom on non-root forum categories.

yoursunny

@davide said:
Someone pranked me!

I read pranked as spanked.

davide

@yoursunny said:
I read pranked as spanked.

Papa Oles would love to spank me until my butt is all nice and red, cooked and Crunchy. They love red in family. Nothing binds a family more than American dollars.

DataRecovery

@davide said:

@yoursunny said:
I read pranked as spanked.

Papa Oles would love to spank me until my butt is all nice and red, cooked and Crunchy. They love red in family. Nothing binds a family more than American dollars.

The Olympic Movement has reached LowEndSpirit:

bikegremlin

@davide said:

@yoursunny said:
I read pranked as spanked.

Papa Oles would love to spank me until my butt is all nice and red, cooked and Crunchy. They love red in family. Nothing binds a family more than American dollars.

Please keep it civil and avoid insults (or allegations - at least without hard evidence).