ZFS on Linux: snapshot directories are empty

Due to a known bug in ZFS for Linux, all file system snapshots created by ZFS thru its userspace utilities cannot be further accessed after their creation. What happens is that ZFS correctly creates the snapshots upon request, but is then unable to mount them on the file system to access their content.

On Debian I use this snapshot utility:

$ cat /etc/cron.d/zfs-auto-snapshot
PATH="/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"

*/15 * * * * root which zfs-auto-snapshot > /dev/null || exit 0 ; zfs-auto-snapshot --quiet --syslog --label=frequent --keep=4 //

which produces periodic snapshots of the ZFS pool mounted on /media/raid1/:

$ zfs list -t snapshot
NAME                                                 USED  AVAIL     REFER  MOUNTPOINT
pool_mirror@zfs-auto-snap_weekly-2024-07-28-0447     136G      -      427G  -
pool_mirror@zfs-auto-snap_daily-2024-08-03-0428     21.3G      -      428G  -
pool_mirror@zfs-auto-snap_daily-2024-08-04-0446     4.99M      -      436G  -
pool_mirror@zfs-auto-snap_weekly-2024-08-04-0447    4.72M      -      436G  -
pool_mirror@zfs-auto-snap_hourly-2024-08-04-2317    27.6M      -      443G  -
[...]
pool_mirror@zfs-auto-snap_frequent-2024-08-05-1045   161M      -      448G  -
pool_mirror@zfs-auto-snap_frequent-2024-08-05-1100   237M      -      448G  -

and I can confirm that the snapshots directories are also created:

root@debianBunker:/media/raid1/.zfs/snapshot# ls -lh
total 0
drwxrwxrwx 1 root root 0 Aug  3 06:28 zfs-auto-snap_daily-2024-08-03-0428
drwxrwxrwx 1 root root 0 Aug  4 06:46 zfs-auto-snap_daily-2024-08-04-0446
drwxrwxrwx 1 root root 0 Aug  5 06:28 zfs-auto-snap_daily-2024-08-05-0428
[...]
drwxrwxrwx 1 root root 0 Jul 28 06:47 zfs-auto-snap_weekly-2024-07-28-0447
drwxrwxrwx 1 root root 0 Aug  4 06:47 zfs-auto-snap_weekly-2024-08-04-0447

that's fine so far. However, if I cd into these snapshot directories, they are all empty.. my data is not here!

root@debianBunker:/media/raid1/.zfs/snapshot# cd zfs-auto-snap_daily-2024-08-03-0428/
root@debianBunker:/media/raid1/.zfs/snapshot/zfs-auto-snap_daily-2024-08-03-0428# ls -alh
total 0
drwxrwxrwx  1 root root 0 Aug  3 06:28 .
drwxrwxrwx 23 root root 2 Aug  5 13:17 ..
root@debianBunker:/media/raid1/.zfs/snapshot/zfs-auto-snap_daily-2024-08-03-0428# 

and this is the corresponding log of the failure to automount:

# tail -n1 /proc/spl/kstat/zfs/dbgmsg
1722857003   ffff88818d57f000 zfs_ctldir.c:1161:zfsctl_snapshot_mount(): Unable to automount /.zfs/snapshot/zfs-auto-snap_daily-2024-08-03-0428 error=512

Doing an ls inside these directories is supposed to trigger ZFS to automount them. But it doesn't happen and so far no one has any clue why. This bug is known to occur frequently on a root ZFS, and seldom on non-root mount points. Mine is not on root.

There are solutions for both cases:

• with ZFS on root, binding root to /sysroot is known to work:
  # mkdir /sysroot
# mount -o bind / /sysroot
  this makes the snapshots accessible at /sysroot/.zfs/, and they are correctly auto-mounted upon ls.

• with ZFS not on root (my case) manually mounting the snapshot without bind works:
  # mount -t zfs pool_mirror@zfs-auto-snap_weekly-2024-07-28-0447 /media/snapshot/
# cd /media/snapshot
# ls
gateway.conf lost+found main_backup ve-103 ve-106 ve-109 ve-111
gateway.conf~ lxc-monitord.log ve-102 ve-104.ovz ve-107 ve-110 ve-200