F39 nfs-ganesha segfault & coredump

Ask Fedora
, , , ,
1

Hey all,

this is my first post in this forum, please let me know if it’s in the wrong place or missing information.

When trying to install nfs-ganesha version V5.5-2 packages on Fedora 39 (beta), it will crash when configured to export CephFS.

Output from /var/log/ganesha/ganesha.log:

09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4673[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4673[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.5
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] load_rados_config :CONFIG :WARN :Missing RADOS URLs backend library
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] fsal_init_fds_limit :MDCACHE LRU :EVENT :Setting the system-imposed limit on FDs to 524288.
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0)
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
09/10/2023 13:58:09 : epoch 6523ead1 : cephfs2.tld : ganesha.nfsd-4674[main] gsh_backtrace :NFS STARTUP :MAJ :stack backtrace follows:
/lib64/libganesha_nfsd.so.5.5(+0x517a5)[0x7f9e3f4357a5]
/lib64/libc.so.6(+0x3e9a0)[0x7f9e3f2409a0]
/lib64/libstdc++.so.6(_ZSt28_Rb_tree_rebalance_for_erasePSt18_Rb_tree_node_baseRS_+0x4)[0x7f9e3b6cd904]
/usr/lib64/ceph/libceph-common.so.2(_ZN15CommonSafeTimerISt5mutexE17cancel_all_eventsEv+0x89)[0x7f9e3bc816f9]
/usr/lib64/ceph/libceph-common.so.2(_ZN15CommonSafeTimerISt5mutexE8shutdownEv+0x41)[0x7f9e3bc819d1]
/usr/lib64/ceph/libceph-common.so.2(_ZN9MonClient8shutdownEv+0x3b1)[0x7f9e3bec6a61]
/usr/lib64/ceph/libceph-common.so.2(_ZN9MonClient21get_monmap_and_configEv+0x627)[0x7f9e3bec7b67]
/lib64/libcephfs.so.2(+0x4fdb9)[0x7f9e3c3aedb9]
/usr/lib64/ganesha/libfsalceph.so(+0x7f46)[0x7f9e3c51ef46]
/lib64/libganesha_nfsd.so.5.5(+0x134e62)[0x7f9e3f518e62]
/lib64/libganesha_nfsd.so.5.5(+0xa5bab)[0x7f9e3f489bab]
/lib64/libganesha_nfsd.so.5.5(+0x691e3)[0x7f9e3f44d1e3]
/lib64/libganesha_nfsd.so.5.5(+0x68989)[0x7f9e3f44c989]
/lib64/libganesha_nfsd.so.5.5(load_config_from_parse+0x115)[0x7f9e3f44d9d5]
/lib64/libganesha_nfsd.so.5.5(ReadExports+0xdd)[0x7f9e3f48aa8d]
/usr/bin/ganesha.nfsd(main+0x674)[0x55d7b258dd64]
/lib64/libc.so.6(+0x2814a)[0x7f9e3f22a14a]
/lib64/libc.so.6(__libc_start_main+0x8b)[0x7f9e3f22a20b]
/usr/bin/ganesha.nfsd(_start+0x25)[0x55d7b258e485]

nfs-ganesha will fail as soon as I add an EXPORT block to the config, e.g.:

NFS_CORE_PARAM
{
        Enable_NLM = false;
        Enable_RQUOTA = false;
        Protocols = 4;
}

NFSv4
{
        Minor_Versions =  1,2;
}

MDCACHE
{
        Dir_Chunk = 0;
}

NFS_KRB5
{
        PrincipalName = "nfs";
        KeytabPath = "/etc/ganesha/keytab";
        Active_krb5 = true;
}

%dir /etc/ganesha/ganesha.conf.d

File /etc/ganesha/ganesha.conf.d/100_erasure.conf:

EXPORT
{
        Export_ID = 100;
        Protocols = 4;
        Transports = TCP;
        Path = /erasure;
        Pseudo = /mpi;
        Access_Type = RW;
        Attr_Expiration_Time = 0;
        Squash = root_squash;
        SecType = krb5;
        FSAL {
                Name = CEPH;
                Filesystem = "cephfs";
                User_Id = "ganesha_dev";
        }
}

Without the EXPORT block it works.

  • CephFS is accessible when mounting it via mount command
  • Ceph config and keyring for ganesha_dev is accessible.

I rolled out the same setup to a Fedora 38 (with Ansible) with nfs-ganesha version V4.4. It starts with no errors.
Packages are installed via dnf:

  • Fedora 38:
root@cephfs3:~# cat /etc/redhat-release
Fedora release 38 (Thirty Eight)
root@cephfs3:~# rpm -qa | grep ganesha
nfs-ganesha-selinux-4.4-2.fc38.noarch
nfs-ganesha-4.4-2.fc38.x86_64
nfs-ganesha-ceph-4.4-2.fc38.x86_64
  • Fedora 39 beta:
root@cephfs2:~# cat /etc/redhat-release
Fedora release 39 (Thirty Nine)
root@cephfs2:~# rpm -qa | grep ganesha
nfs-ganesha-selinux-5.5-2.fc39.noarch
nfs-ganesha-5.5-2.fc39.x86_64
nfs-ganesha-ceph-5.5-2.fc39.x86_64
2

Yes, it was in the wrong place, I moved it to Ask Fedora .

3

This sounds like a bug in nfs-ganesha itself. I’d suggest to open a bug in bugzilla. There is already one for a different use case and a maintainer was asking if using version 5.5 does help, so please add your details there: 2212095 – Ganesha server SIGABRT and coredump