iSCSI Shared LVM

Ask Fedora
, ,
1

I’m attempting to setup a shared volume across three Fedora Server nodes. I’m using a Synology NAS as the iSCSI target. I have all three nodes setup in the iSCSI setup and Synology is showing them as registered. All three nodes see the device as /dev/sda.

Here are the steps that I’ve attempted:

On the first node, create an LVM partition:

node1:$ sudo fdisk /dev/sda

new partition
set type to 44 (Linux LVM)

Installed lvmlockd and sanlock and started the services on all nodes:

node*:$ sudo dnf install -y lvmlockd sanlock
node*:$ sudo systemctl enable --now lvmlockd sanlock

On the first node, created a new shared volume group:

node1:$ sudo vgcreate --shared svg1 /dev/sda1
  Enabling sanlock global lock
  Device /dev/sda1 has no PVID (devices file ch1fvyTrHW6uFbAJXxeDYSgdPOWV55KO)
  Physical volume "/dev/sda1" successfully created.
  Logical volume "lvmlock" created.
  Volume group "svg1" successfully created
  VG svg1 starting sanlock lockspace
  Starting locking.  Waiting until locks are ready...

Checking lock status:

node1:$ sudo lvmlockctl -i
VG svg1 lock_type=sanlock FYsFon-K4no-EikA-zQtX-V22e-V2Kt-b2Pb2U
LS sanlock lvm_svg1
LK VG un ver 0
LK GL un ver 0

Checking for device:

node1:$ sudo lvs -a -o +devices
  Skipping global lock: lockspace not found or started
  Reading VG svg1 without a lock.
  LV        VG         Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices             
  lvol0     fedora     -wi-ao---- <221.89g                                                     /dev/nvme0n1p3(3840)
  root      fedora     -wi-ao----   15.00g                                                     /dev/nvme0n1p3(0)   
  [lvmlock] svg1       -wi-ao----  256.00m                                                     /dev/sda1(0)

And now when I attempt to activate the lock on either node 2 or 3, I get the error below:

node2:$ sudo vgchange --lock-start 
  Skipping global lock: lockspace not found or started

I tried running sudo lvmlockctl --gl-enable resulting the same thing.

pvs on node1 shows the pv but pvs does not show the iscsi pv and says something about the lockspace isn’t found.

node1:$ sudo pvs
  PV             VG         Fmt  Attr PSize    PFree   
  /dev/nvme0n1p3 fedora     lvm2 a--  <236.89g       0 
  /dev/sda1      svg1       lvm2 a--  <200.00g <199.75g

node2:$ sudo pvs
  Skipping global lock: lockspace not found or started
  PV             VG         Fmt  Attr PSize    PFree
  /dev/nvme0n1p3 fedora     lvm2 a--  <236.89g    0

I feel like I’m missing something really stupid. Any ideas?

2

Quick update, realized that I was attempting to create the vg on a partition, rather than just using the entire volume. So removed the partition and started over. Unfortunately, I arrived to the same issue.

What I find weird is that pvs doesn’t see the volume on the other two nodes.

3

Note that LVM does not care if the PV is created in a partition or on a raw device. I usually do use a partition unless I am working with a raid device. Once the PV is created (using pvcreate) then the VG can be created or extended to use that PV (vgcreate or other vg… commands)

This is my system with /home on a full raw raid5 array (7 TB) and / on a partition of an SSD (1.8TB). The SSD also contains the efi and boot partitions.

You can find more info using something like pvdisplay or pvscan than is available with pvs. Similarly with the VG – vgdisplay, vgscan, etc. Similarly for LVs

# pvs
  PV             VG           Fmt  Attr PSize  PFree 
  /dev/md127     fedora_raid1 lvm2 a--  <7.28t     0 
  /dev/nvme0n1p3 fedora_root  lvm2 a--  <1.82t <1.18t
   
# pvscan
  PV /dev/md127       VG fedora_raid1   lvm2 [<7.28 TiB / 0    free]
  PV /dev/nvme0n1p3   VG fedora_root    lvm2 [<1.82 TiB / <1.18 TiB free]
  Total: 2 [<9.10 TiB] / in use: 2 [<9.10 TiB] / in no VG: 0 [0   ]

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md127
  VG Name               fedora_raid1
  PV Size               <7.28 TiB / not usable 0   
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              1907659
  Free PE               0
  Allocated PE          1907659
  PV UUID               JFXU1l-O8R0-q75g-e0iI-oaKY-oWZZ-IWBHje
   
  --- Physical volume ---
  PV Name               /dev/nvme0n1p3
  VG Name               fedora_root
  PV Size               <1.82 TiB / not usable 1.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              476634
  Free PE               308698
  Allocated PE          167936
  PV UUID               l81PQ1-jP4v-u4jv-mXAN-B7dy-X0Tf-SOaUr5

I think you have to create the LV then import the VG & LV into the other node before it is recognized since its identity defines the node where it was originally created. This can be seen with lvdisplay.

4

The man page for lvmlockd has some step-by-step directions. It says you need to set use_lvmlockd = 1 in lvm.conf and configure a unique host id for each of your servers.

1 Like
5

Thanks for the info. When I attempted to use pvcreate it complained about it being a partition and not the device itself. So hence why I changed it.

6

Thanks for the info. Yeah I was following that guide and made the changes to lvm.conf and lvmlocal.conf.

I’m about to give up on sharing using iscsi and clustered lvm.

7

I succeeded once long ago [1] by following the clusters from scratch guide. In the end though, I decided that the configuration was too complex. These days I use AoE (vblade) to share out my storage and I use ZFS to manage the redundancy. What works better will vary depending on what resources you have and what you are trying to do. There is no one correct way.

P.S. Just in case anyone comes across this and decides they want to try running ZFS on AoE storage, one gotcha that took me a while to figure out is that you need to increase the net buffers if you are running AoE over 10G ethernet: [2]

# cat /etc/sysctl.d/99-net_buffers.conf
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
net.core.rmem_default = 131072
net.ipv4.tcp_rmem = 4096 131072 16777216
net.ipv4.tcp_wmem = 4096 131072 16777216

The above has been working on my hardware and OS for many years. I didn’t land on those numbers scientifically though, I just increased them until things started working reliably. This problem was mentioned in another forum discussion somewhere long ago, but I doubt I’d ever be able to find that again.

  1. long ago ≈ 2011: [Linux-cluster] Error: "Failed actions: dlm:1_monitor_0/gfs-control:1_monitor_0 ... not installed". ↩︎

  2. I just noticed that the vblade repo has a comment in its README recommending that rmem_default be set to at least 278528. I’m going to update my configuration even though the lower setting (131072) has been working. ↩︎

1 Like
8

Maybe I should describe what I’m attempting to accomplish.

I have an incus cluster setup and running containers and vms on these three nodes. Currently, I’m using local storage on each node for the containers and vms. My plan was to setup a shared storage location on my Synology NAS, making it easier to migrate containers and vms between the nodes. Also, the local storage is a limited size, so using the NAS storage for more room.

For reference: About storage pools, volumes and buckets - Incus documentation

I’m open to any suggestions on accomplishing my goals.

9

I don’t have any experience with the Incus cluster software. I don’t know what would work best with that system. Sorry.