Hard Drive Internals (Stock Image)
Note: It has been reported that there may be issues doing this with a drive larger than 2 TB. Please see comments below. I have not tested with a larger drive so if you have one to try please do and let us know.
Background
In my home XenServer I have a drive that is dedicated for my OpenMediaVault (OMV) NAS virtual machine (VM). I originally had this drive provisioned as an EXT3 storage repository with the disk allocated to single large VHD that I kept growing until it was almost as large as the physical disk. However, I eventually wanted to do something in order to pass the whole disk through to the VM. I searched and searched online and it seems like there are basically three approaches to this configuration:
- Use PCIe pass-through to give an entire RAID controller to the VM
- Add an entry to udev.d scripts to link a disk into the XenServer removable disk repository
- Create a separate udev repository and manually link the disk to a user created directory
Option (1) was definitely overkill for my home server since I don’t have a nice RAID card, or any plans to put one physically in the XenServer host. Options (2) and (3) were both found at http://techblog.conglomer.net/sata-direct-local-disk-access-on-xenserver/.
Option 2 is basically the following code added to /etc/udev/rules.d/50-udev.rules. These rules take the device and have the server create the link from /dev to the XenServer hosts removable device repository on boot. You just make sure and replace “sdb” with the appropriate /dev entry for your disk.
ACTION=="add", KERNEL=="sdb", SYMLINK+="xapi/block/%k", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'" ACTION=="remove", KERNEL=="sdb", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'"
The reason I don’t love the option (2) approach is that if you stick your disk into the built in removable storage repository you can’t get any separate performance reporting for it, and its just named incorrectly. As a programmer by training it really bothers me when a variable is poorly named.
My Preferred Approach to XenServer Whole Disk Passthrough
I therefore have settled on option (3) as my preferred method. Option (3) is basically a manual version of option (2) where you also create your own storage repository for the disk. Option (3) was offered by the user “makstex” in a comment on the same post at http://techblog.conglomer.net/sata-direct-local-disk-access-on-xenserver/#comment-6404. Option (3) requires running the following commands from the XenServer host shell.
mkdir /srv/YOUR_SR_NAME xe sr-create name-label=”MY BLOCK SR” name-description=”MY BLOCK SR” type=udev content-type=disk device-config:location=/srv/YOUR_SR_NAME ln -s /dev/sdb /srv/YOUR_SR_NAME/sdb xe sr-scan uuid=YOUR_NEW_UUID xe vdi-list sr-uuid=YOUR_NEW_UUID
The actions in the lines above are:
- Create a new directory under /srv for your new storage repository. You can choose another directory location if you prefer.
- Create a new Storage Repository (SR) with the name of your choosing that points to the directory you just created
- Create a soft link from the /dev entry for your disk to the directory of the SR
- Tell XenServer to scan the new repository
- List the disks in the new repository
You can now use the XenCenter GUI or other management tool to add the disk in the SR to the VM of your choice. I have tested this and the storage repository and mapping persist across reboots. However, the configuration won’t persist across all upgrades to make sure and document this so you can put it back after an upgrade.
I can say that the performance in this configuration is pretty close to theoretical maximums for the disk in my testing. The other peace of mind I have is that if the XenServer host gives up the ghost, I can take this disk as is and put it in another server to read from the filesystem on the disk directly.
Hello
Attention!
I lost all my data using your preferred way of passthrough-ing a whole disk to a VM. Why? Cause Xenserver (I used version 7) has a limit of disk-size about 2TB. Each bigger drive will by destroyed without prompting. The data is still there, but the partition-table is corrupted, so you have to use something like testdisk to get your data back.
I had to do long-lasting testdisk-recovery to get my data back. I’m not finished with it, but I have some effort.
So please be sure to only passthrough-ing drives smaller than 2TB!
Best regards
I don’t have a drive larger than 2 TB to test with but I am a little surprised by that. Do you know for sure it wasn’t the guest that corrupted the partition table?
I am currently passing through 4 separate 4TB drives to one (OpenSUSE) VM using your preferred method. It works perfectly and I had no issues with data loss, but the drives are using GPT tables as opposed to MBR partition tables.
Oh … and I am using XCP-ng 8.0…
Thanks for the info
So I have 3 2tb drives and followed your guide. I see one drive but not the other two. Do I need to run this for the other /dev/sd*?
If you are using this method you will need to make a link for each drive in /dev (step 3). You have two options if using multiple disks: you can either place them all in the same directory which means they will wind up in the same storage repository in XenServer. Alternatively you can make a separate directory for each disk and place them in separate SRs.
You will probably want to find some way to easily identify the disks so that you can replicate this through upgrades since you have multiple large disks. Perhaps you can link by UUID of the disk instead.
You sir are a gentleman and a scholar!
Zero issues if your drives are below 2tb each. I think the 2tb limit comes from the 32bit partition limit. My commands are listed below:
mkdir /srv/RAID
xe sr-create name-label=RAID name-description=RAID type=udev content-type=disk device-config:location=/srv/RAID
ln -s /dev/sda /srv/RAID/sda
ln -s /dev/sdb /srv/RAID/sdb
ln -s /dev/sdc /srv/RAID/sdc
No need for step 3 and 4 if your managing from Xenserver windows GUI. Just click rescan in your new SR and they appear.
Next step.
How do we get the drive information to the VM? I have OpenMediaVault like you but its not reporting any information about the drives.
You will still need to attach the disk to your virtual machine in the Xenserver management GUI or through the CLI. Then they should show up as a normal drive in OMV.
hello zero
using your preferred way on xenserver7.0 it works. but it get problem on xenserver7.1(with all hotfix) , when reboot the VM the data in the disk will lost.
I stumbled upon it when someone was talking about it for ESXI as a way to attach disks, so I did a little digging if I could do it in xenserver.
Hi
I was looking for a persistent naming solution, and found out that symbolic linking to /dev/sd* actually doesn’t work at all. For instance, if you do ln -s /dev/sda /srv/RAID/sdb, Xenserver will show /dev/sdb as passthrough disk, and not sda. It seems that it just *reads* the name of the files under /srv/RAID. In fact, when I did a ‘touch /srv/RAID/sdb’, Xenserver used sdb as PT disk. Does anyone has an explanation for this behaviour? This way, it’s impossible to use persistent naming
Regards
Wim
I have a 5.25″ 4×2.5″ HDD Bay in which I can (hot)plug disks in and out
so to not mess up everything, I’m try to make the symlink like “ln -s /dev/disk/by-id/ata-TOSHIBA_MKxxxx… /srv/4×25/Toshi_750GB”.
But if I rescan in XenCenter it says “Unrecognised bus type”.
How may this be done?
I had this too. I was still able to attach the disks to the VM. It’s counter-intuitive because you think it’s a null entry. From the XenServer perspective, I guess it is, but it’s pointing to what’s important, and the VM acts accordingly.
Thank you very much. This How-To was exactly what i searched for.
I wanted to give a Hardware RAID5 over to a vm.
My first tought was just create a VDI with the full size…
Then i had to find a solution to get around that 2TB limit of virtual disks.
After some searching on google i found your site and how-to,
tried it out and it works great.
Now i can easily mount my 22TB RAID under my VM.
Thanks a lot and best regards!
Draven
Hello
I followed the directions from Option 3 to add a 1tb and 5tb disk. The names of each SR shows in XenCentre but no disks are present. Is there another step or steps I need to take? I’m using the latest XenServer 7.6
Appreciate any assistance, thank you.
It’s possible something has changed in Xenserver 7.6 that would interfere. What do you get when you run “ls -Al” in the directory you created for the repository? It sounds like maybe something is wrong with the soft link.
I had the same problem after updating XCP-ng. I had the order of the hard disks shifted where my storage was previously at /dev/sdd it was at /dev/sda after the update. I just had to adjust the soft link to the device and had my partition back. Maybe this will help with the problems 🙂
Hi, after following your i found out that the drives change after reboot. (not good for Spanned Drives). but when I deleted them, one drive started misbehaving.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Error code: SR_BACKEND_FAILURE_77
Error parameters: , Logical Volume group creation failed,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
after entering: xe sr-create name-label=6TB shared=false device-config:device=/dev/sdd type=ext content-type=user
Using xenserver 7.3, and so far so, good.