Computer Build EQUIPMENT GPU LINUX PROJECTS

GPU Passthrough on XCP-ng

USES UBUNTU 18.06.6 SERVER

This is a running log of an attempt to create a hashcat server on an XCP-ng server using PCI passthrough to an Ubuntu virtual machine.

NOTE: This is the GPU I’m using: https://www.nvidia.com/content/grid/pdf/grid_k1_bd-06633-001_v02.pdf

INSTALL ON XCP-NG AS A NEW SERVER

NOTE: Reference Download: https://releases.ubuntu.com/18.04.6/

After downloading the ISO, I simply created a VM with 12 cores and 48 GB RAM

Then, I installed hashcat

sudo apt install hashcat

On the new Ubuntu server VM
NOTE: driver version "367" is needed because of the age of the card

The following commands will download the appropriate driver and install them.

curl -s 'https://us.download.nvidia.com/XFree86/Linux-x86_64/367.134/NVIDIA-Linux-x86_64-367.134.run' -o 'NVIDIA-Linux-x86_64-367.134.run'

apt-get update

apt-get install -y build-essential kmod "linux-headers-$(uname -r)"

bash ./NVIDIA-Linux-x86_64-367.134.run --accept-license --silent --kernel-name "${linux_version}" --kernel-source-path "/usr/src/linux-headers-$(uname -r)"

Show the Hashcat baseline for each hash…

Hashcat Baseline

Use NVTOP to see the temp and load…

NVTOP

Only one (1) GPU core is showing and there are four (4)

NOTE: This is a 4-GPU / 16 GB card

This GPU has x4 cores and 16 GB or RAM. The passthrough is only allowing one core and 4GB of RAM. Soon, I’ll figure out how to passthrough all four cores!

Let’s give PCI Passthrough a try…

On the XCP-ng server (I use SSH)

NOTE: This will attempt to allow the VM to have direct access to the PCI slot
NOTE: Reference: https://xcp-ng.org/docs/compute.html#pci-passthrough

[20:16 DellR730XCPng ~]# lspci | grep NVIDIA
05:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID K1] (rev a1)
06:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID K1] (rev a1)
07:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID K1] (rev a1)
08:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID K1] (rev a1)
[20:17 DellR730XCPng ~]# xl pci-assignable-list
0000:06:00.0
[20:18 DellR730XCPng ~]# xe vm-list
uuid ( RO)           : ff8bc1bd-c42c-7c67-0ffa-03ab3ff94bf7
     name-label ( RW): Ubuntu_Hashcat
    power-state ( RO): halted

[SERVER HOST REDACTED]

[20:19 DellR730XCPng ~]# /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:05.00.0)(0000:06.00.0)(0000:07.00.0)(0000:08.00.0)"
[20:21 DellR730XCPng ~]# reboot

Now let’s try to add it to the VM

On the XCP-ng server (I use SSH)

[20:29 DellR730XCPng ~]# xe vm-param-set other-config:pci=0/0000:05.00.0,0/0000:06.00.0.0,0/0000:07.00.0,0/0000:08.00.0 uuid=ff8bc1bd-c42c-7c67-0ffa-03ab3ff94bf7
[20:29 DellR730XCPng ~]# xe vm-start uuid=ff8bc1bd-c42c-7c67-0ffa-03ab3ff94bf7

Nope, didn’t work yet

On the Ubuntu server

achilles@ubuntuhashcat:~$ sudo hashcat -b
[sudo] password for achilles: 
hashcat (v4.0.1) starting in benchmark mode...

Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.

* Device #1: Not a native Intel OpenCL runtime. Expect massive speed loss.
             You can use --force to override, but do not report related errors.
No devices found/left.

Started: Fri Oct  1 20:32:07 2021
Stopped: Fri Oct  1 20:32:09 2021

Now I have to undo it…

On the XCP-ng server (I use SSH)

/opt/xensource/libexec/xen-cmdline --delete-dom0 xen-pciback.hide

Then reboot

Now let’s try to simply add the four vGPUs to the VM’s group

NOTE: Reference: https://xcp-ng.org/docs/cli_reference.html#gpu-commands

[17:21 DellR730XCPng ~]# xe pgpu-list
uuid ( RO)              : 756d9871-4873-4d61-8957-21f25e323d6f
       vendor-name ( RO): NVIDIA Corporation
       device-name ( RO): GK107GL [GRID K1]
    gpu-group-uuid ( RW): 696971d0-c8a1-3551-709c-22b6df4b771c

uuid ( RO)              : 72b135cc-317a-4517-b255-0b93108fa631
       vendor-name ( RO): NVIDIA Corporation
       device-name ( RO): GK107GL [GRID K1]
    gpu-group-uuid ( RW): 696971d0-c8a1-3551-709c-22b6df4b771c

uuid ( RO)              : a56dcbc8-d732-44ad-8621-1af8ae6da0d5
       vendor-name ( RO): NVIDIA Corporation
       device-name ( RO): GK107GL [GRID K1]
    gpu-group-uuid ( RW): 696971d0-c8a1-3551-709c-22b6df4b771c

uuid ( RO)              : 8da0e340-794c-4fdf-933f-102902e83b99
       vendor-name ( RO): NVIDIA Corporation
       device-name ( RO): GK107GL [GRID K1]
    gpu-group-uuid ( RW): 696971d0-c8a1-3551-709c-22b6df4b771c

uuid ( RO)              : 85e3177c-2220-4050-9b5e-b2655ad38e69
       vendor-name ( RO): Matrox Electronics Systems Ltd.
       device-name ( RO): G200eR2
    gpu-group-uuid ( RW): 0ed92c2c-7d46-e148-a3a7-252c86cb32b6

This shows the four GPU cores. The hashcat command above shows that only 4GB is being issued to each core, and only one is being assigned through the XCP-ng GPU passthrough.

To be continued…

Scroll to top