Title: Nvidia card in eGPU and NixOS

	Title: Nvidia card in eGPU and NixOS
	Author: Solène
	Date: 05 December 2021
	Tags: linux games nixos egpu
	Description: How to setup an eGPU with NixOS using offloading prime
	nvidia feature

	# Updates

	* 2022-01-02: add entry about specialization and how to use the eGPU as
	a display device

	# Introduction

	I previously wrote about using an eGPU on Gentoo Linux. It was working
	when using the eGPU display but I never got it to work for accelerating
	games using the laptop display.

	Now, I'm back on NixOS and I got it to work!

	# What is it about?

	My laptop has a thunderbolt connector and I'm using a Razer Core X
	external GPU case that is connected to the laptop using a thunderbolt
	cable. This allows to use an external "real" GPU on a laptop but it
	has performance trade off and on Linux also compatibility issues.

	There are three ways to use the nvidia eGPU:

	- run the nvidia driver and use it as a normal card with its own
	display connected to the GPU, not always practical with a laptop
	- use optirun / primerun to run programs within a virtual X server on
	that GPU and then display it on the X server (very clunky, originally
	created for Nvidia Optimus laptop)
	- use Nvidia offloading module (it seems recent and I learned about it
	very recently)

	The first case is easy, just install nvidia driver and use the right
	card, it should work on any setup. This is the setup giving best
	performance.

	The most complicated setup is to use the eGPU to render what's
	displayed on the laptop, meaning the video signal has to come back from
	the thunderbolt cable, reducing the bandwidth.

	# Nvidia offloading

	Nvidia made work in their proprietary driver to allow a program to have
	its OpenGL/Vulkan calls to be done in a GPU that is not the one used
	for the display. This allows to throw optirun/primerun for this use
	case, which is good because they added performance penalty, complicated
	setup and many problems.

	Official documentation about offloading with nvidia driver


	# NixOS

	I really love NixOS and for writing articles it's so awesome, because
	instead of a set of instructions depending on conditions, I only have
	to share the piece of config required.

	This is the bits to add to your /etc/nixos/configuration.nix file and
	then rebuild system:

	```nixos configuration
	hardware.nvidia.modesetting.enable = true;
	hardware.nvidia.prime.sync.allowExternalGpu = true;
	hardware.nvidia.prime.offload.enable = true;
	hardware.nvidia.prime.nvidiaBusId = "PCI:10:0:0";
	hardware.nvidia.prime.intelBusId = "PCI:0:2:0";
	services.xserver.videoDrivers = ["nvidia" ];
	```

	A few notes about the previous chunk of config:
	- only add nvidia to the list of video drivers, at first I was adding
	modesetting but this was creating troubles
	- the PCI bus ID can be found with lspci, it has to be translated in
	decimal, here my nvidia id is 10:0:0 but in lspci it's 0a:00:00 with 0a
	being 10 in hexadecimal

	NixOS wiki about nvidia offload mode

	# How to use it

	The use of offloading is controlled by environment variables. What's
	pretty cool is that if you didn't connect the eGPU, it will still work
	(with integrated GPU).

	## Running a command

	We can use glxinfo to be sure it's working, add the environment as a
	prefix:

	```shell
	__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo
	```

	## In Steam

	Modify the command line of each game you want to run with the eGPU
	(it's tedious), by:

	```command line
	__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia %command%
	```

	## In Lutris

	Lutris has a per-game or per-runner setting named "Enable Nvidia
	offloading", you just have to enable it.

	# Advanced usage / boot specialisation

	Previously I only explained how to use the laptop screen and the eGPU
	as a discrete GPU (not doing display). For some reasons, I've
	struggled a LOT to be able to use the eGPU display (which gives more
	performance because it's hitting less thunderbolt limitations).

	I've discovered NixOS "specialisation" feature, allowing to add an
	alternative boot entry to start the system with slight changes, in this
	case, this will create a new "external-display" entry for using the
	eGPU as the primary display device:

	```
	hardware.nvidia.modesetting.enable = true;
	hardware.nvidia.prime.sync.allowExternalGpu = true;
	hardware.nvidia.prime.offload.enable = true;
	hardware.nvidia.prime.nvidiaBusId = "PCI:10:0:0";
	hardware.nvidia.prime.intelBusId = "PCI:0:2:0";
	services.xserver.videoDrivers = ["nvidia" ];

	# external display on the eGPU card
	# otherwise it's discrete mode using laptop screen
	specialisation = {
	external-display.configuration = {
	system.nixos.tags = [ "external-display" ];
	hardware.nvidia.modesetting.enable = pkgs.lib.mkForce false;
	hardware.nvidia.prime.offload.enable = pkgs.lib.mkForce false;
	hardware.nvidia.powerManagement.enable = pkgs.lib.mkForce false;
	services.xserver.config = pkgs.lib.mkOverride 0
	''
	Section "Module"
	Load "modesetting"
	EndSection

	Section "Device"
	Identifier "Device0"
	Driver "nvidia"
	BusID "10:0:0"
	Option "AllowEmptyInitialConfiguration"
	Option "AllowExternalGpus" "True"
	EndSection
	'';
	};
	};
	```

	With this setup, the default boot is the offloading mode but I can
	choose "external-display" to use my nvidia card and the screen attached
	to it, it's very convenient.

	I had to force the xserver configuration file because the one built by
	NixOS was not working for me.