I’m thinking of setting up multi user nix on a compute cluster. The advantage would be to have a shared storage where common packages are reused, this is a great advantage compared to conda where every environment duplicates storage and inodes.
However, the packages are installed as root. As such I’m a bit wary of whether a user installing something could have the system run malware as root by installing a package.
What are the safeguards in place and how do I know I can trust them?
Assuming you’re talking about Nix the package manager,
Nothing is actually “installed as root”, except for stuff that the root user installs themselves. The Nix Store (where all the “derivations”/packages are stored) can be thought of more as a cache, just because a package is there doesn’t mean it’s used anywhere. Users don’t get to choose the cache “key” (i.e. the directory name in /nix/store) either - it is determined by all the build instructions and dependencies needed to build the package, and Nix doesn’t (well, at the very least shouldn’t) give users any control about the package build process after it starts.
When users install something, Nix fetches or builds that package into
/nix/store- which doesn’t affect other users in any way - and then just symlinks that particular package into some user-owned directory in their$PATH(e.g.$HOME/.local/share/nix/profiles/defaultor so), which also doesn’t affect root or other users in any way.So, basically, if some user installs malware only they are affected - the fact that it’s also in the shared
/nix/storeis irrelevant since there’s nothing in other user’s profiles or$PATHor whatever that references it in any way.The most likely vulnerability is something like this:
- The attacker guesses the nix store path of some package that root will use in the future (e.g. glibc from a more up-to-date Nixpkgs version compared to what root currently uses) - note that users can’t replace a derivation that’s already in the store, so they need to guess a future derivation path
- The attacker forces Nix to build that package from source instead of fetching from a trusted substituter
- The attacker finds a way to breach Nix’s sandbox during the build and inject their own backdoor into the resulting package ← this is the difficult part, there are currently no such known sandbox holes
- The attacker then waits until root starts using that package in the store, at which point the backdoor becomes actively installed on the system.
I must add that this is theoretical and I don’t think has ever happened in practice on a multi-user system.
Sorry, I’m pretty bad at explaining stuff, hopefully it makes some sense :)
Note that any user listed in
nix.settings.trusted-userscan place arbitrary directories into the Nix store, and can choose the store key.
Yes, although there have been a few CVE’s related to escalating privileges or breaking out of the sandbox. You’re going to want to keep those updating and keep on top of those.
There is one concern, in that the nix store (
/nix/store) is world readable. It is not world writable, which is good, but there is a problem in that secrets can potentially be copied into the nix store. If you copy a file containing environment variables or the like into the nix store, it could theoretically be found it. This one is on the users of nix to be careful of.With Nix flakes, the entire git repo that the flake originates from is copied into the nix store. Meaning if you have a nix flake in something that is supposed to be a private repo, or contains tracked sensitive data (untracked files are not copied into the nix store), then it could be found and inspected by other users.
My big concern with multi user in this case is not merely the Nix daemon though, but also Linux itself. Linux has been hit with a LOT of privilege escalation and container escape issues over the past few years, and many of them have been zero day’s. Given this, I no longer really have the same level of trust for Linux with regards to multi user isolation, for in cases like these.
Of course, in academic computing, I would probably just do multi user anyways, for simplicity. If you install tracking and monitoring, then you can attach malicious actions to identities. Because every user is registered and operates within the institution, if they break the terms of use for computing equipment, then you can punish them within the institution, or pursue greater legal action.
And then, you also would want to enforce 2FA to help minimize stolen accounts. While that can still be phished via fake login pages that ask for 2FA, it handles most of the things. Isolate the server via firewalls, and then it becomes a small enough target that doesn’t have enough value (assuming of course, your research isn’t too valuable and worth targeting), and you mostly don’t have to worry about it.
Sure, people will root it. But then they’ll send you a nicely worded email explaining how they rooted it and how to fix it. Which is what my friend did at my school, on our shared ssh server. Just keep it updated to handle the low hanging fruit.
However, if you are going to give it to untrusted users with unknown or temporary (not within the institution) identities, then things change, and you have to take it a lot more seriously. I no longer have confidence in just Linux’s user based isolation.
The first line solution I would go to, is to put users in containers and mount the nix store (and nix daemon) in and out. Something like a docker/podman container or Incus container. Of course, container escapes are still possible. If you are even more concerned about those, then you would want virtual machines. It is still possible to share the nix store between virtual machines, but it is more complicated.
In addition to that, virtual machines have a performance tradeoff (usually 95% or more of the host’s performance though), but there might be issues with sharing GPU compute resources among virtual machines, depending on the hardware you are using. Enterprise GPU’s usually support it though.
Do you want explain a bit more?
Linux is Linux. If you’re using the standard install packages like Pam, then it’s built specifically for multiple users.
Elaborate more if you could.
What’s your point? Nix is meant for CI, but even still the defaults are set for multi-user operations.
Nix is absolutely not just for CI, the idea is to use it locally as well, so the CI runs in the same environment.
A user only installs packages into their user environment (and into the nix store). It won’t be accidentally run by root.
However, the packages are installed as root. As such I’m a bit wary of whether a user installing something could have the system run malware as root by installing a package.
I’m pretty sure the packages are built as unprivileged users in a sandbox, and you can also restrict what users can use certain Nix features.
The main security flaw with a multi-user Nix system is that everything you build ends up in a world-readable store.
I’m pretty sure the packages are built as unprivileged users in a sandbox, and you can also restrict what users can use certain Nix features.
Are Nix packages sandboxed by default?
At runtime no, you’d need to use Nixpak or a similar tool
If you mean Nix, the package manager, then no packages are actually installed as root. Every package is installed in the Nix store, but programs that need root privileges can’t be run by any user.
One of the (many) great things about Nix is that users don’t need root privileges to install packages.
Thank you for the explanation. Yes, I’m talking about the nix package manager, not about the operating system.
As far as I understand packages are built by less privileges users, but libraries and binaries are owned by the root user in the nix store.
https://nix.dev/manual/nix/2.34/installation/multi-user.html
What I wonder is wheter this could lead to root executing untrusted code. I will have several users installing packages and I can not trust that they will check every package. Thus I’d like to know whether I can trust the system to be resilient without careful attention.
If one user installs a modified version of a bin or lib it won’t collide with the others.





