Nixifying the Blog
Over the past year or so, I have hopped on the nix train. I am steadily working towards a full NixOS setup, but in the meantime the nix package manager has proved to be the best way I've ever found to manage cross-platform dependencies, whether for my own machines via home-manager or to manage isolated and complete development environments for a variety of projects. At SpecTrust, we have been using nix to manage dependencies for our linux and MacOS engineers for some time now, and it's been a pretty amazing success. I'm sure I'll write something about it in our engineering blog at some point, at which point I'll link to that from here.
Table of Contents
Why Nix
Nix is an optionally declarative, deterministic, cross-platform package manager.
A package installed via nix for a given system type (e.g. linux-amd64) is
guaranteed to be byte-for-byte compatible with the same package installed on
another system of the same type. Nix also supports checking in pinned versions
of package sources, ensuring that all installs over time get the same versions
of specified packages.
What's more, nix package installs are isolated from the rest of the system: a nix package installed and used in a development environment doesn't interfere with any other versions of that package installed on the system, nor does it interfere with any other versions of that package installed by nix. Outside of the context of the development environment, it's like the installed package doesn't even exist.
Essentially, nix brings the declarative nature, isolation, and cross-platform
support of something like npm or pip with virtual environments to system
packages also. This is a huge boon for shared development environments, where
you can be certain that everyone using nix is using exactly the same version of
any required dependencies, including compilers/runtimes (rustc, cargo,
node, python, etc.), classic shell utilities (sed, awk, make, etc.),
modern shell utilities (fd, jq, ripgrep, etc.), infrastructure tooling
(terraform, kubectl, etc.), and background services (postgres, vault,
etc.). It's hard to overstate how nice it is to not need to worry about whether
you can use modern features of bash, make, or awk, since you know that
even engineers running MacOS with its decrepit old BSD forks will have the same
versions of those tools installed that you do. It's also hard to overstate how
much nicer a simple nix invocation for installing all project dependencies
is compared to a long list of projects and utilities to go find in your system
package manager or brew or whatever.
With nix, I can be working on one project that uses Rust 1.53 and switch to another project that uses 1.56 seamlessly, without those installs touching my system rust. With direnv support, which we'll discuss below, this happens automagically in both my shell and my editor, meaning I never have to worry about switching virtual environments or toolchains or worrying about whether I've installed all the dependencies for a given project.
Okay, but it can't all be roses, right? Of couse not, it never is. If you'd like to be talked out of trying nix, see the Why not Nix section at the bottom.
Blogging with Nix
For this blog, I used the upcoming flakes feature, which is a first-party way of pinning dependencies and ensuring hermetic builds. If you'd like to use flakes, you'll need to activate them once you've installed nix. For other projects I have used niv, which is a third-party solution that does the same. Both work well, but flakes are the future, and they're really quite nice. I'll include a section later on how you'd accomplish the same basic thing with niv.
The Flake and Nothing But the Flake
If we could assume we're in the future where everyone is using nix with flakes,
the flake.nix file would be the only one we need. The shell.nix file that
lives alongside is only there for backwards compatibility. So, let's talk first
about the flake.
The general form of a flake is:
{
description = "...";
inputs = {...},
outputs = {...}: {...}
}
The description is just that, a string describing the project. We'll talk about each of inputs and outputs in turn.
Note that nix operations using flakes will only see files that have been staged in git!
inputs
inputs is an attrset (what other languages call a hashmap or a dictionary or
an associative array or an object or...). It describes what sources you'll be
deriving your packages from. For projects that don't need backwards
compatibility, only need to work on one system type, and don't need anything
outside of the standard nixpkgs package set, inputs can be as simple as:
inputs = {
nixpkgs.url = "nixpkgs/nixos-unstable";
}
This will pin your flake (via the flake.lock, which is produced whenever you
run anything) to the current unstable version of nixpkgs, which contains all
of the definitions for all of the packages you can install with nix. You can of
course pin it to any other branch you'd like, such as the most recent stable
branch, although unstable is fairly stable in practice, so for non critical
applications it's probably fine.
Usually, you'll want your flake to apply to more than just one system type (e.g.
linux-x86-64). Specifying that support in the outputs is a bit of a chore, so
you'll also often want to bring in the flake-utils library:
inputs = {
# Provides some nice helpers for multiple system compatibility
flake-utils.url = "github:numtide/flake-utils";
};
We'll talk more about what it does when we get to the output.
For backwards compatibility, there's the flake-compat library, which we will
also pull in:
inputs = {
# Proivdes legacy compatibility for nix-shell
flake-compat = { url = "github:edolstra/flake-compat"; flake = false; };
};
Finally, we use the wonderful rust overlay (an overlay just adds extra packages to nixpkgs or overrides attributes of packages already there) from oxalica, found here. We include the overlay the same way we've included everything else, so our inputs when all is said and done look like:
inputs = {
nixpkgs.url = "nixpkgs/nixos-unstable";
# Proivdes legacy compatibility for nix-shell
flake-compat = { url = "github:edolstra/flake-compat"; flake = false; };
# Provides some nice helpers for multiple system compatibility
flake-utils.url = "github:numtide/flake-utils";
# Provides rust and friends
rust-overlay.url = "github:oxalica/rust-overlay";
};
One thing to remember here is that dotted properties in the nix language are
shorthand for nested attrsets! So nixpkgs.url = "nixpkgs/nixos-unstable" is
equivalent to nixpkgs = { url = "nixpkgs/nixos-unstable"; }.
With our inputs defined, it's time to move on to our outputs!
outputs
The outputs section is a function that returns an attrset. Certain special
properties in the attrset are used by default by the nix commandline tools, but
the attrset can contain any properties, and nix tools can be pointed to those
properties manually if needed.
The general form of a function is arg: body, and outputs look like:
outputs = { self }: { }
This is a function that takes an attrset containing at least a self property.
self is a special property referring to the flake currently being evaluated.
Any other properties will be determined by your inputs: each input will be
passed into the outputs as a property, so nixpkgs will be passed as nixpkgs,
flake-compat as flake-compat, and so on.
There are two ways to access these inputs in the body of the outputs function.
The first is to assign a name to the entire input attrset and then access that,
so you might see something like:
outputs = inputs@{ self, ... }: { pkgs = import inputs.nixpkgs {}; }
The ... means that any attributes can be passed in, and inputs@ assigns the
entire attrset to the variable inputs.
The other way is to unpack the inputs directly:
outputs = { self, nixpkgs }: { pkgs = import nixpkgs {}; }
As far as I know, there's no real difference between the two, and which one you use is just personal preference.
The body of outputs must be an expression that evaluates to an attrset. The
attrset may contain any properties, but certain special properties have special
meanings, which you can see here.
For setting up a development environment, the one we care about is devShell.<system>,
which is used by the nix develop command to enter an isolated development
environment. The value of devShell for a supported system will generally be
the result of calling the mkShell function from nixpkgs with a set of
buildInputs, which are the packages you'd like to have available in your
development environment. For the contents of mkShell, you can search for
documentation on shell.nix, which is the older way of doing this. It uses the
same mkShell function, so anything you find there will apply here too.
The <system> part defines the build shell for a supported system. If I wanted
to support x86 linux and x86 Mac, for example, I would want to do something
like (this is the manual method, which is a pain):
outputs = { self, nixpkgs }:
# the let..in syntax allows you to define variables in the `let` block
# that will be available in the `in` block.
let
# call the `nixpkgs` function with the appropriate `system` to get
# the set of packages for that system
# macOS packages
darwinPkgs = import nixpkgs { system = "x86_64-darwin" };
# linux packages
linuxPkgs = import nixpkgs { system = "x86_64-linux" };
in
{
devShell.x86_64-darwin = darwinPkgs.mkShell {
buildInputs = [ darwinPkgs.bashInteractive darwinPkgs.fd darwinPkgs.gnumake ]
};
devShell.x86_64-linux = darwinPkgs.mkShell {
buildInputs = [ linuxPkgs.bashInteractive linuxPkgs.fd linuxPkgs.gnumake ]
};
};
Of course this is pretty repetitive, and we've got a whole programming language
at our disposal, so we can do better. Using the flake-utils library, which we
specified as part of our inputs, we can automatically define
properties for all systems that nix supports:
# This leads to exactly the same output as above, plus output for any other
# systems that nix supports
outputs = { self, nixpkgs, flake-utils }:
# eachDefaultSystem is a function. It is called with a function that takes
# one argument, system, and returns an attrset. eachDefaultSystem calls that
# function with each supported system, updates the attrsets to add the
# system qualifiers, and then merges the attrsets. so if I return an attrset
# { foo = "foo"; }, it will turn that into
# { foo.x86_64-linux = "foo"; foo.x86_64-darwin = "foo"; ... }
flake-utils.lib.eachDefaultSystem
# here we define a function that takes system as an argument and returns
# an attrset
(system:
# `inherit system` is the same as `system = system`
let pkgs = import nixpkgs { inherit system; };
in
# "unpacks" the pkgs attrset into the parent namespace, so we can say
# `fd` instead of `pkgs.fd`, and so on.
with pkgs;
{
devShell = mkShell {
buildInputs = [ bashInteractive fd gnumake ];
};
}
);
Once this is in place, you're done! You can run nix develop and you will be
dropped into a shell with bash, fd, and gnumake, whose versions will all
be determined by the pinned version of nixpkgs in flake.lock, which gets
created automatically based on the inputs.
The only difference between that and the real flake that the blog uses
is that the latter includes a few extra packages and applies the rust overlay.
The blog's invocation of eachDefaultSystem looks like this:
flake-utils.lib.eachDefaultSystem
(system:
let pkgs = import nixpkgs {
inherit system;
# This is just the `overlay` property on the `rust-overlay` import.
# `overlay` is one of the standard special outpus of flakes, so we're
# accessing that property of the rust-overlay flake!
overlays = [ rust-overlay.overlay ];
};
in
with pkgs;
{
devShell = mkShell {
# Packages required for development.
buildInputs = [
bashInteractive
fd
gnumake
# This actually reads our rust toolchain file to determine which
# version of rust, components, and targets to install.
(rust-bin.fromRustupToolchainFile ./rust-toolchain.toml)
watchexec
];
};
});
Backwards Compatibility
Of course, nix flakes are an upcoming feature, and still labeled as
experimental. Somebody who just rolled in off the street with regular old nix
installed won't be able to run your fancy flake. Luckily, someone has written a
shim for the classic shell.nix file that will evaluate your flake and provide
exactly what you need for a classic nix-shell invocation, which is the old way
to get an isolated local development environment.
The first thing you need to do is include flake-compat in your inputs and
include it as an explicitly named argument in your outputs, e.g.:
inputs = {
nixpkgs.url = "nixpkgs/nixos-unstable";
# Proivdes legacy compatibility for nix-shell
flake-compat = { url = "github:edolstra/flake-compat"; flake = false; };
};
outputs = { self, nixpkgs, flake-compat }: { }
Then the contents of shell.nix should be (you can grab a more recent version
of flake-compat and get its sha256 if desired, but the one below works):
# Legacy compat for folks not on nix with flakes.
#
# flake-compat reads the flake and provides shellNix (for nix-shell) and defaultNix
# (for nix-build). We only need shell here, since we're only using nix for the
# dev environment atm.
(import (
fetchTarball {
url = "https://github.com/edolstra/flake-compat/archive/99f1c2157fba4bfe6211a321fd0ee43199025dbf.tar.gz";
sha256 = "0x2jn3vrawwv9xp15674wjz9pixwjyj3j771izayl962zziivbx2"; }
) {
src = ./.;
}).shellNix
With this in place, you can run nix-shell and get into the same development
environment that you would with nix develop!
No Flakes
If you don't want to use flakes, niv works just as well. You can install niv
for your user with nix-env -iA nixpkgs.niv. You would then follow the
instructions on their repo to init your sources and pin them (essentially just
run niv init), and then you shell.nix would look something like:
# This file returns the evaluation of `mkShell`, which is the same thing we
# set to the `devShell` propety in the flake output above. The difference
# is that instead of our inputs being pinned by a flake, they are pinned in
# the `nix/sources.nix` file that niv generates.
let
sources = import ./nix/sources.nix;
# this is assuming you used the default `nixpkgs` name for nixpkgs. We can
# call this function with no arguments, since we don't have any overlays.
pkgs = import sources.nixpkgs {};
in
# "unpack" the pkgs attrset into the parent namespace, so we can use like
# `mkShell` instead of `pkgs.mkShell`
with pkgs;
mkShell {
buildInputs = [ bashInteractive fd gnumake ];
}
That's all you need. Wow! Isn't this much simpler than flakes? Why would anyone use flakes???
Well, niv hides quite a bit of complexity from you in the automatically
generated sources.nix file, so the flake is arguably simpler overall. In
addition, flakes provide a standard way to share nix packages other than
checking every single thing into nixpkgs, which is pretty nice. Flakes also
play nicely with all of the nix tooling. It also seems to be the general
direction the community is moving, so it is good to know regardless. Finally,
flakes go the extra mile to ensure that your build is hermetic and fully
encapsulated by the flake and its lockfile. You cannot use any inputs not
defined in the flake, and you cannot use impure operations in your evaluations.
While niv does a good job at pinning sources, it does not provide that extra
guarantee around the reproducibility of your build.
Direnv Support
direnv is a really neat program that automatically evaluates an .envrc
file in a directory whenever you enter that directory or any of its children.
To get it working, install direnv, ensuring that you've done the step adding
support for your shell. It is also really nice to add a direnv plugin to your
editor so that your editor environment will also evaluate the .envrc. There
are quality direnv plugins available for emacs, vim, jetbrains editors, and
vscode, as well as others.
The .envrc file for the blog looks like this:
# This file is loaded whenever someone `cd`s into the project directory or when
# an editor plugin does the same.
# Ensure that nix-shell packages and environment variables are set in the environment
if nix flake --help > /dev/null; then
use flake
elif command -v nix-env > /dev/null; then
# Use the `use nix` command from nix-direnv if available.
# nix-direnv caches the shell definition so it doesn't need to
# be calculated every time, significantly speeding up execution
# of the .ennvrc file. This assumes that nix-direnv has been
# installed via `nix-env` (via `make setup`). If you're using
# NixOS or have installed nix-direnv another way, feel free
# to add in checks for config files that might live elsewhere.
if [[ -e ~/.nix-profile/share/nix-direnv/direnvrc ]]; then
source ~/.nix-profile/share/nix-direnv/direnvrc
fi
# fall back to using direnv's builtin nix support
# to prevent bootstrapping problems.
use nix
fi
Update (2022-07-20): previously we defined our own use_flake() in the
.envrc file, but this is builtin to direnv now, so we can just run use flake.
When this file is evaluated, if the user has nix plus flakes installed, it will
evaluate the flake and set up the environment accordingly, meaning you don't
even need to run nix develop to use all your isolated dependencies. If the
user does not have nix with flakes, but does have nix, it will use direnv's
bulitin use nix command to do the same thing using shell.nix.
Finally, if the user does not have flakes and has nix-direnv installed, we are
sure to source it. nix-direnv caches the direnv evaluation so that it is only
re-evaluated if shell.nix or .envrc changes. This can be nice because
evaluation can get slow when your environment has a ton of packages. This isn't
needed for flakes, because flake evaluations are automatically cached.
Why not Nix
There are two main complaints most people have about nix, which I agree with to different degrees.
The first is that the nix expression language is weird. It's a dynamically typed, lazily evaluated, functional programming language. Just that would already be pretty weird for most people, but it also contains some odd syntactic choices. This one isn't a huge problem, though, because while it is confusing at first (and the nix community is awful at commenting their code to explain what's going on to newcomers), the language is relatively simple, with a small surface area and minimal syntactic constructs. It takes a bit to get used to, but once you're used it the language ceases to be a barrier.
The second and more serious complaint is that the documentation is terrible. There's a lot of it, but it can be really hard to find simple documentation for simple tasks you might want to accomplish when you're first getting started with nix. There are some blogs that attempt to fill the void, but there's no getting around the fact that there are a ton of libraries and idioms in use in nixpkgs and almost none of them are documented. This means that you often need to dive in and read code if you want to understand how to configure a package you're installing. Once you know the language this isn't too awful (I highly recommend checking out the nixpkgs repo locally so you can grep through it and read things in the comfortable context of your editor), but it's a hell of a high hill at first. It also doesn't help that the nix community is in the midst of a major shift from classical nix environments to flakes, which are essentially declarative files that specify all your inputs and pin them in a lockfile. I like flakes, but it sucks as a beginner when some examples and documentation are flake-oriented and some aren't, because the two contexts are relatively different.
Essentially, the complaints boil down to nix having a mighty steep learning curve. The technology is amazing, and once it's in place it's solid as a rock, but getting it there is a hassle. You'll be much better off if you've got a "nix person" you're able to ask questions of. The forums are pretty good, and I hear that the folks on Discord are also helpful. Don't hesitate to reach out!
Summary
Despite the poor documentation and the learning curve, nix is one of the best tools I've encountered. It is far and away the easiest way to prevent drift between the systems of different engineers working on a shared codebase that I know of, it's got pretty much every package under the sun, and it goes a long way towards making it easy to share a codebase across Linux and Mac operating systems. I have also moved almost all of my home management (i.e. installing the packages I use on all computers, setting up services, etc.) to nix via home-manager, and it's a really great fit for that, too. I anticipate continuing on in my nix journey for the foreseeable future.