When I first created this blog, I didn’t think too much about how I was going to deploy it. I wanted something simple that just worked, so I went for the smallest VM Scaleway had to offer and called it a day. While it has worked pretty well I’ve forgotten the exact steps to get it to that point, and of course I don’t have any documentation as I was busy writing a blog post instead. To remedy that I’ve decided to move my blog to a new VM, document how (that’s why we’re here) and use NixOS for a portable and declarative deployment.
Why NixOS?
Let me start by quickly explaining what NixOS is. It’s a Linux distribution built around the Nix package manager, that uses the functional Nix language to describe what the system should look like. This enables the ability to create reproducible deployments and gives you a record of everything that is installed on the system. Every rebuild creates a new generation of the system that comes with it’s own boot entry, so in case something breaks it’s possible to boot into one of the previous generations to fix the issue or roll back. It becomes extra powerful when combining it with Git to keep track of the changes made to the system through time - and it makes it extremely easy to get back to a known good state.
NixOS has been my daily driver for more than two years at this point, and while the learning curve of getting into it was pretty steep even for Linux, it has been worth the effort. I don’t think I’ve ever had as few issues with my computer as I do right now. It’s not all roses and rainbows though - upgrading can sometimes be a hassle when breaking changes are introduced. However there’s only a new release every six months and I seldom spend more than a few hours upgrading.
So to answer the question: I chose NixOS because it solves my problem of forgetting how I set up my server, since everything is declared in the configuration. I also discovered that it was extremely easy to deploy to a remote machine which I definitely don’t take for granted. That it’s my preferred distribution which I use for all my other machines is just a nice bonus.
Into the fray
I chose to acquire a new VM for this project as I didn’t want my blog to experience any downtime during this migration. I ended up going with a VPS nano G11s from netcup because it was cheap, hosted in Europe and seems to cover my needs. At the time of writing netcup are also on the list of NixOS friendly hosters, which is where I found them in the first place. You can pick a bunch of different distros to install in the web interface once the VM has been provisioned and even custom images are supported. My machine came with Debian 13 out of the box which is a decent starting point as I’ll be using nixos-anywhere instead of a custom image to deploy the system.
Under the hood nixos-anywhere uses the system call kexec to boot into a NixOS installer from the currently running kernel, then performing the installation on the target host. Debian does not have kexec pre-installed so I need to connect to the remote host and install it manually.
apt install kexec-tools
While I was in there I also added my public key to /root/.ssh/authorized_keys to simplify the installation process
later. The machine is now ready for me to install NixOS via ssh, but I still need to write the configuration i want to
deploy.
Authoring the configuration
With the nixos-anywhere quick start
guide as a starting point, I created a
flake.nix that looks like this
{
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
inputs.disko.url = "github:nix-community/disko";
inputs.disko.inputs.nixpkgs.follows = "nixpkgs";
inputs.sops-nix.url = "github:Mic92/sops-nix";
inputs.sops-nix.inputs.nixpkgs.follows = "nixpkgs";
outputs =
{
nixpkgs,
disko,
sops-nix,
...
}:
{
nixosConfigurations.fafnir = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
disko.nixosModules.disko
sops-nix.nixosModules.sops
./configuration.nix
];
};
};
}It’s more or less a cleaned up version of what the guide provides. I did add sops-nix though as I find manual secret management to be somewhat of a headache and without it, it would be difficult to call this a tryly automated deployment. The other input ‘disko’ I’d heard about before, but never really looked at - it provides a way to declare disk partitioning using the Nix language, which is extremely cool and to be honest a huge relief as partitioning is one of the things I generally dread when installing Linux.
I haven’t yet declared my system though so let me do that now. First I’ll go over how I want the machine to be configured
- I want secrets to be managed by sops-nix
- I want caddy to act as a reverse proxy and expose my blog hosted in a container
- I want caddy to automatically manage Let’s Encrypt certificates for SSL
- I want to automatically maintain DNS records to match the IP of the host
Alright that doesn’t seem like too long of a list. I’ll only put the important parts of the code in this blog post and make the complete version available here. For brevity I’ll skip going into the details of secret management since it’s out of scope for this post, if I had to I’d probably just end up paraphrasing the sops-nix usage example anyway. With that disclaimer out of the way, the first thing on the agenda is to run my blog image in a container on the machine. I’ll create the podman user for running rootless containers, then enable podman and declare the container.
{config, pkgs, ...}:{
users.groups.podman = {
gid = 994;
};
users.users.podman = {
uid = 993;
group = "podman";
linger = true;
autoSubUidGidRange = true;
isSystemUser = true;
createHome = true;
home = "/home/podman";
};
virtualisation.podman.enable = true;
virtualisation.oci-containers = {
backend = "podman";
containers = {
anvo = {
image = "git.gitea.com/andreasvoss/anvo:v0.0.1";
podman.user = "podman";
ports = [
"127.0.0.1:4000:4000"
];
environment = {
TZ = "Etc/UTC";
PHX_HOST = "anvo.dk";
MAILJET_APIKEY_PATH = "/run/secrets/mailjet_apikey";
MAILJET_SECRET_PATH = "/run/secrets/mailjet_secret";
SECRET_KEY_BASE_PATH = "/run/secrets/anvo_secret_key";
};
login = {
registry = "https://git.gitea.com";
username = "andreasvoss";
passwordFile = "/run/secrets/gitea_cr_secret";
};
extraOptions = [
"--secret=anvo_secret_key"
"--secret=mailjet_apikey"
"--secret=mailjet_secret"
];
};
};
};
}
Notice that we are dealing with two different kinds of secrets in the snippet above. Ones that are mounted in the
container through podman using the extraOptions and the gitea_cr_secret that’s on the host system to pull images
from a private container registry. I had some issues mounting secrets from the host to the podman container, so I
created the following systemd service as a workaround. It will create the secrets if they don’t exist so I’m able to
mount them using the --secret flag in the extraOptions.
{config, pkgs, ...}:{
systemd.services.add-podman-secrets = with config.virtualisation.oci-containers; {
serviceConfig.Type = "oneshot";
serviceConfig.User = "podman";
wantedBy = [ "${backend}-anvo.service" ];
script = ''
# If this runs too early there is a binary "newuidmap" missing, so I retry for a bit..
create_podman_secret() {
${pkgs.podman}/bin/podman secret exists $1 || \
${pkgs.podman}/bin/podman secret create --replace $1 /run/secrets/anvo/$1 && \
echo "Successfully ensured existence of podman secret $1"
}
for i in $(seq 1 30); do
if create_podman_secret anvo_secret_key; then
break;
fi
echo "Failed to create podman secret - retrying in 0.5 seconds"
sleep 0.5
done
create_podman_secret mailjet_apikey
create_podman_secret mailjet_secret
'';
};
}I can now expose the app running inside the container using caddy. I’ve installed the package with the caddy-dns plugin which allows me to acquire a Let’s Encrypt certificate through a DNS challenge.
{config, pkgs, ...}:{
services.caddy = {
enable = true;
openFirewall = true;
package = pkgs.caddy.withPlugins {
plugins = [ "github.com/caddy-dns/[email protected]" ];
hash = "sha256-8yZDrejNKsaUnUaTUFYbarWNmxafqp2z2rWo+XRsxV8=";
};
virtualHosts."anvo.dk".extraConfig = ''
reverse_proxy /* localhost:4000
tls {
dns cloudflare {file./run/secrets/cloudflare_token}
resolvers 1.1.1.1
}
'';
};
}
The last piece of the puzzle is to automatically manage my DNS records. There’s a variety of ways to accomplish this,
and I think you can even configure caddy to handle it using another plugin. I went with the services.cloudflare-ddns
available in nixpkgs instead, which seems to work pretty well. I extract the virtualHosts from the caddy configuration
section and create an A record for each of them, so if I add other domains in the future, I don’t need to worry about
creating DNS records which is neat.
{config, pkgs, ...}:{
services.cloudflare-ddns = {
enable = true;
domains = builtins.attrNames config.services.caddy.virtualHosts;
recordComment = "Managed by Nix";
credentialsFile = /run/secrets/cloudflare-ddns-token;
proxied = "true";
updateOnStart = true;
updateCron = "0 */1 * * *";
deleteOnStop = true;
};
}With that my configuration done! The complete configuration is available here.
Now I really hope this works
With all that out of the way, I’m ready to install the operating system on my VM. Well almost. Because I’m using
sops-nix my secrets are encrypted, and I need a way to decrypt them on the server. In other words: I need a key
somewhere. It’s possible to provide the --extra-files flag to nixos-anywhere to copy files from the source to the
target starting at the root directory / on the target machine. I want to create a read-only file with the key at /root/keys.txt so I
run the following commands to create the file locally.
mkdir -p extra/root
echo "AGE-KEY" > extra/root/keys.txt # try to keep the secret out of your shell history please
chmod 400 extra/root/keys.txt
Alright, now I’m really ready to install! Since I installed kexec-tools on the target host earlier, all that remains
is to run nixos-anywhere. Because my local ssh key is encrypted the command will prompt me to unlock it.
nix run github:nix-community/nixos-anywhere -- --extra-files ./extra \
-i /home/andreasvoss/.ssh/id_rsa \
--flake .#fafnir \
--ssh-port 22 \
--target-host root@<HOST-IP>Voilà - after a little while the command does indeed let me know that the installation has completed successfully with the following message.
### Done! ###
The server is now ready! But what do I do in case I made a mistake in my configuration, do I reinstall the OS like that
again? No. Nixos-anywhere is meant for commissioning a new machine, if I do decide run it again all data on the target
will be lost. Instead I’ll run nixos-rebuild like I would for my workstation and provide the extra flag
--target-host.
nixos-rebuild switch --flake .#fafnir --target-host root@<HOST-IP>That’s it. I can now manage my remote machine using Nix.
Why did I wait this long
Well honestly I just thought there would be more friction with getting NixOS deployed in a remote VM. In hindsight it’s crazy that I haven’t attempted this before, because as I mentioned I’ve been using NixOS for a few years and really enjoyed it - with very few exceptions. I encourage you to attempt this if you think automating processes and playing around with Nix is fun. For me personally the satisfaction I get from running a single command then watching everything spin up without manual intervention is immense. I also think the time I invested in setting this up will pay dividends, as I can now freely move my blog wherever I want without worrying about forgetting how I set it up or being vendor locked for that matter! I could even self-host a VM if all other vendors fail me.