A lot has happened since our Hello Hurd
post
beginning of April.
No, not nearly as much as we joked on April 1st
,
but more than enough to share and be proud of.
Building a Hurd virtual machine
As some of you noticed, the previous hacks to build a Hurd virtual
machine (VM) were
removed and no longer work; using Guix you can now build a GNU/Hurd VM
just like you would build a GNU/Linux VM:
guix system disk-image -t hurd-raw bare-hurd.tmpl
This cross-compiles all the relevant packages for GNU/Hurd—specifically the
i586-pc-gnu
triplet—and produces a VM image:
/gnu/store/n7jkfajw0fzp975hv0b9v18r9bbr961q-disk-image
You can build it and start it from your GNU/Linux machine with this
command:
qemu-system-i386 -enable-kvm -m 512 -snapshot -hda \
$(guix system disk-image -t hurd-raw bare-hurd.tmpl)
We are using this ready-made, minimal GNU/Hurd operating system
description
gnu/system/examples/bare-hurd.tmpl
that looks suprisingly familiar:
(use-modules (gnu) (gnu system hurd) (guix utils))
(use-service-modules ssh)
(use-package-modules ssh)
(define %hurd-os
(operating-system
(inherit %hurd-default-operating-system)
(bootloader (bootloader-configuration
(bootloader grub-minimal-bootloader)
(target "/dev/sdX")))
(file-systems (cons (file-system
(device (file-system-label "my-root"))
(mount-point "/")
(type "ext2"))
%base-file-systems))
(host-name "guixygnu")
(timezone "Europe/Amsterdam")
(packages (cons openssh-sans-x %base-packages/hurd))
(services (cons (service openssh-service-type
(openssh-configuration
(openssh openssh-sans-x)
(use-pam? #f)
(port-number 2222)
(permit-root-login #t)
(allow-empty-passwords? #t)
(password-authentication? #t)))
%base-services/hurd))))
%hurd-os
and it can be customized just like a GNU/Linux operating system
description. The end result is a full-blown Guix System with the
Shepherd managing system services
and all that—finally we can run
herd
on the Hurd.
A lot of things had to be in place to support this, we worked on
counting some ~200 patches by ten people over six months; including
generic cross-compilation fixes and support, and Hurd fixes and
support.
Also, we finished the passive translator settings over extended
attributes (xattrs) for the
Hurd
and for
Linux.
You may notice that we are using the new disk-image command rather
than the old vm-image. One of the big hurdles in producing a VM
image was the way Guix produces VM
images: it would run a target
QEMU, e.g. qemu-arm. That does not work for the Hurd, as there is
no qemu-hurd. Without going into the hairy details, when Ludo and
Janneke were—three patch sets, 50 messages and 13 days later—almost
ready to give up, Mathieu came to the rescue with his brand-new
implementation
of the disk-image command. At the time, Hurd work was done on the
wip-hurd branch and the disk-image work on wip-disk-image. Soon
after, Mathieu proposed an explosive
mix
of the two branches; we managed to create the first Hurd system that
really felt like Guix System.
The new implementation of the disk-image command was followed by the
introduction of an --image-type or -t option. This option allows to
produce disk images targeting different supports. The hurd-raw and
hurd-qcow2 image types, producing respectively a raw Hurd disk-image and a
Hurd QCOW2 disk-image were introduced. They can be used this way:
guix system disk-image -t hurd-raw bare-hurd.tmpl
guix system disk-image -t hurd-qcow2 bare-hurd.tmpl
This mechanism providing much more flexibility in the Guix System image
generation will be described in a future blog post.
We also offer downloads of continuously built (actually cross-built)
Guix System on
GNU/Hurd
(~350 MiB).
Substitutes
While amazing to be able to just run the Hurd in a VM, development without
substitutes
is a real pain; a Guix system needs substitutes. How to
go about that? We have a build farm but currently Hurd only runs on
ancient hardware: not an option. We need a machine running Guix/Hurd
to
offload
Hurd build jobs to.
The proposed solution was to automate the building and running of a
Guix VM into a new Guix service: the so-called hurd-vm or childhurd
service. We would add something like:
(service hurd-vm-service-type)
to a couple of our build nodes to add a hird of virtual Hurd machines
to our build farm.
The Childhurd service
The Hurd—being based on a microkernel—has some beautiful built-in
"virtualization" possibilites that have their own naming:
neighborhurds,
subhurds.
Similarly, we are adding
childhurds
to this mix: a childhurd is a GNU/Hurd VM running on GNU/Linux and
managed by Guix.
When you are running Guix System, building a Hurd VM manually is no
longer necessary. Just add the hurd-vm
service
to your operating
system description:
(service hurd-vm-service-type
(hurd-vm-configuration
(disk-size (* 12 (expt 2 30))) ;12GiB
(memory-size 1024))) ; 1GiB
and this will build a childhurd for you when you reconfigure your
system. This childhurd can be stopped and started just like any other
service:
herd stop hurd-vm
herd start childhurd
(childhurd is an alias for hurd-vm).
WARNING
It is highly addictive.
Having shaved this yak, let’s not lose sight that our initial goal was to
offload builds to those GNU/Hurd VMs. The childhurd service does all
the heavy lifting. To offload from GNU/Linux to my childhurd, I added
this to my /etc/guix/machines.scm:
(build-machine
(name "localhost")
(systems (list "i586-gnu"))
(host-key "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHZsrZ63zs+AhWbVJgYq6j1h2rgQGrWKCokpR2/Q/Jzy root@guixygnu")
(port 10022) ;the Hurd VM has SSH listening on that port
(user "root")
(private-key "/home/janneke/.ssh/id_rsa_childhurd"))
That can be used to transparently offload builds from GNU/Linux to the
childhurd:
$ uname -o
GNU/Linux
$ guix build hello -s i586-gnu
The following derivation will be built:
/gnu/store/jqdvjhcxxnbq370y8i2c973c9zfiqrgl-hello-2.10.drv
[…]
guix offload: sending 1 store item (12 MiB) to 'localhost'...
offloading '/gnu/store/jqdvjhcxxnbq370y8i2c973c9zfiqrgl-hello-2.10.drv' to 'localhost'...
[…]
/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
$ file -L $(guix build hello -s i586-gnu)/bin/hello
/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10/bin/hello: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /gnu/store/9vs3gkp6svam82zw7vjlml7iiarcs11c-glibc-2.31/lib/ld.so.1, for GNU/Hurd 0.0.0, not stripped
Hurrah!
Hurd substitutes
Last Friday we produced the very first GNU Hello
substitute
for the Hurd:
StorePath: /gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
URL: nar/gzip/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
Compression: gzip
FileSize: 61822
URL: nar/lzip/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
Compression: lzip
FileSize: 52887
NarHash: sha256:0g4k1kppjs5148ynm4zw4x1kpaby67npc3ws6s7y7hf0il1cgryk
NarSize: 204328
References: 803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10 9vs3gkp6svam82zw7vjlml7iiarcs11c-glibc-2.31 bwvd5338kfm0vsc4i9xvh48vdxr5ywrz-gcc-7.5.0-lib
System: i586-gnu
Deriver: jqdvjhcxxnbq370y8i2c973c9zfiqrgl-hello-2.10.drv
Signature: 1;berlin.guix.gnu.org;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjMDVGOEY5NjMxRUU5QzcxM0REQUNBRTYwNUNCNjJBNzlDNUY4NEVFQTIwMjc5OERBNTQ3NURCOUU2Q0FBRDMwMSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDVBMzkzMTgwOUY1RkQyMTdGMDM4MUVDMTJEODYyNzIyOEYyNjJGRDA4MTcxQjREMzZBNEM0RjBBNjZEQkY4NSMpCiAgIChzICMwQTc2RjZGNENCOTMzQTczNzA4QkNGMzRGREExMzkyOTRGQTQxREQzQTUwQkEwOUE0ODRCQUQyOTA4MjQ5ODIxIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjOEQxNTZGMjk1RDI0QjBEOUE4NkZBNTc0MUE4NDBGRjJEMjRGNjBGN0I2QzQxMzQ4MTRBRDU1NjI1OTcxQjM5NCMpCiAgICkKICApCiApCg==
For development, porting and fixing of packages, you can use a
Childhurd configuration like this:
(use-modules (srfi srfi-1) (guix packages) (guix records))
(use-package-modules base compression file gawk gdb hurd less m4 package-management)
(define guix-packages
(filter-map input->package
(fold alist-delete (package-direct-inputs guix)
'("glibc-utf8-locales" "graphviz" "po4a"))))
(operating-system
(inherit %hurd-vm-operating-system)
(users ...)
(packages
(cons* diffutils file findutils gawk gdb-minimal git-minimal
gnu-make grep gzip less m4 openssh-sans-x tar xz
(append
guix-packages
(delete guile-3.0 %base-packages/hurd)))))
… which allows working from a Git clone:
15:51:05 janneke@dundal:~/src/guix/master [env]
$ ssh -A janneke@childhurd
Last login: Sat Oct 3 15:31:55 2020 from 10.0.2.2
This is the GNU Hurd. Welcome.
janneke@childhurd ~$ git config --global url."git+ssh://git.sv.gnu.org/srv/git/".insteadOf gnu:
janneke@childhurd ~$ git clone gnu:guix
Cloning into 'guix'...
remote: Counting objects: 394436, done.
remote: Compressing objects: 100% (85728/85728), done.
remote: Total 394436 (delta 309572), reused 392294 (delta 307893)
Receiving objects: 100% (394436/394436), 137.05 MiB | 1.18 MiB/s, done.
Resolving deltas: 100% (309572/309572), done.
Updating files: 100% (2199/2199), done.
but before we continue let's first check the
weather:
janneke@childhurd ~$ guix weather
computing 11079 package derivations for i586-gnu...
looking for 11521 store items on https://ci.guix.gnu.org...
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
https://ci.guix.gnu.org
1.5% substitutes available (169 out of 11521)
at least 443.3 MiB of nars (compressed)
966.7 MiB on disk (uncompressed)
0.012 seconds per request (142.2 seconds in total)
81.0 requests per second
[..]
this gives an idea of how young this project is. Any day now, more
Hurd substitutes will
follow. Now let's configure and build Guix:
janneke@childhurd ~$ cd guix
janneke@childhurd ~/guix$ guix environment --bootstrap \
--ad-hoc gcc-toolchain@7 libgcrypt zlib
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
/gnu/store/gxq5flc8kwpn999dw9xxvldy9xfd0q2x-profile.drv
/gnu/store/vfkjnwgl4ckyklrl2z4q8x2vnlrwwyfr-gcc-toolchain-7.5.0.drv
/gnu/store/zh6snj49ayrpw24jn7whpzygj1fpy9cm-module-import-compiled.drv
66.3 MB will be downloaded
[..]
building profile with 3 packages...
janneke@childhurd ~/guix [env]$ ./bootstrap
[..]
janneke@childhurd ~/guix [env]$ ./configure --with-courage\
--localstatedir=/var --sysconfdir=/etc
[..]
janneke@childhurd ~/guix [env]$ make
[..]
janneke@childhurd ~/guix [env]$ ./pre-inst-env guix build hello
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
0.1 MB will be downloaded:
/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
substituting /gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10...
downloading from https://ci.guix.gnu.org/nar/lzip/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10 ...
hello-2.10 52KiB 258KiB/s 00:00 [##################] 100.0%
/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
just like we are used to do…almost. We are using --bootstrap and a
targeted --ad-hoc to avoid dependencies like libx11,
python-minimal, and other packages that do not build yet.
Isolated build environments
To help achieve reproducible
builds, Guix builds
packages in isolated build environments: build environments contain
nothing but the inputs explicitly declared in the package
definition—not even
/bin/sh.
Build environments also lack network access. On GNU/Linux this is
achieved by running builds in separate namespaces. Besides, the
environment contains device nodes and “special” file systems that are
usually expected to be available: /dev/null, /dev/pts, /dev/shm,
/proc, the loopback networking device, and so on. (The exact contents
are
documented.)
On GNU/Linux, these special files and file systems are implemented by
the kernel. Guix only cares about user-land software, meaning that
these devices “leak” from the host kernel instead of being an explicit
“input” of
derivations,
but that’s OK, that’s the deal: the kernel and hardware are considered
outside of Guix’s control.
What about GNU/Hurd, though? In GNU/Hurd, /dev/null, a
Linux-compatible /proc, the TCP/IP stack necessary to implement the
loopback device, and even support for pipes are all implemented in
user-land: writing to /dev/null amounts to talking to the /hurd/null
service (or translator), operations on AF_INET sockets translate to
remote procedure calls (RPCs) to /servers/socket/2, which the
/hurd/pfinet program listens to, and so on.
That raises an interesting question: what should the build environment
contain on GNU/Hurd? So far our GNU/Hurd builds were made in
non-isolated environments; we have just started
implementing support for isolated
builds but we’ll have to answer that question first. If we stick to our
approach—every piece of user-land software must be an explicit input of
the build process—then code that implements TCP/IP, /dev/null, or even
pipe should be an explicit input of any build process that needs those
facilities.
This principled approach can push the notion of controlled, reproducible
build environments to a whole new level. For example, we’ve
had
cases where the choice of the
root file system—e.g., ext4 vs. Btrfs—has an observable effect on
software behavior, leading to concrete issues such as test failures in
one case and not in the other. On GNU/Hurd, build processes could run
their own root file system, doing away with this kind of discrepancy.
On the other hand, there are practical issues that cannot be ignored:
virtually all build processes need these facilities so they’ll need to
be set up one way or another. Also, one could argue that things like
/dev/null have a well-defined interface that’s set in stone and that,
consequently, how they’re implemented does not matter at all. Can we
say the same of the TCP/IP stack though? Maybe not. A line needs to be
drawn somewhere.
We have yet to decide where to draw the line and to precisely define
what the build environment contains on GNU/Hurd. These questions are
closely related to bootstrapping issues we notably discussed at the
2019 Reproducible Builds
Summit.
Tricky, but exciting.
What's next?
In an earlier post we
tried to answer the question “Why bother with the Hurd anyway?” An
obvious question because it is all too easy to get
discouraged, to downplay or underestimate the
potential social impact of GNU and the Hurd.
We tried to make Hurd development as easy and as pleasant as we could.
As you have seen, things start to work pretty nicely and there is
still plenty of work to do in Guix. But in a way this is “merely
packaging” the amazing work of others. Some of the real work that
needs to be done and which is being discussed and is in progress
right now includes:
All these tasks look daunting, and indeed that’s a lot of work ahead.
But the development environment is certainly an advantage. Take an
example: surely anyone who’s hacked on device drivers or file systems
before would have loved to be able to GDB into the code, restart it, add
breakpoints and so on—that’s exactly the experience that the Hurd
offers. As for Guix, it will make it easy to test changes to the
micro-kernel and to the Hurd servers, and that too has the potential to
speed up development and make it a very nice experience.
Join #guix and #hurd on irc.freenode.net or the mailing
lists and get involved!
About GNU Guix
GNU Guix is a transactional package
manager and an advanced distribution of the GNU system that respects
user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, and AArch64 machines.
In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through Guile
programming interfaces and extensions to the
Scheme language.
About the GNU Hurd
The GNU Hurd is the GNU project's
replacement for the Unix kernel. It is a collection of servers that
run on the Mach microkernel to implement file systems, network
protocols, file access control, and other features that are
implemented by the Unix kernel or similar kernels (such as Linux).
More
info.
The mission of the GNU
Hurd
project is to create a general-purpose kernel suitable for the GNU
operating system, which is viable for everyday use, and gives users
and programs as much control over their computing environment as
possible.
