GNU Guix can be used as a “package manager” to install and upgrade
software packages as is familiar to GNU/Linux users, or as an
environment manager, but it can also provision containers or virtual
machines, and manage the operating system running on your machine.
One foundation that sets it apart from other tools in these areas is
reproducibility. From a high-level view, Guix allows users to
declare complete software environments and instantiate them. They can
share those environments with others, who can replicate them or adapt
them to their needs. This aspect is key to reproducible computational
experiments: scientists need to reproduce software environments before
they can reproduce experimental results, and this is one of the things
we are focusing on in the context of the
Guix-HPC effort. At a lower
level, the project, along with others in the Reproducible
Builds community, is working to ensure
that software build outputs are reproducible,
bit for bit.
Work on reproducibility at all levels has been making great progress.
Guix, for instance, allows you to travel back in
time.
That Guix can travel back in time and build software reproducibly is a
great step forward. But there’s still an important piece that’s missing
to make this viable: a stable source code archive. This is where
Software Heritage (SWH for short)
comes in.
When source code vanishes
Guix contains thousands of package definitions. Each package
definition
specifies the package’s source code URL and hash, the package’s
dependencies, and its build procedure. Most of the time, the package’s
source code is an archive (a “tarball”) fetched from a web site, but
more and more frequently the source code is a specific revision checked
out directly from a version control system.
The obvious question, then, is: what happens if the source code URL
becomes unreachable? The whole reproducibility endeavor collapses when
source code disappears. And source code does disappear, or, even
worse, it can be modified in place. At GNU we’re doing a good job of
having stable hosting that keeps releases around “forever”, unchanged
(modulo rare exceptions). But a lot of free software out there is
hosted on personal web pages with a short lifetime and on commercial
hosting services that come and go.
By default Guix would look up source code by hash in the caches of our
build farms. This comes for free: the “substitute”
mechanism
extends to all “build artifacts”, including downloads. However, with
limited capacity, our build farms do not keep all the source code of all
the packages for a long time. Thus, one could very well find oneself
unable to rebuild a package months or years later, simply because its
source code disappeared or moved to a different location.
Connecting to the archive
It quickly became clear that reproducible builds had “reproducible
source code downloads”, so to speak, as a prerequisite. The Software
Heritage archive is the missing piece that would finally allow us to
reproduce software environments years later in spite of the volatility
of code hosting sites. Software Heritage’s mission is to archive
essentially “all” the source code ever published, including version
control history. Its archive already periodically ingests release
tarballs from the GNU servers, repositories from GitHub, packages from
PyPI, and much more.
We quickly settled on a scheme where Guix would fall back to the
Software Heritage archive whenever it fails to download source code from
its original location. That way, package definitions don’t need to be
modified: they still refer to the original source code URL, but the
downloading machinery transparently goes to Software Heritage when
needed.
There are two types of source code downloads in Guix: tarball downloads,
and version control checkouts. In the former case, resorting to
Software Heritage is easy: Guix knows the SHA256 hash of the tarball so
it can look it up by hash using the /content endpoint of the
archive’s
interface.
Fetching version control checkouts is more involved. In this case, the
downloader would first resolve the commit identifier to obtain a
Software Heritage
revision. The
actual code for that revision is then fetched through the
vault.
The vault conveniently allows users to fetch the tarball corresponding
to a revision. However, not all revisions are readily available as
tarballs (understandably), so the vault has an interface that allows you
to request the “cooking” of a specific revision. Cooking is
asynchronous and can take some time. Currently, if a revision hasn’t
been cooked yet, the Guix download machinery will request it and wait
until it is available. The process can take some time but will
eventually succeed.
Success! At this point, we have essentially bridged the gap between the
stable archive that Software Heritage provides and the reproducible
software deployment pipeline of Guix. This code was
integrated in November 2018,
making it the first free software distribution backed by a stable
archive.
The challenges ahead
This milestone was encouraging: we had seemingly achieved our goal, but
we also knew of several shortcomings. First, even though the software
we package is still primarily distributed as tarballs, Software Heritage
keeps relatively few of these tarballs. Software Heritage does ingest
tarballs, notably those found on the GNU
servers, but the primary focus is on
preserving complete version control repositories rather than release
tarballs.
It is not yet clear to us what to do with plain old tarballs. On one
hand, they are here and cannot be ignored. Furthermore, some provide
artifacts that are not in version control, such as configure scripts,
and often enough they are accompanied by a cryptographic signature from
the developers that allows recipients to authenticate the code—an
important piece of information that’s often missing from version control
history. On the other hand, version control tags are increasingly
becoming the mechanism of choice to distribute software releases. It
may be that tags will become the primary mechanism for software release
distribution soon enough.
Version control tags turn out not to be ideal either, because they’re
mutable and per-repository. Conversely, Git commit identifiers are
unambiguous and repository-independent because they’re essentially
content-addressed, but our package definitions often refer to tags, not
commits, because that makes it clearer that we’re providing an actual
release and not an arbitrary revision (this is another illustration of
Zooko’s triangle).
This leads to another limitation that stems from the mismatch between
the way Guix and Software Heritage compute hashes over version control
checkouts: both compute a hash over a serialized representation of the
directory, but they serialize the directory in a different way (SWH
serializes directories as Git trees, while Guix uses “normalized
archives” or Nars, the format the build daemon manipulates, which is
inherited from Nix.) That prevents Guix from looking up revisions by
content hash. The solution will probably involve changing Guix to
support the same method as Software Heritage, and/or adding Guix’s method
to Software Heritage.
Having to wait for “cooking” completion can also be problematic. The
Software Heritage team is investigating the possibility to
automatically cook all version control
tags. That way, relevant
revisions would almost always be readily available through the vault.
Similarly, we have no guarantee that software provided by Guix is
available in the archive. Our plan is to extend Software
Heritage such that it would
periodically archive the source code of software packaged by Guix.
Going further
In the process of adding support for Software Heritage, Guix gained
Guile bindings to the Software Heritage HTTP
interface. Here’s a couple of
things we can do:
(use-modules (guix swh))
;; Check whether SWH has ever crawled our repository.
(define o (lookup-origin "https://git.savannah.gnu.org/git/guix.git"))
⇒ #<<origin> id: 86312956 …>
;; It did! When was its last visit?
(define last-visit
(first (origin-visits o)))
(date->string (visit-date last-visit))
⇒ "Fri Mar 29 10:07:45Z 2019"
;; Does it have our “v0.15.0” Git tag?
(lookup-origin-revision "https://git.savannah.gnu.org/git/guix.git" "v0.15.0")
⇒ #<<revision> id: "359fdda40f754bbf1b5dc261e7427b75463b59be" …>
Guix itself is a Guile library so when we combine the two, there are
interesting things we can do:
(use-modules (guix) (guix swh)
(gnu packages base)
(gnu packages golang))
;; This is our GNU Coreutils package.
coreutils
⇒ #<package coreutils@8.30 gnu/packages/base.scm:342 1c67b40>
;; Does SWH have its tarball?
(lookup-content (origin-sha256 (package-source coreutils))
"sha256")
⇒ #<<content> checksums: (("sha1" …)) data-url: …>
;; Our package for HashiCorp’s Configuration Language (HCL) is
;; built from a Git commit.
(define commit
(git-reference-commit
(origin-uri (package-source go-github-com-hashicorp-hcl))))
;; Is this particular commit available in the archive?
(lookup-revision commit)
⇒ #<<revision> id: "23c074d0eceb2b8a5bfdbb271ab780cde70f05a8" …>
We’re currently using a subset of this interface, but there’s certainly
more we could do. For example, we can compute archive coverage of the
Guix packages; we can also request the archival of each package’s source
code via the “save code”
interface—though
all this is subject to rate
limiting.
Wrap-up
Software Heritage support in Guix creates a bridge from the stable
source code archive to reproducible software deployment with complete
provenance tracking. For the first time it gives us a software package
distribution that can be rebuilt months or years later. This is
particularly beneficial in the context of reproducible science: finally
we can describe reproducible software environments, a prerequisite for
reproducible computational experiments.
Going further, we can provide a complete software supply tool chain with
provenance tracking that links revisions in the archive to
bit-reproducible build artifacts produced by Guix. Oh and Guix itself
is
archived,
so we have this meta-level where we could link Guix revisions to the
revisions of packages it provides… There are still technical challenges
to overcome, but that vision is shaping up.
About GNU Guix
GNU Guix is a transactional package
manager and an advanced distribution of the GNU system that respects
user
freedom.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.
In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through Guile
programming interfaces and extensions to the
Scheme language.
