Planet GNU

Aggregation of development blogs from the GNU Project

April 01, 2023

GNU Guix

Reinstating an iconic error message

Software development is a social process. What might be a “bug” for someone might well be a “feature” for someone else. The Guix project rediscovered it the hard way when, after “fixing a bug” that had been present in Guix System for years, it was confronted with an uproar in its user base.

In this post we look at why developers considered the initial behavior a “bug”, why users on the contrary had come to rely on it, and why developers remained blind to it. A patch to reinstate the initial behavior is being reviewed. This post is also an opportunity for us Guix developers to extend our apologies to our users whose workflow was disrupted.

The crux of the matter

Anyone who’s used Guix System in the past has seen this message on the console during the boot process:

error in finalization thread: Success

The following picture shows a typical boot screen (with additional messages in the same vein):

Picture of a monitor showing the error/success boot message.

If you have never seen it before, it may look surprising to you. Guix System users lived with it literally for years; the message became a hint that the boot process was, indeed, successful.

A few months ago, a contributor sought to satisfy their curiosity by finding the origin of the message. It did look like a spurious error message, after all, and perhaps the right course of action would be to address the problem at its root—or so they thought.

As it turns out, the message originated in Guile—check out the Guile manual if you’re curious about finalization. Investigation revealed two things: first, that this perror call in Guile was presumably reporting the wrong error code—this was fixed.

The second error—the core of the problem—lied in Guix System itself. Remember that, in its quest of memory safety™, statelessness, and fun, Guix System does it all in Guile Scheme—well, except for the kernel (for now). As soon as Linux has booted, Guix System spawns Guile to run boot code that’s in its initial RAM disk (“initrd”). Right before executing shepherd, its service manager, as PID 1, the initrd code would carelessly close all the file descriptors above 2 to make sure they do not leak into PID 1. The problem—you guessed it—is that one of them was the now-famous file descriptor of the finalization thread’s pipe; the finalization thread would quickly notice and boom!

error in finalization thread: Success

Our intrepid developers thought: “hey, we found it! Let’s fix it!”. And so they did.

Breaking user workflows

This could have been the end of the story, but there’s more to it than software. As Xkcd famously captured, this was bound to break someone’s workflow. Indeed, had developers paid more attention to what users had to say, they would have known that the status quo was preferable.

For some time now, users had shown that they held the error/success message deep in their heart. The message was seen on the blackboard at the Ten Years of Guix celebration, as a motto, as a rallying cry, spontaneously put on display:

Picture of a blackboard with the famous message (by Julien Lepiller, under CC0).

What’s more, a fellow NixOS hacker and Guix enthusiast, beguiled by this powerful message, designed stickers and brought them to FOSDEM in February 2023:

Picture of error/success stickers (under CC0).

The sticker design builds upon the “test pilot” graphics made by Luis Felipe for the 1.3.0 release. The test pilot has a bug on its helmet. In a way, the drawing and error message both represent, metaphorically, a core tenet of Guix as a project; just like Haskell is avoiding success at all costs, Guix seems trapped in an error/success quantum state.

Had it gone too far? Was calling it a “bug” the demonstration of the arrogance of developers detached from the reality of the community?

Fixing our mistakes

Those who installed Guix System starting from version 1.4.0 have been missing out on the error/success boot message. The patch submitted today finally reinstates that message. The review process will determine whether consensus is to enable it by default—as part of %base-service—or whether to make it optional—after all, we also need to accommodate the needs of new users who never saw this message. This will allow users to restore their workflow, while also ensuring that those freshly printed stickers remain relevant.

This incident had broader consequences in the project. It led some to suggest that we, finally, set up a request-for-comment (RFC) kind of process that would give all the community a say on important topics—a process most large free software projects have developed in one form or another. Such a process could have prevented this incident: instead of arrogantly labeling it as a “bug”, developers would have proposed an RFC to remove the message; the discussion period, most likely, would have made it clear that removal was not a desirable outcome and we would all have moved on.

This incident made many users uncomfortable, but we are glad that it is now being addressed. The lessons learned will be beneficial to the project for the years to come.

Picture of a metal bird holding an error/success sticker (under CC0).

Credits

Test pilot by Luis Felipe distributed under the terms of CC-BY-SA 4.0; sticker design distributed under CC-BY-SA 4.0 as well. Blackboard picture by Julien Lepiller under CC0; sticker pictures under CC0.

Many thanks to the anonymous sticker provider!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

01 April, 2023 08:57AM by Ludovic Courtès

March 31, 2023

GNUnet News

GNUnet 0.19.4

GNUnet 0.19.4

This is a bugfix release for gnunet 0.19.3. Special thanks goes out to ulfvonbelow who provided an array of patches. This is not an April Fool's joke.

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

A detailed list of changes can be found in the git log , the NEWS and the bug tracker .

31 March, 2023 10:00PM

March 29, 2023

FSF Events

LibrePlanet workshop - April 24 - The immortal cookbook by Adam Monsen

29 March, 2023 08:55PM

March 28, 2023

LibrePlanet Workshop - April 17 - Digital Colonialism, Surveillance Capitalism, and a Libre Software Future by Jose Castro

28 March, 2023 10:55PM

LibrePlanet workshop - April 10 - Newk script: Code katas to learn programming by Reynaldo Cordero

28 March, 2023 10:50PM

parted @ Savannah

parted-3.5.28 released [alpha]

I have released an alpha version of parted-3.5.28

Here are the compressed sources and a GPG detached signature[*]:
  http://alpha.gnu.org/gnu/parted/parted-3.5.28.tar.xz
  http://alpha.gnu.org/gnu/parted/parted-3.5.28.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA256 checksums:

af8a880df2e7b577c99ed9ee27a38e3f645896de8354dbfc05d8e81179a6d6dc  parted-3.5.28.tar.xz
49e8c4fc8aae92d8922f39aaae1fcdb0c8be3f3a80d34e006916e93a4a4852fc  parted-3.5.28.tar.xz.sig

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify parted-3.5.28.tar.xz.sig

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to update
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key bcl@redhat.com

  gpg --recv-keys 117E8C168EFE3A7F

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=parted&download=1' | gpg --import -

This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gettext 0.21
  Gnulib v0.1-5949-g480a59ba60
  Gperf 3.1

NEWS

  • Noteworthy changes in release 3.5.28 (2023-03-24) [alpha]


** New Features

  Support GPT partition attribute bit 63 as no_automount flag.

  Add type commands to set type-id on MS-DOS and type-uuid on GPT.

  Add swap flag support to the dasd disklabel

  Add display of GPT disk and partition UUIDs in JSON output


** Bug Fixes

  Fix use of enums in flag limits by switching to using #define

  Fix ending sector location when using kibi IEC suffix

28 March, 2023 06:32PM by Brian C. Lane

March 26, 2023

a2ps @ Savannah

a2ps 4.15.3 released [stable]


GNU a2ps is an Any to PostScript filter.  Of course it processes plain
text files, but also pretty prints quite a few popular languages.

For more information, see https://www.gnu.org/software/a2ps/

This release is a minor bug-fix release; no pressing need to update unless
you’re affected by a bug it fixes (see the end of this message for details).


Here are the compressed sources and a GPG detached signature:
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.3.tar.gz
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.3.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

b2ae4016b789a198c50a2f1dc0fefc11bda18ebe  a2ps-4.15.3.tar.gz
0A6B4OtNy/LUlj2J4d8rtm9x5m1ztBUsQ8+YOOaq98c  a2ps-4.15.3.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.15.3.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.15.3.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5892-g83006fa8c9

NEWS

* Noteworthy changes in release 4.15.3 (2023-03-26) [stable]
 * Bug fixes:
   - Fix fixps to use GhostScript’s ps2write device instead of defunct
     pswrite.
 * Build:
   - Fix a problem building PDF version of manual.


26 March, 2023 08:39PM by Reuben Thomas

texinfo @ Savannah

Texinfo 7.0.3 released

We have released version 7.0.3 of Texinfo, the GNU documentation format. This is a minor bug-fix release.

It's available via a mirror (xz is much smaller than gz, but gz is available too just in case):

http://ftpmirror.gnu.org/texinfo/texinfo-7.0.3.tar.xz
http://ftpmirror.gnu.org/texinfo/texinfo-7.0.3.tar.gz

Please send any comments to bug-texinfo@gnu.org.

Full announcement:

https://lists.gnu.org/archive/html/bug-texinfo/2023-03/msg00087.html

26 March, 2023 12:49PM by Gavin D. Smith

March 25, 2023

Greg Casamento

Swift->ObjC interop

Some interesting notes. I will update this posting as i find more: * https://dart.dev/guides/libraries/objective-c-interop

25 March, 2023 10:27AM by Unknown (noreply@blogger.com)

Compatibility project almost complete

As the much villified theme for star trek enterprise says "its been a long road getting from there to here" i am almost done with all of the work that needed to be done to get us to Catalina compatibility in GNUstep. The reason this is still significant is because Apple hasn't made many changes to either the Foundation or AppKit APIs since then. I have been workinf hard over the last three years. All of the new classes are fully tested. Once this effort is completed I am going to focus on printing, which has always been a problem in GS. And possibly a "reference" distribution.

25 March, 2023 10:13AM by Unknown (noreply@blogger.com)

March 23, 2023

grep @ Savannah

grep-3.10 released [stable]


This is to announce grep-3.10, a stable release,
fixing a bug with -P and \d. TL;DR, grep-3.9 would do this:

  $ LC_ALL=en_US.UTF-8 grep -P '\d' <<< ٠١٢٣٤٥٦٧٨٩
  ٠١٢٣٤٥٦٧٨٩

It should print nothing, like it has always done.
For more detail, see https://lists.gnu.org/r/bug-grep/2023-03/msg00005.html

Thanks to Paul Eggert for catching the \D variant and to Bruno Haible
for assiduously tending gnulib and for testing grep on so many
different systems.

There have been 12 commits by 2 people in the 17 days since 3.9.

See the NEWS below for a brief summary.

Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Jim Meyering (8)
  Paul Eggert (4)

Jim
 [on behalf of the grep maintainers]
==================================================================

Here is the GNU grep home page:
    http://gnu.org/s/grep/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=grep.git;a=shortlog;h=v3.10
or run this command from a git-cloned grep directory:
  git shortlog v3.9..v3.10

Here are the compressed sources:
  https://ftp.gnu.org/gnu/grep/grep-3.10.tar.gz   (2.7MB)
  https://ftp.gnu.org/gnu/grep/grep-3.10.tar.xz   (1.7MB)

Here are the GPG detached signatures:
  https://ftp.gnu.org/gnu/grep/grep-3.10.tar.gz.sig
  https://ftp.gnu.org/gnu/grep/grep-3.10.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

  7d3d830703183532f0b66619f0b148827e86eda7  grep-3.10.tar.gz
  3nsh2OM0jqZWnG/Vc06QoxFp72JCnqPc5Ipvwd2F0mA=  grep-3.10.tar.gz
  b8413017681fcd6249e0d0fb9c78225944074f23  grep-3.10.tar.xz
  JO+ltZX7WnEAh5tRuIaKC7h6ccGD0CxMYCYzuIr2hVs=  grep-3.10.tar.xz

Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify grep-3.10.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096/0x7FD9FCCB000BEEEE 2010-06-14 [SCEA]
        Key fingerprint = 155D 3FC5 00C8 3448 6D1E  EA67 7FD9 FCCB 000B EEEE
  uid                   [ unknown] Jim Meyering <jim@meyering.net>
  uid                   [ unknown] Jim Meyering <meyering@fb.com>
  uid                   [ unknown] Jim Meyering <meyering@gnu.org>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key jim@meyering.net

  gpg --recv-keys 7FD9FCCB000BEEEE

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=grep&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify grep-3.10.tar.gz.sig

This release was bootstrapped with the following tools:
  Autoconf 2.72a.92-8db0
  Automake 1.16i
  Gnulib v0.1-5916-gf61570c0ef

NEWS

* Noteworthy changes in release 3.10 (2023-03-22) [stable]

** Bug fixes

  With -P, \d now matches only ASCII digits, regardless of PCRE
  options/modes. The changes in grep-3.9 to make \b and \w work
  properly had the undesirable side effect of making \d also match
  e.g., the Arabic digits: ٠١٢٣٤٥٦٧٨٩.  With grep-3.9, -P '\d+'
  would match that ten-digit (20-byte) string. Now, to match such
  a digit, you would use \p{Nd}. Similarly, \D is now mapped to [^0-9].
  [bug introduced in grep 3.9]


23 March, 2023 01:31AM by Jim Meyering

parallel @ Savannah

GNU Parallel 20230322 ('Arrest Warrant') released

GNU Parallel 20230322 ('Arrest Warrant') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  GNU parallel is magic, half of my work uses it, to the point where they're referenced and thanked in my thesis
    -- Best Catboy Key Grip @alamogordoglass@twitter

New in this release:

  • Better support for wide characters in --latest-line.
  • Support for rsync 3.2.7.
  • Bug fixes and man page updates.

News about GNU Parallel:

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel


GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:


About GNU SQL


GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients.

So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.


About GNU Niceload


GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

23 March, 2023 12:12AM by Ole Tange

March 20, 2023

FSF Blogs

coreutils @ Savannah

coreutils-9.2 released [stable]


This is to announce coreutils-9.2, a stable release.
See the NEWS below for a brief summary.

Thanks to everyone who has contributed!
There have been 209 commits by 14 people in the 48 weeks since 9.1.


Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Arsen Arsenović (1)     Jim Meyering (7)
  Bernhard Voelker (3)    Paul Eggert (90)
  Bruno Haible (1)        Pierre Marsais (1)
  Carl Edquist (2)        Pádraig Brady (98)
  ChuanGang Jiang (2)     Rasmus Villemoes (1)
  Dennis Williamson (1)   Stefan Kangas (1)
  Ivan Radić (1)          Álvar Ibeas (1)


Pádraig [on behalf of the coreutils maintainers]

==================================================================

Here is the GNU coreutils home page:
    http://gnu.org/s/coreutils/

For a summary of changes and contributors, see:
    http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=shortlog;h=v9.2
or run this command from a git-cloned coreutils directory:
    git shortlog v9.1..v9.2

To summarize the 665 gnulib-related changes, run these commands
from a git-cloned coreutils directory:
     git checkout v9.2
     git submodule summary v9.1

==================================================================

Here are the compressed sources:
  https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.gz   (14MB)
  https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.xz   (5.6MB)

Here are the GPG detached signatures:
  https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.gz.sig
  https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

  6afa9ce3729afc82965a33d02ad585d1571cdeef  coreutils-9.2.tar.gz
  ebWNqhmcY84g95GRF3NLISOUnJLReVZPkI4yiQFZzUg=  coreutils-9.2.tar.gz
  3769071b357890dc36d820c597c1c626a1073fcb  coreutils-9.2.tar.xz
  aIX/R7nNshHeR9NowXhT9Abar5ixSKrs3xDeKcwEsLM=  coreutils-9.2.tar.xz

Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify coreutils-9.2.tar.xz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096 2011-09-23 [SC]
        6C37 DC12 121A 5006 BC1D  B804 DF6F D971 3060 37D9
  uid           [ unknown] Pádraig Brady <P@draigBrady.com>
  uid           [ unknown] Pádraig Brady <pixelbeat@gnu.org>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key P@draigBrady.com

  gpg --recv-keys DF6FD971306037D9

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=coreutils&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify coreutils-9.2.tar.gz.sig

This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5857-gf17d397771
  Bison 3.8.2

==================================================================

NEWS

* Noteworthy changes in release 9.2 (2023-03-20) [stable]

** Bug fixes

  'comm --output-delimiter="" --total' now delimits columns in the total
  line with the NUL character, consistent with NUL column delimiters in
  the rest of the output.  Previously no delimiters were used for the
  total line in this case.
  [bug introduced with the --total option in coreutils-8.26]

  'cp -p' no longer has a security hole when cloning into a dangling
  symbolic link on macOS 10.12 and later.
  [bug introduced in coreutils-9.1]

  'cp -rx / /mnt' no longer complains "cannot create directory /mnt/".
  [bug introduced in coreutils-9.1]

  cp, mv, and install avoid allocating too much memory, and possibly
  triggering "memory exhausted" failures, on file systems like ZFS,
  which can return varied file system I/O block size values for files.
  [bug introduced in coreutils-6.0]

  cp, mv, and install now immediately acknowledge transient errors
  when creating copy-on-write or cloned reflink files, on supporting
  file systems like XFS, BTRFS, APFS, etc.
  Previously they would have tried again with other copy methods
  which may have resulted in data corruption.
  [bug introduced in coreutils-7.5 and enabled by default in coreutils-9.0]

  cp, mv, and install now handle ENOENT failures across CIFS file systems,
  falling back from copy_file_range to a better supported standard copy.
  [issue introduced in coreutils-9.0]

  'mv --backup=simple f d/' no longer mistakenly backs up d/f to f~.
  [bug introduced in coreutils-9.1]

  rm now fails gracefully when memory is exhausted.
  Previously it may have aborted with a failed assertion in some cases.
  [This bug was present in "the beginning".]

  rm -d (--dir) now properly handles unreadable empty directories.
  E.g., before, this would fail to remove d: mkdir -m0 d; src/rm -d d
  [bug introduced in v8.19 with the addition of this option]

  runcon --compute no longer looks up the specified command in the $PATH
  so that there is no mismatch between the inspected and executed file.
  [bug introduced when runcon was introduced in coreutils-6.9.90]

  'sort -g' no longer infloops when given multiple NaNs on platforms
  like x86_64 where 'long double' has padding bits in memory.
  Although the fix alters sort -g's NaN ordering, that ordering has
  long been documented to be platform-dependent.
  [bug introduced 1999-05-02 and only partly fixed in coreutils-8.14]

  stty ispeed and ospeed options no longer accept and silently ignore
  invalid speed arguments, or give false warnings for valid speeds.
  Now they're validated against both the general accepted set,
  and the system supported set of valid speeds.
  [This bug was present in "the beginning".]

  stty now wraps output appropriately for the terminal width.
  Previously it may have output 1 character too wide for certain widths.
  [bug introduced in coreutils-5.3]

  tail --follow=name works again with non seekable files.  Previously it
  exited with an "Illegal seek" error when such a file was replaced.
  [bug introduced in fileutils-4.1.6]

  'wc -c' will again efficiently determine the size of large files
  on all systems.  It no longer redundantly reads data from certain
  sized files larger than SIZE_MAX.
  [bug introduced in coreutils-8.24]

** Changes in behavior

  Programs now support the new Ronna (R), and Quetta (Q) SI prefixes,
  corresponding to 10^27 and 10^30 respectively,
  along with their binary counterparts Ri (2^90) and Qi (2^100).
  In some cases (e.g., 'sort -h') these new prefixes simply work;
  in others, where they exceed integer width limits, they now elicit
  the same integer overflow diagnostics as other large prefixes.

  'cp --reflink=always A B' no longer leaves behind a newly created
  empty file B merely because copy-on-write clones are not supported.

  'cp -n' and 'mv -n' now exit with nonzero status if they skip their
  action because the destination exists, and likewise for 'cp -i',
  'ln -i', and 'mv -i' when the user declines.  (POSIX specifies this
  for 'cp -i' and 'mv -i'.)

  cp, mv, and install again read in multiples of the reported block size,
  to support unusual devices that may have this constraint.
  [behavior inadvertently changed in coreutils-7.2]

  du --apparent now counts apparent sizes only of regular files and
  symbolic links.  POSIX does not specify the meaning of apparent
  sizes (i.e., st_size) for other file types, and counting those sizes
  could cause confusing and unwanted size mismatches.

  'ls -v' and 'sort -V' go back to sorting ".0" before ".A",
  reverting to the behavior in coreutils-9.0 and earlier.
  This behavior is now documented.

  ls --color now matches a file extension case sensitively
  if there are different sequences defined for separate cases.

  printf unicode \uNNNN, \UNNNNNNNN syntax, now supports all valid
  unicode code points.  Previously is was restricted to the C
  universal character subset, which restricted most points <= 0x9F.

  runcon now exits with status 125 for internal errors.  Previously upon
  internal errors it would exit with status 1, which was less distinguishable
  from errors from the invoked command.

  'split -n N' now splits more evenly when the input size is not a
  multiple of N, by creating N output files whose sizes differ by at
  most 1 byte.  Formerly, it did this only when the input size was
  less than N.

  'stat -c %s' now prints sizes as unsigned, consistent with 'ls'.

** New Features

  cksum now accepts the --base64 (-b) option to print base64-encoded
  checksums.  It also accepts/checks such checksums.

  cksum now accepts the --raw option to output a raw binary checksum.
  No file name or other information is output in this mode.

  cp, mv, and install now accept the --debug option to
  print details on how a file is being copied.

  factor now accepts the --exponents (-h) option to print factors
  in the form p^e, rather than repeating the prime p, e times.

  ls now supports the --time=modification option, to explicitly
  select the default mtime timestamp for display and sorting.

  mv now supports the --no-copy option, which causes it to fail when
  asked to move a file to a different file system.

  split now accepts options like '-n SIZE' that exceed machine integer
  range, when they can be implemented as if they were infinity.

  split -n now accepts piped input even when not in round-robin mode,
  by first copying input to a temporary file to determine its size.

  wc now accepts the --total={auto,never,always,only} option
  to give explicit control over when the total is output.

** Improvements

  cp --sparse=auto (the default), mv, and install,
  will use the copy_file_range syscall now also with sparse files.
  This may be more efficient, by avoiding user space copies,
  and possibly employing copy offloading or reflinking,
  for the non sparse portion of such sparse files.

  On macOS, cp creates a copy-on-write clone in more cases.
  Previously cp would only do this when preserving mode and timestamps.

  date --debug now diagnoses if multiple --date or --set options are
  specified, as only the last specified is significant in that case.

  rm outputs more accurate diagnostics in the presence of errors
  when removing directories.  For example EIO will be faithfully
  diagnosed, rather than being conflated with ENOTEMPTY.

  tail --follow=name now works with single non regular files even
  when their modification time doesn't change when new data is available.
  Previously tail would not show any new data in this case.

  tee -p detects when all remaining outputs have become broken pipes, and
  exits, rather than waiting for more input to induce an exit when written.

  tee now handles non blocking outputs, which can be seen for example with
  telnet or mpirun piping through tee to a terminal.
  Previously tee could truncate data written to such an output and fail,
  and also potentially output a "Resource temporarily unavailable" error.


20 March, 2023 03:53PM by Pádraig Brady

Andy Wingo

a world to win: webassembly for the rest of us

Good day, comrades!

Today I'd like to share the good news that WebAssembly is finally coming for the rest of us weirdos.

A world to win

WebAssembly for the rest of us

17 Mar 2023 – BOB 2023

Andy Wingo

Igalia, S.L.

This is a transcript-alike of a talk that I gave last week at BOB 2023, a gathering in Berlin of people that are using "technologies beyond the mainstream" to get things done: Haskell, Clojure, Elixir, and so on. PDF slides here, and I'll link the video too when it becomes available.

WebAssembly, the story

WebAssembly is an exciting new universal compute platform

WebAssembly: what even is it? Not a programming language that you would write software in, but rather a compilation target: a sort of assembly language, if you will.

WebAssembly, the pitch

Predictable portable performance

  • Low-level
  • Within 10% of native

Reliable composition via isolation

  • Modules share nothing by default
  • No nasal demons
  • Memory sandboxing

Compile your code to WebAssembly for easier distribution and composition

If you look at what the characteristics of WebAssembly are as an abstract machine, to me there are two main areas in which it is an advance over the alternatives.

Firstly it's "close to the metal" -- if you compile for example an image-processing library to WebAssembly and run it, you'll get similar performance when compared to compiling it to x86-64 or ARMv8 or what have you. (For image processing in particular, native still generally wins because the SIMD primitives in WebAssembly are more narrow and because getting the image into and out of WebAssembly may imply a copy, but the general point remains.) WebAssembly's instruction set covers a broad range of low-level operations that allows compilers to produce efficient code.

The novelty here is that WebAssembly is both portable while also being successful. We language weirdos know that it's not enough to do something technically better: you have to also succeed in getting traction for your alternative.

The second interesting characteristic is that WebAssembly is (generally speaking) a principle-of-least-authority architecture: a WebAssembly module starts with access to nothing but itself. Any capabilities that an instance of a module has must be explicitly shared with it by the host at instantiation-time. This is unlike DLLs which have access to all of main memory, or JavaScript libraries which can mutate global objects. This characteristic allows WebAssembly modules to be reliably composed into larger systems.

WebAssembly, the hype

It’s in all browsers! Serve your code to anyone in the world!

It’s on the edge! Run code from your web site close to your users!

Compose a library (eg: Expat) into your program (eg: Firefox), without risk!

It’s the new lightweight virtualization: Wasm is what containers were to VMs! Give me that Kubernetes cash!!!

Again, the remarkable thing about WebAssembly is that it is succeeding! It's on all of your phones, all your desktop web browsers, all of the content distribution networks, and in some cases it seems set to replace containers in the cloud. Launch the rocket emojis!

WebAssembly, the reality

WebAssembly is a weird backend for a C compiler

Only some source languages are having success on WebAssembly

What about Haskell, Ocaml, Scheme, F#, and so on – what about us?

Are we just lazy? (Well...)

So why aren't we there? Where is Clojure-on-WebAssembly? Where are the F#, the Elixir, the Haskell compilers? Some early efforts exist, but they aren't really succeeding. Why is that? Are we just not putting in the effort? Why is it that Rust gets to ride on the rocket ship but Scheme does not?

WebAssembly, the reality (2)

WebAssembly (1.0, 2.0) is not well-suited to garbage-collected languages

Let’s look into why

As it turns out, there is a reason that there is no good Scheme implementation on WebAssembly: the initial version of WebAssembly is a terrible target if your language relies on the presence of a garbage collector. There have been some advances but this observation still applies to the current standardized and deployed versions of WebAssembly. To better understand this issue, let's dig into the guts of the system to see what the limitations are.

GC and WebAssembly 1.0

Where do garbage-collected values live?

For WebAssembly 1.0, only possible answer: linear memory

(module
  (global $hp (mut i32) (i32.const 0))
  (memory $mem 10)) ;; 640 kB

The primitive that WebAssembly 1.0 gives you to represent your data is what is called linear memory: just a buffer of bytes to which you can read and write. It's pretty much like what you get when compiling natively, except that the memory layout is more simple. You can obtain this memory in units of 64-kilobyte pages. In the example above we're going to request 10 pages, for 640 kB. Should be enough, right? We'll just use it all for the garbage collector, with a bump-pointer allocator. The heap pointer / allocation pointer is kept in the mutable global variable $hp.

(func $alloc (param $size i32) (result i32)
  (local $ret i32)
  (loop $retry
    (local.set $ret (global.get $hp))
    (global.set $hp
      (i32.add (local.get $size) (local.get $ret)))

    (br_if 1
      (i32.lt_u (i32.shr_u (global.get $hp) 16)
                (memory.size))
      (local.get $ret))

    (call $gc)
    (br $retry)))

Here's what an allocation function might look like. The allocation function $alloc is like malloc: it takes a number of bytes and returns a pointer. In WebAssembly, a pointer to memory is just an offset, which is a 32-bit integer (i32). (Having the option of a 64-bit address space is planned but not yet standard.)

If this is your first time seeing the text representation of a WebAssembly function, you're in for a treat, but that's not the point of the presentation :) What I'd like to focus on is the (call $gc) -- what happens when the allocation pointer reaches the end of the region?

GC and WebAssembly 1.0 (2)

What hides behind (call $gc) ?

Ship a GC over linear memory

Stop-the-world, not parallel, not concurrent

But... roots.

The first thing to note is that you have to provide the $gc yourself. Of course, this is doable -- this is what we do when compiling to a native target.

Unfortunately though the multithreading support in WebAssembly is somewhat underpowered; it lets you share memory and use atomic operations but you have to create the threads outside WebAssembly. In practice probably the GC that you ship will not take advantage of threads and so it will be rather primitive, deferring all collection work to a stop-the-world phase.

GC and WebAssembly 1.0 (3)

Live objects are

  • the roots
  • any object referenced by a live object

Roots are globals and locals in active stack frames

No way to visit active stack frames

What's worse though is that you have no access to roots on the stack. A GC has to keep live objects, as defined circularly as any object referenced by a root, or any object referenced by a live object. It starts with the roots: global variables and any GC-managed object referenced by an active stack frame.

But there we run into problems, because in WebAssembly (any version, not just 1.0) you can't iterate over the stack, so you can't find active stack frames, so you can't find the stack roots. (Sometimes people want to support this as a low-level capability but generally speaking the consensus would appear to be that overall performance will be better if the engine is the one that is responsible for implementing the GC; but that is foreshadowing!)

GC and WebAssembly 1.0 (3)

Workarounds

  • handle stack for precise roots
  • spill all possibly-pointer values to linear memory and collect conservatively

Handle book-keeping a drag for compiled code

Given the noniterability of the stack, there are basically two work-arounds. One is to have the compiler and run-time maintain an explicit stack of object roots, which the garbage collector can know for sure are pointers. This is nice because it lets you move objects. But, maintaining the stack is overhead; the state of the art solution is rather to create a side table (a "stack map") associating each potential point at which GC can be called with instructions on how to find the roots.

The other workaround is to spill the whole stack to memory. Or, possibly just pointer-like values; anyway, you conservatively scan all words for things that might be roots. But instead of having access to the memory to which the WebAssembly implementation would spill your stack, you have to do it yourself. This can be OK but it's sub-optimal; see my recent post on the Whippet garbage collector for a deeper discussion of the implications of conservative root-finding.

GC and WebAssembly 1.0 (4)

Cycles with external objects (e.g. JavaScript) uncollectable

A pointer to a GC-managed object is an offset to linear memory, need capability over linear memory to read/write object from outside world

No way to give back memory to the OS

Gut check: gut says no

If that were all, it would already be not so great, but it gets worse! Another problem with linear-memory GC is that it limits the potential for composing a number of modules and the host together, because the garbage collector that manages JavaScript objects in a web browser knows nothing about your garbage collector over your linear memory. You can easily create memory leaks in a system like that.

Also, it's pretty gross that a reference to an object in linear memory requires arbitrary read-write access over all of linear memory in order to read or write object fields. How do you build a reliable system without invariants?

Finally, once you collect garbage, and maybe you manage to compact memory, you can't give anything back to the OS. There are proposals in the works but they are not there yet.

If the BOB audience had to choose between Worse is Better and The Right Thing, I think the BOB audience is much closer to the Right Thing. People like that feel instinctual revulsion to ugly systems and I think GC over linear memory describes an ugly system.

GC and WebAssembly 1.0 (5)

There is already a high-performance concurrent parallel compacting GC in the browser

Halftime: C++ N – Altlangs 0

The kicker is that WebAssembly 1.0 requires you to write and deliver a terrible GC when there is already probably a great GC just sitting there in the host, one that has hundreds of person-years of effort invested in it, one that will surely do a better job than you could ever do. WebAssembly as hosted in a web browser should have access to the browser's garbage collector!

I have the feeling that while those of us with a soft spot for languages with garbage collection have been standing on the sidelines, Rust and C++ people have been busy on the playing field scoring goals. Tripping over the ball, yes, but eventually they do manage to make within striking distance.

Change is coming!

Support for built-in GC set to ship in Q4 2023

With GC, the material conditions are now in place

Let’s compile our languages to WebAssembly

But to continue the sportsball metaphor, I think in the second half our players will finally be able to get out on the pitch and give it the proverbial 110%. Support for garbage collection is coming to WebAssembly users, and I think even by the end of the year it will be shipping in major browsers. This is going to be big! We have a chance and we need to sieze it.

Scheme to Wasm

Spritely + Igalia working on Scheme to WebAssembly

Avoid truncating language to platform; bring whole self

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Even with GC, though, WebAssembly is still a weird machine. It would help to see the concrete approaches that some languages of interest manage to take when compiling to WebAssembly.

In that spirit, the rest of this article/presentation is a walkthough of the approach that I am taking as I work on a WebAssembly compiler for Scheme. (Thanks to Spritely for supporting this work!)

Before diving in, a meta-note: when you go to compile a language to, say, JavaScript, you are mightily tempted to cut corners. For example you might implement numbers as JavaScript numbers, or you might omit implementing continuations. In this work I am trying to not cut corners, and instead to implement the language faithfully. Sometimes this means I have to work around weirdness in WebAssembly, and that's OK.

When thinking about Scheme, I'd like to highlight a few specific areas that have interesting translations. We'll start with value representation, which stays in the GC theme from the introduction.

Scheme to Wasm: Values

;;       any  extern  func
;;        |
;;        eq
;;     /  |   \
;; i31 struct  array

The unitype: (ref eq)

Immediate values in (ref i31)

  • fixnums with 30-bit range
  • chars, bools, etc

Explicit nullability: (ref null eq) vs (ref eq)

The GC extensions for WebAssembly are phrased in terms of a type system. Oddly, there are three top types; as far as I understand it, this is the result of a compromise about how WebAssembly engines might want to represent these different kinds of values. For example, an opaque JavaScript value flowing into a WebAssembly program would have type (ref extern). On a system with NaN boxing, you would need 64 bits to represent a JS value. On the other hand a native WebAssembly object would be a subtype of (ref any), and might be representable in 32 bits, either because it's a 32-bit system or because of pointer compression.

Anyway, three top types. The user can define subtypes of struct and array, instantiate values of those types, and access their fields. The life cycle of reference-typed objects is automatically managed by the run-time, which is just another way of saying they are garbage-collected.

For Scheme, we need a common supertype for all values: the unitype, in Bob Harper's memorable formulation. We can use (ref any), but actually we'll use (ref eq) -- this is the supertype of values that can be compared by (pointer) identity. So now we can code up eq?:

(func $eq? (param (ref eq) (ref eq))
           (result i32)
  (ref.eq (local.get a) (local.get b)))

Generally speaking in a Scheme implementation there are immediates and heap objects. Immediates can be encoded in the bits of a value, whereas for heap object the bits of a value encode a reference (pointer) to an object on the garbage-collected heap. We usually represent small integers as immediates, as well as booleans and other oddball values.

Happily, WebAssembly gives us an immediate value type, i31. We'll encode our immediates there, and otherwise represent heap objects as instances of struct subtypes.

Scheme to Wasm: Values (2)

Heap objects subtypes of struct; concretely:

(struct $heap-object
  (struct (field $tag-and-hash i32)))
(struct $pair
  (sub $heap-object
    (struct i32 (ref eq) (ref eq))))

GC proposal allows subtyping on structs, functions, arrays

Structural type equivalance: explicit tag useful

We actually need to have a common struct supertype as well, for two reasons. One is that we need to be able to hash Scheme values by identity, but for this we need an embedded lazily-initialized hash code. It's a bit annoying to take the per-object memory hit but it's a reality, and the JVM does it this way, so it must not be so terrible.

The other reason is more subtle: WebAssembly's type system is built in such a way that types that are "structurally" equivalent are indistinguishable. So a pair has two fields, besides the hash, but there might be a number of other fundamental object types that have the same shape; you can't fully rely on WebAssembly's dynamic type checks (ref.test et al) to be able to query the type of a value. Instead we re-use the low bits of the hash word to include a type tag, which might be 1 for pairs, 2 for vectors, 3 for closures, and so on.

Scheme to Wasm: Values (3)

(func $cons (param (ref eq)
                   (ref eq))
            (result (ref $pair))
  (struct.new_canon $pair
    ;; Assume heap tag for pairs is 1.
    (i32.const 1)
    ;; Car and cdr.
    (local.get 0)
    (local.get 1)))

(func $%car (param (ref $pair))
            (result (ref eq))
  (struct.get $pair 1 (local.get 0)))

With this knowledge we can define cons, as a simple call to struct.new_canon pair.

I didn't have time for this in the talk, but there is a ghost haunting this code: the ghost of nominal typing. See, in a web browser at least, every heap object will have its first word point to its "hidden class" / "structure" / "map" word. If the engine ever needs to check that a value is of a specific shape, it can do a quick check on the map word's value; if it needs to do deeper introspection, it can dereference that word to get more details.

Under the hood, testing whether a (ref eq) is a pair or not should be a simple check that it's a (ref struct) (and not a fixnum), and then a comparison of its map word to the run-time type corresponding to $pair. If subtyping of $pair is allowed, we start to want inline caches to handle polymorphism, but the checking the map word is still the basic mechanism.

However, as I mentioned, we only have structural equality of types; two (struct (ref eq)) type definitions will define the same type and have the same map word (run-time type / RTT). Hence the _canon in the name of struct.new_canon $pair: we create an instance of $pair, with the canonical run-time-type for objects having $pair-shape.

In earlier drafts of the WebAssembly GC extensions, users could define their own RTTs, which effectively amounts to nominal typing: not only does this object have the right structure, but was it created with respect to this particular RTT. But, this facility was cut from the first release, and it left ghosts in the form of these _canon suffixes on type constructor instructions.

For the Scheme-to-WebAssembly effort, we effectively add back in a degree of nominal typing via type tags. For better or for worse this results in a so-called "open-world" system: you can instantiate a separately-compiled WebAssembly module that happens to define the same types and use the same type tags and it will be able to happily access the contents of Scheme values from another module. If you were to use nominal types, you would't be able to do so, unless there were some common base module that defined and exported the types of interests, and which any extension module would need to import.

(func $car (param (ref eq)) (result (ref eq))
  (local (ref $pair))
  (block $not-pair
    (br_if $not-pair
      (i32.eqz (ref.test $pair (local.get 0))))
    (local.set 1 (ref.cast $pair) (local.get 0))
    (br_if $not-pair
      (i32.ne
        (i32.const 1)
        (i32.and
          (i32.const 0xff)
          (struct.get $heap-object 0 (local.get 1)))))
    (return_call $%car (local.get 1)))

  (call $type-error)
  (unreachable))

In the previous example we had $%car, with a funny % in the name, taking a (ref $pair) as an argument. But in the general case (barring compiler heroics) car will take an instance of the unitype (ref eq). To know that it's actually a pair we have to make two checks: one, that it is a struct and has the $pair shape, and two, that it has the right tag. Oh well!

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

But with all of that I think we have a solid story on how to represent values. I went through all of the basic value types in Guile and checked that they could all be represented using GC types, and it seems that all is good. Now on to the next point: varargs.

Scheme to Wasm: Varargs

(list 'hey)      ;; => (hey)
(list 'hey 'bob) ;; => (hey bob)

Problem: Wasm functions strongly typed

(func $list (param ???) (result (ref eq))
  ???)

Solution: Virtualize calling convention

In WebAssembly, you define functions with a type, and it is impossible to call them in an unsound way. You must call $car exactly 2 arguments or it will not compile, and those arguments have to be of specific types, and so on. But Scheme doesn't enforce these restrictions on the language level, bless its little miscreant heart. You can call car with 5 arguments, and you'll get a run-time error. There are some functions that can take a variable number of arguments, doing different things depending on incoming argument count.

How do we square these two approaches to function types?

;; "Registers" for args 0 to 3
(global $arg0 (mut (ref eq)) (i31.new (i32.const 0)))
(global $arg1 (mut (ref eq)) (i31.new (i32.const 0)))
(global $arg2 (mut (ref eq)) (i31.new (i32.const 0)))
(global $arg3 (mut (ref eq)) (i31.new (i32.const 0)))

;; "Memory" for the rest
(type $argv (array (ref eq)))
(global $argN (ref $argv)
        (array.new_canon_default
          $argv (i31.const 42) (i31.new (i32.const 0))))

Uniform function type: argument count as sole parameter

Callee moves args to locals, possibly clearing roots

The approach we are taking is to virtualize the calling convention. In the same way that when calling an x86-64 function, you pass the first argument in $rdi, then $rsi, and eventually if you run out of registers you put arguments in memory, in the same way we'll pass the first argument in the $arg0 global, then $arg1, and eventually in memory if needed. The function will receive the number of incoming arguments as its sole parameter; in fact, all functions will be of type (func (param i32)).

The expectation is that after checking argument count, the callee will load its arguments from globals / memory to locals, which the compiler can do a better job on than globals. We might not even emit code to null out the argument globals; might leak a little memory but probably would be a win.

You can imagine a world in which $arg0 actually gets globally allocated to $rdi, because it is only live during the call sequence; but I don't think that world is this one :)

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Great, two points out of the way! Next up, tail calls.

Scheme to Wasm: Tail calls

;; Call known function
(return_call $f arg ...)

;; Call function by value
(return_call_ref $type callee arg ...)

Friends -- I almost cried making this slide. We Schemers are used to working around the lack of tail calls, and I could have done so here, but it's just such a relief that these functions are just going to be there and I don't have to think much more about them. Technically speaking the proposal isn't merged yet; checking the phases document it's at the last station before headed to the great depot in the sky. But, soon soon it will be present and enabled in all WebAssembly implementations, and we should build systems now that rely on it.

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Next up, my favorite favorite topic: delimited continuations.

Scheme to Wasm: Prompts (1)

Problem: Lightweight threads/fibers, exceptions

Possible solutions

  • Eventually, built-in coroutines
  • binaryen’s asyncify (not yet ready for GC); see Julia
  • Delimited continuations

“Bring your whole self”

Before diving in though, one might wonder why bother. Delimited continuations are a building-block that one can use to build other, more useful things, notably exceptions and light-weight threading / fibers. Could there be another way of achieving these end goals without having to implement this relatively uncommon primitive?

For fibers, it is possible to implement them in terms of a built-in coroutine facility. The standards body seems willing to include a coroutine primitive, but it seems far off to me; not within the next 3-4 years I would say. So let's put that to one side.

There is a more near-term solution, to use asyncify to implement coroutines somehow; but my understanding is that asyncify is not ready for GC yet.

For the Guile flavor of Scheme at least, delimited continuations are table stakes of their own right, so given that we will have them on WebAssembly, we might as well use them to implement fibers and exceptions in the same way as we do on native targets. Why compromise if you don't have to?

Scheme to Wasm: Prompts (2)

Prompts delimit continuations

(define k
  (call-with-prompt ’foo
    ; body
    (lambda ()
      (+ 34 (abort-to-prompt 'foo)))
    ; handler
    (lambda (continuation)
      continuation)))

(k 10)       ;; ⇒ 44
(- (k 10) 2) ;; ⇒ 42

k is the _ in (lambda () (+ 34 _))

There are a few ways to implement delimited continuations, but my usual way of thinking about them is that a delimited continuation is a slice of the stack. One end of the slice is the prompt established by call-with-prompt, and the other by the continuation of the call to abort-to-prompt. Capturing a slice pops it off the stack, copying it out to the heap as a callable function. Calling that function splats the captured slice back on the stack and resumes it where it left off.

Scheme to Wasm: Prompts (3)

Delimited continuations are stack slices

Make stack explicit via minimal continuation-passing-style conversion

  • Turn all calls into tail calls
  • Allocate return continuations on explicit stack
  • Breaks functions into pieces at non-tail calls

This low-level intuition of what a delimited continuation is leads naturally to an implementation; the only problem is that we can't slice the WebAssembly call stack. The workaround here is similar to the varargs case: we virtualize the stack.

The mechanism to do so is a continuation-passing-style (CPS) transformation of each function. Functions that make no calls, such as leaf functions, don't need to change at all. The same goes for functions that make only tail calls. For functions that make non-tail calls, we split them into pieces that preserve the only-tail-calls property.

Scheme to Wasm: Prompts (4)

Before a non-tail-call:

  • Push live-out vars on stacks (one stack per top type)
  • Push continuation as funcref
  • Tail-call callee

Return from call via pop and tail call:

(return_call_ref (call $pop-return)
                 (i32.const 0))

After return, continuation pops state from stacks

Consider a simple function:

(define (f x y)
  (+ x (g y))

Before making a non-tail call, a "tailified" function will instead push all live data onto an explicitly-managed stack and tail-call the callee. It also pushes on the return continuation. Returning from the callee pops the return continuation and tail-calls it. The return continuation pops the previously-saved live data and continues.

In this concrete case, tailification would split f into two pieces:

(define (f x y)
  (push! x)
  (push-return! f-return-continuation-0)
  (g y))

(define (f-return-continuation-0 g-of-y)
  (define k (pop-return!))
  (define x (pop! x))
  (k (+ x g-of-y)))

Now there are no non-tail calls, besides calls to run-time routines like push! and + and so on. This transformation is implemented by tailify.scm.

Scheme to Wasm: Prompts (5)

abort-to-prompt:

  • Pop stack slice to reified continuation object
  • Tail-call new top of stack: prompt handler

Calling a reified continuation:

  • Push stack slice
  • Tail-call new top of stack

No need to wait for effect handlers proposal; you can have it all now!

The salient point is that the stack on which push! operates (in reality, probably four or five stacks: one in linear memory or an array for types like i32 or f64, three for each of the managed top types any, extern, and func, and one for the stack of return continuations) are managed by us, so we can slice them.

Someone asked in the talk about whether the explicit memory traffic and avoiding the return-address-buffer branch prediction is a source of inefficiency in the transformation and I have to say, yes, but I don't know by how much. I guess we'll find out soon.

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Okeydokes, last point!

Scheme to Wasm: Numbers

Numbers can be immediate: fixnums

Or on the heap: bignums, fractions, flonums, complex

Supertype is still ref eq

Consider imports to implement bignums

  • On web: BigInt
  • On edge: Wasm support module (mini-gmp?)

Dynamic dispatch for polymorphic ops, as usual

First, I would note that sometimes the compiler can unbox numeric operations. For example if it infers that a result will be an inexact real, it can use unboxed f64 instead of library routines working on heap flonums ((struct i32 f64); the initial i32 is for the hash and tag). But we still need a story for the general case that involves dynamic type checks.

The basic idea is that we get to have fixnums and heap numbers. Fixnums will handle most of the integer arithmetic that we need, and will avoid allocation. We'll inline most fixnum operations as a fast path and call out to library routines otherwise. Of course fixnum inputs may produce a bignum output as well, so the fast path sometimes includes another slow-path callout.

We want to minimize binary module size. In an ideal compile-to-WebAssembly situation, a small program will have a small module size, down to a minimum of a kilobyte or so; larger programs can be megabytes, if the user experience allows for the download delay. Binary module size will be dominated by code, so that means we need to plan for aggressive dead-code elimination, minimize the size of fast paths, and also minimize the size of the standard library.

For numbers, we try to keep module size down by leaning on the platform. In the case of bignums, we can punt some of this work to the host; on a JavaScript host, we would use BigInt, and on a WASI host we'd compile an external bignum library. So that's the general story: inlined fixnum fast paths with dynamic checks, and otherwise library routine callouts, combined with aggressive whole-program dead-code elimination.

Scheme to Wasm

  • Value representation
  • Varargs
  • Tail calls
  • Delimited continuations
  • Numeric tower

Hey I think we did it! Always before when I thought about compiling Scheme or Guile to the web, I got stuck on some point or another, was tempted down the corner-cutting alleys, and eventually gave up before starting. But finally it would seem that the stars are aligned: we get to have our Scheme and run it too.

Miscellenea

Debugging: The wild west of DWARF; prompts

Strings: stringref host strings spark joy

JS interop: Export accessors; Wasm objects opaque to JS. externref.

JIT: A whole ’nother talk!

AOT: wasm2c

Of course, like I said, WebAssembly is still a weird machine: as a compilation target but also at run-time. Debugging is a right proper mess; perhaps some other article on that some time.

How to represent strings is a surprisingly gnarly question; there is tension within the WebAssembly standards community between those that think that it's possible for JavaScript and WebAssembly to share an underlying string representation, and those that think that it's a fool's errand and that copying is the only way to go. I don't know which side will prevail; perhaps more on that as well later on.

Similarly the whole interoperation with JavaScript question is very much in its early stages, with the current situation choosing to err on the side of nothing rather than the wrong thing. You can pass a WebAssembly (ref eq) to JavaScript, but JavaScript can't do anything with it: it has no prototype. The state of the art is to also ship a JS run-time that wraps each wasm object, proxying exported functions from the wasm module as object methods.

Finally, some language implementations really need JIT support, like PyPy. There, that's a whole 'nother talk!

WebAssembly for the rest of us

With GC, WebAssembly is now ready for us

Getting our languages on WebAssembly now a S.M.O.P.

Let’s score some goals in the second half!

(visit-links
 "gitlab.com/spritely/guile-hoot-updates"
 "wingolog.org"
 "wingo@igalia.com"
 "igalia.com"
 "mastodon.social/@wingo")

WebAssembly has proven to have some great wins for C, C++, Rust, and so on -- but now it's our turn to get in the game. GC is coming and we as a community need to be getting our compilers and language run-times ready. Let's put on the coffee and bang some bytes together; it's still early days and there's a world to win out there for the language community with the best WebAssembly experience. The game is afoot: happy consing!

20 March, 2023 09:06AM by Andy Wingo

Amin Bandali

LibrePlanet 2023: What's new in Jami

Update: Jami has won this year's Award for Project of Social Benefit, presented by the Free Software Foundation "to a project or team responsible for applying free software, or the ideas of the free software movement, to intentionally and significantly benefit society. This award stresses the use of free software in service to humanity."

Today I gave a talk at LibrePlanet 2023 on what's new in and about Jami since my Jami and how it empowers users talk for LibrePlanet 2021.

Here is the abstract for my talk, also available on the LibrePlanet 2023's speakers page:

Jami is free/libre software for universal communication that respects the freedoms and privacy of its users. An official GNU package, Jami is an end-to-end encrypted secure and distributed communication tool for calling, conferencing, messaging, and file transfer. Jami has end-user applications across multiple operating systems and platforms, as well as multiple APIs and a plugin system for building upon and extending Jami as a framework for secure and private communication.

This talk gives an update on what's new in and about Jami since bandali's "Jami and how it empowers users" talk at LibrePlanet 2021.

Presentation slides: pdf (with notes, only notes) | bib
LaTeX sources: tar.gz | zip
Video: coming soon

I will add the presentation video once the conference recordings have been processed and published by the Free Software Foundation.

LibrePlanet is a conference about software freedom, happening on March 19-20, 2023. The event is hosted by the Free Software Foundation, and brings together software developers, law and policy experts, activists, students, and computer users to learn skills, celebrate free software accomplishments, and face upcoming challenges. Newcomers are always welcome, and LibrePlanet 2023 will feature programming for all ages and experience levels.

20 March, 2023 01:16AM

March 19, 2023

a2ps @ Savannah

a2ps 4.15.2 released [stable]


GNU a2ps is an Any to PostScript filter. Of course it processes plain text
files, but also pretty prints quite a few popular languages.

More detailed web pages about GNU a2ps is available at
https://savannah.gnu.org/projects/a2ps/.

This release is a minor bug-fix release. It fixes a long-standing but rare
crash, makes a minor fix to the build system, and finally puts the manual
online; see:

https://gnu.org/software/a2ps/manual/

Here are the compressed sources and a GPG detached signature:
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.2.tar.gz
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.2.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

b02c9f4066ebb2899f7615b93b354fb77192377c  a2ps-4.15.2.tar.gz
7FKQSp+sEmQWsyrJokBfPWF92pfWUpnhpBVVT+Az0iU  a2ps-4.15.2.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.15.2.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.15.2.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5853-ge0aefd96b6

NEWS

* Noteworthy changes in release 4.15.2 (2023-03-19) [stable]
 * Bug fixes:
   - Fix old crash when using --stdin="".
 * Build
   - Make configure stop if libpaper is not found.
   - Enable building the manual for gnu.org.


19 March, 2023 07:06PM by Reuben Thomas

Trisquel GNU/Linux

Trisquel 11.0 "Aramo" release announcement

Our most ambitious release to date, Trisquel 11.0 Aramo is out! After extensive work and thorough testing, we are proud to declare Aramo to be production-ready. This release brings many improvements and covers more grounds both in terms of machines supported and in installation options. Here are some highlights of the main accomplishments included in this release:

Major achievements

  • New supported architectures. Following the addition of 32bit ARM support in Trisquel 10, we now introduce support for 64bit ARM and POWER architectures, to expand the options for hardware compatibility.
  • d-i/Netinstall (also called "debian-installer") is the text-mode installer for Trisquel, which allows for advanced and custom installations, often used for servers. After Ubuntu dropped support for this installation method, we stepped in to add any removed pieces and make it functional again, often from upstream Debian sources.
  • Browser packaging: as Ubuntu continues to shift towards snap packaging, we took on the task of continuing to package key components such as Abrowser (our improved Firefox derivative) as a standard .deb package. Abrowser continues to bring a fully free browser that balances privacy and usability.
  • Compatibility with AMD/ATI graphics cards. We made a specific effort to improve the support for these cards without requiring non-free firmware blobs. The result is a default configuration that should make most of these types of cards work at a basic level, without 2D/3D acceleration. Please report if you test it with one of those cards!

Aramo is based on Ubuntu 22.04LTS and will receive support until 2027. Users of Trisquel 10 Nabia can upgrade directly using the update-manager or do-release-upgrade commands at a console terminal.

Editions

  • Trisquel. We keep MATE (version 1.26 on this release) as the default desktop environment due to its great accessibility support, simple user interface and no dependency on 3D acceleration.
  • Triskel. Our KDE (v5.92) based edition now as mature as our MATE release is excellent for customizing the design and functionality in fine detail.
  • Trisquel Mini. Running LXDE (v0.99.2), the Mini edition is a lightweight desktop perfect for netbooks, old computers and users with minimal resource usage needs.
  • Trisquel Sugar or Trisquel On A Sugar Toast (TOAST): Based on the Sugar learning platform (v0.119), TOAST comes with dozens of educational activities for children.
  • Network installer image: To deploy with a command-line install interface, it is ideal for servers and advanced users who want to explore custom designed environments. GNOME users will be happy to find that Trisquel GNOME environment has been added to the tasksel step on the d-i installer.

In addition, this release had put some additional work on Budgie and Kylin alternative desktop environments for users that would like them to test them out, help us with some feedback.

Upcoming changes

One pending task that will get into motion soon is the main website redesign and improvement on l10n support and as we transition to the trisquel.org domain.

During Aramo's lifespan we'll continue to improve installation support for ARM and POWERPC whose base rootfs are available at our cdimage archive.

With the continuous support form our community and contributors we will keep providing one of the best fully free operating systems, and tackle the big technical challenges ahead. Trisquel is a non-profit project, you can help sustain it by becoming a member, donating or buying from our store.

We can't finish this great release without thanking for all the donors that keep the project going, and to the hackers Amin Bandali, bill-auger, David L, David Philipe Gil, Denis "GNUtoo" Carikli, "dragestil", Francis Meetze, Jason Self, Joshua Aspinall , Keno Goertz, "knife", "Lappi", Legimet, Mason Hock, Pablo Correa, "Parodper", Simon Josefsson, and many others for all the code, patches, bug reports, translations, and advice. Special thanks to Luis "Ark74" Guzmán, for his very prolific contributions, and to the wonderful community that keeps the project going.

19 March, 2023 06:58PM by quidam

FSF Latin America

The TRApp Trap

The TRApp Trap

Mobile phone apps, that our board member Alexandre Oliva calls TRApps in his new article, have replaced, not very spontaneously, web sites that adhered to international standards and were compatible with free systems, TRApping people in a duopoly of proprietary and invasive systems.

When private businesses do so, it's bad; but when governments impose on citizens the use of proprietary operating systems and programs, to get public services or to comply with legal obligations, we denounce them as imposed taxing software.

They're "imposed" in the sense that you can't avoid them, and "taxing" in that they charge you and take from you your most valuable good: your freedom.

We call for consumers, citizens and users at large to resist these impositions and insist that public and private services be available through sites that will work properly when accessed with a standard browser on a free operating system, without installing freedom-depriving programs, not even those that even standard browsers themselves would install and run automatically from visited sites. And, when it's necessary to run software on the service recipient's computer, the software ought to be free.

Read the full article on our site, without TRApps or proprietary JavaScript.
https://www.fsfla.org/texto/TRApps


About FSFLA

Free Software Foundation Latin America joined in 2005 the international FSF network, previously formed by Free Software Foundations in the United States, in Europe and in India. These sister organizations work in their corresponding geographies towards promoting the same Free Software ideals and defending the same freedoms for software users and developers, working locally but cooperating globally.
https://www.fsfla.org/


Copyright 2023 FSFLA

Permission is granted to make and distribute verbatim copies of this entire document without royalty, provided the copyright notice, the document's official URL, and this permission notice are preserved.

Permission is also granted to make and distribute verbatim copies of individual sections of this document worldwide without royalty provided the copyright notice and the permission notice above are preserved, and the document's official URL is preserved or replaced by the individual section's official URL.

https://www.fsfla.org/anuncio/2023-03-TRApps

19 March, 2023 03:12PM

FSF Blogs

Reporting back from day one of LibrePlanet: Charting the Course

19 March, 2023 02:55AM

FSF News

Free Software Awards winners announced: Eli Zaretskii, Tad (SkewedZeppelin), GNU Jami

BOSTON, Massachusetts, USA -- Saturday, March 18, 2023 -- The Free Software Foundation (FSF) today announced the recipients of the 2022 Free Software Awards, which are given annually at the FSF's LibrePlanet conference to groups and individuals in the free software community who have made significant contributions to the cause for software freedom. This year's recipients of the awards are Eli Zaretskii, Tad (SkewedZeppelin), and GNU Jami. As LibrePlanet 2023 is a hybrid in-person and online conference this year, the ceremony was conducted both in person and virtually.

19 March, 2023 01:05AM

March 18, 2023

FSF Latin America

Linux-libre turns 15!

Linux-libre turns 15!

It was February 2008 when Jeff Moe announced Linux-libre, a project to share the efforts that freedom-respecting distros had to undertake to drop the nonfree bits distributed as part of the kernel Linux.
https://web.archive.org/web/1/lists.autistici.org/message/20080221.002845.467ba592.en.html

"For fifteen years, the Linux-libre project has remained dedicated to providing a kernel that respects everyone's freedom and has become an essential part of the free software movement. Linux-libre is widely used by those who value their freedom to use, study, change, and share software without restrictions or limitations. These freedoms are essential to creating a just society."
-- Jason Self

Since around 1996, Linux has carried sourceless firmware encoded as sequences of numbers disguised as source code. UTUTO and gNewSense pioneered the efforts of removing them. Cleaning Linux up is a substantial amount of work, so the existence of Linux-libre has alleviated one of the main difficulties in maintaining GNU+Linux distros that abide by the GNU Free Software Distribution Guidelines. The Linux-libre compiled kernel distributions maintained by Jason Self, Freesh (.deb), liberRTy (low-latency .deb) and RPMFreedom (.rpm), make it easy for users of other GNU+Linux distros to take a step towards freedom when their hardware is not too user-hostile.

"Thanks to Linux-libre, we have entirely libre GNU+Linux distros. Thanks to Linux-libre, people like me who are not kernel hackers can install one of those distros and have a computer which never runs a nonfree program on the CPU. (Provided we use LibreJS as well to reject nonfree Javascript programs that web sites send us.)"
-- Richard Stallman

Early pieces of firmware in Linux ran peripheral devices, but some of the blobs loaded by Linux nowadays reconfigure the primary central processing units and others contain an entire operating system for the peripherals' CPUs, including a copy of the kernel Linux itself and several other freedom-depriving programs!

After years of our denouncing the social, technical, and legal risks out of Linux's misbehavior, most of the blobs got moved to separate files, still part of the kernel Linux, and then to separate packages, which mitigates some of the legal risks, but the problem keeps growing: more and more devices depend on nonfree firmware and thus remain under exclusive and proprietary control by their suppliers.

Challenge

For 27 years, the nonfree versions of Linux have shown that tolerating blobs and making it easy for users to install and accept them makes users increasingly dependent on user-hostile, blob-requiring devices for their computing. Refusing to give these devices' suppliers what they wish, namely your money and control over your computing, is more likely to succeed at changing their practices if more users refuse.

If you're the kind of software freedom supporter who demands respect for your freedom, keep on enjoying the instant gratification that GNU Linux-libre affords you, and supporting (or being!) those who refurbish old computers and build new ones to respect our autonomy.

However, if you're of the kind for whom last-generation computers are hard to resist, even though you'd prefer if they were more respectful of your freedom, you may wish to consider a delayed gratification challenge: if you and your friends resist hostile computers now, you may get more respectful ones later, for yourselves and for all of us; if you don't, the next generations will likely be even more hostile. Are you up for the challenge?
https://en.wikipedia.org/wiki/Delayed_gratification

Present and Future

GNU Linux-libre releases are currently prepared with scripts that automate the cleaning-up and part of the verification. For each upstream major and stable release, we run the scripts, updating them as needed, and publish them, along with the cleaning-up logs and the cleaned-up sources, in a git repository. Each source release is an independent tag, as in, there are no branches for cleaned-up sources. This is so we can quickly retract releases if freedom bugs are found.

We have plans to change the cleaning-up process and the repository structure in the future: we're (slowly) preparing to move to a rewritten git repository, in which, for each commit in upstream Linux main and stable repositories, there will be a corresponding cleaned-up commit in ours. Undesirable bits are going to be cleaned up at the commit corresponding to the one in which upstream introduced or modified them, and other modifications will be checked and integrated unchanged, mirroring the upstream commit graph, with "git replace" mappings for individual commits and, perhaps, also for cleaned-up files.

This is expected to enable us to track upstream development very closely, to get stable and major releases out nearly instantly and often automatically and to enable Linux developers to clone our freed repository instead of our upstream to write and test their changes. The same techniques used to create the cleaned-up repository can be used to fix freedom bugs in it.

Artwork

Jason Self has made several beautiful pictures of his version of Freedo, our light-blue penguin mascot, and we've used them for our recent releases.

Marking the beginning of the week in which we celebrate 15 years of Linux-libre, we had the pleasure of publishing a major release, 6.2-gnu, codenamed "la quinceañera", with a picture of Freedo dressed up for the occasion.
https://www.fsfla.org/pipermail/linux-libre/2023-February/003502.html

But there's more! He also made a commemorative black-and-white wallpaper with classic Freedo, also dressed up for the occasion. Check them out, and feel free to tune the colors to your liking!
https://linux-libre.fsfla.org/#news

He also modeled a 3D Freedo in Blender, and we're looking for someone who could 3D-print it and get it to the FSF office in time for the LibrePlanet conference. Rumor has it that Richard Stallman is going to auction it off to raise funds for the FSF! Can you help?
https://libreplanet.org/2023/


About GNU Linux-libre

GNU Linux-libre is a GNU package maintained by Alexandre Oliva, on behalf of FSFLA, and by Jason Self. It releases cleaned-up versions of Linux, suitable for use in distributions that comply with the Free Software Distribution Guidelines published by the GNU project, and by users who wish to run Free versions of Linux on their GNU systems. The project offers cleaning-up scripts, Free sources, binaries for some GNU+Linux distributions, and artwork with GNU and the Linux-libre mascot: Freedo, the clean, Free and user-friendly light-blue penguin. Visit our web site and Be Free!
https://linux-libre.fsfla.org/
https://www.gnu.org/distros/

About the GNU Operating System and Linux

Richard Stallman announced in September 1983 the plan to develop a Free Software Unix-like operating system called GNU. GNU is the only operating system developed specifically for the sake of users' freedom.
https://www.gnu.org/
https://www.gnu.org/gnu/the-gnu-project.html

In 1992, the essential components of GNU were complete, except for one, the kernel. When in 1992 the kernel Linux was re-released under the GNU GPL, making it Free Software, the combination of GNU and Linux formed a complete Free operating system, which made it possible for the first time to run a PC without non-Free Software. This combination is the GNU+Linux system.
https://www.gnu.org/gnu/gnu-linux-faq.html

About FSFLA

Free Software Foundation Latin America joined in 2005 the international FSF network, previously formed by Free Software Foundations in the United States, in Europe and in India. These sister organizations work in their corresponding geographies towards promoting the same Free Software ideals and defending the same freedoms for software users and developers, working locally but cooperating globally.
https://www.fsfla.org/


Copyright 2023 FSFLA

Permission is granted to make and distribute verbatim copies of this entire document without royalty, provided the copyright notice, the document's official URL, and this permission notice are preserved.

Permission is also granted to make and distribute verbatim copies of individual sections of this document worldwide without royalty provided the copyright notice and the permission notice above are preserved, and the document's official URL is preserved or replaced by the individual section's official URL.

https://www.fsfla.org/anuncio/2023-02-Linux-libre-15

18 March, 2023 06:13PM

GNU Health

Leading Public Mental Health Hospital in Argentina embraces GNU Health

The World Health Organization defines health as a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity.

Unfortunately, this definition is far from being a reality in our societies. Instead of embracing the system of health, we live in the system of disease, ruled by a reactive, reductionist and unsustainable model of healthcare. The beautiful noble art and science of medicine is ill. Financial institutions and giant technological corporations are removing the human factor from medicine, transforming people and patients into clients. They are reducing the non-negotiable human right to healthcare to a privilege of a few.

Coming back to the formal definition of health, in the current system of disease very little is taken into account from the social and mental well-being . Today, many people with mental health conditions not only have to deal with the physiopathological aspects of the disorder, but also with the stigma, exclusion and invisibilization from the society.

But there is hope. Medicine is a social science, and GNUHealth is a social project with some technology behind. That feeling of optimism and hope has been reinforced in last week trip to Argentina and their people. In the end, medicine is about people interacting and taking care of people. Is about people before patients. I know them well, because I did my medical career in Argentina.

Group picture with health professionals from HESM, UNER, Government officials and GNU Solidario at the entrance of the leading Public Mental Health Hospital in Entre Ríos, Argentina

The Mental Health Hospital has chosen GNUHealth to improve the management of the institution resources, as well as to provide the best medical care for their community, both in outpatient and inpatient settings. Being able to properly identify every person who needs attention, and knowing the socio-sanitary, medical and clinical history in real time will make a big difference in the care of the individual.

The implementation of GNUHealth in this health institution will be lead by Prof. Dr. Fernando Sassetti and the department of public health studies of the University of Entre Ríos in the context of the GNU Health Alliance of Academic and Research Institutions agreement signed with GNU Solidario.

Health is an equilibrium of the inseparable and interconnected physical, social, mental and spiritual domains. Medicine is about taking into consideration and maintaining this body-mind-spirit-environment balance. This holistic approach to medicine is encoded in the genome of every nurse, psychologist, social worker and doctor from the Mental Health Hospital and the Primary care centers I got to know during these years in Entre Ríos, Argentina.

Links / References

Un software Libre para mejorar las políticas de salud: https://www.eldiario.com.ar/253548-un-software-para-mejorar-las-politicas-de-salud/

Hospital Escuela de Salud Mental : http://www.hesm.gob.ar/

Audiovisual institucional Hospital Escuela de Salud Mental: https://www.youtube.com/watch?v=Jx08WyfKRIE&t=12s

GNU Health: https://www.gnuhealth.org

18 March, 2023 12:53AM by Luis Falcon

March 16, 2023

FSF News

Gary Benson

Which APT repository did a package come from?

$ apt policy wget
wget:
  Installed: 1.21.2-2ubuntu1
  Candidate: 1.21.2-2ubuntu1
  Version table:
 *** 1.21.2-2ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

16 March, 2023 01:18PM by gbenson

Python debugger

By the way, if you’ve never used Python’s debugger, it really is as simple as adding a call to the built-in function breakpoint() at the point you want it to stop.

16 March, 2023 10:09AM by gbenson

March 15, 2023

GNU Guix

Building Toolchains with Guix

In order to deploy embedded software using Guix we first need to teach Guix how to cross-compile it. Since Guix builds everything from source, this means we must teach Guix how to build our cross-compilation toolchain.

The Zephyr Project uses its own fork of GCC with custom configs for the architectures supported by the project. In this article, we describe the cross-compilation toolchain we defined for Zephyr; it is implemented as a Guix channel.

About Zephyr

Zephyr is a real-time operating system from the Linux Foundation. It aims to provide a common environment which can target even the most resource constrained devices.

Zephyr introduces a module system which allows third parties to share code in a uniform way. Zephyr uses CMake to perform physical component composition of these modules. It searches the filesystem and generates scripts which the toolchain will use to successfully combine those components into a firmware image.

The fact that Zephyr provides this mechanism is one reason I chose to target it in the first place.

This separation of modules in an embedded context is a really great thing. It brings many of the advantages that it brings to the Linux world such as code re-use, smaller binaries, more efficient cache/RAM usage, etc. It also allows us to work as independent groups and compose contributions from many teams.

It also brings all of the complexity. Suddenly most of the problems that plague traditional deployment now apply to our embedded system. The fact that the libraries are statically linked at compile time instead of dynamically at runtime is simply an implementation detail. I say most because everything is statically linked so there is no runtime component discovery that needs to be accounted for.

Anatomy of a Toolchain

Toolchains are responsible for taking high level descriptions of programs and lowering them down to a series of equivalent machine instructions. This process involves more than just a compiler. The compiler uses the GNU Binutils to manipulate its internal representation down to a given architecture. It also needs the use of the C standard library as well as a few other libraries needed for some compiler optimizations.

The C library provides the interface to the underlying kernel. System calls like write and read are provided by GNU C Library (glibc) on most distributions.

In embedded systems, smaller implementations like RedHat's newlib and newlib-nano are used.

Bootstrapping a Toolchain

In order to compile GCC we need a C library that's been compiled for our target architecture. How can we cross compile our C library if we need our C library to build a cross compiler? The solution is to build a simpler compiler that doesn't require the C library to function. It will not be capable of as many optimizations and it will be very slow, however it will be able to build the C libraries as well as the complete version of GCC.

In order to build the simpler compiler we need to compile the Binutils to work with our target architecture. Binutils can be bootstrapped with our host GCC and have no target dependencies. More information is available in this article.

Doesn't sound so bad right? It isn't... in theory. However internet forums since time immemorial have been littered with the laments of those who came before. From incorrect versions of ISL to the wrong C library being linked or the host linker being used, etc. The one commonality between all of these issues is the environment. Building GCC is difficult because isolating build environments is hard.

In fact as of v0.14.2, the Zephyr “software development kit” (SDK) repository took down the build instructions and posted a sign that read "Building this is too complicated, don't worry about it." (I'm paraphrasing, but not by much.)

We will neatly sidestep all of these problems and not risk destroying or polluting our host system with garbage by using Guix to manage our environments for us.

Our toolchain only requires the first pass compiler because newlib(-nano) is statically linked and introduced to the toolchain by normal package composition.

Defining the Packages

All of the base packages are defined in zephyr/packages/zephyr.scm. Zephyr modules (coming soon!) are defined in zephyr/packages/zephyr-xyz.scm, following the pattern of other module systems implemented by Guix.

Binutils

First thing we need to build is the arm-zephyr-eabi binutils. This is very easy in Guix.

(define-public arm-zephyr-eabi-binutils
  (let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
    (package
      (inherit xbinutils)
      (name "arm-zephyr-eabi-binutils")
      (version "2.38")
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://github.com/zephyrproject-rtos/binutils-gdb")
                      (commit "6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
                (file-name (git-file-name name version))
                (sha256 (base32 "0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcyh0isr"))))
      (arguments
       `(#:tests? #f
         ,@(substitute-keyword-arguments (package-arguments xbinutils)
             ((#:configure-flags flags)
          `(cons "--program-prefix=arm-zephyr-eabi-" ,flags)))))
      (native-inputs
       (modify-inputs (package-native-inputs xbinutils)
         (prepend texinfo bison flex gmp dejagnu)))
      (home-page "https://zephyrproject.org")
      (synopsis "Binutils for the Zephyr RTOS"))))

The function cross-binutils returns a package which has been configured for the given GNU triplet. We simply inherit that package and replace the source. The Zephyr build system expects the binutils to be prefixed with arm-zephyr-eabi- which is accomplished by adding another flag to the #:configure-flags argument.

We can test our package definition using the -L flag with guix build to add our packages.

$ guix build -L guix-zephyr zephyr-binutils

/gnu/store/...-zephyr-binutils-2.38

This directory contains the results of make install.

GCC sans libc

This one is a bit more involved. Don't be afraid! This version of GCC wants ISL version 0.15. It's easy enough to make that happen. Inherit the current version of ISL and swap out the source and update the version. For most packages the build process doesn't change that much between versions.

(define-public isl-0.15
  (package
    (inherit isl)
    (version "0.15")
    (source (origin
              (method url-fetch)
              (uri (list (string-append "mirror://sourceforge/libisl/isl-"
                            version ".tar.gz")))
              (sha256
               (base32
                "11vrpznpdh7w8jp4wm4i8zqhzq2h7nix71xfdddp8xnzhz26gyq2"))))))

Like the binutils, there is a cross-gcc function for creating cross-GCC packages. This one accepts keywords specifying which binutils and libc to use. If libc isn't given (like here), gcc is configured with many options disabled to facilitate being built without libc. Therefore we need to add the extra options we want (I got them from the SDK configuration scripts in the sdk-ng Git repository as well as the commits to use for each of the tools).

(define-public gcc-arm-zephyr-eabi-12
  (let ((xgcc (cross-gcc "arm-zephyr-eabi"
                         #:xbinutils zephyr-binutils)))
    (package
      (inherit xgcc)
      (version "12.1.0")
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://github.com/zephyrproject-rtos/gcc")
                      (commit "0218469df050c33479a1d5be3e5239ac0eb351bf")))
                (file-name (git-file-name (package-name xgcc) version))
                (sha256
                 (base32
                  "1s409qmidlvzaw1ns6jaanigh3azcxisjplzwn7j2n3s33b76zjk"))
                (patches (search-patches
                          "gcc-12-cross-environment-variables.patch"
                          "gcc-cross-gxx-include-dir.patch"))))
      (native-inputs (modify-inputs (package-native-inputs xgcc)
                       ;; Get rid of stock ISL
                       (delete "isl")
                       ;; Add additional dependencies that xgcc doesn't have
                       ;; including our special ISL
                       (prepend flex
                                isl-0.15)))
      (arguments
       (substitute-keyword-arguments (package-arguments xgcc)
         ((#:phases phases)
          `(modify-phases ,phases
             (add-after 'unpack 'fix-genmultilib
               (lambda _
                 (patch-shebang "gcc/genmultilib")))

             (add-after 'set-paths 'augment-CPLUS_INCLUDE_PATH
               (lambda* (#:key inputs #:allow-other-keys)
                 (let ((gcc (assoc-ref inputs "gcc")))
                   ;; Remove the default compiler from CPLUS_INCLUDE_PATH to
                   ;; prevent header conflict with the GCC from native-inputs.
                   (setenv "CPLUS_INCLUDE_PATH"
                           (string-join (delete (string-append gcc
                                                               "/include/c++")
                                                (string-split (getenv
                                                               "CPLUS_INCLUDE_PATH")
                                                              #\:)) ":"))
                   (format #t
                    "environment variable `CPLUS_INCLUDE_PATH' changed to `a`%"
                    (getenv "CPLUS_INCLUDE_PATH")))))))

         ((#:configure-flags flags)
          ;; The configure flags are largely identical to the flags used by the
          ;; "GCC ARM embedded" project.
          `(append (list
                    "--enable-multilib"
                    "--with-newlib"
                    "--with-multilib-list=rmprofile"
                    "--with-host-libstdcxx=-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm"
                    "--enable-plugins"
                    "--disable-decimal-float"
                    "--disable-libffi"
                    "--disable-libgomp"
                    "--disable-libmudflap"
                    "--disable-libquadmath"
                    "--disable-libssp"
                    "--disable-libstdcxx-pch"
                    "--disable-nls"
                    "--disable-shared"
                    "--disable-threads"
                    "--disable-tls"
                    "--with-gnu-ld"
                    "--with-gnu-as"
                    "--enable-initfini-array")
                   (delete "--disable-multilib"
                           ,flags)))))
      (native-search-paths
       (list (search-path-specification
              (variable "CROSS_C_INCLUDE_PATH")
              (files '("arm-zephyr-eabi/include")))
             (search-path-specification
              (variable "CROSS_CPLUS_INCLUDE_PATH")
              (files '("arm-zephyr-eabi/include" "arm-zephyr-eabi/c++"
                       "arm-zephyr-eabi/c++/arm-zephyr-eabi")))
             (search-path-specification
              (variable "CROSS_LIBRARY_PATH")
              (files '("arm-zephyr-eabi/lib")))))
      (home-page "https://zephyrproject.org")
      (synopsis "GCC for the Zephyr RTOS"))))

This GCC can be built like so.

$ guix build -L guix-zephyr gcc-cross-sans-libc-arm-zephyr-eabi

/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0-lib
/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0

Great! We now have our stage-1 compiler.

Newlib(-nano)

The newlib package package is quite straight forward (relatively). It is mostly adding in the relevent configuration flags and patching the files the patch-shebangs phase missed.

(define-public zephyr-newlib
  (package
    (name "zephyr-newlib")
    (version "3.3")
    (source (origin
          (method git-fetch)
          (uri (git-reference
            (url "https://github.com/zephyrproject-rtos/newlib-cygwin")
            (commit "4e150303bcc1e44f4d90f3489a4417433980d5ff")))
          (sha256
           (base32 "08qwjpj5jhpc3p7a5mbl7n6z7rav5yqlydqanm6nny42qpa8kxij"))))
    (build-system gnu-build-system)
    (arguments
     `(#:out-of-source? #t
       #:configure-flags '("--target=arm-zephyr-eabi"
               "--enable-newlib-io-long-long"
               "--enable-newlib-io-float"
               "--enable-newlib-io-c99-formats"
               "--enable-newlib-retargetable-locking"
               "--enable-newlib-lite-exit"
               "--enable-newlib-multithread"
               "--enable-newlib-register-fini"
               "--enable-newlib-extra-sections"
               "--disable-newlib-wide-orient"
               "--disable-newlib-fseek-optimization"
               "--disable-newlib-supplied-syscalls"
               "--disable-newlib-target-optspace"
               "--disable-nls")
       #:phases
       (modify-phases %standard-phases
     (add-after 'unpack 'fix-references-to-/bin/sh
       (lambda _
         (substitute# '("libgloss/arm/cpu-init/Makefile.in"
                "libgloss/arm/Makefile.in"
                "libgloss/libnosys/Makefile.in"
                "libgloss/Makefile.in")
           (("/bin/sh") (which "sh")))
         #t)))))
    (native-inputs
     `(("xbinutils" ,zephyr-binutils)
       ("xgcc" ,gcc-arm-zephyr-eabi-12)
       ("texinfo" ,texinfo)))
    (home-page "https://www.sourceware.org/newlib/")
    (synopsis "C library for use on embedded systems")
    (description "Newlib is a C library intended for use on embedded
systems.  It is a conglomeration of several library parts that are easily
usable on embedded products.")
    (license (license:non-copyleft
          "https://www.sourceware.org/newlib/COPYING.NEWLIB"))))

And the build.

$ guix build -L guix-zephyr zephyr-newlib

/gnu/store/...-zephyr-newlib-3.3

Complete Toolchain

Mostly complete. libstdc++ does not build because arm-zephyr-eabi is not arm-none-eabi so a dynamic link check is performed/failed. I cannot figure out how crosstool-ng handles this.

Now that we've got the individual tools it's time to create our complete toolchain. For this we need to do some package transformations. Because these transformations are going to have to be done for every combination of binutils/gcc/newlib it is best to create a function which we can reuse for every version of the SDK.

(define (arm-zephyr-eabi-toolchain xgcc newlib version)
  "Produce a cross-compiler zephyr toolchain package with the compiler XGCC and the C\n  library variant NEWLIB."
  (let ((newlib-with-xgcc
         (package
           (inherit newlib)
           (native-inputs
            (modify-inputs (package-native-inputs newlib)
              (replace "xgcc" xgcc))))))
    (package
      (name (string-append "arm-zephyr-eabi"
                           (if (string=? (package-name newlib-with-xgcc)
                                         "newlib-nano")
                               "-nano"
                               "")
                           "-toolchain"))
      (version version)
      (source #f)
      (build-system trivial-build-system)
      (arguments
       '(#:modules ((guix build union)
                    (guix build utils))
         #:builder (begin
                     (use-modules (ice-9 match)
                                  (guix build union)
                                  (guix build utils))
                     (let ((out (assoc-ref %outputs "out")))
                       (mkdir-p out)
                       (match %build-inputs
                         (((names . directories) ...)
                          (union-build (string-append out "/arm-zephyr-eabi")
                                       directories)))))))
      (inputs `(("binutils" ,zephyr-binutils)
                ("gcc" ,xgcc)
                ("newlib" ,newlib-with-xgcc)))
      (synopsis "Complete GCC tool chain for ARM zephyrRTOS development")
      (description
       "This package provides a complete GCC tool chain for ARM
  bare metal development with zephyr rtos.  This includes the GCC arm-zephyr-eabi cross compiler
  and newlib (or newlib-nano) as the C library.  The supported programming
  language is C.")
      (home-page (package-home-page xgcc))
      (license (package-license xgcc)))))

This function creates a special package which consists of the toolchain in a special directory hierarchy, i.e arm-zephyr-eabi/. Our complete toolchain definition looks like this.

(define-public arm-zephyr-eabi-toolchain-0.15.0
  (arm-zephyr-eabi-toolchain gcc-arm-zephyr-eabi-12 zephyr-newlib
                             "0.15.0"))

To build:

$ guix build -L guix-zephyr arm-zephyr-eabi-toolchain
/gnu/store/...-arm-zephyr-eabi-toolchain-0.15.0

Note: Guix now includes a mechanism to describe platforms at a high level, and which the --system and --target build options build upon. It is not used here but could be a way to better integrate Zephyr support in the future.

Integrating with Zephyr Build System

Zephyr uses CMake as its build system. It contains numerous CMake files in both the so-called ZEPHYR_BASE, the zephyr source code repository, as well as a handful in the SDK which help select the correct toolchain for a given board.

There are standard locations the build system will look for the SDK. We are not using any of them. Our SDK lives in the store, immutable forever. According to the Zephyr documentation, the variable ZEPHYR_SDK_INSTALL_DIR needs to point to our custom spot.

We also need to grab the CMake files from the repository and create a file, sdk_version, which contains the version string ZEPHYR_BASE uses to find a compatible SDK.

Along with the SDK proper we need to include a number of python packages required by the build system.

(define-public zephyr-sdk
  (package
    (name "zephyr-sdk")
    (version "0.15.0")
    (home-page "https://zephyrproject.org")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/zephyrproject-rtos/sdk-ng")
                    (commit "v0.15.0")))
              (file-name (git-file-name name version))
              (sha256
               (base32
                "04gsvh20y820dkv5lrwppbj7w3wdqvd8hcanm8hl4wi907lwlmwi"))))
    (build-system trivial-build-system)
    (arguments
     `(#:modules ((guix build union)
                  (guix build utils))
       #:builder (begin
                   (use-modules (guix build union)
                                (ice-9 match)
                                (guix build utils))
                   (let ((out (assoc-ref %outputs "out"))
                         (cmake-scripts (string-append (assoc-ref
                                                        %build-inputs
                                                        "source")
                                                       "/cmake"))
                         (sdk-out (string-append out "/zephyr-sdk-0.15.0")))
                     (mkdir-p out)

                     (match (assoc-remove! %build-inputs "source")
                       (((names . directories) ...)
                        (union-build sdk-out directories)))

                     (copy-recursively cmake-scripts
                                       (string-append sdk-out "/cmake"))

                     (with-directory-excursion sdk-out
                       (call-with-output-file "sdk_version"
                         (lambda (p)
                           (format p "0.15.0"))))))))
    (propagated-inputs (list arm-zephyr-eabi-toolchain-0.15.0
                             zephyr-binutils
                             dtc
                             python-3
                             python-pyelftools
                             python-pykwalify
                             python-pyyaml
                             python-packaging))
    (native-search-paths
     (list (search-path-specification
            (variable "ZEPHYR_SDK_INSTALL_DIR")
            (separator #f)
            (files '("")))))
    (synopsis "Zephyr SDK")
    (description
     "zephyr-sdk contains bundles a complete gcc toolchain as well
as host tools like dtc, openocd, qemu, and required python packages.")
    (license license:apsl2)))

Testing

In order to test we will need an environment with the SDK installed. We can take advantage of guix shell to avoid installing test packages into our home environment. This way if it causes problems we can just exit the shell and try again.

guix shell -L guix-zephyr zephyr-sdk cmake ninja git

ZEPHYR_BASE can be cloned into a temporary workspace to test our toolchain functionality. (For now. Eventually we will need to create a package for zephyr-base that our Guix zephyr-build-system can use.)

mkdir /tmp/zephyr-project
cd /tmp/zephyr-project
git clone https://github.com/zephyrproject-rtos/zephyr
export ZEPHYR_BASE=/tmp/zephyr-project/zephyr

In order to build for the test board (k64f in this case) we need to get a hold of the vendor Hardware Abstraction Layers and CMSIS. (These will also need to become Guix packages to allow the build system to compose modules).

git clone https://github.com/zephyrproject-rtos/hal_nxp && \
git clone https://github.com/zephyrproject-rtos/cmsis

To inform the build system about this module we pass it in with -DZEPHYR_MODULES= which is a semicolon separated list of paths containing a module.yml file.

To build the hello world sample we use the following incantation.

cmake -Bbuild $ZEPHYR_BASE/samples/hello_world \
    -GNinja \
    -DBOARD=frdm_k64f \
    -DBUILD_VERSION=3.1.0 \
    -DZEPHYR_MODULES="/tmp/zephyr-project/hal_nxp;/tmp/zephyr-project/cmsis" \
      && ninja -Cbuild

If everything is set up correctly we will end up with a ./build directory with all our build artifacts. The SDK is correctly installed!

Conclusion

A customized cross toolchain is one of the most difficult pieces of software to build. Using Guix, we do not need to be afraid of the complexity! We can fiddle with settings, swap out components, and do the most brain dead things to our environments without a care in the world. Just exit the environment and it's like it never happened at all.

It highlights one of my favorite aspects of Guix, every package is a working reference design for you to modify and learn from.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

15 March, 2023 04:00PM by Mitchell Schmeisser

March 12, 2023

a2ps @ Savannah

a2ps 4.15.1 released [stable]


GNU a2ps is a filter which generates PostScript from various formats,
with pretty-printing features, strong support for many alphabets, and
customizable layout.

See https://www.gnu.org/software/a2ps/ for more information.

This is a bug-fix release. Users of 4.15 should upgrade. See below for more
details.


Here are the compressed sources and a GPG detached signature:
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.1.tar.gz
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.1.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

8674b90626d6d1505af8b2ae392f2495b589a052  a2ps-4.15.1.tar.gz
l5dwi6AoBa/DtbkeBsuOrJe4WEOpDmbP3mp8Y8oEKyo  a2ps-4.15.1.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.15.1.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.15.1.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5853-ge0aefd96b6

NEWS

* Noteworthy changes in release 4.15.1 (2023-03-12) [stable]
 * Bug fixes:
   - Use “grep -F” rather than obsolete fgrep.
   - Fix broken a2ps-lpr-wrapper script, and translate to sh for
     portability.


12 March, 2023 03:06PM by Reuben Thomas

March 10, 2023

Andy Wingo

pre-initialization of garbage-collected webassembly heaps

Hey comrades, I just had an idea that I won't be able to work on in the next couple months and wanted to release it into the wild. They say if you love your ideas, you should let them go and see if they come back to you, right? In that spirit I abandon this idea to the woods.

Basically the idea is Wizer-like pre-initialization of WebAssembly modules, but for modules that store their data on the GC-managed heap instead of just in linear memory.

Say you have a WebAssembly module with GC types. It might look like this:

(module
  (type $t0 (struct (ref eq)))
  (type $t1 (struct (ref $t0) i32))
  (type $t2 (array (mut (ref $t1))))
  ...
  (global $g0 (ref null eq)
    (ref.null eq))
  (global $g1 (ref $t1)
    (array.new_canon $t0 (i31.new (i32.const 42))))
  ...
  (function $f0 ...)
  ...)

You define some struct and array types, there are some global variables, and some functions to actually do the work. (There are probably also tables and other things but I am simplifying.)

If you consider the object graph of an instantiated module, you will have some set of roots R that point to GC-managed objects. The live objects in the heap are the roots and any object referenced by a live object.

Let us assume a standalone WebAssembly module. In that case the set of types T of all objects in the heap is closed: it can only be one of the types $t0, $t1, and so on that are defined in the module. These types have a partial order and can thus be sorted from most to least specific. Let's assume that this sort order is just the reverse of the definition order, for now. Therefore we can write a general type introspection function for any object in the graph:

(func $introspect (param $obj anyref)
  (block $L2 (ref $t2)
    (block $L1 (ref $t1)
      (block $L0 (ref $t0)
        (br_on_cast $L2 $t2 (local.get $obj))
        (br_on_cast $L1 $t1 (local.get $obj))
        (br_on_cast $L0 $t0 (local.get $obj))
        (unreachable))
      ;; Do $t0 things...
      (return))
    ;; Do $t1 things...
    (return))
  ;; Do $t2 things...
  (return))

In particular, given a WebAssembly module, we can generate a function to trace edges in an object graph of its types. Using this, we can identify all live objects, and what's more, we can take a snapshot of those objects:

(func $snapshot (result (ref (array (mut anyref))))
  ;; Start from roots, use introspect to find concrete types
  ;; and trace edges, use a worklist, return an array of
  ;; all live objects in topological sort order
  )

Having a heap snapshot is interesting for introspection purposes, but my interest is in having fast start-up. Many programs have a kind of "initialization" phase where they get the system up and running, and only then proceed to actually work on the problem at hand. For example, when you run python3 foo.py, Python will first spend some time parsing and byte-compiling foo.py, importing the modules it uses and so on, and then will actually run foo.py's code. Wizer lets you snapshot the state of a module after initialization but before the real work begins, which can save on startup time.

For a GC heap, we actually have similar possibilities, but the mechanism is different. Instead of generating an array of all live objects, we could generate a serialized state of the heap as bytecode, and another function to read the bytecode and reload the heap:

(func $pickle (result (ref (array (mut i8))))
  ;; Return an array of bytecode which, when interpreted,
  ;; can reconstruct the object graph and set the roots
  )
(func $unpickle (param (ref (array (mut i8))))
  ;; Interpret the bytecode, building object graph in
  ;; topological order
  )

The unpickler is module-dependent: it will need one case to construct each concrete type $tN in the module. Therefore the bytecode grammar would be module-dependent too.

What you would get with a bytecode-based $pickle/$unpickle pair would be the ability to serialize and reload heap state many times. But for the pre-initialization case, probably that's not precisely what you want: you want to residualize a new WebAssembly module that, when loaded, will rehydrate the heap. In that case you want a function like:

(func $make-init (result (ref (array (mut i8))))
  ;; Return an array of WebAssembly code which, when
  ;; added to the module as a function and invoked, 
  ;; can reconstruct the object graph and set the roots.
  )

Then you would use binary tools to add that newly generated function to the module.

In short, there is a space open for a tool which takes a WebAssembly+GC module M and produces M', a module which contains a $make-init function. Then you use a WebAssembly+GC host to load the module and call the $make-init function, resulting in a WebAssembly function $init which you then patch in to the original M to make M'', which is M pre-initialized for a given task.

Optimizations

Some of the object graph is constant; for example, an instance of a struct type that has no mutable fields. These objects don't have to be created in the init function; they can be declared as new constant global variables, which an engine may be able to initialize more efficiently.

The pre-initialized module will still have an initialization phase in which it builds the heap. This is a constant function and it would be nice to avoid it. Some WebAssembly hosts will be able to run pre-initialization and then snapshot the GC heap using lower-level facilities (copy-on-write mappings, pointer compression and relocatable cages, pre-initialization on an internal level...). This would potentially decrease latency and may allow for cross-instance memory sharing.

Limitations

There are five preconditions to be able to pickle and unpickle the GC heap:

  1. The set of concrete types in a module must be closed.

  2. The roots of the GC graph must be enumerable.

  3. The object-graph edges from each live object must be enumerable.

  4. To prevent cycles, we have to know when an object has been visited: objects must have identity.

  5. We must be able to create each type in a module.

I think there are three limitations to this pre-initialization idea in practice.

One is externref; these values come from the host and are by definition not introspectable by WebAssembly. Let's keep the closed-world assumption and consider the case where the set of external reference types is closed also. In that case if a module allows for external references, we can perhaps make its pickling routines call out to the host to (2) provide any external roots (3) identify edges on externref values (4) compare externref values for identity and (5) indicate some imported functions which can be called to re-create exernal objects.

Another limitation is funcref. In practice in the current state of WebAssembly and GC, you will only have a funcref which is created by ref.func, and which (3) therefore has no edges and (5) can be re-created by ref.func. However neither WebAssembly nor the JS API has no way of knowing which function index corresponds to a given funcref. Including function references in the graph would therefore require some sort of host-specific API. Relatedly, function references are not comparable for equality (func is not a subtype of eq), which is a little annoying but not so bad considering that function references can't participate in a cycle. Perhaps a solution though would be to assume (!) that the host representation of a funcref is constant: the JavaScript (e.g.) representations of (ref.func 0) and (ref.func 0) are the same value (in terms of ===). Then you could compare a given function reference against a set of known values to determine its index. Note, when function references are expanded to include closures, we will have more problems in this area.

Finally, there is the question of roots. Given a module, we can generate a function to read the values of all reference-typed globals and of all entries in all tables. What we can't get at are any references from the stack, so our object graph may be incomplete. Perhaps this is not a problem though, because when we unpickle the graph we won't be able to re-create the stack anyway.

OK, that's my idea. Have at it, hackers!

10 March, 2023 09:20AM by Andy Wingo

March 07, 2023

a2ps @ Savannah

a2ps 4.15 released [stable]


I am delighted to announce the first stable release of GNU a2ps since 2007!

This release contains few user-visible changes. It does however contain a
lot of changes “under the hood”: code clean-up, etc. Therefore, it’s likely
that there are new bugs. Do report them to Savannah[1], or the mailing list
please!

A big thank-you to all those who tested pre-releases, and especially to
Bruno Haible’s tireless work to promote portability: he both tested a2ps on
many systems and found lots of minor portability problems, and advised on
their solution (often, gnulib code that he wrote). Remaining problems are of
course mine!

[1] https://savannah.gnu.org/projects/a2ps


Here are the compressed sources and a GPG detached signature:
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.tar.gz
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

807667f838c29bde73bb91fae60ef98826bd460e  a2ps-4.15.tar.gz
pa3FqSIvmESKV8a162lItydD6vmjDGehNN8ILpnHZlI  a2ps-4.15.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.15.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.15.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5857-gf17d397771

NEWS

* Noteworthy changes in release 4.15 (2023-03-07) [stable]
 * New maintainer, Reuben Thomas.
 * Features:
   - Replace the 'psmandup' utility with simpler 'lp2' to directly print
     documents to a simplex printer.
   - Remove the outdated 'psset' and 'fixnt', and simplify 'fixps' to
     always process its input with Ghostscript.
   - Use libpaper's paper sizes. This includes user-defined paper sizes
     when using libpaper 2. It is still possible to define custom margins
     using "Medium:" specifications in the configuration file, and the
     one size defined by a2ps that libpaper does not know about, Quarto, is
     retained for backwards compatiblity, and as an example.
 * Documentation
   - Remove some obsolete explanations.
   - Reformat --help output consistently to 80 columns.
   - Some English fixes.
 * Bug fixes:
   - Avoid a crash when a medium is not specified; instead, use the default
     libpaper size (configured by the user or sysadmin, or the locale
     default).
   - Fix some other potential crashes and compiler warnings.
   - Fixes for security bugs CVE-2001-1593, CVE-2015-8107 and CVE-2014-0466.
   - Minor bugs fixed.
 * Predefined delegations:
   - Remove support for defunct Netscape and proprietary Acrobat Reader.
   - Add lpr wrapper for automatic detection of different printing systems,
     including CUPS support.
 * Encodings:
   - Use libre fonts for KOI-8.
   - Composite fonts support.
 * Build
   - Update build system to more recent autotools and gettext versions.
   - Build man pages in a simpler and more robust way.
   - Document runtime dependencies.
   - Minor code quality improvements.
   - Minor tidy up and removal of obsolete code.
   - Require libpaper.
   - Remove OS/2 support.


07 March, 2023 06:20PM by Reuben Thomas

March 05, 2023

grep @ Savannah

grep-3.9 released [stable]


This is to announce grep-3.9, a stable release.

The NEWS below describes the two main bug fixes since 3.8.

There have been 38 commits by 4 people in the 26 weeks since 3.8.

Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Bruno Haible (2)
  Carlo Marcelo Arenas Belón (2)
  Jim Meyering (11)
  Paul Eggert (23)

Jim
 [on behalf of the grep maintainers]
==================================================================

Here is the GNU grep home page:
    http://gnu.org/s/grep/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=grep.git;a=shortlog;h=v3.9
or run this command from a git-cloned grep directory:
  git shortlog v3.8..v3.9

Here are the compressed sources:
  https://ftp.gnu.org/gnu/grep/grep-3.9.tar.gz   (2.7MB)
  https://ftp.gnu.org/gnu/grep/grep-3.9.tar.xz   (1.7MB)

Here are the GPG detached signatures:
  https://ftp.gnu.org/gnu/grep/grep-3.9.tar.gz.sig
  https://ftp.gnu.org/gnu/grep/grep-3.9.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

  f84afbfc8d6e38e422f1f2fc458b0ccdbfaeb392  grep-3.9.tar.gz
  7ZF6C+5DtxJS9cpR1IwLjQ7/kAfSpJCCbEJb9wmfWT8=  grep-3.9.tar.gz
  bcaa3f0c4b81ae4192c8d0a2be3571a14ea27383  grep-3.9.tar.xz
  q80RQJ7iPUyvNf60IuU7ushnAUz+7TE7tfSIrKFwtZk=  grep-3.9.tar.xz

Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify grep-3.9.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096/0x7FD9FCCB000BEEEE 2010-06-14 [SCEA]
        Key fingerprint = 155D 3FC5 00C8 3448 6D1E  EA67 7FD9 FCCB 000B EEEE
  uid                   [ unknown] Jim Meyering <jim@meyering.net>
  uid                   [ unknown] Jim Meyering <meyering@fb.com>
  uid                   [ unknown] Jim Meyering <meyering@gnu.org>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key jim@meyering.net

  gpg --recv-keys 7FD9FCCB000BEEEE

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=grep&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify grep-3.9.tar.gz.sig

This release was bootstrapped with the following tools:
  Autoconf 2.72a.65-d081
  Automake 1.16i
  Gnulib v0.1-5861-g2ba7c75ed1

NEWS

* Noteworthy changes in release 3.9 (2023-03-05) [stable]

** Bug fixes

  With -P, some non-ASCII UTF8 characters were not recognized as
  word-constituent due to our omission of the PCRE2_UCP flag. E.g.,
  given f(){ echo Perú|LC_ALL=en_US.UTF-8 grep -Po "$1"; } and
  this command, echo $(f 'r\w'):$(f '.\b'), before it would print ":r".
  After the fix, it prints the correct results: "rú:ú".

  When given multiple patterns the last of which has a back-reference,
  grep no longer sometimes mistakenly matches lines in some cases.
  [Bug#36148#13 introduced in grep 3.4]


05 March, 2023 04:09PM by Jim Meyering

March 02, 2023

FSF News

Right to repair advocate Elizabeth Chamberlain to keynote FSF's LibrePlanet

BOSTON, Massachusetts, USA -- March 2, 2023 -- The Free Software Foundation (FSF) today announced the director of sustainability at iFixit, Elizabeth Chamberlain, as its closing keynote for LibrePlanet 2023, the fifteenth edition of the FSF's conference on ethical technology and user freedom. The annual technology and social justice conference will be held March 18 and 19, 2023 at the Boston Convention and Exhibition Center as well as online.

02 March, 2023 05:15PM

March 01, 2023

FSF Blogs

guile-cv @ Savannah

Guile-CV version 0.4.0

Guile-CV version 0.4.0 is released! (February 2023)

This is a maintenance release, which introduces new interfaces.

Changes since the previous version

For a list of changes since the previous version, visit the NEWS file. For a complete description, consult the git summary and git log

01 March, 2023 04:26AM by David Pirotte

February 26, 2023

make @ Savannah

GNU Make 4.4.1 Released!

The next stable version of GNU Make, version 4.4.1, has been released and is available for download from https://ftp.gnu.org/gnu/make/?C=M;O=D

Please see the NEWS file that comes with the GNU Make distribution for details on user-visible changes.

26 February, 2023 08:13PM by Paul D. Smith

www @ Savannah

Linux-libre turns 15!

Linux-libre turns 15!

Shared from <http://www.fsfla.org/anuncio/2023-02-Linux-libre-15>

It was February 2008 when Jeff Moe announced Linux-libre, a project to share the efforts that freedom-respecting distros had to undertake to drop the nonfree bits distributed as part of the kernel Linux.

> "For fifteen years, the Linux-libre project has remained dedicated
> to providing a kernel that respects everyone's freedom and has
> become an essential part of the free software movement. Linux-libre
> is widely used by those who value their freedom to use, study,
> change, and share software without restrictions or limitations.
> These freedoms are essential to creating a just society."
> -- Jason Self


Since around 1996, Linux has carried sourceless firmware encoded as sequences of numbers disguised as source code. UTUTO and gNewSense pioneered the efforts of removing them. Cleaning Linux up is a substantial amount of work, so the existence of Linux-libre has alleviated one of the main difficulties in maintaining GNU+Linux distros that abide by the GNU Free Software Distribution Guidelines. The Linux-libre compiled kernel distributions maintained by Jason Self, Freesh (.deb), liberRTy (low-latency .deb) and RPMFreedom (.rpm), make it easy for users of other GNU+Linux distros to take a step towards freedom when their hardware is not too user-hostile.

> "Thanks to Linux-libre, we have entirely libre GNU+Linux distros.
> Thanks to Linux-libre, people like me who are not kernel hackers can
> install one of those distros and have a computer which never runs a
> nonfree program on the CPU. (Provided we use LibreJS as well to
> reject nonfree Javascript programs that web sites send us.)"
> -- Richard Stallman


Early pieces of firmware in Linux ran peripheral devices, but some of the blobs loaded by Linux nowadays reconfigure the primary central processing units and others contain an entire operating system for the peripherals' CPUs, including a copy of the kernel Linux itself and several other freedom-depriving programs!

After years of our denouncing the social, technical, and legal risks out of Linux's misbehavior, most of the blobs got moved to separate files, still part of the kernel Linux, and then to separate packages, which mitigates some of the legal risks, but the problem keeps growing: more and more devices depend on nonfree firmware and thus remain under exclusive and proprietary control by their suppliers.

Challenge

For 27 years, the nonfree versions of Linux have shown that tolerating blobs and making it easy for users to install and accept them makes users increasingly dependent on user-hostile, blob-requiring devices for their computing. Refusing to give these devices' suppliers what they wish, namely your money and control over your computing, is more likely to succeed at changing their practices if more users refuse.

If you're the kind of software freedom supporter who demands respect for your freedom, keep on enjoying the instant gratification that GNU Linux-libre affords you, and supporting (or being!) those who refurbish old computers and build new ones to respect our autonomy.

However, if you're of the kind for whom last-generation computers are hard to resist, even though you'd prefer if they were more respectful of your freedom, you may wish to consider a delayed gratification challenge: if you and your friends resist hostile computers now, you may get more respectful ones later, for yourselves and for all of us; if you don't, the next generations will likely be even more hostile. Are you up for the challenge?

Present and Future

GNU Linux-libre releases are currently prepared with scripts that automate the cleaning-up and part of the verification. For each upstream major and stable release, we run the scripts, updating them as needed, and publish them, along with the cleaning-up logs and the cleaned-up sources, in a git repository. Each source release is an independent tag, as in, there are no branches for cleaned-up sources. This is so we can quickly retract releases if freedom bugs are found.

We have plans to change the cleaning-up process and the repository structure in the future: we're (slowly) preparing to move to a rewritten git repository, in which, for each commit in upstream Linux main and stable repositories, there will be a corresponding cleaned-up commit in ours. Undesirable bits are going to be cleaned up at the commit corresponding to the one in which upstream introduced or modified them, and other modifications will be checked and integrated unchanged, mirroring the upstream commit graph, with "git replace" mappings for individual commits and, perhaps, also for cleaned-up files.

This is expected to enable us to track upstream development very closely, to get stable and major releases out nearly instantly and often automatically and to enable Linux developers to clone our freed repository instead of our upstream to write and test their changes. The same techniques used to create the cleaned-up repository can be used to fix freedom bugs in it.

Artwork

Jason Self has made several beautiful pictures of his version of Freedo, our light-blue penguin mascot, and we've used them for our recent releases.

Marking the beginning of the week in which we celebrate 15 years of Linux-libre, we had the pleasure of publishing a major release, 6.2-gnu, codenamed "la quinceañera", with a picture of Freedo dressed up for the occasion: <https://www.fsfla.org/pipermail/linux-libre/2023-February/003502.html>

But there's more! He also made a commemorative black-and-white wallpaper with classic Freedo, also dressed up for the occasion. Check them out, and feel free to tune the colors to your liking! <https://linux-libre.fsfla.org/#news>

He also modeled a 3D Freedo in Blender, and we're looking for someone who could 3D-print it and get it to the FSF office in time for the LibrePlanet conference. Rumor has it that Richard Stallman is going to auction it off to raise funds for the FSF! Can you help?


About GNU Linux-libre

GNU Linux-libre is a GNU package maintained by Alexandre Oliva, on behalf of FSFLA, and by Jason Self. It releases cleaned-up versions of Linux, suitable for use in distributions that comply with the Free Software Distribution Guidelines published by the GNU project, and by users who wish to run Free versions of Linux on their GNU systems. The project offers cleaning-up scripts, Free sources, binaries for some GNU+Linux distributions, and artwork with GNU and the Linux-libre mascot: Freedo, the clean, Free and user-friendly light-blue penguin. Visit our web site and Be Free!

About the GNU Operating System and Linux

Richard Stallman announced in September 1983 the plan to develop a Free Software Unix-like operating system called GNU. GNU is the only
operating system developed specifically for the sake of users' freedom: <http://www.gnu.org/gnu/the-gnu-project.html>

In 1992, the essential components of GNU were complete, except for one, the kernel. When in 1992 the kernel Linux was re-released under the GNU GPL, making it Free Software, the combination of GNU and Linux formed a complete Free operating system, which made it possible for the first time to run a PC without non-Free Software. This combination is the GNU+Linux system: <http://www.gnu.org/gnu/gnu-linux-faq.html>

About FSFLA

Free Software Foundation Latin America joined in 2005 the international FSF network, previously formed by Free Software Foundations in the United States, in Europe and in India. These sister organizations work in their corresponding geographies towards promoting the same Free Software ideals and defending the same freedoms for software users and developers, working locally but cooperating globally.


Copyright 2023 FSFLA

Permission is granted to make and distribute verbatim copies of this entire document without royalty, provided the copyright notice, the document's official URL, and this permission notice are preserved.

Permission is also granted to make and distribute verbatim copies of individual sections of this document worldwide without royalty provided the copyright notice and the permission notice above are preserved, and the document's official URL is preserved or replaced by the individual section's official URL.

26 February, 2023 02:46AM by Jason Self

February 24, 2023

GNU Health

Fundación La Vicuña joins GNU Health

On Thursday, Feb 23rd, 2023, GNU Solidario and the Spanish NGO Fundación La Vicuña ORL have signed a cooperation agreement to promote and implement the Health and Hospital Management component from GNUHealth in those areas and institutions where Fundación La Vicuña has activities, mainly Spain and countries in Africa.

Fundación La Vicuña is a non-profit organization founded 15 years ago by a group of physicians, mostly ear, nose and throat specialists in Cadiz, Spain.

GNU Solidario and Fundacion La Vicuña share the goal of improving the lives of the underprivileged, through Social Medicine and universal access to healthcare. GNU Health will be a very valuable tool to assess the socioeconomic determinants of health and to minimize the impact in the vulnerable population, both in Spain and in the African continent. GNU Health will improve the management of health institutions and the daily medical practice where Fundación La Vicuña has missions. Patient evaluations, medical records, prescriptions, laboratory, surgeries and inpatient/hospitalization will be some of the areas that will benefit from GNU Health HMIS.

Casimiro García, president and founder of Fundación La Vicuña and Luis Falcón, founder and president of GNU Solidario, formalized the cooperation agreement this Thursday. In the coming weeks, GNU Solidario will train the team from Fnd. La Vicuña in the use of GNUHealth, and a development environment will be rolled out.

We are thrilled and looking forward to working hand in hand with Fundación la Vicuña, to put into practice the philosophy of open science and Libre software in healthcare for the betterment of our societies, delivering Social Medicine and dignity to those who need it most.

For more information you can visit Fundación la Vicuña homepage (in Spanish): http://www.fundacionlavicuna.org/

24 February, 2023 06:54PM by GNU Solidario

February 23, 2023

FSF Blogs

February 22, 2023

parallel @ Savannah

GNU Parallel 20230222 ('Gaziantep') released

GNU Parallel 20230222 ('Gaziantep') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  Praise GNU parallel, though. That gets me pretty far.
    -- Your Obed. Servant, J. B. @Jeffinatorator

New in this release:

  • parsort: --parallel now does closer to what you expect.


  • parallel: --files0 is --files but \0 separated.


  • Bug fixes and man page updates.


News about GNU Parallel:


GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.


About GNU Parallel


GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference


If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)


If GNU Parallel saves you money:



About GNU SQL


GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.


About GNU Niceload


GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

22 February, 2023 10:41PM by Ole Tange

February 21, 2023

FSF Blogs

February 20, 2023

GNU Guix

Dissecting Guix, Part 2: The Store Monad

Hello again!

In the last post, we briefly mentioned the with-store and run-with-store macros. Today, we'll be looking at those in further detail, along with the related monad library and the %store-monad!

Typically, we use monads to chain operations together, and the %store-monad is no different; it's used to combine operations that work on the Guix store (for instance, creating derivations, building derivations, or adding data files to the store).

However, monads are a little hard to explain, and from a distance, they seem to be quite incomprehensible. So, I want you to erase them from your mind for now. We'll come back to them later. And be aware that if you can't seem to get your head around them, it's okay; you can understand most of the architecture of Guix without understanding monads.

Yes, No, Maybe So

Let's instead implement another M of functional programming, maybe values, representing a value that may or may not exist. For instance, there could be a procedure that attempts to pop a stack, returning the result if there is one, or nothing if the stack has no elements.

maybe is a very common feature of statically-typed functional languages, and you'll see it all over the place in Haskell and OCaml code. However, Guile is dynamically typed, so we usually use ad-hoc #f values as the "null value" instead of a proper "nothing" or "none".

Just for fun, though, we'll implement a proper maybe in Guile. Fire up that REPL once again, and let's import a bunch of modules that we'll need:

(use-modules (ice-9 match)
             (srfi srfi-9))

We'll implement maybe as a record with two fields, is? and value. If the value contains something, is? will be #t and value will contain the thing in question, and if it's empty, is?'ll be #f.

(define-record-type <maybe>
  (make-maybe is? value)
  maybe?
  (is? maybe-is?)
  (value maybe-value))

Now we'll define constructors for the two possible states:

(define (something value)
  (make-maybe #t value))

(define (nothing)
  (make-maybe #f #f)) ;the value here doesn't matter; we'll just use #f

And make some silly functions that return optional values:

(define (remove-a str)
  (if (eq? (string-ref str 0) #\a)
      (something (substring str 1))
      (nothing)))

(define (remove-b str)
  (if (eq? (string-ref str 0) #\b)
      (something (substring str 1))
      (nothing)))

(remove-a "ahh")
⇒ #<<maybe> is?: #t value: "hh">

(remove-a "ooh")
⇒ #<<maybe> is?: #f value: #f>

(remove-b "bad")
⇒ #<<maybe> is?: #t value: "ad">

But what if we want to compose the results of these functions?

Keeping Your Composure

As you might have guessed, this is not fun. Cosplaying as a compiler backend typically isn't.

(let ((t1 (remove-a "abcd")))
  (if (maybe-is? t1)
      (remove-b (maybe-value t1))
      (nothing)))
⇒ #<<maybe> is?: #t value: "cd">

(let ((t1 (remove-a "bbcd")))
  (if (maybe-is? t1)
      (remove-b (maybe-value t1))
      (nothing)))
⇒ #<<maybe> is?: #f value: #f>

I can almost hear the heckling. Even worse, composing three:

(let* ((t1 (remove-a "abad"))
       (t2 (if (maybe-is? t1)
               (remove-b (maybe-value t1))
               (nothing))))
  (if (maybe-is? t2)
      (remove-a (maybe-value t2))
      (nothing)))
⇒ #<<maybe> is?: #t value: "d">

So, how do we go about making this more bearable? Well, one way could be to make remove-a and remove-b accept maybes:

(define (remove-a ?str)
  (match ?str
    (($ <maybe> #t str)
     (if (eq? (string-ref str 0) #\a)
         (something (substring str 1))
         (nothing)))
    (_ (nothing))))

(define (remove-b ?str)
  (match ?str
    (($ <maybe> #t str)
     (if (eq? (string-ref str 0) #\b)
         (something (substring str 1))
         (nothing)))
    (_ (nothing))))

Not at all pretty, but it works!

(remove-b (remove-a (something "abc")))
⇒ #<<maybe> is?: #t value: "c">

Still, our procedures now require quite a bit of boilerplate. Might there be a better way?

The Ties That >>= Us

First of all, we'll revert to our original definitions of remove-a and remove-b, that is to say, the ones that take a regular value and return a maybe.

(define (remove-a str)
  (if (eq? (string-ref str 0) #\a)
      (something (substring str 1))
      (nothing)))

(define (remove-b str)
  (if (eq? (string-ref str 0) #\b)
      (something (substring str 1))
      (nothing)))

What if tried introducing higher-order procedures (procedures that accept other procedures as arguments) into the equation? Because we're functional programmers and we have an unhealthy obsession with that sort of thing.

(define (maybe-chain maybe proc)
  (if (maybe-is? maybe)
      (proc (maybe-value maybe))
      (nothing)))

(maybe-chain (something "abc")
             remove-a)
⇒ #<<maybe> is?: #t value: "bc">

(maybe-chain (nothing)
             remove-a)
⇒ #<<maybe> is?: #f value: #f>

It lives! To make it easier to compose procedures like this, we'll define a macro that allows us to perform any number of sequenced operations with only one composition form:

(define-syntax maybe-chain*
  (syntax-rules ()
    ((_ maybe proc)
     (maybe-chain maybe proc))
    ((_ maybe proc rest ...)
     (maybe-chain* (maybe-chain maybe proc)
                   rest ...))))

(maybe-chain* (something "abad")
              remove-a
              remove-b
              remove-a)
⇒ #<<maybe> is?: #t value: "d">

Congratulations, you've just implemented the bind operation, commonly written as >>=, for our maybe type. And it turns out that a monad is just any container-like value for which >>= (along with another procedure called return, which wraps a given value in the simplest possible form of a monad) has been implemented.

A more formal definition would be that a monad is a mathematical object composed of three parts: a type, a bind function, and a return function. So, how do monads relate to Guix?

New Wheel, Old Wheel

Now that we've reinvented the wheel, we'd better learn to use the original wheel. Guix provides a generic, high-level monads library, along with the two generic monads %identity-monad and %state-monad, and the Guix-specific %store-monad. Since maybe is not one of them, let's integrate our version into the Guix monad system!

First we'll import the module that provides the aforementioned library:

(use-modules (guix monads))

To define a monad's behaviour in Guix, we simply use the define-monad macro, and provide two procedures: bind, and return.

(define-monad %maybe-monad
  (bind maybe-chain)
  (return something))

bind is just the procedure that we use to compose monadic procedure calls together, and return is the procedure that wraps values in the most basic form of the monad. A properly implemented bind and return must follow the so-called monad laws:

  1. (bind (return x) proc) must be equivalent to (proc x).
  2. (bind monad return) must be equivalent to just monad.
  3. (bind (bind monad proc-1) proc-2) must be equivalent to (bind monad (lambda (x) (bind (proc-1 x) proc-2))).

Let's verify that our maybe-chain and something procedures adhere to the monad laws:

(define (mlaws-proc-1 x)
  (something (+ x 1)))

(define (mlaws-proc-2 x)
  (something (+ x 2)))

;; First law: the left identity.
(equal? (maybe-chain (something 0)
                     mlaws-proc-1)
        (mlaws-proc-1 0))
⇒ #t

;; Second law: the right identity.
(equal? (maybe-chain (something 0)
                     something)
        (something 0))
⇒ #t

;; Third law: associativity.
(equal? (maybe-chain (maybe-chain (something 0)
                                  mlaws-proc-1)
                     mlaws-proc-2)
        (maybe-chain (something 0)
                     (lambda (x)
                       (maybe-chain (mlaws-proc-1 x)
                                    mlaws-proc-2))))
⇒ #t

Now that we know they're valid, we can use the with-monad macro to tell Guix to use these specific implementations of bind and return, and the >>= macro to thread monads through procedure calls!

(with-monad %maybe-monad
  (>>= (something "aabbc")
       remove-a
       remove-a
       remove-b
       remove-b))
⇒ #<<maybe> is?: #t value: "c">

We can also now use return:

(with-monad %maybe-monad
  (return 32))
⇒ #<<maybe> is?: #t value: 32>

But Guix provides many higher-level interfaces than >>= and return, as we will see. There's mbegin, which evaluates monadic expressions without binding them to symbols, returning the last one. This, however, isn't particularly useful with our %maybe-monad, as it's only really usable if the monadic operations within have side effects, just like the non-monadic begin.

There's also mlet and mlet*, which do bind the results of monadic expressions to symbols, and are essentially equivalent to a chain of (>>= MEXPR (lambda (BINDING) ...)):

;; This is equivalent...
(mlet* %maybe-monad ((str -> "abad") ;non-monadic binding uses the -> symbol
                     (str1 (remove-a str))
                     (str2 (remove-b str)))
  (remove-a str))
⇒ #<<maybe> is?: #t value: "d">

;; ...to this:
(with-monad %maybe-monad
  (>>= (return "abad")
       (lambda (str)
         (remove-a str))
       (lambda (str1)
         (remove-b str))
       (lambda (str2)
         (remove-a str))))

Various abstractions over these two exist too, such as mwhen (a when plus an mbegin), munless (an unless plus an mbegin), and mparameterize (dynamically-scoped value rebinding, like parameterize, in a monadic context). lift takes a procedure and a monad and creates a new procedure that returns a monadic value.

There are also interfaces for manipulating lists wrapped in monads; listm creates such a list, sequence turns a list of monads into a list wrapped in a monad, and the anym, mapm, and foldm procedures are like their non-monadic equivalents, except that they return lists wrapped in monads.

This is all well and good, you may be thinking, but why does Guix need a monad library, anyway? The answer is technically that it doesn't. But building on the monad API makes a lot of things much easier, and to learn why, we're going to look at one of Guix's built-in monads.

In a State

Guix implements a monad called %state-monad, and it works with single-argument procedures returning two values. Behold:

(with-monad %state-monad
  (return 33))
⇒ #<procedure 21dc9a0 at <unknown port>:1106:22 (state)>

The run-with-state value turns this procedure into an actually useful value, or, rather, two values:

(run-with-state (with-monad %state-monad (return 33))
  (list "foo" "bar" "baz"))
⇒ 33
⇒ ("foo" "bar" "baz")

What can this actually do for us, though? Well, it gets interesting if we do some >>=ing:

(define state-seq
  (mlet* %state-monad ((number (return 33)))
    (state-push number)))
result
⇒ #<procedure 7fcb6f466960 at <unknown port>:1484:24 (state)>

(run-with-state state-seq (list 32))
⇒ (32)
⇒ (33 32)

(run-with-state state-seq (list 30 99))
⇒ (30 99)
⇒ (33 30 99)

What is state-push? It's a monadic procedure for %state-monad that takes whatever's currently in the first value (the primary value) and pushes it onto the second value (the state value), which is assumed to be a list, returning the old state value as the primary value and the new list as the state value.

So, when we do (run-with-state result (list 32)), we're passing (list 32) as the initial state value, and then the >>= form passes that and 33 to state-push. What %state-monad allows us to do is thread together some procedures that require some kind of state, while essentially pretending the state value is stored globally, like you might do in, say, C, and then retrieve both the final state and the result at the end!

If you're a bit confused, don't worry. We'll write some of our own %state-monad-based monadic procedures and hopefully all will become clear. Consider, for instance, the Fibonacci sequence, in which each value is computed by adding the previous two. We could use the %state-monad to compute Fibonacci numbers by storing the previous number as the primary value and the number before that as the state value:

(define (fibonacci-thing value)
  (lambda (state)
    (values (+ value state)
            value)))

Now we can feed our Fibonacci-generating procedure the first value using run-with-state and the second using return:

(run-with-state
    (mlet* %state-monad ((starting (return 1))
                         (n1 (fibonacci-thing starting))
                         (n2 (fibonacci-thing n1)))
      (fibonacci-thing n2))
  0)
⇒ 3
⇒ 2

(run-with-state
    (mlet* %state-monad ((starting (return 1))
                         (n1 (fibonacci-thing starting))
                         (n2 (fibonacci-thing n1))
                         (n3 (fibonacci-thing n2))
                         (n4 (fibonacci-thing n3))
                         (n5 (fibonacci-thing n4)))
      (fibonacci-thing n5))
  0)
⇒ 13
⇒ 8

This is all very nifty, and possibly useful in general, but what does this have to do with Guix? Well, many Guix store-based operations are meant to be used in concert with yet another monad, called the %store-monad. But if we look at (guix store), where %store-monad is defined...

(define-alias %store-monad %state-monad)
(define-alias store-return state-return)
(define-alias store-bind state-bind)

It was all a shallow façade! All the "store monad" is is a special case of the state monad, where a value representing the store is passed as the state value.

Lies, Damned Lies, and Abstractions

We mentioned that, technically, we didn't need monads for Guix. Indeed, many (now deprecated) procedures take a store value as the argument, such as build-expression->derivation. However, monads are far more elegant and simplify store code by quite a bit.

build-expression->derivation, being deprecated, should never of course be used. For one thing, it uses the "quoted build expression" style, rather than G-expressions (we'll discuss gexps another time). The best way to create a derivation from some basic build code is to use the new-fangled gexp->derivation procedure:

(use-modules (guix gexp)
             (gnu packages irc))

(define symlink-irssi
  (gexp->derivation "link-to-irssi"
    #~(symlink #$(file-append irssi "/bin/irssi") #$output)))
⇒ #<procedure 7fddcc7b81e0 at guix/gexp.scm:1180:2 (state)>

You don't have to understand the #~(...) form yet, only everything surrounding it. We can see that this gexp->derivation returns a procedure taking the initial state (store), just like our %state-monad procedures did, and like we used run-with-state to pass the initial state to a %state-monad monadic value, we use our old friend run-with-store when we have a %store-monad monadic value!

(define symlink-irssi-drv
  (with-store store
    (run-with-store store
      symlink-irssi)))
⇒ #<derivation /gnu/store/q7kwwl4z6psifnv4di1p1kpvlx06fmyq-link-to-irssi.drv => /gnu/store/6a94niigx4ii0ldjdy33wx9anhifr25x-link-to-irssi 7fddb7ef52d0>

Let's just check this derivation is as expected by reading the code from the builder script.

(define symlink-irssi-builder
  (list-ref (derivation-builder-arguments symlink-irssi-drv) 1))

(call-with-input-file symlink-irssi-builder
  (lambda (port)
    (read port)))

⇒ (symlink
   "/gnu/store/hrlmypx1lrdjlxpkqy88bfrzg5p0bn6d-irssi-1.4.3/bin/irssi"
   ((@ (guile) getenv) "out"))

And indeed, it symlinks the irssi binary to the output path. Some other, higher-level, monadic procedures include interned-file, which copies a file from outside the store into it, and text-file, which copies some text into it. Generally, these procedures aren't used, as there are higher-level procedures that perform similar functions (which we will discuss later), but for the sake of this blog post, here's an example:

(with-store store
  (run-with-store store
    (text-file "unmatched-paren"
      "( <paren@disroot.org>")))
⇒ "/gnu/store/v6smacxvdk4yvaa3s3wmd54lixn1dp3y-unmatched-paren"

Conclusion

What have we learned about monads? The key points we can take away are:

  1. Monads are a way of composing together procedures and values that are wrapped in containers that give them extra context, like maybe values.
  2. Guix provides a high-level monad library that compensates for Guile's lack of static typing or an interface-like system.
  3. The (guix monads) module provides the state monad, which allows you to thread state through procedures, allowing you to essentially pretend it's a global variable that's modified by each procedure.
  4. Guix uses the store monad frequently to thread a store connection through procedures that need it.
  5. The store monad is really just the state monad in disguise, where the state value is used to thread the store object through monadic procedures.

If you've read this post in its entirety but still don't yet quite get it, don't worry. Try to modify and tinker about with the examples, and ask any questions on the IRC channel #guix:libera.chat and mailing list at help-guix@gnu.org, and hopefully it will all click eventually!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

20 February, 2023 09:30PM by (

a2ps @ Savannah

a2ps 4.14.95 released [alpha]


This alpha release benefits from feedback from the platform-testers list
(mostly Bruno Haible, thanks Bruno!) The work is all on the build system.
If you have not tried a previous alpha release for functionality, now is the
time!

Here are the compressed sources and a GPG detached signature:
  https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.95.tar.gz
  https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.95.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

3169e01029bb2eec80feb488bafdd417fb35c7d5  a2ps-4.14.95.tar.gz
pP7eBLeaAn/4x48sq8548vTAkj0rpMi2yToQuCRRgvg  a2ps-4.14.95.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.14.95.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.14.95.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5836-geecd8afd69

NEWS

* Noteworthy changes in release 4.14.95 (2023-02-20) [alpha]
 * Build
  - Build man pages in a simpler and more robust way, using x-to-1 from
    gnulib.
  - Don't install useless liba2ps.h.
  - Other minor build system improvements, including updating to more recent
    autoconf and gettext.
  - Don't require help2man or gperf to build from source.
  - Document runtime dependencies.
  - Minor code quality improvements.


20 February, 2023 09:03PM by Reuben Thomas

February 17, 2023

lightning @ Savannah

GNU lightning 2.2.1 release

GNU lightning is a library to aid in making portable programs
that compile assembly code at run time.

Development:
http://git.savannah.gnu.org/cgit/lightning.git

Download release:
ftp://ftp.gnu.org/gnu/lightning/lightning-2.2.1.tar.gz

  GNU Lightning 2.2.1 main new features:

  • Variable stack framesize implemented for aarch64, arm, i686, mips, riscv, loongarch and x86_64. This means function calls use only the minimum required stack space for prolog and epilog.
  • Optimization of prolog and epilog to not create a frame pointer if not required, and not even save and restore the stack pointer if not required on a leaf function. These features implemented for the ports with variable stack framesize.
  • New clor, czr, ctor and ctzr instructions, that count leading/trailing zeros/ones. These use hardware implementation when available, otherwise fallback to a software implementation.
  • Correct several bugs with jit_arg_register_p and jit_putarg{r,i}{_f,_d}. These bugs were not noticed earlier due to an incorrect check for correctness in check/carg.c.
  • Add rip relative addressing support for x86_64 and shorter signed 64 bit constant load if the constant fits in a signed 32 bit integer. This significantly reduces code size generation.
  • Correct bugs in branch generation code for pppc and sparc.
  • Correct bug in signed 32 bit integer load in ppc 64 bits.
  • Add short relative unconditional branches and calls to mips, reducing code size generation.
  • And several extra minor optimizations.

17 February, 2023 07:27PM by Paulo César Pereira de Andrade

February 12, 2023

Parabola GNU/Linux-libre

[From Arch] Switch to the base-devel meta package requires manual intervention

The base-devel package group has recently been replaced by a meta package of the same name.
People that had the base-devel package group installed (meaning people that installed base-devel before February 2nd) have to explicitly re-install it to get the new base-devel package installed on their system:

pacman -Syu base-devel

12 February, 2023 06:50PM by bill auger

February 11, 2023

health @ Savannah

GNU Health Hospital Management 4.2 series released!

Dear community:

I am excited to announce the release of series 4.2 from the GNU Health Hospital Management Information System (HMIS) component!

The 4.2 series is the result of one year of work and cooperation with the community. Many new exciting features are included, bugs fixed and new translations are now in place.

What is new in GNUHealth Hospital Management 4.2 series

The following is a summary of the main new features included in GH 4.2 .

  • Enhanced Medical Imaging functionality and ergonomics.
  • Introduced GNU Health "Focus on" section on patient main form. Key health indicators that put our attention on the patient main health conditions. The main indicators are Cardiovascular (excl hbp), HBP, Nutrition, Cognitive, Social, Cancer and Immuno.
  • Surgery package has been vastly revised and enhanced. Thanks to the collaboration with our colleagues from Cirugia Solidaria, the surgery package is now being used in thousands of operations and all the new functionality is available in this 4.2.
  • Enhanced Insurance and billing functionality. Now, in one view we can integrate all product policies from a particular insurance company plan. The 4.2 release also allows to include a fixed prices on product or category.
  • The Vital Record System (VRS) can now issue reports on birth and death certificates.
  • Demographics can now accept entering estimate age / DoB.
  • Health services has now the functionality of "grouping" all the tests from a single order - lab and medical imaging -. It allows automatically updating the service document directly right from the request order wizard. There a new "ungroup" checkbox that, when set, it will behave as today, ie, giving the option to the manager.
  • Improved Patient encounter / evaluation. Medical interventions, DDx and secondary conditions. A new report is now available that summarizes the key information from the evaluation.
  • Weblate translations holds 35 languages!
  • Instance and connection information visible at the GTK client title
  • On the technical side, we have improved unit testing on each package, speedup load times on large datafiles and using python-sql for most queries.
  • Last but not least.... GNU Health is now REUSE (Free Software Foundation Europe) compliant! This is a great step forward, since REUSE facilitates documenting and sharing licenses of Libre projects like GNU Health. It's been quite a bit of work, but definitely worth!

Upgrading from GNU Health 4.0

The GNUHealth 4.2 will benefit from the stability of using Tryton 6.0! Still, at GH level there are significant changes on the data dictionary and kernel.

As usual: 

  • Make a FULL BACKUP your kernel, database and attach directories !!!
  • Follow the instructions on the Wikibooks.

Development focus

In addition of the GH HMIS server, we will focus the development in the
following  areas of the GNU Health ecosystem:

  • The Documentation Portal. We now have a dedicated server that will host the documentation for the GNUHealth ecosystem components.

  The docmentation portal is a read-only resource, focusing on stability and high-quality. We will also keep using Wikibooks as a community wiki, as well as development.

  • MyGNUHealth: The GNU Health app for desktop and mobile devices
  • Thalamus and the Federation Portal. The GNU Health Federation integrates information from many health institutions and individuals from a region or country. The GH Federation portal will  allow to manage resources, as well as the main point for **analytics** and **reporting** of massive demographics and epidemiological data generated nationwide. People, health centers and research institutions will benefit from the GNU Health Federation and the GNU Health ecosystem in general.

As always, no matter how hard we try to avoid them, there will be bugs, so please test the new system, upgrade process, languages, and give us your feedback via them via health@gnu.org

The community server has been already migrated to 4.2.0, so you just need to download the GNU Health HMIS client.

Happy and Healthy Hacking !

--
Dr. Luis Falcon, M.D.
President, GNU Solidario
Advancing Social Medicine
https://www.gnuhealth.org

11 February, 2023 11:55PM by Luis Falcon

February 09, 2023

GNUnet News

Messenger-GTK 0.8.0

Messenger-GTK 0.8.0

Following the new release of "libgnunetchat" there have been some changes regarding the applications utilizing it. So we are pleased to announce the new release of the Messenger-GTK application. This release will be compatible with libgnunetchat 0.1.3 and GNUnet 0.19.3 upwards.

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

Noteworthy changes in 0.8.0

  • A simple media player has been added to the user interface to play received audio and video files without a third-party application.
  • Multiple crashes and breaking issues have been fixed.

A detailed list of changes can be found in the ChangeLog .

Known Issues

  • Chats still require a reliable connection between GNUnet peers. So this still depends on the upcoming NAT traversal to be used outside of local networks for most users (see #5710 ).
  • File sharing via the FS service should work in a GNUnet single-user setup but a multi-user setup breaks it (see #7355 )
  • Mobile devices might come with multiple sensors per camera which doesn't get recogniced by the GStreamer pipeline. As soon as libcamera supports those devices, there will be changes to utilize that fixing this issue.

In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org .

messenger-cli 0.1.1

There's also a new release of the terminal application using the GNUnet Messenger service. But since that's only a release to provide compatibility with changes in its dependencies it is only a minor release without adding new features.

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

09 February, 2023 11:00PM

February 08, 2023

Gary Benson

Flask on Elastic Beanstalk

I had a play with Elastic Beanstalk the other day. It’s one of those things people turn their noses up at, but it seems pretty good for prototyping and small things. My biggest issue so far has been that, for Python applications, it expects a WSGI callable called application in the file application.py… but I was using Flask, and every single Flask application ever has a WSGI callable called app in the file app.py. I tried to not care, but it got too much after about an hour so I went and found how to override it:

$ cat .ebextensions/01_wsgi.config
option_settings:
  aws:elasticbeanstalk:container:python:
    WSGIPath: "app:app"

Thank you Nik Tomazic for that! (=⌒‿‿⌒=)

08 February, 2023 11:35PM by gbenson

February 07, 2023

The other AWS IAM SSO problem

The other thing you’ll run into using IAM users in AWS CLI is that a lot of things don’t support SSO sessions anyway. If you configure an IAM user with an SSO session name as recommended you’ll get errors like this:

$ eb init -p python-3.8 eb-flask-app
ERROR: InvalidConfigError - The profile "default" is configured to use SSO but is missing required configuration: sso_start_url, sso_region

and this:

$ terraform apply
| Error: configuring Terraform AWS Provider: loading configuration: profile "default" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

You can fix these by configuring without an SSO session:

$ aws configure sso
SSO session name (Recommended): 
WARNING: Configuring using legacy format (e.g. without an SSO session).
Consider re-running "configure sso" command and providing a session name.
SSO start URL [None]: https://whatever.awsapps.com/start
SSO region [None]: us-east-1
 ...

You can also fix them by just editing your ~/.aws/config, and copying the sso_start_url and sso_region keys from the [sso-session ...] section into the relevant user’s section, but that might be a hack too far!

07 February, 2023 02:44PM by gbenson

Andy Wingo

whippet: towards a new local maximum

Friends, you might have noted, but over the last year or so I really caught the GC bug. Today's post sums up that year, in the form of a talk I gave yesterday at FOSDEM. It's long! If you prefer video, you can have a look instead to the at the FOSDEM event page.

Whippet: A New GC for Guile

4 Feb 2023 – FOSDEM

Andy Wingo

Guile is...

Mostly written in Scheme

Also a 30 year old C library

// API
SCM scm_cons (SCM car, SCM cdr);

// Many third-party users
SCM x = scm_cons (a, b);

So the context for the whole effort is that Guile has this part of its implementation which is in C. It also exposes a lot of that implementation to users as an API.

Putting the C into GC

SCM x = scm_cons (a, b);

Live objects: the roots, plus anything a live object refers to

How to include x into roots?

  • Refcounting
  • Register (& later unregister) &x with gc
  • Conservative roots

So what contraints does this kind of API impose on the garbage collector?

Let's start by considering the simple cons call above. In a garbage-collected environment, the GC is responsible for reclaiming unused memory. How does the GC know that the result of a scm_cons call is in use?

Generally speaking there are two main strategies for automatic memory management. One is reference counting: you associate a count with an object, incremented once for each referrer; in this case, the stack would hold a reference to x. When removing the reference, you decrement the count, and if it goes to 0 the object is unused and can be freed.

We GC people used to laugh at reference-counting as a memory management solution because it over-approximates the live object set in the presence of cycles, but it would seem that refcounting is coming back. Anyway, this isn't what Guile does, not right now anyway.

The other strategy we can use is tracing: the garbage collector periodically finds all of the live objects on the system and then recycles the memory for everything else. But how to actually find the first live objects to trace?

One way is to inform the garbage collector of the locations of all roots: references to objects originating from outside the heap. This can be done explicitly, as in V8's Handle<> API, or implicitly, in the form of a side table generated by the compiler associating code locations with root locations. This is called precise rooting: the GC is aware of all root locations at all code positions where GC might happen. Generally speaking you want the side table approach, in which the compiler writes out root locations to stack maps, because it doesn't impose any overhead at run-time to register and unregister locations. However for run-time routines implemented in C or C++, you won't be able to get the C compiler to do this for you, so you need the explicit approach if you want precise roots.

Conservative roots

Treat every word in stack as potential root; over-approximate live object set

1993: Bespoke GC inherited from SCM

2006 (1.8): Added pthreads, bugs

2009 (2.0): Switch to BDW-GC

BDW-GC: Roots also from extern SCM foo;, etc

The other way to find roots is very much not The Right Thing. Call it cheeky, call it sloppy, call it yolo, call it what you like, but in the trade it's known as conservative root-finding. This strategy looks like this:

uintptr_t *limit = stack_base_for_platform();
uintptr_t *sp = __builtin_frame_address();
for (; sp < limit; sp++) {
  void *obj = object_at_address(*sp);
  if (obj) add_to_live_objects(obj);
}

You just look at every word on the stack and pretend it's a pointer. If it happens to point to an object in the heap, we add that object to the live set. Of course this algorithm can find a spicy integer whose value just happens to correspond to an object's address, even if that object wouldn't have been counted as live otherwise. This approach doesn't compute the minimal live set, but rather a conservative over-approximation. Oh well. In practice this doesn't seem to be a big deal?

Guile has used conservative root-finding since its beginnings, 30 years ago and more. We had our own bespoke mark-sweep GC in the beginning, but it's now going on 15 years or so that we switched to the third-party Boehm-Demers-Weiser (BDW) collector. It's been good to us! It's better than what we had, it's mostly just worked, and it works correctly with threads.

Conservative roots

+: Ergonomic, eliminates class of bugs (handle registration), no compiler constraints

-: Potential leakage, no compaction / object motion; no bump-pointer allocation, calcifies GC choice

Conservative root-finding does have advantages. It's quite pleasant to program with, in environments in which the compiler is unable to produce stack maps for you, as it eliminates a set of potential bugs related to explicit handle registration and unregistration. Like stack maps, it also doesn't impose run-time overhead on the user program. And although the compiler isn't constrained to emit code to clear roots, it generally does, and sometimes does so more promptly than would be the case with explicit handle deregistration.

But, there are disadvantages too. The potential for leaks is one, though I have to say that in 20 years of using conservative-roots systems, I have not found this to be a problem. It's a source of anxiety whenever a program has memory consumption issues but I've never identified it as being the culprit.

The more serious disadvantage, though, is that conservative edges prevent objects from being moved by the GC. If you know that a location holds a pointer, you can update that location to point to a new location for an object. But if a location only might be a pointer, you can't do that.

In the end, the ergonomics of conservative collection lead to a kind of calcification in Guile, that we thought that BDW was as good as we could get given the constraints, and that changing to anything else would require precise roots, and thus an API and ABI change, losing users, and so on.

What if I told you

You can find roots conservatively and

  • move objects and compact the heap
  • do fast bump-pointer allocation
  • incrementally migrate to precise roots

BDW is not the local maximum

But it turns out, that's not true! There is a way to have conservative roots and also use more optimal GC algorithms, and one which preserves the ability to incrementally refactor the system to have more precision if that's what you want.

Immix

Fundamental GC algorithms

  • mark-compact
  • mark-sweep
  • evacuation
  • mark-region

Immix is a mark-region collector

Let's back up to a high level. Garbage collector implementations are assembled from instances of algorithms, and there are only so many kinds of algorithms out there.

There's mark-compact, in which the collector traverses the object graph once to find live objects, then once again to slide them down to one end of the space they are in.

There's mark-sweep, where the collector traverses the graph once to find live objects, then traverses the whole heap, sweeping dead objects into free lists to be used for future allocations.

There's evacuation, where the collector does a single pass over the object graph, copying the objects outside their space and leaving a forwarding pointer behind.

The BDW collector used by Guile is a mark-sweep collector, and its use of free lists means that allocation isn't as fast as it could be. We want bump-pointer allocation and all the other algorithms give it to us.

Then in 2008, Stephen Blackburn and Kathryn McKinley put out their Immix paper that identified a new kind of collection algorithm, mark-region. A mark-region collector will mark the object graph and then sweep the whole heap for unmarked regions, which can then be reused for allocating new objects.

Allocate: Bump-pointer into holes in thread-local block, objects can span lines but not blocks

Trace: Mark objects and lines

Sweep: Coarse eager scan over line mark bytes

Blackburn and McKinley's paper also describes a new mark-region GC algorithm, Immix, which is interesting because it gives us bump-pointer allocation without requiring that objects be moveable. The diagram above, from the paper, shows the organization of an Immix heap. Allocating threads (mutators) obtain 64-kilobyte blocks from the heap. Blocks contains 128-byte lines. When Immix traces the object graph, it marks both objects and the line the object is on. (Usually blocks are part of 2MB aligned slabs, with line mark bits/bytes are stored in a packed array at the start of the slab. When marking an object, it's easy to find the associated line mark just with address arithmetic.)

Immix reclaims memory in units of lines. A set of contiguous lines that were not marked in the previous collection form a hole (a region). Allocation proceeds into holes, in the usual bump-pointer fashion, giving us good locality for contemporaneously-allocated objects, unlike freelist allocation. The slow path, if the object doesn't fit in the hole, is to look for the next hole in the block, or if needed to acquire another block, or to stop for collection if there are no more blocks.

Immix: Opportunistic evacuation

Before trace, determine if compaction needed. If not, mark as usual

If so, select candidate blocks and evacuation target blocks. When tracing in that block, try to evacuate, fall back to mark

The neat thing that Immix adds is a way to compact the heap via opportunistic evacuation. As Immix allocates, it can end up skipping over holes and leaving them unpopulated, and as subsequent cycles of GC occur, it could be that a block ends up with many small holes. If that happens to many blocks it could be time to compact.

To fight fragmentation, Immix decides at the beginning of a GC cycle whether to try to compact or not. If things aren't fragmented, Immix marks in place; it's cheaper that way. But if compaction is needed, Immix selects a set of blocks needing evacuation and another set of empty blocks to evacuate into. (Immix has to keep around a couple percent of memory in empty blocks in reserve for this purpose.)

As Immix traverses the object graph, if it finds that an object is in a block that needs evacuation, it will try to evacuate instead of marking. It may or may not succeed, depending on how much space is available to evacuate into. Maybe it will succeed for all objects in that block, and you will be left with an empty block, which might even be given back to the OS.

Immix: Guile

Opportunistic evacuation compatible with conservative roots!

Bump-pointer allocation

Compaction!

1 year ago: start work on WIP GC implementation

Tying this back to Guile, this gives us all of our desiderata: we can evacuate, but we don't have to, allowing us to cause referents of conservative roots to be marked in place instead of moved; we can bump-pointer allocate; and we are back on the train of modern GC implementations. I could no longer restrain myself: I started hacking on a work-in-progress garbage collector workbench about a year ago, and ended up with something that seems to take us in the right direction.

Whippet vs Immix: Tiny lines

Immix: 128B lines + mark bit in object

Whippet: 16B “lines”; mark byte in side table

More size overhead: 1/16 vs 1/128

Less fragmentation (1 live obj = 2 lines retained)

More alloc overhead? More small holes

What I ended up building wasn't quite Immix. Guile's object representation is very thin and doesn't currently have space for a mark bit, for example, so I would have to have a side table of mark bits. (I could have changed Guile's object representation but I didn't want to require it.) I actually chose mark bytes instead of bits because both the Immix line marks and BDW's own side table of marks were bytes, to allow for parallel markers to race when setting marks.

Then, given that you have a contiguous table of mark bytes, why not remove the idea of lines altogether? Or what amounts to the same thing, why not make line size to be 16 bytes and do away with per-object mark bits? You can then bump-pointer into holes in the mark byte array. The only thing you need to do to that is to be able to cheaply find the end of an object, so you can skip to the next hole while sweeping; you don't want to have to chase pointers to do that. But consider, you've already paid the cost of having a mark byte associated with every possible start of an object, so if your basic object alignment is 16 bytes, that's a memory overhead of 1/16, or 6.25%; OK. Let's put that mark byte to work and include an "end" bit, indicating the end of the object. Allocating an object has to store into the mark byte array to initialize this "end" marker, but you need to write the mark byte anyway to allow for conservative roots ("does this address hold an object?"); writing the end at the same time isn't so bad, perhaps.

The expected outcome would be that relative to 128-byte lines, Whippet ends up with more, smaller holes. Such a block would be a prime target for evacuation, of course, but during allocation this is overhead. Or, it could be a source of memory efficiency; who knows. There is some science yet to do to properly compare this tactic to original Immix, but I don't think I will get around to it.

While I am here and I remember these things, I need to mention two more details. If you read the Immix paper, it describes "conservative line marking", which is related to how you find the end of an object; basically Immix always marks the line an object is on and the next one, in case the object spans the line boundary. Only objects larger than a line have to precisely mark the line mark array when they are traced. Whippet doesn't do this because we have the end bit.

The other detail is the overflow allocator; in the original Immix paper, if you allocate an object that's smallish but still larger than a line or two, but there's no hole big enough in the block, Immix keeps around a completely empty block per mutator in which to bump-pointer-allocate these medium-sized objects. Whippet doesn't do that either, instead relying on such failure to allocate in a block to cause fragmentation and thus hurry along the process of compaction.

Whippet vs Immix: Lazy sweeping

Immix: “cheap” eager coarse sweep

Whippet: just-in-time lazy fine-grained sweep

Corrolary: Data computed by sweep available when sweep complete

Live data at previous GC only known before next GC

Empty blocks discovered by sweeping

Having a fine-grained line mark array means that it's no longer a win to do an eager sweep of all blocks after collecting. Instead Whippet applies the classic "lazy sweeping" optimization to make mutators sweep their blocks just before allocating into them. This introduces a delay in the collection algorithm: Whippet doesn't find out about e.g. fragmentation until the whole heap is swept, but by the time we fully sweep the heap, we've exhausted it via allocation. It introduces a different flavor to the GC, not entirely unlike original Immix, but foreign.

Whippet vs BDW

Compaction/defrag/pinning, heap shrinking, sticky-mark generational GC, threads/contention/allocation, ephemerons, precision, tools

Right! With that out of the way, let's talk about what Whippet gives to Guile, relative to BDW-GC.

Whippet vs BDW: Motion

Heap-conservative tracing: no object moveable

Stack-conservative tracing: stack referents pinned, others not

Whippet: If whole-heap fragmentation exceeds threshold, evacuate most-fragmented blocks

Stack roots scanned first; marked instead of evacuated, implicitly pinned

Explicit pinning: bit in mark byte

If all edges in the heap are conservative, then you can't move anything, because you don't know if an edge is a pointer that can be updated or just a spicy integer. But most systems aren't actually like this: you have conservative edges from the stack, but you can precisely enumerate intra-object edges on the heap. In that case, you have a known set of conservative edges, and you can simply visit those edges first, marking their referents in place instead of evacuating. (Marking an object instead of evacuating implicitly pins it for the duration of the current GC cycle.) Then you visit heap edges precisely, possibly evacuating objects.

I should note that Whippet has a bit in the mark byte for use in explicitly pinning an object. I'm not sure how to manage who is responsible for setting that bit, or what the policy will be; the current idea is to set it for any object whose identity-hash value is taken. We'll see.

Whippet vs BDW: Shrinking

Lazy sweeping finds empty blocks: potentially give back to OS

Need empty blocks? Do evacuating collection

Possibility to do http://marisa.moe/balancer.html

With the BDW collector, your heap can only grow; it will never shrink (unless you enable a non-default option and you happen to have verrry low fragmentation). But with Whippet and evacuation, we can rearrange objects so as to produce empty blocks, which can then be returned to the OS if so desired.

In one of my microbenchmarks I have the system allocating long-lived data, interspersed with garbage (objects that are dead after allocation) whose size is in a power-law distribution. This should produce quite some fragmentation, eventually, and it does. But then Whippet decides to defragment, and it works great! Since Whippet doesn't keep a whole 2x reserve like a semi-space collector, it usually takes more than one GC cycle to fully compact the heap; usually about 3 cycles, from what I can see. I should do some more measurements here.

Of course, this is just mechanism; choosing the right heap sizing policy is a different question.

wingolog.org/archives/2022/10/22/the-sticky-mark-bit-algorithm

Card marking barrier (256B); compare to BDW mprotect / SIGSEGV

The Boehm collector also has a non-default mode in which it uses mprotect and a SIGSEGV handler to enable sticky-mark-bit generational collection. I haven't done a serious investigation, but I see it actually increasing run-time by 20% on one of my microbenchmarks that is actually generation-friendly. I know that Azul's C4 collector used to use page protection tricks but I can only assume that BDW's algorithm just doesn't work very well. (BDW's page barriers have another purpose, to enable incremental collection, in which marking is interleaved with allocation, but this mode is off if parallel markers are supported, and I don't know how well it works.)

Anyway, it seems we can do better. The ideal would be a semi-space nursery, which is the usual solution, but because of conservative roots we are limited to the sticky mark-bit algorithm. Some benchmarks aren't very generation-friendly; the first pair of bars in the chart above shows the mt-gcbench microbenchmark running with and without generational collection, and there's no difference. But in the second, for the quads benchmark, we see a 2x speedup or so.

Of course, to get generational collection to work, we require mutators to use write barriers, which are little bits of code that run when an object is mutated that tell the GC where it might find links from old objects to new objects. Right now in Guile we don't do this, but this benchmark shows what can happen if we do.

Whippet vs BDW: Scale

BDW: TLS segregated-size freelists, lock to refill freelists, SIGPWR for stop

Whippet: thread-local block, sweep without contention, wait-free acquisition of next block, safepoints to stop with ragged marking

Both: parallel markers

Another thing Whippet can do better than BDW is performance when there are multiple allocating threads. The Immix heap organization facilitates minimal coordination between mutators, and maximum locality for each mutator. Sweeping is naturally parallelized according to how many threads are allocating. For BDW, on the other hand, every time an mutator needs to refill its thread-local free lists, it grabs a global lock; sweeping is lazy but serial.

Here's a chart showing whippet versus BDW on one microbenchmark. On the X axis I add more mutator threads; each mutator does the same amount of allocation, so I'm increasing the heap size also by the same factor as the number of mutators. For simplicity I'm running both whippet and BDW with a single marker thread, so I expect to see a linear increase in elapsed time as the heap gets larger (as with 4 mutators there are roughly 4 times the number of live objects to trace). This test is run on a Xeon Silver 4114, taskset to free cores on a single socket.

What we see is that as I add workers, elapsed time increases linearly for both collectors, but more steeply for BDW. I think (but am not sure) that this is because whippet effectively parallelizes sweeping and allocation, whereas BDW has to contend over a global lock to sweep and refill free lists. Both have the linear factor of tracing the object graph, but BDW has the additional linear factor of sweeping, whereas whippet scales with mutator count.

Incidentally you might notice that at 4 mutator threads, BDW randomly crashed, when constrained to a fixed heap size. I have noticed that if you fix the heap size, BDW sometimes (and somewhat randomly) fails. I suspect the crash due to fragmentation and inability to compact, but who knows; multiple threads allocating is a source of indeterminism. Usually when you run BDW you let it choose its own heap size, but for these experiments I needed to have a fixed heap size instead.

Another measure of scalability is, how does the collector do as you add marker threads? This chart shows that for both collectors, runtime decreases as you add threads. It also shows that whippet is significantly slower than BDW on this benchmark, which is Very Weird, and I didn't have access to the machine on which these benchmarks were run when preparing the slides in the train... so, let's call this chart a good reminder that Whippet is a WIP :)

While in the train to Brussels I re-ran this test on the 4-core laptop I had on hand, and got the results that I expected: that whippet performed similarly to BDW, and that adding markers improved things, albeit marginally. Perhaps I should look on a different microbenchmark.

Incidentally, when you configure Whippet for parallel marking at build-time, it uses a different implementation of the mark stack when compared to the parallel marker, even when only 1 marker is enabled. Certainly the parallel marker could use some tuning.

Whippet vs BDW: Ephemerons

BDW: No ephemerons

Whippet: Yes

Another deep irritation I have with BDW is that it doesn't support ephemerons. In Guile we have a number of facilities (finalizers, guardians, the symbol table, weak maps, et al) built on what BDW does have (finalizers, weak references), but the implementations of these facilities in Guile are hacky, slow, sometimes buggy, and don't compose (try putting an object in a guardian and giving it a finalizer to see what I mean). It would be much better if the collector API supported ephemerons natively, specifying their relationship to finalizers and other facilities, allowing us to build what we need in terms of those primitives. With our own GC, we can do that, and do it in such a way that it doesn't depend on the details of the specific collection algorithm. The exception of course is that as BDW doesn't support ephemerons per se, what we get is actually a weak-key association instead, whose value can keep the key alive. Oh well, it's no worse than the current situation.

Whippet vs BDW: Precision

BDW: ~Always stack-conservative, often heap-conservative

Whippet: Fully configurable (at compile-time)

Guile in mid/near-term: C stack conservative, Scheme stack precise, heap precise

Possibly fully precise: unlock semi-space nursery

Conservative tracing is a fundamental design feature of the BDW collector, both of roots and of inter-heap edges. You can tell BDW how to trace specific kinds of heap values, but the default is to do a conservative scan, and the stack is always scanned conservatively. In contrast, these tradeoffs are all configurable in Whippet. You can scan the stack and heap precisely, or stack conservatively and heap precisely, or vice versa (though that doesn't make much sense), or both conservatively.

The long-term future in Guile is probably to continue to scan the C stack conservatively, to continue to scan the Scheme stack precisely (even with BDW-GC, the Scheme compiler emits stack maps and installs a custom mark routine), but to scan the heap as precisely as possible. It could be that a user uses some of our hoary ancient APIs to allocate an object that Whippet can't trace precisely; in that case we'd have to disable evacuation / object motion, but we could still trace other objects precisely.

If Guile ever moved to a fully precise world, that would be a boon for performance, in two ways: first that we would get the ability to use a semi-space nursery instead of the sticky-mark-bit algorithm, and relatedly that we wouldn't need to initialize mark bytes when allocating objects. Second, we'd gain the option to use must-move algorithms for the old space as well (mark-compact, semi-space) if we wanted to. But it's just an option, one that that Whippet opens up for us.

Whippet vs BDW: Tools?

Can build heap tracers and profilers moer easily

More hackable

(BDW-GC has as many preprocessor directives as whippet has source lines)

Finally, relative to BDW-GC, whippet has a more intangible advantage: I can actually hack on it. Just as an indication, 15% of BDW source lines are pre-processor directives, and there is one file that has like 150 #ifdef's, not counting #elseif's, many of them nested. I haven't done all that much to BDW itself, but I personally find it excruciating to work on.

Hackability opens up the possibility to build more tools to help us diagnose memory use problems. They aren't in Whippet yet, but there can be!

Engineering Whippet

Embed-only, abstractions, migration, modern; timeline

OK, that rounds out the comparison between BDW and Whippet, at least on a design level. Now I have a few words about how to actually get this new collector into Guile without breaking the bug budget. I try to arrange my work areas on Guile in such a way that I spend a minimum of time on bugs. Part of my strategy is negligence, I will admit, but part also is anticipating problems and avoiding them ahead of time, even if it takes more work up front.

Engineering Whippet: Embed-only

github.com/wingo/whippet-gc/

Semi: 6 kB; Whippet: 22 kB; BDW: 184 kB

Compile-time specialization:

  • for embedder (e.g. how to forward objects)
  • for selected GC algorithm (e.g. semi-space vs whippet)

Built apart, but with LTO to remove library overhead

So the BDW collector is typically shipped as a shared library that you dynamically link to. I should say that we've had an overall good experience with upgrading BDW-GC in the past; its maintainer (Ivan Maidanski) does a great and responsible job on a hard project. It's been many, many years since we had a bug in BDW-GC. But still, BDW is dependency, and all things being equal we prefer to remove moving parts.

The approach that Whippet is taking is to be an embed-only library: it's designed to be compiled into your project. It's not an include-only library; it still has to be compiled, but with link-time-optimization and a judicious selection of fast-path interfaces, Whippet is mostly able to avoid abstractions being a performance barrier.

The result is that Whippet is small, both in source and in binary, which minimizes its maintenance overhead. Taking additional stripped optimized binary size as the metric, by my calculations a semi-space collector (with a large object space and ephemeron support) takes about 6 kB of object file size, whereas Whippet takes 22 and BDW takes 184. Part of how Whippet gets so small is that it is is configured in major ways at compile-time (choice of main GC algorithm), and specialized against the program it's embedding against (e.g. how to patch in a forwarding pointer). Having all API being internal and visible to LTO instead of going through ELF symbol resolution helps in a minor way as well.

Engineering Whippet: Abstract performance

User API abstracts over GC algorithm, e.g. semi-space or whippet

Expose enough info to allow JIT to open-code fast paths

Inspired by mmtk.io

Abstractions permit change: of algorithm, over time

From a composition standpoint, Whippet is actually a few things. Firstly there is an abstract API to make a heap, create per-thread mutators for a heap, and allocate objects for a mutator. There is the aforementioned embedder API, for having the embedding program indicate how to trace objects and install forwarding pointers. Then there is some common code (for example ephemeron support). There are implementations of the different spaces: semi-space, large object, whippet/immix; and finally collector implementations that tie together the spaces into a full implementation of the abstract API. (In practice the more iconic spaces are intertwingled with the collector implementations they define.)

I don't think I would have gone down this route without seeing some prior work, for example libpas, but it was really MMTk that convinced me that it was worth spending a little time thinking about the GC not as a structureless blob but as a system made of parts and exposing a minimal interface. In particular, I was inspired by seeing that MMTk is able to get good performance while also being abstract, exposing representation details such as how to tell a JIT compiler about allocation fast-paths, but in a principled way. So, thanks MMTk people, for this and so many things!

I'm particularly happy that the API is abstract enough that it frees up not only the garbage collector to change implementations, but also Guile and other embedders, in that they don't have to bake in a dependency on specific collectors. The semi-space collector has been particularly useful here in ensuring that the abstractions don't accidentally rely on support for object pinning.

Engineering Whippet: Migration

API implementable by BDW-GC (except ephemerons)

First step for Guile: BDW behind Whippet API

Then switch to whippet/immix (by default)

The collector API can actually be implemented by the BDW collector. Whippet includes a collector that is a thin wrapper around the BDW API, with support for fast-path allocation via thread-local freelists. In this way we can always check the performance of any given collector against an external fixed point (BDW) as well as a theoretically known point (the semi-space collector).

Indeed I think the first step for Guile is precisely this: refactor Guile to allocate through the Whippet API, but using the BDW collector as the implementation. This will ensure that the Whippet API is sufficient, and then allow an incremental switch to other collectors.

Incidentally, when it comes to integrating Whippet, there are some choices to be made. I mentioned that it's quite configurable, and this chart can give you some idea. On the left side is one microbenchmark (mt-gcbench) and on the right is another (quads). The first generates a lot of fragmentation and has a wide range of object sizes, including some very large objects. The second is very uniform and many allocations die young.

(I know these images are small; right-click to open in new tab or pinch to zoom to see more detail.)

Within each set of bars we have 10 different scenarios, corresponding to different Whippet configurations. (All of these tests are run on my old 4-core laptop with 4 markers if parallel marking is supported, and a 2x heap.)

The first bar in each side is serial whippet: one marker. Then we see parallel whippet: four markers. Great. Then there's generational whippet: one marker, but just scanning objects allocated in the current cycle, hoping that produces enough holes. Then generational parallel whippet: the same as before, but with 4 markers.

The next 4 bars are the same: serial, parallel, generational, parallel-generational, but with one difference: the stack is scanned conservatively instead of precisely. You might be surprised but all of these configurations actually perform better than their precise counterparts. I think the reason is that the microbenchmark uses explicit handle registration and deregistration (it's a stack) instead of compiler-generated stack maps in a side table, but I'm not precisely (ahem) sure.

Finally the next 2 bars are serial and parallel collectors, but marking everything conservatively. I have generational measurements for this configuration but it really doesn't make much sense to assume that you can emit write barriers in this context. These runs are slower than the previous configuration, mostly because there are some non-pointer locations that get scanned conservatively that wouldn't get scanned precisely. I think conservative heap scanning is less efficient than precise but I'm honestly not sure, there are some instruction locality arguments in the other direction. For mt-gcbench though there's a big array of floating-point values that a precise scan will omit, which causes significant overhead there. Probably for this configuration to be viable Whippet would need the equivalent of BDW's API to allocate known-pointerless objects.

Engineering Whippet: Modern

stdatomic

constexpr-ish

pthreads (for parallel markers)

No void*; instead struct types: gc_ref, gc_edge, gc_conservative_ref, etc

Embed-only lib avoids any returns-struct-by-value ABI issue

Rust? MMTk; supply chain concerns

Platform abstraction for conservative root finding

I know it's a sin, but Whippet is implemented in C. I know. The thing is, in the Guile context I need to not introduce wild compile-time dependencies, because of bootstrapping. And I know that Rust is a fine language to use for GC implementation, so if that's what you want, please do go take a look at MMTk! It's a fantastic project, written in Rust, and it can just slot into your project, regardless of the language your project is written in.

But if what you're looking for is something in C, well then you have to pick and choose your C. In the case of Whippet I try to use the limited abilities of C to help prevent bugs; for example, I generally avoid void* and instead wrap pointers or addresses into single-field structs that can't be automatically cast, for example to prevent a struct gc_ref that denotes an object reference (or NULL; it's an option type) from being confused with a struct gc_conservative_ref, which might not point to an object at all.

(Of course, by "C" I mean "C as compiled by gcc and clang with -fno-strict-aliasing". I don't know if it's possible to implement even a simple semi-space collector in C without aliasing violations. Can you access a Foo* object within a mmap'd heap through its new address after it has been moved via memcpy? Maybe not, right? Thoughts are welcome.)

As a project written in the 2020s instead of the 1990s, Whippet gets to assume a competent C compiler, for example relying on the compiler to inline and fold branches where appropriate. As in libpas, Whippet liberally passes functions as values to inline functions, and relies on the compiler to boil away function calls. Whippet only uses the C preprocessor when it absolutely has to.

Finally, there is a clean abstraction for anything that's platform-specific, for example finding the current stack bounds. I haven't compiled this code on Windows or MacOS yet, but I am not anticipating too many troubles.

Engineering Whippet: Timeline

As time permits

Whippet TODO: heap growth/shrinking, finalizers, safepoint API

Guile TODO: safepoints; heap-conservative first

Precise heap TODO: gc_trace_object, SMOBs, user structs with raw ptr fields, user gc_malloc usage; 3.2

6 months for 3.1.1; 12 for 3.2.0 ?

So where does this get us? Where are we now?

For Whippet itself, I think it's mostly done -- enough to start shifting focus to some different phase. It's missing some needed features, notably the ability to grow the heap at all, as I've been in fixed-heap-size-only mode during development. It's also missing finalizers. And, something needs to be done to unify Guile's handling of safepoints and processing of asynchronous signals with Whippet's need to stop all mutators. Some details remain.

But, I think we are close to ready to start integrating in Guile. At first this is just porting Guile to use the Whippet API to access BDW instead of using BDW directly. This whole thing is a side project for me that I work on when I can, so it doesn't exactly proceed at full pace. Perhaps this takes 6 months. Then we can cut a new unstable release, and hopefully release 3.2 withe support for the Immix-flavored collector in another 6 or 9 months.

I thought that we would be forced to make ABI changes, if only because of some legacy APIs assume conservative tracing of object contents. But after a discussion at FOSDEM with Carlo Piovesan I realized this isn't true: because the decision to evacuate or not is made on a collection-by-collection basis, I could simply disable evacuation if the user ever uses a facility that might prohibit object motion, for example if they ever define a SMOB type. If the user wants evacuation, they need to be more precise with their data types, but either way Guile is ready.

Whippet: A Better GC?

An Immix-derived GC

github.com/wingo/whippet-gc/

https://wingolog.org/tags/gc/

Guile 3.2 ?

Thanks to MMTk authors for inspiration!

And that's it! Thanks for reading all the way here. Comments are quite welcome.

As I mentioned in the very beginning, this talk was really about Whippet in the context of Guile. There is a different talk to be made about Guile+Whippet versus other language implementations, for example those with concurrent marking or semi-space nurseries or the like. Yet another talk is Whippet in the context of other GC algorithms. But this is a start. It's something I've been working on for a while now already and I'm pleased that it's gotten to a point where it seems to be at least OK, at least an improvement with respect to BDW-GC in some ways.

But before leaving you, another chart, to give a more global idea of the state of things. Here we compare a single mutator thread performing a specific microbenchmark that makes trees and also lots of fragmentation, across three different GC implementations and a range of heap sizes. The heap size multipliers in this and in all the other tests in this post are calculated analytically based on what the test thinks its maximum heap size should be, not by measuring minimum heap sizes that work. This size is surely lower than the actual maximum required heap size due to internal fragmentation, but the tests don't know about this.

The three collectors are BDW, a semi-space collector, and whippet. Semi-space manages to squeeze in less than 2x of a heap multiplier because it has (and whippet has) a separate large object space that isn't ever evacuated.

What we expect is that tighter heaps impose more GC time, and indeed we see that times are higher on the left side than the right.

Whippet is the only implementation that manages to run at a 1.3x heap, but it takes some time. It's slower than BDW at a 1.5x heap but better there on out, until what appears to be a bug or pathology makes it take longer at 5x. Adding memory should always decrease run time.

The semi-space collector starts working at 1.75x and then surpasses all collectors from 2.5x onwards. We expect the semi-space collector to win for big heaps, because its overhead is proportional to live data only, whereas mark-sweep and mark-region collectors have to sweep, which is proportional to heap size, and indeed that's what we see.

I think this chart shows we have some tuning yet to do. The range between 2x and 3x is quite acceptable, but we need to see what's causing Whippet to be slower than BDW at 1.5x. I haven't done as much performance tuning as I would like to but am happy to finally be able to know where we stand.

And that's it! Happy hacking, friends, and may your heap sizes be ever righteous.

07 February, 2023 01:14PM by Andy Wingo

February 06, 2023

Gary Benson

Using AWS CLI with IAM users

When you configure AWS CLI to use an IAM user, the first thing it asks for is an SSO session name. Don’t put whitespace or punctuation in it. The command doesn’t tell you this, but it’s going to use what you enter as an identifier, and fail with a cryptic error:

$ aws configure sso
SSO session name (Recommended): AWS CLI on Gary's Chromebook
SSO start URL [None]: https://whatever.awsapps.com/start
SSO region [None]: us-east-1
SSO registration scopes [sso:account:access]:

An error occurred (InvalidClientMetadataException) when calling the RegisterClient operation:

You need to put something like “gbenson” in there.

06 February, 2023 11:43AM by gbenson

February 05, 2023

GNUnet News

libgnunetchat 0.1.3

libgnunetchat 0.1.3 released

This is mostly a bugfix and compatibility release for libgnunetchat 0.1.2. But it will also change the build process of libgnunetchat to use Meson instead of GNU Automake and it will ensure compatibility with latest changes in GNUnet 0.19.0. The required version of GNUnet to build this library is 0.19.3 for compatibility reasons though.

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

05 February, 2023 11:00PM

GNUnet 0.19.3

GNUnet 0.19.3

This is a bugfix release for gnunet 0.19.2. Note that starting with this release, we will no longer ship a verbose ChangeLog file in the tarball. The git log serves this purpose now.

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

A detailed list of changes can be found in the git log , the NEWS and the bug tracker .

05 February, 2023 11:00PM

gpaint @ Savannah

version 0.3.4 released

Hi, a new release,.

Version 0.3.4 is now available as

https://alpha.gnu.org/gnu/gpaint/gpaint-2-0.3.4.tar.gz

This release combines existing patches from GNU/Linux distributions into an official release.  In addition,
the build infrastructure is modernized to be based on
current versions of the GNU Autotools

Changes in this version:

    * adding guix.scm, development under GNU Guix support

    Debian patches from Goedson Teixeira Paixao <goedson> incorporatd into main release
    includng patches for
    * fixing missing heds libs by <goedson>
    * Fix toolbar behaviour so that gpaint follows the style set in
      the user preferences by <goedson>
    * Fix foreground/background color selection by <goedson>
    * Fix crash when saving in unsupported format by <goedson>
    * Add accelerator keys to common functions by Matt Wheeler <m@funkyhat.org>
    * Ignore non-printable characters on text input by Ying-Chun Liu (PaulLiu) <grandpaul@gmail.com>
    * fix crash on fill button click by <goedson>
    * Fix line width combo box by Thomas Viehmann <tv@beamnet.de>
    * Fixes rotation operations: Implement the rotation in multiples of 90 degrees using the
      gdk_pixbuf_rotate_simple function by <goedson>
    * Avoids crash on font selection by <goedson>
    * Fixes the gpaint.desktop file by <goedson>
    * Removes reference to non-existent menu.h file by <goedson>
    * Fixes compiling with recent versions of libgtk by <goedson>

This release represents gpaint is resuming active development.

Roadmap is detailed in README file but suggestions are welcome.

05 February, 2023 06:57AM by Andy Tai

February 02, 2023

libc @ Savannah

The GNU C Library version 2.37 is now available

The GNU C Library
=================

The GNU C Library version 2.37 is now available.

The GNU C Library is used as the C library in the GNU system and
in GNU/Linux systems, as well as many other systems that use Linux
as the kernel.

The GNU C Library is primarily designed to be a portable
and high performance C library.  It follows all relevant
standards including ISO C11 and POSIX.1-2017.  It is also
internationalized and has one of the most complete
internationalization interfaces known.

The GNU C Library webpage is at http://www.gnu.org/software/libc/

Packages for the 2.37 release may be downloaded from:
        http://ftpmirror.gnu.org/libc/
        http://ftp.gnu.org/gnu/libc/

The mirror list is at http://www.gnu.org/order/ftp.html

Distributions are encouraged to track the release/* branches
corresponding to the releases they are using.  The release
branches will be updated with conservative bug fixes and new
features while retaining backwards compatibility.

NEWS for version 2.37
=====================

Major new features:

  • The getent tool now supports the --no-addrconfig option. The output of

  getent with --no-addrconfig may contain addresses of families not
  configured on the current host i.e. as-if you had not passed
  AI_ADDRCONFIG to getaddrinfo calls.

Deprecated and removed features, and other changes affecting compatibility:

  • The dynamic linker no longer loads shared objects from the "tls"

  subdirectories on the library search path or the subdirectory that
  corresponds to the AT_PLATFORM system name, or employs the legacy AT_HWCAP
  search mechanism, which was deprecated in version 2.33.

Security related changes:

  CVE-2022-39046: When the syslog function is passed a crafted input
  string larger than 1024 bytes, it reads uninitialized memory from the
  heap and prints it to the target log file, potentially revealing a
  portion of the contents of the heap.

The following bugs are resolved with this release:

  [12154] network: Cannot resolve hosts which have wildcard aliases
  [12165] libc: readdir: Do not skip entries with zero d_ino values
  [19444] build: build failures with -O1 due to -Wmaybe-uninitialized
  [24774] nptl: pthread_rwlock_timedwrlock stalls on ARM
  [24816] nss: nss/tst-nss-files-hosts-long fails when no interface has
    AF_INET6 address (ie docker)
  [27087] stdio: PowerPC: Redefinition error with Clang from IEEE
    redirection headers
  [28846] network: CMSG_NXTHDR may trigger -Wstrict-overflow warning
  [28937] dynamic-link: New DSO dependency sorter does not put new map
    first if in a cycle
  [29249] libc: csu/libc-tls.c:202: undefined reference to
    `_startup_fatal_not_constant'
  [29305] network: Inefficient buffer space usage in nss_dns for
    gethostbyname and other functions
  [29375] libc: don't hide MAP_ANONYMOUS behind _GNU_SOURCE
  [29402] nscd: nscd: No such file or directory
  [29415] nscd: getaddrinfo with AI_ADDRCONFIG returns addresses with
    wrong family
  [29427] dynamic-link: Inconsistency detected by ld.so: dl-printf.c:
    200: _dl_debug_vdprintf: Assertion `! "invalid format specifier"'
    failed!
  [29463] math: math/test-float128-y1 fails on x86_64
  [29485] build: Make hangs when the test misc/tst-pidfile returns
    FAIL_UNSUPPORTED
  [29490] dynamic-link: [bisected] new __brk_call causes dynamic loader
    segfault on alpha
  [29499] build: Check failed on misc/tst-glibcsyscalls while building
    for RISCV64 on a unmatched hardware
  [29501] build: Check failed on stdlib/tst-strfrom while building for
    RISCV64 on a unmatched hardware
  [29502] libc: alpha sys/acct.h out of date
  [29514] build: Need to use -fPIE not -fpie
  [29528] dynamic-link: __libc_early_init not called after dlmopen that
    reuses namespace
  [29536] libc: syslog fail to create large messages (CVE-2022-39046)
  [29537] libc: [2.34 regression]: Alignment issue on m68k when using
    futexes on qemu-user
  [29539] libc: LD_TRACE_LOADED_OBJECTS changed how vDSO library are
    printed
  [29544] libc: Regression in syslog(3) calls breaks RFC due to extra
    whitespace
  [29564] build: Incorrect way to change MAKEFLAGS in Makerules
  [29576] build: librtld.os: in function `_dl_start_profile':
    (.text+0x9444): undefined reference to `strcpy'
  [29578] libc: Definition of SUN_LEN() is wrong
  [29583] build: iconv failures on 32bit platform due to missing large
    file support
  [29600] dynamic-link: dlmopen hangs after loading certain libraries
  [29604] localedata: Update locale data to Unicode 15.0.0
  [29605] nscd: Regression in NSCD backend of getaddrinfo
  [29607] nscd: nscd repeatably crashes calling __strlen_avx2 when hosts
    cache is enabled
  [29611] string: Optimized AVX2 string functions unconditionally use
    BMI2 instructions
  [29624] malloc: errno is not cleared when entering main
  [29638] libc: stdlib: arc4random fallback is never used
  [29657] libc: Incorrect struct stat for 64-bit time on linux/generic
    platforms
  [29698] build: Configuring for AArch32 on ARMv8+ disables
    optimizations
  [29727] locale: __strtol_internal out-of-bounds read when parsing
    thousands grouping
  [29730] libc: broken y2038 support in fstatat on MIPS N64
  [29746] libc: ppoll() does not switch to __ppoll64 when
    -D_TIME_BITS=64 and -D_FORTIFY_SOURCE=2 is given on 32bit
  [29771] libc: Restore IPC_64 support in sysvipc *ctl functions
  [29780] build: possible parallel make issue in glibc-2.36 (siglist-
    aux.S: No such file or directory)
  [29864] libc: __libc_start_main() should obtain program headers
    address (_dl_phdr) from the auxv, not the ELF header.
  [29951] time: daylight variable not set correctly if last DST change
    coincides with offset change
  [30039] stdio: __vsprintf_internal does not handle unspecified buffer
    length in fortify mode

Release Notes
=============

https://sourceware.org/glibc/wiki/Release/2.37

Contributors
============

This release was made possible by the contributions of many people.
The maintainers are grateful to everyone who has contributed
changes or bug reports.  These include:

Adhemerval Zanella
Adhemerval Zanella Netto
Alan Modra
Alistair Francis
Andreas K. Hüttel
Andreas Schwab
Arjun Shankar
Aurelien Jarno
Carlos Eduardo Seo
Carlos O'Donell
Chenghua Xu
Cristian Rodríguez
Damien Zammit
Fabian Vogt
Fangrui Song
Felix Riemann
Flavio Cruz
Florian Weimer
H.J. Lu
Jakub Wilk
Javier Pello
John David Anglin
Joseph Myers
Jörg Sonnenberger
Kito Cheng
Letu Ren
Lucas A. M. Magalhaes
Ludovic Courtès
Martin Jansa
Martin Joerg
Michael Hudson-Doyle
Mike FABIAN
Noah Goldstein
Paul Eggert
Paul Pluzhnikov
Qingqing Li
Rajalakshmi Srinivasaraghavan
Raphael Moreira Zinsly
Richard Henderson
Sajan Karumanchi
Samuel Thibault
Sergei Trofimovich
Sergey Bugaev
Shahab Vahedi
Siddhesh Poyarekar
Stefan Liebler
Sunil K Pandey
Szabolcs Nagy
Tom Honermann
Tulio Magno Quites Machado Filho
Vladislav Khmelevsky
Wilco Dijkstra
Xi Ruoyao
Xiaolin Tang
Xiaoming Ni
Xing Li
Yu Chien Peter Lin
YunQiang Su
Zong Li
caiyinyu
fanquake
Łukasz Stelmach
наб

We would like to call out the following and thank them for their
tireless patch review:

Adhemerval Zanella
Arjun Shankar
Aurelien Jarno
Carlos O'Donell
Cristian Rodríguez
DJ Delorie
Fangrui Song
Florian Weimer
H.J. Lu
Noah Goldstein
Palmer Dabbelt
Paul E. Murphy
Philippe Mathieu-Daudé
Premachandra Mallappa
Sam James
Siddhesh Poyarekar
Sunil K Pandey
Szabolcs Nagy
Tulio Magno Quites Machado Filho
Wilco Dijkstra
Yann Droneaud

02 February, 2023 03:12AM by Carlos O'Donell

January 31, 2023

health @ Savannah

GNU Health Federation community server updated

Dear community

The GNU Health federation community server (federation.gnuhealth.org) has been updated! These are some of the main improvements:

- NGINX and WSGI: The Hospital Management component instance is now running behind uWSGI and NGINX.

- HMIS 4.2 Release Candidate 2: We are very close to HMIS 4.2 stable. This pre-release version will help us to test the upcoming features and find bugs before the release. :)

- Secure connection: You can test the demo web client using TLS. Please check the demo database section on the GNUHealth Wikibooks.

- Thalamus, the GNU Health Federation message and authentication server still runs on Gunicorn, also using https.

More info and resources:

Happy and healthy hacking!
Luis

31 January, 2023 11:44PM by Luis Falcon

January 28, 2023

Applied Pokology

Applied Pokology - Interesting poke idiom: sparse tables

During tonight poke online office hours our friend hdzki came with an interesting use case. He is poking at some binary structures that are like sparse tables whose entries are distributed in the file in an arbitrary way. Each sparse table is characterized by an array of consecutive non-NULL pointers. Each pointer points to an entry in the table. The table entries can be anywhere in the IO space, and are not necessarily consecutive, nor be in order.

Full text...

28 January, 2023 12:00AM

January 27, 2023

Andy Wingo

three approaches to heap sizing

How much memory should a program get? Tonight, a quick note on sizing for garbage-collected heaps. There are a few possible answers, depending on what your goals are for the system.

you: doctor science

Sometimes you build a system and you want to study it: to identify its principal components and see how they work together, or to isolate the effect of altering a single component. In that case, what you want is a fixed heap size. You run your program a few times and determine a heap size that is sufficient for your problem, and then in future run the program with that new fixed heap size. This allows you to concentrate on the other components of the system.

A good approach to choosing the fixed heap size for a program is to determine the minimum heap size a program can have by bisection, then multiplying that size by a constant factor. Garbage collection is a space/time tradeoff: the factor you choose represents a point on the space/time tradeoff curve. I would choose 1.5 in general, but this is arbitrary; I'd go more with 3 or even 5 if memory isn't scarce and I'm really optimizing for throughput.

Note that a fixed-size heap is not generally what you want. It's not good user experience for running ./foo at the command line, for example. The reason for this is that program memory use is usually a function of the program's input, and only in some cases do you know what the input might look like, and until you run the program you don't know what the exact effect of input on memory is. Still, if you have a team of operations people that knows what input patterns look like and has experience with a GC-using server-side process, fixed heap sizes could be a good solution there too.

you: average josé/fina

On the other end of the spectrum is the average user. You just want to run your program. The program should have the memory it needs! Not too much of course; that would be wasteful. Not too little either; I can tell you, my house is less than 100m², and I spend way too much time shuffling things from one surface to another. If I had more space I could avoid this wasted effort, and in a similar way, you don't want to be too stingy with a program's heap. Do the right thing!

Of course, you probably have multiple programs running on a system that are making similar heap sizing choices at the same time, and the relative needs and importances of these programs could change over time, for example as you switch tabs in a web browser, so the right thing really refers to overall system performance, whereas what you are controlling is just one process' heap size; what is the Right Thing, anyway?

My corner of the GC discourse agrees that something like the right solution was outlined by Kirisame, Shenoy, and Panchekha in a 2022 OOPSLA paper, in which the optimum heap size depends on the allocation rate and the gc cost for a process, which you measure on an ongoing basis. Interestingly, their formulation of heap size calculation can be made by each process without coordination, but results in a whole-system optimum.

There are some details but you can imagine some instinctive results: for example, when a program stops allocating because it's waiting for some external event like user input, it doesn't need so much memory, so it can start shrinking its heap. After all, it might be quite a while before the program has new input. If the program starts allocating again, perhaps because there is new input, it can grow its heap rapidly, and might then shrink again later. The mechanism by which this happens is pleasantly simple, and I salute (again!) the authors for identifying the practical benefits that an abstract model brings to the problem domain.

you: a damaged, suspicious individual

Hoo, friends-- I don't know. I've seen some things. Not to exaggerate, I like to think I'm a well-balanced sort of fellow, but there's some suspicion too, right? So when I imagine a background thread determining that my web server hasn't gotten so much action in the last 100ms and that really what it needs to be doing is shrinking its heap, kicking off additional work to mark-compact it or whatever, when the whole point of the virtual machine is to run that web server and not much else, only to have to probably give it more heap 50ms later, I-- well, again, I exaggerate. The MemBalancer paper has a heartbeat period of 1 Hz and a smoothing function for the heap size, but it just smells like danger. Do I need danger? I mean, maybe? Probably in most cases? But maybe it would be better to avoid danger if I can. Heap growth is usually both necessary and cheap when it happens, but shrinkage is never necessary and is sometimes expensive because you have to shuffle around data.

So, I think there is probably a case for a third mode: not fixed, not adaptive like the MemBalancer approach, but just growable: grow the heap when and if its size is less than a configurable multiplier (e.g. 1.5) of live data. Never shrink the heap. If you ever notice that a process is taking too much memory, manually kill it and start over, or whatever. Default to adaptive, of course, but when you start to troubleshoot a high GC overhead in a long-lived proess, perhaps switch to growable to see its effect.

unavoidable badness

There is some heuristic badness that one cannot avoid: even with the adaptive MemBalancer approach, you have to choose a point on the space/time tradeoff curve. Regardless of what you do, your system will grow a hairy nest of knobs and dials, and if your system is successful there will be a lively aftermarket industry of tuning articles: "Are you experiencing poor object transit? One knob you must know"; "Four knobs to heaven"; "It's raining knobs"; "GC engineers DO NOT want you to grab this knob!!"; etc. (I hope that my British readers are enjoying this.)

These ad-hoc heuristics are just part of the domain. What I want to say though is that having a general framework for how you approach heap sizing can limit knob profusion, and can help you organize what you have into a structure of sorts.

At least, this is what I tell myself; inshallah. Now I have told you too. Until next time, happy hacking!

27 January, 2023 09:45PM by Andy Wingo

January 26, 2023

GNU Taler news

GNU Taler v0.9.1 released

We are happy to announce the release of GNU Taler v0.9.1.

26 January, 2023 11:00PM

January 25, 2023

poke @ Savannah

GNU poke 3.0 released

I am happy to announce a new major release of GNU poke, version 3.0.

This release is the result of a year of development.  A lot of things have changed and improved with respect to the 2.x series; we have fixed many bugs and added quite a lot of new exciting and useful features.  See below for a description of many of them.

From now on, we intend to do not one but two major releases of poke every year.  What is moving us to change this is the realization that users have to wait for too long to enjoy new features, which are continuously being added in a project this young and active.

The tarball poke-3.0.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-3.0.tar.gz.

> GNU poke (http://www.jemarch.net/poke) is an interactive, extensible editor for binary data.  Not limited to editing basic entities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them.


Thanks to the people who contributed with code and/or documentation to this release.  In certain but no significant order they are:

   Mohammad-Reza Nabipoor
   Arsen Arsenović
   Luca Saiu
   Bruno Haible
   apache2
   Indu Bhagat
   Agathe Porte
   Alfred M. Szmidt
   Daiki Ueno
   Darshit Shah
   Jan Seeger
   Sergio Durigan Junior

   ... and yours truly

As always, thank you all!

But wait, this time we also have special thanks:

To Bruno Haible for his invaluable advise and his help in throughfully testing this new release in many different platforms and configurations.

To the Sourceware overseers, Mark Wielaard, Arsen Arsenović, and Sam James for their help in setting up the buildbots we are using for CI at sourceware.

What is new in this release:

User interface updates

  • A screen pager has been added to the poke application.  If enabled with the `.set pager yes' option, output will be paged one screenful at a time.
  • A tracer has been added to libpoke and the poke application. If enabled with the `.set tracer yes' option, subsequent loaded Poke types will be instrumentalized so calls to user-defined handlers are executed when certain events happen:
    • Every time a field gets mapped.
    • Every time a struct/union gets mapped.
    • Every time a field gets constructed.
    • Every time a struct/union gets constructed.
    • Every time an optional field is omitted when mapping or constructing.
  • A new command sdiff (for "structured diff") has been added to the poke application, that provides a way to generate patchable diffs of mapped structured Poke values.  This command is an interface to the structured diffs provided by the new diff.pk pickle.
  • When no name is passed to the .mem command, an unique name for the memory IOS with the form N will be used automatically, where N is a positive integer.
  • Auto-completion of 'attributes is now available in the poke application.
  • Constraint errors now contain details on the location (which field) where the constraint error happens, along with the particular expression that failed.
  • Inline assembler expressions and statements are now supported:

    ,----
    | asm (TEMPLATE [: OUTPUTS [: INPUTS]])
    | asm TYPE: (TEMPLATE [: INPUTS])
    `----

  • Both `printf' and `format' now support printing values of type `any'.
  • Both `printf' and `format' now support printing integral values interpreted as floating-point values encoded in IEEE 754.  Format tags %f, %g and %e are supported.  This feature, along with the new ieee754.pk pickle, eases dealing with floating-point data in binary data.
  • Pre-conditional optional fields are added to complement the currently supported post-conditional optional fields. A pre-conditional optional field like the following makes FNAME optional based on the evaluation of CONDITION.  But the field itself is not mapped if the condition evaluates to false:

    ,----
    | if (CONDITION)
    |   TYPE FNAME;
    `----

  • A new option `.set autoremap no' can be used in order to tell poke to not remap mapped values automatically.  This greatly speeds up things, but assumes that the contents of the IO space are not updated out of the control of the user.  See the manual for details.
  • The :to argument to the `extract' command is now optional, and defaults to the empty string.
  • ${XDG_CONFIG_HOME:-$HOME/.config} is now preferred to XDG_CONFIG_DIRS.

Poke Language updates

  • Array and struct constructors are now primaries in the Poke syntax. This means that it is no longer necessary to enclose them between parenthesis in constructions like:

    ,----
    | (Packet {}).field
    `----

    and this is now accepted:
    ,----
    | Packet {}.field
    `----

  • Bit-concatenation is now supported in l-values.  After executing the following code the value of `a' is 0x1N and the value of `b' is (uint<28>)0x2345678:

    ,----
    | var a = 0 as int<4>;
    | var b = 0 as uint<28>;
    |
    | a:::b = 0x12345678;
    `----

  • Arrays can now be indented by size, by specifying an offset as an index.  This is particularly useful for accessing structures such as string tables without having to explicitly iterate on the array's elements.
  • Union types can now be declared as "integral".  The same features of integral structs are now available for unions: integration, deintegration, the ability of being used in contexts where an integer is expected, etc.
  • Support for "computed fields" has been added to struct and union types.  Computed fields are accessed just like regular fields, but the semantics of referring to them and of assigning to them are specified by the user by the way of defining getter and setter methods.
  • This version introduces three new Poke attributes that work on values of type `any':

    ,----
    | VAL'elem (N)
    |    evaluates to the Nth element in VAL, as a value of type `any'.
    |
    | VAL'eoffset (N)
    |    evaluates to the offset of the Nth element in VAL.
    |
    | VAL'esize (N)
    |    evaluates to the size of the Nth element in VAL.
    |
    | VAL'ename (N)
    |    attribute evaluates to the name of the Nth element in VAL.
    `----

  • Two new operators have been introduced to facilitate operating Poke array as stacks in an efficient way: apush and apop.  Since these operators change the size of the involved arrays, they are only allowed in unbounded arrays.
  • Poke programs can now hook in the IO subsystem by installing functions that will be invoked when certain operations on IO spaces are being performed:

    ,----
    | ios_open_hook
    |   Functions in this hook are invoked once a new IO space has been
    |   opened.
    |
    | ios_set_hook
    |   Functions in this hook are invoked once the current IO space
    |   changes.
    |
    | ios_close_pre_hook
    | ios_close_hook
    |   Functions in these hooks are invoked before and after an IO space is
    |   closed, respectively.
    `----

  • The 'length attribute is now valid in values of type `any'.
  • Poke declarations can now be annotated as `immutable'.  It is not allowed to re-define immutable definitions.
  • A new compiler built-in `iolist' has been introduced, that returns an array with the IO space identifiers of currently open IOS.
  • We have changed the logic of the EXCOND operator ?!.  It now evaluates to 1 (true) if the execution of the first operand raises the specified exception, and to 0 (false) otherwise.  We profusedly apologize for the backwards incompatibility, but this is way better than the previous (reversed) logic.
  • The containing struct or union value can now be refered as SELF in the body of methods.  SELF is of type `any'.
  • Integer literal suffixes (B, H, U, etc) are case-insensitive. But until now little-case `b' wasn't being recognized as such.  Now `1B' is the same than `1b'.
  • Casting to union types now raise a compile-time error.
  • If no explicit message is specified in calls to `assert', a default one showing the source code of the failing condition is constructed and used instead.
  • An operator `remap' has been used in order to force a re-map of some mapped Poke value.
  • Signed integral types of one bit are not allowed.  How could they be, in two's complement?
  • The built-in function get_time has been renamed to gettime, to follow the usual naming of the corresponding standard C function.

Standard Poke Library updates

  • New standard functions:

    ,----
    | eoffset (V, N)
    |   Given a value of type `any' and a name, returns the offset of
    |   the element having that name.
    |
    | openset (HANDLER, [FLAGS])
    |   Open an IO space and make it the current IO space.
    |
    | with_temp_ios ([HANDLER], [FLAGS], [DO], [ENDIAN])
    |   Execute some code with a temporary IO space.
    |
    | with_cur_ios (IOS, [DO], [ENDIAN])
    |   Execute some code on some given IO space.
    `----

libpoke updates

  • New API function pk_struct_ref_set_field_value.
  • New API function pk_type_name.

Pickles updates

  • New pickles provided in the poke distribution:

    ,----
    | diff.pk
    |   Useful binary diffing utilities.  In particular, it implements
    |   the "structured diff" format as described in
    |   https://binary-tools.net/bindiff.pdf.
    |
    | io.pk
    |   Facilities to dump data to the terminal.
    |
    | pk-table.pk
    |   Convenient facilities to Poke programs to print tabulated data.
    |
    | openpgp.pk
    |   Pickle to poke at OpenPGP RFC 4880 data.
    |
    | sframe.pk
    | sframe-dump.pk
    |   Pickles for the SFrame unwinding format, and related dump
    |   utilities.
    |
    | search.pk
    |   Utility for searching data in IO spaces that conform to some
    |   given Poke type.
    |
    | riscv.pk
    |   Pickle to poke at instructions encoded in the RISC-V instruction
    |   set (RV32I).  It also provides methods to generate assembly
    |   language.
    |
    | coff.pk
    | coff-aarch64.pk
    | coff-i386.pk
    |   COFF object files.
    |
    | pe.pk
    | pe-amd64.pk
    | pe-arm.pk
    | pe-arm64.pk
    | pe-debug.pk
    | pe-i386.pk
    | pe-ia64.pk
    | pe-m32r.pk
    | pe-mips.pk
    | pe-ppc.pk
    | pe-riscv.pk
    | pe-sh3.pk
    |   PE/COFF object files.
    |
    | pcap.pk
    |   Capture file format.
    |
    | uuid.pk
    |   Universally Unique Identifier (UUID) as defined by RFC4122.
    |
    | redoxfs.pk
    |   RedoxFS files ystem of Redox OS.
    |
    | ieee754.pk
    |   IEEE Standard for Floating-Point Arithmetic.
    `----

  • The ELF pickle now provides functions implementing ELF hashing.

Build system updates

  • It is now supported to configure the poke sources with --disable-hserver.

Documentation updates

  • Documentation for the `format' language construction has been added to the poke manual.

Other updates

  • A new program poked, for "poke daemon", has been contributed to the poke distribution by Mohammad-Reza Nabipoor.  poked links with libpoke and uses Unix sockets to act as a broker to communicate with an instance of a Poke incremental compiler.  This is already used by several user interfaces to poke.
  • The machine-interface subsystem has been removed from poke, in favor of the poked approach.
  • The example GUI that was intended to be a test tool for the machine interface has been removed from the poke distribution.
  • Many bugs have been fixed.

--
Jose E. Marchesi
Frankfurt am Main
26 January 2023

25 January, 2023 07:08PM by Jose E. Marchesi