Planet GNU

Aggregation of development blogs from the GNU Project

September 22, 2022

parallel @ Savannah

GNU Parallel 20220922 ('Elizabeth') released

GNU Parallel 20220922 ('Elizabeth') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  reduced our backend test pipelines from 4 to 1.30 hrs. gnu parallel for the win!!!
     -- Swapnil Sahu @CaffeinatedWryy@twitter

New in this release:

  • --colour-failed only changes output for failing jobs.
  • Password for --sshlogin can be put in $SSHPASS.
  • Examples are moved from `man parallel` to `man parallel_examples`.
  • Bug fixes and man page updates.

News about GNU Parallel:

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

22 September, 2022 07:49PM by Ole Tange

FSF Blogs

Interview with Martin Dougiamas of Moodle

This is the latest installment of our Licensing and Compliance Lab's series on free software developers who choose GNU licenses for their works.

22 September, 2022 06:12PM

September 21, 2022

Free Software Awards: Nominate those who have charted a course to freedom by November 30

Announcement of this year's Free Software Awards. Read more about how to nominate individuals and projects who have made an impact in free software.

21 September, 2022 08:40PM

September 20, 2022

Simon Josefsson

Privilege separation of GSS-API credentials for Apache

To protect web resources with Kerberos you may use Apache HTTPD with mod_auth_gssapi — however, all web scripts (e.g., PHP) run under Apache will have access to the Kerberos long-term symmetric secret credential (keytab). If someone can get it, they can impersonate your server, which is bad.

The gssproxy project makes it possible to introduce privilege separation to reduce the attack surface. There is a tutorial for RPM-based distributions (Fedora, RHEL, AlmaLinux, etc), but I wanted to get this to work on a DPKG-based distribution (Debian, Ubuntu, Trisquel, PureOS, etc) and found it worthwhile to document the process. I’m using Ubuntu 22.04 below, but have tested it on Debian 11 as well. I have adopted the gssproxy package in Debian, and testing this setup is part of the scripted autopkgtest/debci regression testing.

First install the required packages:

root@foo:~# apt-get update
root@foo:~# apt-get install -y apache2 libapache2-mod-auth-gssapi gssproxy curl

This should give you a working and running web server. Verify it is operational under the proper hostname, I’ll use foo.sjd.se in this writeup.

root@foo:~# curl --head http://foo.sjd.se/
HTTP/1.1 200 OK

The next step is to create a keytab containing the Kerberos V5 secrets for your host, the exact steps depends on your environment (usually kadmin ktadd or ipa-getkeytab), but use the string “HTTP/foo.sjd.se” and then confirm using something like the following.

root@foo:~# ls -la /etc/gssproxy/httpd.keytab
-rw------- 1 root root 176 Sep 18 06:44 /etc/gssproxy/httpd.keytab
root@foo:~# klist -k /etc/gssproxy/httpd.keytab -e
Keytab name: FILE:/etc/gssproxy/httpd.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 HTTP/foo.sjd.se@GSSPROXY.EXAMPLE.ORG (aes256-cts-hmac-sha1-96) 
   2 HTTP/foo.sjd.se@GSSPROXY.EXAMPLE.ORG (aes128-cts-hmac-sha1-96) 
root@foo:~# 

The file should be owned by root and not be in the default /etc/krb5.keytab location, so Apache’s libapache2-mod-auth-gssapi will have to use gssproxy to use it.

Then configure gssproxy to find the credential and use it with Apache.

root@foo:~# cat<<EOF > /etc/gssproxy/80-httpd.conf
[service/HTTP]
mechs = krb5
cred_store = keytab:/etc/gssproxy/httpd.keytab
cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
euid = www-data
process = /usr/sbin/apache2
EOF

For debugging, it may be useful to enable more gssproxy logging:

root@foo:~# cat<<EOF > /etc/gssproxy/gssproxy.conf
[gssproxy]
debug_level = 1
EOF
root@foo:~#

Restart gssproxy so it finds the new configuration, and monitor syslog as follows:

root@foo:~# tail -F /var/log/syslog &
root@foo:~# systemctl restart gssproxy

You should see something like this in the log file:

Sep 18 07:03:15 foo gssproxy[4076]: [2022/09/18 05:03:15]: Exiting after receiving a signal
Sep 18 07:03:15 foo systemd[1]: Stopping GSSAPI Proxy Daemon…
Sep 18 07:03:15 foo systemd[1]: gssproxy.service: Deactivated successfully.
Sep 18 07:03:15 foo systemd[1]: Stopped GSSAPI Proxy Daemon.
Sep 18 07:03:15 foo gssproxy[4092]: [2022/09/18 05:03:15]: Debug Enabled (level: 1)
Sep 18 07:03:15 foo systemd[1]: Starting GSSAPI Proxy Daemon…
Sep 18 07:03:15 foo gssproxy[4093]: [2022/09/18 05:03:15]: Kernel doesn't support GSS-Proxy (can't open /proc/net/rpc/use-gss-proxy: 2 (No such file or directory))
Sep 18 07:03:15 foo gssproxy[4093]: [2022/09/18 05:03:15]: Problem with kernel communication! NFS server will not work
Sep 18 07:03:15 foo systemd[1]: Started GSSAPI Proxy Daemon.
Sep 18 07:03:15 foo gssproxy[4093]: [2022/09/18 05:03:15]: Initialization complete.

The NFS-related errors is due to a default gssproxy configuration file, it is harmless and if you don’t use NFS with GSS-API you can silence it like this:

root@foo:~# rm /etc/gssproxy/24-nfs-server.conf
root@foo:~# systemctl try-reload-or-restart gssproxy

The log should now indicate that it loaded the keytab:

Sep 18 07:18:59 foo systemd[1]: Reloading GSSAPI Proxy Daemon…
Sep 18 07:18:59 foo gssproxy[4182]: [2022/09/18 05:18:59]: Received SIGHUP; re-reading config.
Sep 18 07:18:59 foo gssproxy[4182]: [2022/09/18 05:18:59]: Service: HTTP, Keytab: /etc/gssproxy/httpd.keytab, Enctype: 18
Sep 18 07:18:59 foo gssproxy[4182]: [2022/09/18 05:18:59]: New config loaded successfully.
Sep 18 07:18:59 foo systemd[1]: Reloaded GSSAPI Proxy Daemon.

To instruct Apache — or actually, the MIT Kerberos V5 GSS-API library used by mod_auth_gssap loaded by Apache — to use gssproxy instead of using /etc/krb5.keytab as usual, Apache needs to be started in an environment that has GSS_USE_PROXY=1 set. The background is covered by the gssproxy-mech(8) man page and explained by the gssproxy README.

When systemd is used the following can be used to set the environment variable, note the final command to reload systemd.

root@foo:~# mkdir -p /etc/systemd/system/apache2.service.d
root@foo:~# cat<<EOF > /etc/systemd/system/apache2.service.d/gssproxy.conf
[Service]
Environment=GSS_USE_PROXY=1
EOF
root@foo:~# systemctl daemon-reload

The next step is to configure a GSS-API protected Apache resource:

root@foo:~# cat<<EOF > /etc/apache2/conf-available/private.conf
<Location /private>
  AuthType GSSAPI
  AuthName "GSSAPI Login"
  Require valid-user
</Location>

Enable the configuration and restart Apache — the suggested use of reload is not sufficient, because then it won’t be restarted with the newly introduced GSS_USE_PROXY variable. This just applies to the first time, after the first restart you may use reload again.

root@foo:~# a2enconf private
Enabling conf private.
To activate the new configuration, you need to run:
systemctl reload apache2
root@foo:~# systemctl restart apache2

When you have debug messages enabled, the log may look like this:

Sep 18 07:32:23 foo systemd[1]: Stopping The Apache HTTP Server…
Sep 18 07:32:23 foo gssproxy[4182]: [2022/09/18 05:32:23]: Client [2022/09/18 05:32:23]: (/usr/sbin/apache2) [2022/09/18 05:32:23]: connected (fd = 10)[2022/09/18 05:32:23]: (pid = 4651) (uid = 0) (gid = 0)[2022/09/18 05:32:23]:
Sep 18 07:32:23 foo gssproxy[4182]: message repeated 4 times: [ [2022/09/18 05:32:23]: Client [2022/09/18 05:32:23]: (/usr/sbin/apache2) [2022/09/18 05:32:23]: connected (fd = 10)[2022/09/18 05:32:23]: (pid = 4651) (uid = 0) (gid = 0)[2022/09/18 05:32:23]:]
Sep 18 07:32:23 foo systemd[1]: apache2.service: Deactivated successfully.
Sep 18 07:32:23 foo systemd[1]: Stopped The Apache HTTP Server.
Sep 18 07:32:23 foo systemd[1]: Starting The Apache HTTP Server…
Sep 18 07:32:23 foo gssproxy[4182]: [2022/09/18 05:32:23]: Client [2022/09/18 05:32:23]: (/usr/sbin/apache2) [2022/09/18 05:32:23]: connected (fd = 10)[2022/09/18 05:32:23]: (pid = 4657) (uid = 0) (gid = 0)[2022/09/18 05:32:23]:
root@foo:~# Sep 18 07:32:23 foo gssproxy[4182]: message repeated 8 times: [ [2022/09/18 05:32:23]: Client [2022/09/18 05:32:23]: (/usr/sbin/apache2) [2022/09/18 05:32:23]: connected (fd = 10)[2022/09/18 05:32:23]: (pid = 4657) (uid = 0) (gid = 0)[2022/09/18 05:32:23]:]
Sep 18 07:32:23 foo systemd[1]: Started The Apache HTTP Server.

Finally, set up a dummy test page on the server:

root@foo:~# echo OK > /var/www/html/private

To verify that the server is working properly you may acquire tickets locally and then use curl to retrieve the GSS-API protected resource. The "--negotiate" enables SPNEGO and "--user :" asks curl to use username from the environment.

root@foo:~# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: jas@GSSPROXY.EXAMPLE.ORG

Valid starting Expires Service principal
09/18/22 07:40:37 09/19/22 07:40:37 krbtgt/GSSPROXY.EXAMPLE.ORG@GSSPROXY.EXAMPLE.ORG
root@foo:~# curl --negotiate --user : http://foo.sjd.se/private
OK
root@foo:~#

The log should contain something like this:

Sep 18 07:56:00 foo gssproxy[4872]: [2022/09/18 05:56:00]: Client [2022/09/18 05:56:00]: (/usr/sbin/apache2) [2022/09/18 05:56:00]: connected (fd = 10)[2022/09/18 05:56:00]: (pid = 5042) (uid = 33) (gid = 33)[2022/09/18 05:56:00]:
Sep 18 07:56:00 foo gssproxy[4872]: [CID 10][2022/09/18 05:56:00]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 33,socket: (null)
Sep 18 07:56:00 foo gssproxy[4872]: [CID 10][2022/09/18 05:56:00]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 33,socket: (null)
Sep 18 07:56:00 foo gssproxy[4872]: [CID 10][2022/09/18 05:56:00]: gp_rpc_execute: executing 1 (GSSX_INDICATE_MECHS) for service "HTTP", euid: 33,socket: (null)
Sep 18 07:56:00 foo gssproxy[4872]: [CID 10][2022/09/18 05:56:00]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 33,socket: (null)
Sep 18 07:56:00 foo gssproxy[4872]: [CID 10][2022/09/18 05:56:00]: gp_rpc_execute: executing 9 (GSSX_ACCEPT_SEC_CONTEXT) for service "HTTP", euid: 33,socket: (null)

The Apache log will look like this, notice the authenticated username shown.

127.0.0.1 - jas@GSSPROXY.EXAMPLE.ORG [18/Sep/2022:07:56:00 +0200] "GET /private HTTP/1.1" 200 481 "-" "curl/7.81.0"

Congratulations, and happy hacking!

20 September, 2022 06:40AM by simon

September 19, 2022

poke @ Savannah

[VIDEO] Testing the toolchain with GNU poke: assembling RISC-V instructions

Don't miss this little talk from Mohammad-Reza Nabipoor about leveraging GNU poke as a test tool in the assembler.   He uses RISC-V to explore how to better write pickles for instruction sets.  Looks promising!

https://www.youtube.com/watch?v=n09mhw4-m_E

19 September, 2022 11:52AM by Jose E. Marchesi

September 16, 2022

FSF Blogs

FSD meeting recap 2022-09-16

Check out the great work our volunteers accomplished at today's Free Software Directory (FSD) IRC meeting.

16 September, 2022 09:00PM

September 14, 2022

FSF Events

LibrePlanet 2022 CFS office hours (2022-09-22)

Need help getting your session proposal in good shape? We're holding office hours in #LibrePlanet on Libera.chat Thursdays at 13:00 EDT.

14 September, 2022 10:55PM

LibrePlanet 2022 CFS office hours (2022-10-27)

Need help getting your session proposal in good shape? We're holding office hours in #LibrePlanet on Libera.chat Thursdays at 13:00 EDT.

14 September, 2022 10:55PM

LibrePlanet 2022 CFS office hours (2022-10-20)

Need help getting your session proposal in good shape? We're holding office hours in #LibrePlanet on Libera.chat Thursdays at 13:00 EDT.

14 September, 2022 10:55PM

LibrePlanet 2022 CFS office hours (2022-10-13)

Need help getting your session proposal in good shape? We're holding office hours in #LibrePlanet on Libera.chat Thursdays at 13:00 EDT.

14 September, 2022 10:55PM

LibrePlanet 2022 CFS office hours (2022-09-29)

Need help getting your session proposal in good shape? We're holding office hours in #LibrePlanet on Libera.chat Thursdays at 13:00 EDT.

14 September, 2022 10:55PM

FSF Blogs

LibrePlanet 2023: Let's chart the course together! Submit your session by November 2

We are proud to announce the fifteenth edition of the Free Software Foundation's (FSF) conference on ethical technology and user freedom, which will be held in spring 2023, both online and in Boston (exact venue TBD). In these fifteen years, LibrePlanet has always been a community that brings together concerned users of all varieties to carve out the direction of software freedom for today as well as for years to come. The call for sessions is now open and will close on November 2, 2022. Potential talks should examine free software through the lens of the theme "Charting the Course."

14 September, 2022 10:55PM

September 13, 2022

unifont @ Savannah

Unifont 15.0.01 Released

13 September 2022 Unifont 15.0.01 is now available.  This is a major release corresponding to today's Unicode 15.0.0 release.

Download this release from GNU server mirrors at:

     https://ftpmirror.gnu.org/unifont/unifont-15.0.01/

or if that fails,

     https://ftp.gnu.org/gnu/unifont/unifont-15.0.01/

or, as a last resort,

     ftp://ftp.gnu.org/gnu/unifont/unifont-15.0.01/

These files are also available on the unifoundry.com website:

     https://unifoundry.com/pub/unifont/unifont-15.0.01/

Font files are in the subdirectory

     https://unifoundry.com/pub/unifont/unifont-15.0.01/font-builds/

A more detailed description of font changes is available at

      https://unifoundry.com/unifont/index.html

and of utility program changes at

      http://unifoundry.com/unifont/unifont-utilities.html

13 September, 2022 06:03PM by Paul Hardy

September 12, 2022

Luca Saiu

GNU Hackers' Meeting 2022: Call for presentations, even remote

The GNU Hackers’ Meetings or or “GHMs” are a friendly and informal venue to discuss technical topics related to GNU (https://www.gnu.org) and free software (https://www.gnu.org/philosophy/free-sw.html); anybody is welcome to register and attend. The GNU Hackers’ Meeting 2022 will take place on October 1st and October 2st in İzmir, Turkey; see the event home page at . We decided to help students who wish to attend by contributing 50€ out of their 60€ attendance fee (required by the hotel for use of the conference room, coffee and snacks) so that students will need to only pay 10€, upon presenting proof of ... [Read more]

12 September, 2022 04:05PM by Luca Saiu (positron@gnu.org)

FSF Blogs

The need for free software education now

Making the choice to use free software in the classroom helps us create learning environments that foreground questions of value. This can be illustrated by a passage in Confucius's "Analects" where the skilled mechanic is illustrated as sharpening their tools before they are able to do their work well. In the 21st century, and in the context of digital learning environments, to sharpen tools would mean to know not just "how" the tool works but "what" the tool means with respect to the place of the human in the world. This will be explored by considering the five Ws: what free software education is, who it is for, when and where it takes place, and why our active role as digital makers, not just passive users, is central to the meaning of free software education.

12 September, 2022 03:55PM

September 08, 2022

GNUnet News

messenger-cli 0.1.0

messenger-cli 0.1.0 released

We are pleased to announce the release of the messenger-cli application.
The application is a counterpart for the terminal to the previous release of the GTK application using the GNUnet Messenger service. The goal is to provide private and secure communication between any group of devices. So server admins or users relying on a terminal focused window manager have now a proper option to utilize the service as well.

The application provides the following features:

  • Creating direct chats and group chats
  • Sending text messages
  • Sharing files privately
  • Deleting messages
  • Verifying contact identities
  • Switching between different accounts

The application utilizes the previously released library "libgnunetchat" in an user interface built with ncurses. It will adapt its different views depending on the terminal size to show most important information. The navigation is done via arrow-, ESCAPE, TAB, ENTER and DELETE keys. More information about that can be found here .

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links might be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

Noteworthy changes in 0.1.0

  • It is possible to create direct chats and group chats via lobbies, shared keys or invitations
  • Members of a chats can be observed
  • Chats allow sending text messages or files
  • Messages can be deleted in any chat locally
  • Switching between different accounts can be done during runtime

A detailed list of changes can be found in the ChangeLog .

Known Issues

  • It is still difficult to get reliable chats between different devices. This might change with the upcoming changes on the GNUnet transport layer though.
  • It might happen that the FS service is not connected which might stop any file upload or stall it forever.

In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org .

08 September, 2022 10:00PM

September 04, 2022

Luca Saiu

The GNU Hackers' Meeting 2022 is less than one month away

The GNU Hackers’ Meetings are a venue to discuss technical topics related to GNU and free software. GNU Hackers’ Meetings have been taking place since 2007: you may want to look at the pages documenting most past editions (https://www.gnu.org/ghm/previous.html) which in many cases also include presentation slides and video recordings. The event atmosphere is always friendly and informal. Anybody is welcome to register and attend, including newcomers. The next GNU Hackers’ Meeting will take place in İzmir, Turkey on Saturday 1st and Sunday 2nd October 2022. We updated the GHM 2022 web page (https://www.gnu.org/ghm/2022) with information about the venue, accommodation ... [Read more]

04 September, 2022 06:21PM by Luca Saiu (positron@gnu.org)

September 03, 2022

GNUnet News

GNUnet 0.17.5

03 September, 2022 10:00PM

grep @ Savannah

grep-3.8 released [stable]

This is to announce grep-3.8, a stable release.
Special thanks to Carlo Arenas for adding PCRE2 support
and to Paul Eggert for his many fine changes.

There have been 104 commits by 6 people in the 55 weeks since 3.7.
See the NEWS below for a brief summary.

Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Carlo Marcelo Arenas Belón (2)
  Helge Kreutzmann (1)
  Jim Meyering (27)
  Ondřej Fiala (1)
  Paul Eggert (71)
  Ulrich Eckhardt (2)

Jim [on behalf of the grep maintainers]
==================================================================

Here is the GNU grep home page:
    http://gnu.org/s/grep/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=grep.git;a=shortlog;h=v3.8
or run this command from a git-cloned grep directory:
  git shortlog v3.7..v3.8

To summarize the 432 gnulib-related changes, run these commands
from a git-cloned grep directory:
  git checkout v3.8
  git submodule summary v3.7

==================================================================
Here are the compressed sources:
  https://ftp.gnu.org/gnu/grep/grep-3.8.tar.gz   (2.8MB)
  https://ftp.gnu.org/gnu/grep/grep-3.8.tar.xz   (1.7MB)

Here are the GPG detached signatures:
  https://ftp.gnu.org/gnu/grep/grep-3.8.tar.gz.sig
  https://ftp.gnu.org/gnu/grep/grep-3.8.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:
eb3bf741fefb2d64e67d9ea6d74c723ea0efddb6  grep-3.8.tar.gz
jeYKUWnAwf3YFwvZO72ldbh7/Pp95jGbi9YNwgvi+5c  grep-3.8.tar.gz
6d0d32cabaf44efac9e1d2c449eb041525c54b2e  grep-3.8.tar.xz
SY18wbT7CBkE2HND/rtzR1z3ceQk+35hQa/2YBOrw4I  grep-3.8.tar.xz

Each SHA256 checksum is base64 encoded, preferred over the much
longer hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify grep-3.8.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096/0x7FD9FCCB000BEEEE 2010-06-14 [SCEA]
        Key fingerprint = 155D 3FC5 00C8 3448 6D1E  EA67 7FD9 FCCB 000B EEEE
  uid   Jim Meyering <jim@meyering.net>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key jim@meyering.net
  gpg --recv-keys 7FD9FCCB000BEEEE
  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=grep&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify grep-3.8.tar.gz.sig

This release was bootstrapped with the following tools:
  Autoconf 2.72a.55-bc66c
  Automake 1.16i
  Gnulib v0.1-5279-g19435dc207

==================================================================
NEWS

* Noteworthy changes in release 3.8 (2022-09-02) [stable]

** Changes in behavior

  The -P option is now based on PCRE2 instead of the older PCRE,
  thanks to code contributed by Carlo Arenas.

  The egrep and fgrep commands, which have been deprecated since
  release 2.5.3 (2007), now warn that they are obsolescent and should
  be replaced by grep -E and grep -F.

  The confusing GREP_COLOR environment variable is now obsolescent.
  Instead of GREP_COLOR='xxx', use GREP_COLORS='mt=xxx'.  grep now
  warns if GREP_COLOR is used and is not overridden by GREP_COLORS.
  Also, grep now treats GREP_COLOR like GREP_COLORS by silently
  ignoring it if it attempts to inject ANSI terminal escapes.

  Regular expressions with stray backslashes now cause warnings, as
  their unspecified behavior can lead to unexpected results.
  For example, '\a' and 'a' are not always equivalent
  <https://bugs.gnu.org/39678>.  Similarly, regular expressions or
  subexpressions that start with a repetition operator now also cause
  warnings due to their unspecified behavior; for example, *a(+b|{1}c)
  now has three reasons to warn.  The warnings are intended as a
  transition aid; they are likely to be errors in future releases.

  Regular expressions like [:space:] are now errors even if
  POSIXLY_CORRECT is set, since POSIX now allows the GNU behavior.

** Bug fixes

  In locales using UTF-8 encoding, the regular expression '.' no
  longer sometimes fails to match Unicode characters U+D400 through
  U+D7FF (some Hangul Syllables, and Hangul Jamo Extended-B) and
  Unicode characters U+108000 through U+10FFFF (half of Supplemental
  Private Use Area plane B).
  [bug introduced in grep 3.4]

  The -s option no longer suppresses "binary file matches" messages.
  [Bug#51860 introduced in grep 3.5]

** Documentation improvements

  The manual now covers unspecified behavior in patterns like \x, (+),
  and range expressions outside the POSIX locale.

03 September, 2022 08:04AM by Jim Meyering

September 02, 2022

remotecontrol @ Savannah

September 01, 2022

Andy Wingo

new month, new brainworm

Today, a brainworm! I had a thought a few days ago and can't get it out of my head, so I need to pass it on to another host.

So, imagine a world in which there is a a drive to build a kind of Kubernetes on top of WebAssembly. Kubernetes nodes are generally containers, associated with additional metadata indicating their place in overall system topology (network connections and so on). (I am not a Kubernetes specialist, as you can see; corrections welcome.) Now in a WebAssembly cloud, the nodes would be components, probably also with additional topological metadata. VC-backed companies will duke it out for dominance of the WebAssembly cloud space, and in a couple years we will probably emerge with an open source project that has become a de-facto standard (though it might be dominated by one or two players).

In this world, Kubernetes and Spiffy-Wasm-Cloud will coexist. One of the success factors for Kubernetes was that you can just put your old database binary inside a container: it's the same ABI as when you run your database in a virtual machine, or on (so-called!) bare metal. The means of composition are TCP and UDP network connections between containers, possibly facilitated by some kind of network fabric. In contrast, in Spiffy-Wasm-Cloud we aren't starting from the kernel ABI, with processes and such: instead there's WASI, which is more of a kind of specialized and limited libc. You can't just drop in your database binary, you have to write code to get it to conform to the new interfaces.

One consequence of this situation is that I expect WASI and the component model to develop a rich network API, to allow WebAssembly components to interoperate not just with end-users but also other (micro-)services running in the same cloud. Likewise there is room here for a company to develop some complicated network fabrics for linking these things together.

However, WebAssembly-to-WebAssembly links are better expressed via typed functional interfaces; it's more expressive and can be faster. Not only can you end up having fine-grained composition that looks more like lightweight Erlang processes, you can also string together components in a pipeline with communications overhead approaching that of a simple function call. Relative to Kubernetes, there are potential 10x-100x improvements to be had, in throughput and in memory footprint, at least in some cases. It's the promise of this kind of improvement that can drive investment in this area, and eventually adoption.

But, you still have some legacy things running in containers. What to do? Well... Maybe recompile them to WebAssembly? That's my brain-worm.

A container is a file system image containing executable files and data. Starting with the executable files, they are in machine code, generally x64, and interoperate with system libraries and the run-time via an ABI. You could compile them to WebAssembly instead. You could interpret them as data, or JIT-compile them as webvm does, or directly compile them to WebAssembly. This is the sort of thing you hire Fabrice Bellard to do ;) Then you have the filesystem. Let's assume it is stateless: any change to the filesystem at runtime doesn't need to be preserved. (I understand this is a goal, though I could be wrong.) So you could put the filesystem in memory, as some kind of addressable data structure, and you make the libc interface access that data structure. It's something like the microkernel approach. And then you translate whatever topological connectivity metadata you had for Kubernetes to your Spiffy-Wasm-Cloud's format.

Anyway in the end you have a WebAssembly module and some metadata, and you can run it in your WebAssembly cloud. Or on the more basic level, you have a container and you can now run it on any machine with a WebAssembly implementation, even on other architectures (coucou RISC-V!).

Anyway, that's the tweet. Have fun, whoever gets to work on this :)

01 September, 2022 10:12AM by Andy Wingo

August 31, 2022

freeipmi @ Savannah

FreeIPMI 1.6.10 Released

o Support IPv6 Lan configuration in ipmi-config.  IPv6
  configuration is supported in the new Lan6_Conf section.
o Fix static compilation issues by renaming a number of internal
  functions.
o Misc documentation corrections.

https://ftp.gnu.org/gnu/freeipmi/freeipmi-1.6.10.tar.gz

31 August, 2022 05:13PM by Albert Chu

August 30, 2022

Parabola GNU/Linux-libre

Grub bootloader upgrade and configuration incompatibilities

2022-08-30 - Christian Hesse

Recent changes in grub added a new command option to fwsetup and changed the way the command is invoked in the generated boot configuration. Depending on your system hardware and setup this could cause an unbootable system due to incompatibilities between the installed bootloader and configuration. After a grub package update it is advised to run both, installation and regeneration of configuration:

grub-install ...
grub-mkconfig -o /boot/grub/grub.cfg

30 August, 2022 09:22PM by bill auger

August 23, 2022

Andy Wingo

accessing webassembly reference-typed arrays from c++

The WebAssembly garbage collection proposal is coming soonish (really!) and will extend WebAssembly with the the capability to create and access arrays whose memory is automatically managed by the host. As long as some system component has a reference to an array, it will be kept alive, and as soon as nobody references it any more, it becomes "garbage" and is thus eligible for collection.

(In a way it's funny to define the proposal this way, in terms of what happens to garbage objects that by definition aren't part of the program's future any more; really the interesting thing is the new things you can do with live data, defining new data types and representing them outside of linear memory and passing them between components without copying. But "extensible-arrays-structs-and-other-data-types" just isn't as catchy as "GC". Anyway, I digress!)

One potential use case for garbage-collected arrays is for passing large buffers between parts of a WebAssembly system. For example, a webcam driver could produce a stream of frames as reference-typed arrays of bytes, and then pass them by reference to a sandboxed WebAssembly instance to, I don't know, identify cats in the images or something. You get the idea. Reference-typed arrays let you avoid copying large video frames.

A lot of image-processing code is written in C++ or Rust. With WebAssembly 1.0, you just have linear memory and no reference-typed values, which works well for these languages that like to think of memory as having a single address space. But once you get reference-typed arrays in the mix, you effectively have multiple address spaces: you can't address the contents of the array using a normal pointer, as you might be able to do if you mmap'd the buffer into a program's address space. So what do you do?

reference-typed values are special

The broader question of C++ and GC-managed arrays is, well, too broad for today. The set of array types is infinite, because it's not just arrays of i32, it's also arrays of arrays of i32, and arrays of those, and arrays of records, and so on.

So let's limit the question to just arrays of i8, to see if we can make some progress. So imagine a C function that takes an array of i8:

void process(array_of_i8 array) {
  // ?
}

If you know WebAssembly, there's a clear translation of the sort of code that we want:

(func (param $array (ref (array i8)))
  ; operate on local 0
  )

The WebAssembly function will have an array as a parameter. But, here we start to run into more problems with the LLVM toolchain that we use to compile C and other languages to WebAssembly. When the C front-end of LLVM (clang) compiles a function to the LLVM middle-end's intermediate representation (IR), it models all local variables (including function parameters) as mutable memory locations created with alloca. Later optimizations might turn these memory locations back to SSA variables and thence to registers or stack slots. But, a reference-typed value has no bit representation, and it can't be stored to linear memory: there is no alloca that can hold it.

Incidentally this problem is not isolated to future extensions to WebAssembly; the externref and funcref data types that landed in WebAssembly 2.0 and in all browsers are also reference types that can't be written to main memory. Similarly, the table data type which is also part of shipping WebAssembly is not dissimilar to GC-managed arrays, except that they are statically allocated at compile-time.

At Igalia, my colleagues Paulo Matos and Alex Bradbury have been hard at work to solve this gnarly problem and finally expose reference-typed values to C. The full details and final vision are probably a bit too much for this article, but some bits on the mechanism will help.

Firstly, note that LLVM has a fairly traditional breakdown between front-end (clang), middle-end ("the IR layer"), and back-end ("the MC layer"). The back-end can be quite target-specific, and though it can be annoying, we've managed to get fairly good support for reference types there.

In the IR layer, we are currently representing GC-managed values as opaque pointers into non-default, non-integral address spaces. LLVM attaches an address space (an integer less than 224 or so) to each pointer, mostly for OpenCL and GPU sorts of use-cases, and we abuse this to prevent LLVM from doing much reasoning about these values.

This is a bit of a theme, incidentally: get the IR layer to avoid assuming anything about reference-typed values. We're fighting the system, in a way. As another example, because LLVM is really oriented towards lowering high-level constructs to low-level machine operations, it doesn't necessarily preserve types attached to pointers on the IR layer. Whereas for WebAssembly, we need exactly that: we reify types when we write out WebAssembly object files, and we need LLVM to pass some types through from front-end to back-end unmolested. We've had to change tack a number of times to get a good way to preserve data from front-end to back-end, and probably will have to do so again before we end up with a final design.

Finally on the front-end we need to generate an alloca in different address spaces depending on the type being allocated. And because reference-typed arrays can't be stored to main memory, there are semantic restrictions as to how they can be used, which need to be enforced by clang. Fortunately, this set of restrictions is similar enough to what is imposed by the ARM C Language Extensions (ACLE) for scalable vector (SVE) values, which also don't have a known bit representation at compile-time, so we can piggy-back on those. This patch hasn't landed yet, but who knows, it might land soon; in the mean-time we are going to run ahead of upstream a bit to show how you might define and use an array type definition. Further tacks here are also expected, as we try to thread the needle between exposing these features to users and not imposing too much of a burden on clang maintenance.

accessing array contents

All this is a bit basic, though; it just gives you enough to have a local variable or a function parameter of a reference-valued type. Let's continue our example:

void process(array_of_i8 array) {
  uint32_t sum;
  for (size_t idx = 0; i < __builtin_wasm_array_length(array); i++)
    sum += (uint8_t)__builtin_wasm_array_ref_i8(array, idx);
  // ...
}

The most basic way to extend C to access these otherwise opaque values is to expose some builtins, say __builtin_wasm_array_length and so on. Probably you need different intrinsics for each scalar array element type (i8, i16, and so on), and one for arrays which return reference-typed values. We'll talk about arrays of references another day, but focusing on the i8 case, the C builtin then lowers to a dedicated LLVM intrinsic, which passes through the middle layer unscathed.

In C++ I think we can provide some nicer syntax which preserves the syntactic illusion of array access.

I think this is going to be sufficient as an MVP, but there's one caveat: SIMD. You can indeed have an array of i128 values, but you can only access that array's elements as i128; worse, you can't load multiple data from an i8 array as i128 or even i32.

Compare this to to the memory control proposal, which instead proposes to map buffers to non-default memories. In WebAssembly, you can in theory (and perhaps soon in practice) have multiple memories. The easiest way I can see on the toolchain side is to use the address space feature in clang:

void process(uint8_t *array __attribute__((address_space(42))),
             size_t len) {
  uint32_t sum;
  for (size_t idx = 0; i < len; i++)
    sum += array[idx];
  // ...
}

How exactly to plumb the mapping between address spaces which can only be specified by number from the front-end to the back-end is a little gnarly; really you'd like to declare the set of address spaces that a compilation unit uses symbolically, and then have the linker produce a final allocation of memory indices. But I digress, it's clear that with this solution we can use SIMD instructions to load multiple bytes from memory at a time, so it's a winner with respect to accessing GC arrays.

Or is it? Perhaps there could be SIMD extensions for packed GC arrays. I think it makes sense, but it's a fair amount of (admittedly somewhat mechanical) specification and implementation work.

& future

In some future bloggies we'll talk about how we will declare new reference types: first some basics, then some more integrated visions for reference types and C++. Lots going on, and this is just a brain-dump of the current state of things; thoughts are very much welcome.

23 August, 2022 10:35AM by Andy Wingo

August 22, 2022

parallel @ Savannah

GNU Parallel 20220822 ('Rushdie') released

GNU Parallel 20220822 ('Rushdie') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  Parallel is Good Stuff (tm)
    -- bloopernova@ycombinator

New in this release:

  • --header 0 allows using {filename} as replacement string
  • Bug fixes and man page updates.

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

22 August, 2022 08:25PM by Ole Tange

August 18, 2022

Andy Wingo

just-in-time code generation within webassembly

Just-in-time (JIT) code generation is an important tactic when implementing a programming language. Generating code at run-time allows a program to specialize itself against the specific data it is run against. For a program that implements a programming language, that specialization is with respect to the program being run, and possibly with respect to the data that program uses.

The way this typically works is that the program generates bytes for the instruction set of the machine it's running on, and then transfers control to those instructions.

Usually the program has to put its generated code in memory that is specially marked as executable. However, this capability is missing in WebAssembly. How, then, to do just-in-time compilation in WebAssembly?

webassembly as a harvard architecture

In a von Neumman machine, like the ones that you are probably reading this on, code and data share an address space. There's only one kind of pointer, and it can point to anything: the bytes that implement the sin function, the number 42, the characters in "biscuits", or anything at all. WebAssembly is different in that its code is not addressable at run-time. Functions in a WebAssembly module are numbered sequentially from 0, and the WebAssembly call instruction takes the callee as an immediate parameter.

So, to add code to a WebAssembly program, somehow you'd have to augment the program with more functions. Let's assume we will make that possible somehow -- that your WebAssembly module that had N functions will now have N+1 functions, and with function N being the new one your program generated. How would we call it? Given that the call instructions hard-code the callee, the existing functions 0 to N-1 won't call it.

Here the answer is call_indirect. A bit of a reminder, this instruction take the callee as an operand, not an immediate parameter, allowing it to choose the callee function at run-time. The callee operand is an index into a table of functions. Conventionally, table 0 is called the indirect function table as it contains an entry for each function which might ever be the target of an indirect call.

With this in mind, our problem has two parts, then: (1) how to augment a WebAssembly module with a new function, and (2) how to get the original module to call the new code.

late linking of auxiliary webassembly modules

The key idea here is that to add code, the main program should generate a new WebAssembly module containing that code. Then we run a linking phase to actually bring that new code to life and make it available.

System linkers like ld typically require a complete set of symbols and relocations to resolve inter-archive references. However when performing a late link of JIT-generated code, we can take a short-cut: the main program can embed memory addresses directly into the code it generates. Therefore the generated module would import memory from the main module. All references from the generated code to the main module can be directly embedded in this way.

The generated module would also import the indirect function table from the main module. (We would ensure that the main module exports its memory and indirect function table via the toolchain.) When the main module makes the generated module, it also embeds a special patch function in the generated module. This function would add the new functions to the main module's indirect function table, and perform any relocations onto the main module's memory. All references from the main module to generated functions are installed via the patch function.

We plan on two implementations of late linking, but both share the fundamental mechanism of a generated WebAssembly module with a patch function.

dynamic linking via the run-time

One implementation of a linker is for the main module to cause the run-time to dynamically instantiate a new WebAssembly module. The run-time would provide the memory and indirect function table from the main module as imports when instantiating the generated module.

The advantage of dynamic linking is that it can update a live WebAssembly module without any need for re-instantiation or special run-time checkpointing support.

In the context of the web, JIT compilation can be triggered by the WebAssembly module in question, by calling out to functionality from JavaScript, or we can use a "pull-based" model to allow the JavaScript host to poll the WebAssembly instance for any pending JIT code.

For WASI deployments, you need a capability from the host. Either you import a module that provides run-time JIT capability, or you rely on the host to poll you for data.

static linking via wizer

Another idea is to build on Wizer's ability to take a snapshot of a WebAssembly module. You could extend Wizer to also be able to augment a module with new code. In this role, Wizer is effectively a late linker, linking in a new archive to an existing object.

Wizer already needs the ability to instantiate a WebAssembly module and to run its code. Causing Wizer to ask the module if it has any generated auxiliary module that should be instantiated, patched, and incorporated into the main module should not be a huge deal. Wizer can already run the patch function, to perform relocations to patch in access to the new functions. After having done that, Wizer (or some other tool) would need to snapshot the module, as usual, but also adding in the extra code.

As a technical detail, in the simplest case in which code is generated in units of functions which don't directly call each other, this is as simple as just appending the functions to the code section and then and appending the generated element segments to the main module's element segment, updating the appended function references to their new values by adding the total number of functions in the module before the new module was concatenated to each function reference.

late linking appears to be async codegen

From the perspective of a main program, WebAssembly JIT code generation via late linking appears the same as aynchronous code generation.

For example, take the C program:

struct Value;
struct Func {
  struct Expr *body;
  void *jitCode;
};

void recordJitCandidate(struct Func *func);
uint8_t* flushJitCode(); // Call to actually generate JIT code.

struct Value* interpretCall(struct Expr *body,
                            struct Value *arg);

struct Value* call(struct Func *func,
                   struct Value* val) {
  if (func->jitCode) {
    struct Value* (*f)(struct Value*) = jitCode;
    return f(val);
  } else {
    recordJitCandidate(func);
    return interpretCall(func->body, val);
  }
}

Here the C program allows for the possibility of JIT code generation: there is a slot in a Func instance to fill in with a code pointer. If this program generates code for a given Func, it won't be able to fill in the pointer -- it can't add new code to the image. But, it could tell Wizer to do so, and Wizer could snapshot the program, link in the new function, and patch &func->jitCode. From the program's perspective, it's as if the code becomes available asynchronously.

demo!

So many words, right? Let's see some code! As a sketch for other JIT compiler work, I implemented a little Scheme interpreter and JIT compiler, targetting WebAssembly. See interp.cc for the source. You compile it like this:

$ /opt/wasi-sdk/bin/clang++ -O2 -Wall \
   -mexec-model=reactor \
   -Wl,--growable-table \
   -Wl,--export-table \
   -DLIBRARY=1 \
   -fno-exceptions \
   interp.cc -o interplib.wasm

Here we are compiling with WASI SDK. I have version 14.

The -mexec-model=reactor argument means that this WASI module isn't just a run-once thing, after which its state is torn down; rather it's a multiple-entry component.

The two -Wl, options tell the linker to export the indirect function table, and to allow the indirect function table to be augmented by the JIT module.

The -DLIBRARY=1 is used by interp.cc; you can actually run and debug it natively but that's just for development. We're instead compiling to wasm and running with a WASI environment, giving us fprintf and other debugging niceties.

The -fno-exceptions is because WASI doesn't support exceptions currently. Also we don't need them.

WASI is mainly for non-browser use cases, but this module does so little that it doesn't need much from WASI and I can just polyfill it in browser JavaScript. So that's what we have here:

loading wasm-jit...

&&<<<<><>>&&<&>>>><><><><><><>>><>>

Each time you enter a Scheme expression, it will be parsed to an internal tree-like intermediate language. You can then run a recursive interpreter over that tree by pressing the "Evaluate" button. Press it a number of times, you should get the same result.

As the interpreter runs, it records any closures that it created. The Func instances attached to the closures have a slot for a C++ function pointer, which is initially NULL. Function pointers in WebAssembly are indexes into the indirect function table; the first slot is kept empty so that calling a NULL pointer (a pointer with value 0) causes an error. If the interpreter gets to a closure call and the closure's function's JIT code pointer is NULL, it will interpret the closure's body. Otherwise it will call the function pointer.

If you then press the "JIT" button above, the module will assemble a fresh WebAssembly module containing JIT code for the closures that it saw at run-time. Obviously that's just one heuristic: you could be more eager or more lazy; this is just a detail.

Although the particular JIT compiler isn't much of interest---the point being to see JIT code generation at all---it's nice to see that the fibonacci example sees a good speedup; try it yourself, and try it on different browsers if you can. Neat stuff!

not just the web

I was wondering how to get something like this working in a non-webby environment and it turns out that the Python interface to wasmtime is just the thing. I wrote a little interp.py harness that can do the same thing that we can do on the web; just run as `python3 interp.py`, after having `pip3 install wasmtime`:

$ python3 interp.py
...
Calling eval(0x11eb0) 5 times took 1.716s.
Calling jitModule()
jitModule result: <wasmtime._module.Module object at 0x7f2bef0821c0>
Instantiating and patching in JIT module
... 
Calling eval(0x11eb0) 5 times took 1.161s.

Interestingly it would appear that the performance of wasmtime's code (0.232s/invocation) is somewhat better than both SpiderMonkey (0.392s) and V8 (0.729s).

reflections

This work is just a proof of concept, but it's a step in a particular direction. As part of previous work with Fastly, we enabled the SpiderMonkey JavaScript engine to run on top of WebAssembly. When combined with pre-initialization via Wizer, you end up with a system that can start in microseconds: fast enough to instantiate a fresh, shared-nothing module on every HTTP request, for example.

The SpiderMonkey-on-WASI work left out JIT compilation, though, because, you know, WebAssembly doesn't support JIT compilation. JavaScript code actually ran via the C++ bytecode interpreter. But as we just found out, actually you can compile the bytecode: just-in-time, but at a different time-scale. What if you took a SpiderMonkey interpreter, pre-generated WebAssembly code for a user's JavaScript file, and then combined them into a single freeze-dried WebAssembly module via Wizer? You get the benefits of fast startup while also getting decent baseline performance. There are many engineering considerations here, but as part of work sponsored by Shopify, we have made good progress in this regard; details in another missive.

I think a kind of "offline JIT" has a lot of value for deployment environments like Shopify's and Fastly's, and you don't have to limit yourself to "total" optimizations: you can still collect and incorporate type feedback, and you get the benefit of taking advantage of adaptive optimization without having to actually run the JIT compiler at run-time.

But if we think of more traditional "online JIT" use cases, it's clear that relying on host JIT capabilities, while a good MVP, is not optimal. For one, you would like to be able to freely emit direct calls from generated code to existing code, instead of having to call indirectly or via imports. I think it still might make sense to have a language run-time express its generated code in the form of a WebAssembly module, though really you might want native support for compiling that code (asynchronously) from within WebAssembly itself, without calling out to a run-time. Most people I have talked to that work on WebAssembly implementations in JS engines believe that a JIT proposal will come some day, but it's good to know that we don't have to wait for it to start generating code and taking advantage of it.

& out

If you want to play around with the demo, do take a look at the wasm-jit Github project; it's fun stuff. Happy hacking, and until next time!

18 August, 2022 03:36PM by Andy Wingo

August 11, 2022

a2ps @ Savannah

a2ps 4.14.92 released [alpha]

This alpha release reverts extensive whitespace changes to --help output, so
as not to annoy translators (thanks, Benno Schulenberg!).  These will be
restored before the next stable release in a “whitespace-only” change.

11 August, 2022 03:06PM by Reuben Thomas

August 10, 2022

GNUnet News

GNUnet 0.17.4

10 August, 2022 10:00PM

August 09, 2022

GNU Taler news

Zero-Knowledge Age Restriction for GNU Taler

We propose a design for a privacy-friendly method of age restriction in e-commerce that is aligned with the principle of subsidiarity. The design is presented as an extension of a privacy-friendly payment protocol with a zero-knowledge scheme that cryprographically augments coins for this purpose. Our scheme enables buyers to prove to be of sufficient age for a particular transaction without disclosing it. Our modification preserves the privacy and security properties of the payment system such as the anonymity of minors as buyers as well as unlinkability of transactions. We show how our scheme can be instantiated with ECDSA as well with a variant of EdDSA, respectively, and how it can be integrated with the GNU Taler payment system. We provide formal proofs and implementation of our proposal. Key performance measurements for various CPU architectures and implementations are presented.

09 August, 2022 10:00PM

GNU Health

Cirugía Solidaria chooses GNU Health

The GNU Health community keeps growing, and that makes us very proud! This time, the Spanish non-profit organization Cirugía Solidaria has chosen GNU Health as their Hospital and Lab Management system.

Cirugía Solidaria was born in 2000 by a team of surgeons, anesthetists and nurses from “Virgen de la Arrixaca Hospital”, in Murcia, Spain, with the goal to provide medical assistance and to perform surgeries to underprivileged population and those in risk of social exclusion. Currently, Cirugía Solidaria counts with a multi-disciplinary team of health professionals around Spain that just made its 20th anniversary of cooperation.

GNUHealth Hospital Management client for Cirugía Solidaria

Around a month ago I received a message from Dr. Cerezuela, expressing their willingness to be part of the GNU Health community. Their main missions currently are focused, but not limited, to the African continent.

Source: Cirugía Solidaria

After several conferences and meetings, this August 1st 2022, Cirugía Solidaria and GNU Solidario signed an agreement to cooperate in the implementation, training and maintenance of the GNU Health Hospital Management and Lab Information System in those countries and health institutions where Cirugía Solidaria will be present.

Source: Cirugía Solidaria

This is very exciting. We have many projects in different countries from Africa, and working with Cirugía Solidaria will help to generate more local capacity, to cover the needs of those health professionals and their population.

This is not just about surgeries or health informatics. GNU Health will allow Cirugía Solidaria to create sustainable projects. They will have unified clinical and surgical histories, telemedicine; assess the nutritional and educational status of the population, and many other socioeconomic determinants of health and disease.

I want to give our warmest welcome to the team of Cirurgía Solidaria, and we are very much looking forward to cooperating with this great organization, for the betterment our our societies, and for those that need it most.

About GNU Health

The GNU Health project provides the tools for individuals, health professionals, institutions and governments to proactively assess and improve the underlying determinants of health, from the socioeconomic agents to the molecular basis of disease. From primary health care to precision medicine.

GNU Health is a Libre, community driven project from GNU Solidario, a non-profit humanitarian organization focused on Social Medicine. Our project has been adopted by public and private health institutions and laboratories, multilateral organizations and national public health systems around the world.

The GNU Health project provides the tools for individuals, health professionals, institutions and governments to proactively assess and improve the underlying determinants of health, from the socioeconomic agents to the molecular basis of disease. From primary health care to precision medicine.

The following are the main components that make up the GNU Health ecosystem:

  • Social Medicine and Public HealthHospital Management (HMIS)
  • Laboratory Management (Occhiolino)
  • Personal Health Record (MyGNUHealth)
  • Bioinformatics and Medical Genetics
  • Thalamus and Federated health networks
  • GNU Health embedded on Single Board devices

GNU Health is a GNU (www.gnu.org) official package, awarded with the Free Software Foundation award of Social benefit, among others. GNU Health has been adopted by many hospitals, governments and multilateral organizations around the globe.

See also:

GNU Health : https://www.gnuhealth.org

GNU Solidario : https://www.gnusolidario.org

Digital Public Good Alliance: https://digitalpublicgoods.net/

Original post : https://my.gnusolidario.org/2022/08/09/cirugia-solidaria-chooses-gnu-health/

09 August, 2022 05:22PM by Luis Falcon

August 08, 2022

a2ps @ Savannah

a2ps 4.14.91 released [alpha]

This alpha release marks the return of GNU a2ps to the Translation Project.

Some other minor issues have also been fixed.


Here are the compressed sources and a GPG detached signature:
  https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.91.tar.gz
  https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.91.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

36c2514304132eb2eb8921252145ced28f209182  a2ps-4.14.91.tar.gz
1LQ+pPTsYhMbt09CdSMrTaMP55VIi0MP7oaa+zDvRG0  a2ps-4.14.91.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.14.91.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.14.91.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.69
  Automake 1.16.1
  Gnulib v0.1-5347-gc0c72120f0

NEWS

* Noteworthy changes in release 4.14.91 (2022-08-08) [alpha]
 * Build:
   - Re-add a2ps to the Translation Project, and remove po files from git.
 * Bug fixes:
   - Remove reference to @COM_distill@ variable in a2ps_cfg.in.
 * Documentation:
   - Format --help output consistently to 80 columns.
   - Fix a couple of message typos.

08 August, 2022 07:47PM by Reuben Thomas

August 07, 2022

Andy Wingo

coarse or lazy?

sweeping, coarse and lazy

One of the things that had perplexed me about the Immix collector was how to effectively defragment the heap via evacuation while keeping just 2-3% of space as free blocks for an evacuation reserve. The original Immix paper states:

To evacuate the object, the collector uses the same allocator as the mutator, continuing allocation right where the mutator left off. Once it exhausts any unused recyclable blocks, it uses any completely free blocks. By default, immix sets aside a small number of free blocks that it never returns to the global allocator and only ever uses for evacuating. This headroom eases defragmentation and is counted against immix's overall heap budget. By default immix reserves 2.5% of the heap as compaction headroom, but [...] is fairly insensitive to values ranging between 1 and 3%.

To Immix, a "recyclable" block is partially full: it contains surviving data from a previous collection, but also some holes in which to allocate. But when would you have recyclable blocks at evacuation-time? Evacuation occurs as part of collection. Collection usually occurs when there's no more memory in which to allocate. At that point any recyclable block would have been allocated into already, and won't become recyclable again until the next trace of the heap identifies the block's surviving data. Of course after the next trace they could become "empty", if no object survives, or "full", if all lines have survivor objects.

In general, after a full allocation cycle, you don't know much about the heap. If you could easily know where the live data and the holes were, a garbage collector's job would be much easier :) Any algorithm that starts from the assumption that you know where the holes are can't be used before a heap trace. So, I was not sure what the Immix paper is meaning here about allocating into recyclable blocks.

Thinking on it again, I realized that Immix might trigger collection early sometimes, before it has exhausted the previous cycle's set of blocks in which to allocate. As we discussed earlier, there is a case in which you might want to trigger an early compaction: when a large object allocator runs out of blocks to decommission from the immix space. And if one evacuating collection didn't yield enough free blocks, you might trigger the next one early, reserving some recyclable and empty blocks as evacuation targets.

when do you know what you know: lazy and eager

Consider a basic question, such as "how many bytes in the heap are used by live objects". In general you don't know! Indeed you often never know precisely. For example, concurrent collectors often have some amount of "floating garbage" which is unreachable data but which survives across a collection. And of course you don't know the difference between floating garbage and precious data: if you did, you would have collected the garbage.

Even the idea of "when" is tricky in systems that allow parallel mutator threads. Unless the program has a total ordering of mutations of the object graph, there's no one timeline with respect to which you can measure the heap. Still, Immix is a stop-the-world collector, and since such collectors synchronously trace the heap while mutators are stopped, these are times when you can exactly compute properties about the heap.

Let's retake the question of measuring live bytes. For an evacuating semi-space, knowing the number of live bytes after a collection is trivial: all survivors are packed into to-space. But for a mark-sweep space, you would have to compute this information. You could compute it at mark-time, while tracing the graph, but doing so takes time, which means delaying the time at which mutators can start again.

Alternately, for a mark-sweep collector, you can compute free bytes at sweep-time. This is the phase in which you go through the whole heap and return any space that wasn't marked in the last collection to the allocator, allowing it to be used for fresh allocations. This is the point in the garbage collection cycle in which you can answer questions such as "what is the set of recyclable blocks": you know what is garbage and you know what is not.

Though you could sweep during the stop-the-world pause, you don't have to; sweeping only touches dead objects, so it is correct to allow mutators to continue and then sweep as the mutators run. There are two general strategies: spawn a thread that sweeps as fast as it can (concurrent sweeping), or make mutators sweep as needed, just before they allocate (lazy sweeping). But this introduces a lag between when you know and what you know—your count of total live heap bytes describes a time in the past, not the present, because mutators have moved on since then.

For most collectors with a sweep phase, deciding between eager (during the stop-the-world phase) and deferred (concurrent or lazy) sweeping is very easy. You don't immediately need the information that sweeping allows you to compute; it's quite sufficient to wait until the next cycle. Moving work out of the stop-the-world phase is a win for mutator responsiveness (latency). Usually people implement lazy sweeping, as it is naturally incremental with the mutator, naturally parallel for parallel mutators, and any sweeping overhead due to cache misses can be mitigated by immediately using swept space for allocation. The case for concurrent sweeping is less clear to me, but if you have cores that would otherwise be idle, sure.

eager coarse sweeping

Immix is interesting in that it chooses to sweep eagerly, during the stop-the-world phase. Instead of sweeping irregularly-sized objects, however, it sweeps over its "line mark" array: one byte for each 128-byte "line" in the mark space. For 32 kB blocks, there will be 256 bytes per block, and line mark bytes in each 4 MB slab of the heap are packed contiguously. Therefore you get relatively good locality, but this just mitigates a cost that other collectors don't have to pay. So what does eager marking over these coarse 128-byte regions buy Immix?

Firstly, eager sweeping buys you eager identification of empty blocks. If your large object space needs to steal blocks from the mark space, but the mark space doesn't have enough empties, it can just trigger collection and then it knows if enough blocks are available. If no blocks are available, you can grow the heap or signal out-of-memory. If the lospace (large object space) runs out of blocks before the mark space has used all recyclable blocks, that's no problem: evacuation can move the survivors of fragmented blocks into these recyclable blocks, which have also already been identified by the eager coarse sweep.

Without eager empty block identification, if the lospace runs out of blocks, firstly you don't know how many empty blocks the mark space has. Sweeping is a kind of wavefront that moves through the whole heap; empty blocks behind the wavefront will be identified, but those ahead of the wavefront will not. Such a lospace allocation would then have to either wait for a concurrent sweeper to advance, or perform some lazy sweeping work. The expected latency of a lospace allocation would thus be higher, without eager identification of empty blocks.

Secondly, eager sweeping might reduce allocation overhead for mutators. If allocation just has to identify holes and not compute information or decide on what to do with a block, maybe it go brr? Not sure.

lines, lines, lines

The original Immix paper also notes a relative insensitivity of the collector to line size: 64 or 256 bytes could have worked just as well. This was a somewhat surprising result to me but I think I didn't appreciate all the roles that lines play in Immix.

Obviously line size affect the worst-case fragmentation, though this is mitigated by evacuation (which evacuates objects, not lines). This I got from the paper. In this case, smaller lines are better.

Line size affects allocation-time overhead for mutators, though which way I don't know: scanning for holes will be easier with fewer lines in a block, but smaller lines would contain more free space and thus result in fewer collections. I can only imagine though that with smaller line sizes, average hole size would decrease and thus medium-sized allocations would be harder to service. Something of a wash, perhaps.

However if we ask ourselves the thought experiment, why not just have 16-byte lines? How crazy would that be? I think the impediment to having such a precise line size would mainly be Immix's eager sweep, as a fine-grained traversal of the heap would process much more data and incur possibly-unacceptable pause time overheads. But, in such a design you would do away with some other downsides of coarse-grained lines: a side table of mark bytes would make the line mark table redundant, and you eliminate much possible "dark matter" hidden by internal fragmentation in lines. You'd need to defer sweeping. But then you lose eager identification of empty blocks, and perhaps also the ability to evacuate into recyclable blocks. What would such a system look like?

Readers that have gotten this far will be pleased to hear that I have made some investigations in this area. But, this post is already long, so let's revisit this in another dispatch. Until then, happy allocations in all regions.

07 August, 2022 09:44AM by Andy Wingo

August 06, 2022

Christine Lemmer-Webber

A Guile Steel smelting pot

Last month I made a blogpost titled Guile Steel: A Proposal for a Systems Lisp. It got more attention than I anticipated, which is both a blessing and and curse. I mean, mostly the former, the curse isn't so serious, it's mostly that the post was aimed at a specific community and got more coverage than that, and funny things happen when things leave their intended context.

The blessing is that real, actual progress has happened, in terms of organization, actual development (thanks to others mostly!), and a compilation of potential directions. In many ways "Guile Steel" was meant to be a meta project, somewhat biased around Guile but more so a clever name to start brewing some ideas (and gathering intelligence) around, a call-to-arms for those who are likeminded, a test even to see if there are enough likeminded people out there. The answer to that one is: yes, and there's actually a lot that's happening or has happened historically. I actually think Lisp is going through a quiet renaissance and is on the verge or a major revival, but that's a topic for another post. The goal of this post is to give a lay of the landscape, as I've seen it since then. There's a lot out there.

If you enjoy this post by the way, there's an IRC channel: #guile-steel on irc.libera.chat. It's surprisingly well populated given that people have only shown up through word of mouth.

First, an aside on language (again)

Also by-the-way, it's debatable what "systems language" even means, and the previous post spent some time debating that. Language is mostly fuzzy, and subject to the constant battle between fuzzy and crisp systems, and "systems language" is something people evoke to make themselves sound like very crispy people, even though the term could hardly be fuzzier.

We're embracing the hand-waviness here; I've previously mentioned that "Blockchain" is to "Bitcoin" what "Roguelike" is to "Rogue". Similarly, "Systems Language" is to "C/Rust" what "Roguelike" is to "Rogue".

My friend Technomancy put things about as well or as succinctly as you could: "low-level enough to bootstrap a runtime". We'll extend "runtime" to not only mean "programming language runtime" but also "operating system runtime", and that's the scope of this post.

With that, let's start diving into directions.

Carp and Scopes

I was unaware at the time of writing the previous post of two remarkable "systems lisps" that already existed, are here, now, today, and you can and maybe should use them: Carp and Scopes. Both of them are statically typed, and both perform automatic memory management without the overhead of a garbage collector or reference counting in a style familiar to Rust.

They are also both kind of similar yet different. Carp is written on top of Haskell, looks a lot like Clojure in style. Scopes is written in C++, looks a lot like Scheme, and has an optional, non-parenthetical whitespace syntax which reminds me a lot of Wisp, so is maybe more amenable to the type of people who fear parentheses or must work with people who do.

I can't make a judgement about either; I would like to find some time to try each of them. Scopes looks more up my alley of the two. If someone packaged either of these languages for Guix I would try it in a heartbeat.

Anyway, Carp and Scopes already are systems lisps of sorts you can try today. (If you've made something cool with either of them, let me know.)

Pre-Scheme

There's a lot to say on this one, despite its obscurity, enough that I'm going to give it several sub-headings. I'll get the big one up front: Andrew Whatson is doing an incredible job porting Pre-Scheme to Guile. But I'll get more to that below.

What the heck is Pre-Scheme anyway?

PreScheme (or is it Pre-Scheme or prescheme or what? nobody is consistent, and I won't be either) is a "systems lisp" that is used to bootstrap the incredible but "sleeper hit" (or shall we say "cult classic"?) of programming language runtimes, Scheme48. PreScheme compiles to C, is statically typed with type inference based on a modified version of Hindley-Milner, and uses manual memory management for heap-allocated resources (much like C) rather than garbage collection. (C is just the current main target, compiling directly to native architectures or WebAssembly is also possible.)

The wild things about PreScheme are that unlike C or Rust, you can hack on it live at the REPL just like Scheme, and you can even incorporate a certain amount of Scheme, and it mostly looks like Scheme. But it still compiles down efficiently to low-level code.

It's used to implement Scheme48's virtual machine and garbage collector, and is bootstrappable from a working Scheme48, but there's also apparently a version sitting around somewhere on top of some metacircular Scheme which Jonathan Rees wrote on top of Common Lisp, giving it a good bootstrapping story. While used for Scheme48, and usable from it today, there's no reason you can't use it for other things, and a few smaller projects have.

What's more wild about PreScheme is how incredibly good of an idea it is, how long it's sat around (since the 80s, with a lot of work happening in the 90s!), and how little attention it's gotten. PreScheme's thoughtful design actually follows from Richard Kelsey's amazing PhD dissertation, Compilation By Program Transformation, which really feels like the kind of obscure CS thing that, if you've made it this far in this writeup, you probably would love reading. (Thank you to Olin Shivers for reviving this dissertation in LaTeX, which otherwise would have been lost to history.)

guile-prescheme

Now I did mention prescheme and how I thought that was a fairly interesting starting point on the last Guile Steel blogpost, and I actually got several people reaching out to me saying they wanted to take up this initiative, and a few of them suggested maybe they should start porting PreScheme to Guile, and I said "yes you should!" to all of them, but one person took up the initiative quickly and has been doing a straight and faithful port to Guile named guile-prescheme.

The emulator (which isn't too much code really) has already worked for a couple of weeks (which means you can already hack PreScheme at Guile's REPL, and Andrew Whatson says that the "compile to C" compiler is already well on its way, and will likely be there in about a month.

The main challenge apparently is the conversion of macros, which are stuck in the r4rs era of Scheme. Andrew has been slowly converting everything to syntax-case. syntax-case is encoded in r6rs and even more appealingly r7rs-small, which begs the question: how general of a port is this? Does it really have to just be to Guile? And that brings us to our next subsection...

The Secret Society of PreScheme Revivalists

Okay there's not really a secret society, we just have an email thread going and I organized a video call recently, and we're likely to do another one (I hope). This one was really good, very productive. (We didn't record it, sadly. Maybe we should have.)

On said video call we got Andrew Whatson of course, who's doing the current porting effort, but also Richard Kelsey (the original brain behind PreScheme, co-author of much of Scheme48), Michael Sperber (current maintainer of Scheme48 and also someone who has used PreScheme previously commercially, to do some monte carlo simulation things for some financial firm or something curious like that), and Jonathan Rees (co-author of Scheme48, and one of my friends who I like to call up and talk about all sorts of curious topics). There were a few others, all cool people, and also me, hand-waving excitedly as usual.

As an aside, my wife Morgan says my superpower is that I'm good at "showing up in a room and being excited and getting everyone else excited", and she's right, I think. And... well there's just a lot of interesting stuff in computer science history, amazing people whose work has just been mostly ignored, stuff left on the shelf. It's not what the Tech Influencers (TM) are currently touting, it's not what a FAANG company is going to currently hire you to do, but if you're trying to solve the hard problems people don't even realize they have, your best bet is to scour history.

I don't know if it's true or not but this felt like one of those times where the folks who have worked on PreScheme historically seemed kind of surprised that here we had a gathering of people who are extremely interested in the stuff they've done, but also happy about it. Anyway, that seemed like my reading, I like to think so anyway. Andrew (I think?) said some nice things about how it was just exciting to be able to talk to the people who have done these things, and I agree. It is cool stuff. We are grateful to be able to talk about it.

The conversation was really nice, we got some interesting historical information (some of that which I've conveyed here), and Richard Kelsey indicated that he's been doing work on microcontrollers and wishes he could be using PreScheme, but the things clients/employers get nervous about is "will be able to actually hire anyone to work on this stuff who isn't just you?" I'd like to think that we're building up enough enthusiasm where we can demonstrate in the affirmative, but that's going to take some time.

Anyway, I hinted in the last part that some of the more interesting conversation came to, just how portable is this port? Andrew indicated that he thought that the port to Guile as he was doing it was already helping to make things more portable. Andrew is just focusing on Guile first, but is avoiding the Guile-specific ice-9 namespace of Guile modules (which in this case, from a standardization perspective, becomes a little bit too appropriate) and is using as much generic Scheme and SRFI extensions as possible. Once the Guile version gets working, the goal is then to try porting to a more standardized form of Scheme (probably r7rs-small), and then that would mean that any Scheme following that standard could use the same version of PreScheme. Michael Sperber seemed to indicate that maybe Scheme48 could use this version too.

This would actually be pretty incredible because it would mean that any version of Scheme following the Scheme standard would suddenly have access to PreScheme, and any of those could also be used to bootstrap a PreScheme based Scheme.

A PreScheme JIT?

I thought Andrew Whatson (flatwhatson here) said this well enough himself so I'm just going to quote it verbatim:

<flatwhatson> re: pre-scheme interest for bootstrapping, i think it's more
              interesting than just "compiling to C"
<flatwhatson> Michael Sperber's rejected paper "A Tractable Native-Code Scheme
              System" describes repurposing the pre-scheme compiler (more
              accurately called the transformational compiler) as a jit
              byte-code optimizer and native-code emitter
<flatwhatson> the prescheme compiler basically lowers prescheme code to a
              virtual machine-code and then emits that as C
<flatwhatson> i think it would be feasible to directly emit native code at
              that point instead
<flatwhatson> https://www.deinprogramm.de/sperber/papers/tractable-native-code-scheme-system.pdf
<flatwhatson> Kelsey's dissertation describes transforming a high-level
              language to a low-level language, not specifically scheme to C.
<flatwhatson> > The machine language is an assembly language written in the
              syntax of the intermediate language and has a much simpler
              semantics. The machine is assumed to be a Von Neumann machine
              with a store and register-to-register instructions. Identifiers
              represent the machine’s registers and primitive procedures are
              the machine’s instructions.
<flatwhatson> Also, we have code for the unreleased byte-code jit-compiling
              native-emitting version of Scheme 48:
              https://www.s48.org/cgi-bin/hgwebdir.cgi/s48-compiler/

(How the hell that paper was rejected btw, I have no idea. It's great.)

Future directions for PreScheme

One obvious improvement to PreScheme is: compile to WebAssembly (aka WASM)! This would be pretty neat and maybe, maybe, maybe could mean a good path to getting more Schemes in the browser without using Emscripten (which is a bit heavy-handed of an approach). Andrew and I both think this is a fun idea, worth exploring. I think once the "compile to C" part of the port to Guile is done, it's worth beginning to look at in earnest.

Relatedly, it would also, I think, be pretty neat if guile-prescheme was compelling enough for more of Guile to be rewritten in it. This would improve Guile's already-better-than-most bootstrapping story and also make hacking on certain parts of Guile's internals more accessible and pleasant to a larger part of Guile's existing userbase.

The other obvious improvement to PreScheme is exploring (handwave handwave handwave) the kinds of automated memory management which have become popular with Rust's borrow checker and also appear in Carp and Scopes, as discussed above.

3L: The Computing System of the Future (???)

I mentioned that an appealing use of PreScheme might be to write not just a language runtime, but also an operating system. A very interesting project called 3L exists and is real and does just that. In fact, it's also a capability-secure operating system, and it cites all the right stuff and has all the right ideas going for it. And it's using PreScheme!

Now the problem is, seemingly nobody I know who would be interested in exactly a project like this even had heard of it before (except for the incredible and incredibly nice hacker pukkamustard is the one who made me even aware of it by mentioning it in the #guile-steel chatroom), and I couldn't even find the actual code on the main webpage. But there actually is source code, not a lot of it, but it's there, and in a way "not a lot of it" is not a bad thing here, because what's there looks stunningly similar to a very familiar metacircular evaluator, which begs the question, is that really enough, though?

And actually maybe it is, because hey look there's a demo video and a nice talk. And it's using Scheme48!

(As a complete aside: I'd be much more likely to play with Scheme48 if someone Geiser support for it... that's something I've poked at doing every now and then but I haven't had enough of a dedicated block of time. If you, dear reader, feel inspired enough to add such support, or actually if you give 3L a try, let me know).

Anyway, cool stuff, I've been meaning to reach out to the author, maybe I will after I post this. I wonder what's come of it. (It's also missing a license file or any indicators, but maybe we could get that fixed easily enough?)

WebAssembly

I had a call with someone recently who said WebAssembly was really only useful for C/Rust users, and I thought this was fairly surprising/confusing, but maybe that's because I think WebAssembly is pretty cool and have hand-coded a small amount of it for fun. Its text-based syntax is S-Expression based which makes it appealing for lispy type folks, and just easy to parse and work with in general.

It's stretching it a bit to call WebAssembly a Lisp, it's really just something that's designed to be an intermediate language (eg in GCC), a place where compiler authors often deign it okay/acceptable to use s-expressions because they don't fear that they'll scare off non-lispers or PLT people, because hey most users aren't going to touch this stuff anyway, right?

I dunno, I consider it a win at least that s-expressions have survived here. I showed up to an in-person WebAssembly meeting once and talked to one of the developers about it, praised them for this choice, and they said "Oh... yeah, well, we initially did it because it was the easiest thing to start with, and then eventually we came to like it, which I guess is the usual thing that happens with S-Expressions." (Literally true, look up the history of M-Expressions vs S-Expressions.)

At any rate, most people aren't coding WebAssembly by hand. However, you could, and if you're going to, a Lisp based environment is actually a really good choice. wasm-adventure is a really cool little demo game (try it!), all hand-written in WebAssembly kinda sorta. The README gives its motivations as "Is it possible (and enjoyable) to write a game directly in web assembly's text format? Eventually, would it be cool to generate wat from Scheme code using the Racket lang system?", and the answer becomes an emphatic "yes". What's interesting is that Lisp's venerable quasiquote does much of the heavy lifting to make, without too much work, a little DSL for authoring WebAssembly which results in some surprisingly easy to read code compared to generic WebAssembly. (The author, Zoé Martin, is another one of those quiet geniuses you run into on the internet; she has a lovely homebrew computer design too.)

So what I'd really like is to see more languages compiling directly to WebAssembly without emscripten as an intermediate hack. Guile especially, of course. Andy Wingo gave an amazing little talk on this where he does a little (quasi, pre-recorded) live coding demo of compiling to WebAssembly and I thought "YES!!! Compiling to WASM is probably right around the corner" and it turns out that's probably not the case because Wingo would like to see some important extensions to WASM land, and I guess, yes that probably makes sense, and also he's working on a new garbage collector which seems damn cool and like it'll be really good for Guile and maybe even could help the compiling to WASM story even before the much desired WASM-GC extension we all want lands, but who knows. I mean it would also be nice to have like, the tail call elimination extension, etc etc etc. But see also Wingo's writeup about targeting the web from last year, etc etc. (And on that note, I mean, is Webassembly the new Kubernetes?)

As another aside, there are two interesting Schemes which are actually written in WebAssembly, or rather, one written directly in hand-coded WASM named scheme.wasm, and one which compiles itself to Webassembly called Schism (which has a cool paper, but sadly hasn't been updated in a couple of years).

As another another aside, I was on a video call with Douglas Crockford at one point and mentioned WebAssembly and how cool I thought it was, and Crock kinda went "meh" to it, and I was like what? I mean it has ocap people you and I have both collaborated on it with, overall its design seems pretty good, better than most of the things of its ilk that have been tried before, why are you meh'ing WebAssembly? And Crock said that well, it's not that WebAssembly is bad, it's just that it felt like an opportunity to do something impactful, and it's "just another von neumann architecture", like, boring, can't we do better? But when I asked for specific alternatives, Crock didn't have a specific design in mind, just thought that maybe we could do better, maybe it could even incorporate actors at a more fundamental level.

Well... it turns out we both know someone who did just that, and (so I hear) both recently got nerdsniped by that very same person who had just such an architecture...

Mycelia and uFork

So I had a call with sorta-colleague, sorta-friend I guess? I'm talking about Dale Schumacher, and I don't know him super well, we don't get to talk that much, but I've enjoyed the amount we have. Dale has been trying to get me to have a video call for a while, we finally did, and I was expecting us to talk about our respective actor'y system projects, and we did... but the big surprise was hearing about Mycelia, Dale's ocap-secure hybrid-actor-model-lisp-machine-lambda-calculus operating system, and its equally astounding, actually maybe more astounding, virtual machine and maybe potentially CPU architecture design, uFork. We're going to take a major digression but I promise that it ties back in.

This isn't the first time Dale's reached out and it's resulted in me being surprised and nerdsniped. A few years ago Dale reached out to me to talk about this programming language he wrote called Humus. What's astounding personally about Humus is that it has an eerie amount of similarity to Spritely Goblins, the ocap distributed object architecture I've been working on the last few years, despite that we fully designed our systems independently. Dale beat me to it, but it was an independent reinvention in the sense that I simply wasn't aware of Humus until Dale started emailing me.

The eerie similarity is because I think Dale and I's systems are the most seriously true-to-form implementations of the "Classic Actor Model" that have been implemented in recent times (more true than say, Erlang, which does some other things, and "Classic" thrown on there because Carl Hewitt has some new ideas that he feels strongly should now be associated with "Actor Model" that can be layered on Dale and I's systems, but are not there at the base layer). (Actually, Goblins supports one other thing that makes it more the vat model of computation, but that isn't important for this post.) The Classic Actor Model says (hand-waving past pre-determinism in the general case, at least from the perspective of a single actor, due to ordering concerns... but those too can be layered on) that you can do pretty much all computation in terms of just actors, which are these funky distributed objects which handle messages one at a time, and while handling them are only able to do some combination of three things: (1) send messages to actors they know about, (2) create new actors (and get their address in the process, which they can share with other actors should they choose... argument passing, basically), and (3) designate their behavior for the next time they are handling a message. It's pretty common to use "become" for last that operation, but the curious thing that both Dale and I did was use lambdas as the thing you become. (By the way, those familiar with Scheme history should notice something interesting and familiar here, and for that same reason Dale and I are also in the shared company of being scolded by Carl Hewitt for saying our actors are made out of lambdas, despite him liking our systems otherwise, I think...)

I remarked off-hand that "well I guess one of the main differences between our systems, and maybe a thing you might not like, is that mine is lispy / based on Scheme, and..."

Dale waved his hand. "That's mostly surface..."

"Surface syntax, yeah I know. So I guess it doesn't..."

"No wait it does matter. What I'm trying to show you is that I actually do like that kind of stuff. In fact I have some projects which use it at a fundamental level. Here... let me show you..." And that's when we started talking about uFork, the instruction architecture he was working on, which I later found was actually part of a larger thing called Mycelia.

Well I'm glad I got the lecture directly from Dale because, let's see, how does the Mycelia project brand itself (at the time of writing)? "A bare-metal actor operating system for Raspberry Pi." Well, maybe this underwhelming self-description is why seemingly nobody I know (yes, like 3L above) has seemingly heard about it, despite it being extremely up the alley of the kind of programming people I tend to hang out with.

Mycelia is not just some throwaway raspberry pi project (which is certainly the impression I would have gotten from lazily scanning that page). Most of those are like, some cute repackaging of some existing FOSS POSIX'y thing. But Mycelia is an actually-working, you-can-actually-boot-it-on-real-hardware open source operating system (under Apache v2) with a ton of novel ideas which happens to be targeting the Raspberry Pi, but it could be ported to run on anything.

Anyway, there are a lot of interesting stuff in there, but here's a bulleted list summary. For Mycelia:

  • It's is an object-capability-secure operating system
  • It has a Lisp-like language for coding, pretty much Scheme-like, to hack around on
  • The Kernel language / Vau calculus show up, which is... wild
  • It encodes the Actor model and the Lambda calculus in a way that is sensible and coherent afaict
  • It is a "Lisp Machine" in many senses of the term.

But the uFork virtual machine / abstract idea for a CPU also are curious on their own. I dunno, I spent the other night reading it kind of wildly after our call. It also encodes the lambda calculus / actor model in fundamental instructions.

Dale was telling me he'd like to build an actual, physical CPU, but of course that takes a lot of resources, so he might settle for an FPGA for now. The architecture, should it be built, also somehow encodes a hardware garbage collector, which I haven't heard of anything doing that since the actual physical Lisp Machines died out.

At any rate, Dale was really excited to tell me about why his system encoded instructions operating on memory split in quads. He asked me why I thought that would be; I'm not honestly sharp enough in this kind of area to know, sadly, though I said "I hope it's not because you're planning on encoding RDF at the CPU layer". Thankfully it's not that, but then he started mentioning how his system encodes a stream of continuations...

Wait, that sounds familiar. "Have you ever heard of something called sectorlisp?" I asked, with a raised eyebrow.

"Scroll to the bottom of the document," Dale said, grinning.

Oh, there it was. Of course.

sectorlisp

The most technically impressive thing I think I've ever seen is John McCarthy's "Lisp implemented in Lisp", also known as a "metacircular evaluator". If you aren't familiar with it, it's been summarized well in the talk The Most Beautiful Program Ever Written by William Byrd. I think the best way to understand it really, and (I'm biased) the easiest to read version of things is in the Scheme in Scheme section of A Scheme Primer (though I wrote that for my work, and as said, I'm biased... I don't think I did anything new there, just explained ideas as simply as I could).

The second most technically impressive thing I've ever seen is sectorlisp, and the two are directly related. According to its README, "sectorlisp is a 512-byte implementation of LISP that's able to bootstrap John McCarthy's meta-circular evaluator on bare metal." Where traditional metacircular evaluator examples can be misconstrued as being the stuff of pure abstractlandia, sectorlisp gets brutally direct about things. In one sector (half a kilobyte!!!), sectorlisp manages to encode a whole-ass lisp system that actually runs. And yet, the nature of the metacircular evaluator persists. (Take that, person on Hacker News who called metacircular evaluators "cheating"! Even if you think mine was, I don't think you can accuse sectorlisp's of that.)

If you do nothing else, watch the sectorlisp blinkenlights demo, even just as an astounding visual demo alone. (Blinkenlights is another project of Justine's, and also wildly impressive.) I highly recommend the following blogposts of Justine's: SectorLISP Now Fits in One Sector, Lisp with GC in 436 bytes, and especially Lambda Calculus in 383 Bytes. Hikaru Ikuta (woodrush) has also written some amazing blogposts, including Extending SectorLISP to Implement BASIC REPLs and Games and Building a Neural Network in Pure Lisp Without Built-In Numbers Using Only Atoms and Lists (and related, but not part of sectorlisp: A Lisp Interpreter Implemented in Conway's Game of Life, which gave off strong Wireworld Computer vibes for me). If you are only going to lazily scan through one of those blogposts, I recommend it be Lambda Calculus in 383 Bytes, which has some wild representations of the ideas (including visually), a bit too advanced for me at present admittedly, though I stare at them in wonder.

I had a bunch more stuff here, partly because the author is someone I find both impressive technically but who has also said some fairly controversial things... to say the least. But I think it was too much of a digression for this article. The short version is that Justine's stuff is probably the smartest, most mind-blowing tech I've ever seen, kinda scarily and intimidatingly smart, and it's hard to mentally reconcile that with some of those statements. I don't know, maybe she wants to move past that phase, I'd like to think so. I think she hasn't said anything like that in a long time, and it feels out of phase with the rest of this post but... it feels like something that needs to be acknowledged.

GOOL, GOAL, and OpenGOAL

Every now and then when people say Lisp couldn't possibly be performant, Lisp people like to bring up that Naughty Dog famously had its own Lisp implementations for most of its earlier games. Andy Gavin has written about GOOL, which was a mix of lisp and assembly and of course lisp generating assembly, and I don't think much was written about its followup GOAL until OpenGOAL came up, which... I haven't looked at it too much tbh. I guess it's getting interesting for some people for the ability to play Jak and Daxter on modern hardware (which I've never played but looked fun), but I'm more curious if someone's gonna poke at it to do something completely different.

But I do know some of the vague legends. I don't remember if this is true or where I read it but one of them that Sony accused Naughty Dog of accessing some devkit or APIs they weren't supposed to have access to because they were pulling off a bunch of features performantly in Crash Bandicoot that were assumed otherwise not possible. But nope, just lisp nerds making DSLs that pull off some crazy shit I guess.

BONUS: Shoutout to Kandria

Oh, speaking of games written in Lisp, I guess Kandria's engine is gonna be FOSS, and that game looks wicked cool, maybe support their Kickstarter while you still can. It's not really in the theme of this post from the definition of "systems lisp" I gave earlier, but this blogpost about its lispy internals is really neat.

Okay here's everything else we're done

This was a long-ass post. There's other things maybe that could be discussed, so I'll just dump briefly:

Okay, that's it. Hopefully you found something interesting out of this post. Meanwhile, I was just gonna spend an hour on this. I spent my whole day! Eeek!

06 August, 2022 06:35PM by Christine Lemmer-Webber (cwebber@dustycloud.org)

August 04, 2022

GNUnet News

GNUnet 0.17.3

GNUnet 0.17.3

This is a bugfix release for gnunet 0.17.2. In addition to the fixed in the source, the documentation websites including the handbook have been updated and consolidated: https://docs.gnunet.org .

Notably, the GNUnet project now publishes a GNS zone for its websites which can be used to test resolution on any installation. For example:

$ gnunet-gns -t ANY -u www.gnunet.org

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

Noteworthy changes in 0.17.3 (since 0.17.2)

  • DHT : Various bugfixes in the protocol.
  • TRANSPORT : Fix HTTPS tests. #7257
  • DOCUMENTATION :
    • Migrate from texinfo to sphinx.
    • Dropped dependency on texinfo.
    • Added dependency on sphinx.

A detailed list of changes can be found in the ChangeLog and the bugtracker .

04 August, 2022 10:00PM

August 02, 2022

libc @ Savannah

The GNU C Library version 2.36 is now available

The GNU C Library
=================

The GNU C Library version 2.36 is now available.

The GNU C Library is used as the C library in the GNU system and
in GNU/Linux systems, as well as many other systems that use Linux
as the kernel.

The GNU C Library is primarily designed to be a portable
and high performance C library.  It follows all relevant
standards including ISO C11 and POSIX.1-2017.  It is also
internationalized and has one of the most complete
internationalization interfaces known.

The GNU C Library webpage is at http://www.gnu.org/software/libc/

Packages for the 2.36 release may be downloaded from:
        http://ftpmirror.gnu.org/libc/
        http://ftp.gnu.org/gnu/libc/

The mirror list is at http://www.gnu.org/order/ftp.html

NEWS for version 2.36
=====================

Major new features:

  • Support for DT_RELR relative relocation format has been added to

  glibc.  This is a new ELF dynamic tag that improves the size of
  relative relocations in shared object files and position independent
  executables (PIE).  DT_RELR generation requires linker support for
  -z pack-relative-relocs option, which is supported for some targets
  in recent binutils versions.  Lazy binding doesn't apply to DT_RELR.

  • On Linux, the pidfd_open, pidfd_getfd, and pidfd_send_signal functions

  have been added.  The pidfd functionality provides access to a process
  while avoiding the issue of PID reuse on tranditional Unix systems.

  • On Linux, the process_madvise function has been added. It has the

  same functionality as madvise but alters the target process identified
  by the pidfd.

  • On Linux, the process_mrelease function has been added.  It allows a

  caller to release the memory of a dying process.  The release of the
  memory is carried out in the context of the caller, using the caller's
  CPU affinity, and priority with CPU usage accounted to the caller.

  • The “no-aaaa” DNS stub resolver option has been added.  System

  administrators can use it to suppress AAAA queries made by the stub
  resolver, including AAAA lookups triggered by NSS-based interfaces
  such as getaddrinfo.  Only DNS lookups are affected: IPv6 data in
  /etc/hosts is still used, getaddrinfo with AI_PASSIVE will still
  produce IPv6 addresses, and configured IPv6 name servers are still
  used.  To produce correct Name Error (NXDOMAIN) results, AAAA queries
  are translated to A queries.  The new resolver option is intended
  primarily for diagnostic purposes, to rule out that AAAA DNS queries
  have adverse impact.  It is incompatible with EDNS0 usage and DNSSEC
  validation by applications.

  • On Linux, the fsopen, fsmount, move_mount, fsconfig, fspick, open_tree,

  and mount_setattr have been added.  They are part of the new Linux kernel
  mount APIs that allow applications to more flexibly configure and operate
  on filesystem mounts.  The new mount APIs are specifically designed to work
  with namespaces.

  • localedef now accepts locale definition files encoded in UTF-8.

  Previously, input bytes not within the ASCII range resulted in
  unpredictable output.

  • Support for the mbrtoc8 and c8rtomb multibyte/UTF-8 character conversion

  functions has been added per the ISO C2X N2653 and C++20 P0482R6 proposals.
  Support for the char8_t typedef has been added per the ISO C2X N2653
  proposal.  The functions are declared in uchar.h in C2X mode or when the
  _GNU_SOURCE macro or C++20 __cpp_char8_t feature test macro is defined.
  The char8_t typedef is declared in uchar.h in C2X mode or when the
  _GNU_SOURCE macro is defined and the C++20 __cpp_char8_t feature test macro
  is not defined (if __cpp_char8_t is defined, then char8_t is a builtin type).

  • The functions arc4random, arc4random_buf, and arc4random_uniform have been

  added.  The functions wrap getrandom and/or /dev/urandom to return high-
  quality randomness from the kernel.

  • Support for LoongArch running on Linux has been added.  This port requires

  as least binutils 2.38, GCC 12, and Linux 5.19.  Currently only hard-float
  ABI is supported:

    - loongarch64-linux-gnu

  The LoongArch ABI is 64-bit little-endian.

Deprecated and removed features, and other changes affecting compatibility:

  • Support for prelink will be removed in the next release; this includes

  removal of the LD_TRACE_PRELINKING, and LD_USE_LOAD_BIAS, environment
  variables and their functionality in the dynamic loader.

  • The Linux kernel version check has been removed along with the

  LD_ASSUME_KERNEL environment variable.  The minimum kernel used to built
  glibc is still provided through NT_GNU_ABI_TAG ELF note and also printed
  when libc.so is issued directly.

  • On Linux, The LD_LIBRARY_VERSION environment variable has been removed.

The following bugs are resolved with this release:

  [14932] dynamic-link: dlsym(handle, "foo") and dlsym(RTLD_NEXT, "foo")
    return different result with versioned "foo"
  [16355] libc: syslog.h's SYSLOG_NAMES namespace violation and utter
    mess
  [23293] dynamic-link: aarch64: getauxval is broken when run as ld.so
    ./exe and ld.so adjusts argv on the stack
  [24595] nptl: [2.28 Regression]: Deadlock in atfork handler which
    calls dlclose
  [25744] locale: mbrtowc with Big5-HKSCS returns 2 instead of 1 when
    consuming the second byte of certain double byte characters
  [25812] stdio: Libio vtable protection is sometimes only partially
    enforced
  [27054] libc: pthread_atfork handlers that call pthread_atfork
    deadlock
  [27924] dynamic-link: ld.so: Support DT_RELR relative relocation
    format
  [28128] build: declare_symbol_alias doesn't work for assembly codes
  [28566] network: getnameinfo with NI_NOFQDN is not thread safe
  [28752] nss: Segfault in getpwuid when stat fails
  [28815] libc: realpath should not copy to resolved buffer on error
  [28828] stdio: fputwc crashes
  [28838] libc: FAIL: elf/tst-p_align3
  [28845] locale: ld-monetary.c should be updated to match ISO C and
    other standards.
  [28850] libc: linux: __get_nprocs_sched reads uninitialized memory
    from the stack
  [28852] libc: getaddrinfo leaks memory with AI_ALL
  [28853] libc: tst-spawn6 changes current foreground process group
    (breaks test isolation)
  [28857] libc: FAIL: elf/tst-audit24a
  [28860] build: --enable-kernel=5.1.0 build fails because of missing
    __convert_scm_timestamps
  [28865] libc: linux: _SC_NPROCESSORS_CONF and _SC_NPROCESSORS_ONLN are
    inaccurate without /sys and /proc
  [28868] dynamic-link: Dynamic loader DFS algorithm segfaults on
    missing libraries
  [28880] libc: Program crashes if date beyone 2038
  [28883] libc: sysdeps/unix/sysv/linux/select.c: __select64
    !__ASSUME_TIME64_SYSCALLS && !__ASSUME_PSELECT fails on Microblaze
  [28896] string: strncmp-avx2-rtm and wcsncmp-avx2-rtm fallback on non-
    rtm variants when avoiding overflow
  [28922] build: The .d dependency files aren't always generated
  [28931] libc: hosts lookup broken for SUCCESS=CONTINUE and
    SUCCESS=MERGE
  [28936] build: nm: No such file
  [28950] localedata: Add locale for ISO code "tok" (Toki Pona)
  [28953] nss: NSS lookup result can be incorrect if function lookup
    clobbers errno
  [28970] math: benchtest: libmvec benchmark doesn't build with make
    bench.
  [28991] libc: sysconf(_SC_NPROCESSORS_CONF) should read
    /sys/devices/system/cpu/possible
  [28993] libc: closefrom() iterates until max int if no access to
    /proc/self/fd/
  [28996] libc: realpath fails to copy partial result to resolved buffer
    on ENOENT and EACCES
  [29027] math: [ia64] fabs fails with sNAN input
  [29029] nptl: poll() spuriously returns EINTR during thread
    cancellation and with cancellation disabled
  [29030] string: GLIBC 2.35 regression - Fortify crash on certain valid
    uses of mbsrtowcs (*** buffer overflow detected ***: terminated)
  [29062] dynamic-link: Memory leak in _dl_find_object_update if object
    is promoted to global scope
  [29069] libc: fstatat64_time64_statx wrapper broken on MIPS N32 with
    -D_FILE_OFFSET_BITS=64 and -D_TIME_BITS=64
  [29071] dynamic-link: m68k: Removal of ELF_DURING_STARTUP optimization
    broke ld.so
  [29097] time: fchmodat does not handle 64 bit time_t for
    AT_SYMLINK_NOFOLLOW
  [29109] libc: posix_spawn() always returns 1 (EPERM) on clone()
    failure
  [29141] libc: _FORTIFY_SOURCE=3 fail for gcc 12/glibc 2.35
  [29162] string: [PATCH] string.h syntactic error:
    include/bits/string_fortified.h:110: error: expected ',' or ';'
    before '__fortified_attr_access'
  [29165] libc: [Regression] broken argv adjustment
  [29187] dynamic-link: [regression] broken argv adjustment for nios2
  [29193] math: sincos produces a different output than sin/cos
  [29197] string: __strncpy_power9() uses uninitialised register vs18
    value for filling after \0
  [29203] libc: daemon is not y2038 aware
  [29204] libc: getusershell is not 2038 aware
  [29207] libc: posix_fallocate fallback implementation is not y2038
    aware
  [29208] libc: fpathconf(_PC_ASYNC_IO) is not y2038 aware
  [29209] libc: isfdtype is not y2038 aware
  [29210] network: ruserpass is not y2038 aware
  [29211] libc: __open_catalog is not y2038 aware
  [29213] libc: gconv_parseconfdir is not y2038 aware
  [29214] nptl: pthread_setcanceltype fails to set type
  [29225] network: Mistyped define statement in socket/sys/socket.h in
    line 184
  [29274] nptl: __read_chk is not a cancellation point
  [29279] libc: undefined reference to `mbstowcs_chk' after
    464d189b9622932a75302290625de84931656ec0
  [29304] libc: mq_timedreceive does not handle 64 bit syscall return
    correct for !__ASSUME_TIME64_SYSCALLS
  [29403] libc: st_atim, st_mtim, st_ctim stat struct members are
    missing on microblaze with largefile

Release Notes
=============

https://sourceware.org/glibc/wiki/Release/2.36

Contributors
============

This release was made possible by the contributions of many people.
The maintainers are grateful to everyone who has contributed
changes or bug reports.  These include:

=Joshua Kinard
Adhemerval Zanella
Adhemerval Zanella Netto
Alan Modra
Andreas Schwab
Arjun Shankar
Arnout Vandecappelle (Essensium/Mind)
Carlos O'Donell
Cristian Rodríguez
DJ Delorie
Danila Kutenin
Darius Rad
Dmitriy Fedchenko
Dmitry V. Levin
Emil Soleyman-Zomalan
Fangrui Song
Florian Weimer
Gleb Fotengauer-Malinovskiy
Guilherme Janczak
H.J. Lu
Ilyahoo Proshel
Jason A. Donenfeld
Joan Bruguera
John David Anglin
Jonathan Wakely
Joseph Myers
José Bollo
Kito Cheng
Maciej W. Rozycki
Mark Wielaard
Matheus Castanho
Max Gautier
Michael Hudson-Doyle
Nicholas Guriev
Noah Goldstein
Paul E. Murphy
Raghuveer Devulapalli
Ricardo Bittencourt
Sam James
Samuel Thibault
Sergei Trofimovich
Siddhesh Poyarekar
Stafford Horne
Stefan Liebler
Steve Grubb
Su Lifan
Sunil K Pandey
Szabolcs Nagy
Tejas Belagod
Tom Coldrick
Tom Honermann
Tulio Magno Quites Machado Filho
WANG Xuerui
Wangyang Guo
Wilco Dijkstra
Xi Ruoyao
Xiaoming Ni
Yang Yanchao
caiyinyu

02 August, 2022 01:17AM by Carlos O'Donell

July 31, 2022

gcide @ Savannah

GCIDE version 0.53

GCIDE version 0.53 is available for download.

31 July, 2022 07:13AM by Sergey Poznyakoff

July 27, 2022

FSF News

July 25, 2022

poke @ Savannah

GNU poke 2.4 released

I am happy to announce a new release of GNU poke, version 2.4.

This is a bugfix release in the poke 2.x series.

See the file NEWS in the distribution tarball for a list of issues fixed in this release.

The tarball poke-2.4.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-2.4.tar.gz.

> GNU poke (http://www.jemarch.net/poke) is an interactive, extensible editor for binary data.  Not limited to editing basic entities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them.


Happy poking!

--
Jose E. Marchesi
Frankfurt am Main
25 July 2022

25 July, 2022 02:21PM by Jose E. Marchesi

July 24, 2022

parallel @ Savannah

GNU Parallel 20220722 ('Roe vs Wade') released

GNU Parallel 20220722 ('Roe vs Wade') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  The syntax for GNU Parallel is so slick that I often use it just to make my script read nicer, and the parallelism is a cherry on top.
    -- Epistaxis@reddit

New in this release:

  • --colour-failed will color output red if the job fails.
  • sql: support for InfluxDB.
  • Polarhome.com is dead, so these OSs are no longer supported: AIX HPUX IRIX Minix OPENSTEP OpenIndiana OpenServer QNX Solaris Syllable Tru64 Ultrix UnixWare.
  • Bug fixes and man page updates.

News about GNU Parallel:

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

24 July, 2022 07:25AM by Ole Tange

July 23, 2022

datamash @ Savannah

GNU Datamash 1.8 released

This is to announce datamash-1.8, a new release.

Datamash is a command-line program which performs basic numeric, textual and
statistical operations on input textual data.


This is the first release for new maintainer Tim Rice, with much appreciation
to Shawn Wagner and Erik Auerswald for their help. See the AUTHORS and THANKS
files for additional credits and acknowledgements.


GNU Datamash home page:
   https://www.gnu.org/software/datamash/

Please report any problem you may experience to the bug-datamash@gnu.org
mailing list.

Happy Hacking!
- Tim Rice

==================================================================

Here are the compressed sources and a GPG detached signature[*]:
https://ftp.gnu.org/gnu/datamash/datamash-1.8.tar.gz
https://ftp.gnu.org/gnu/datamash/datamash-1.8.tar.gz.sig

Use a mirror for higher download bandwidth:
https://ftpmirror.gnu.org/datamash/datamash-1.8.tar.gz
https://ftpmirror.gnu.org/datamash/datamash-1.8.tar.gz.sig

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  For instructions about how to do this, please
refer to https://ftp.gnu.org/README.  (In particular you will need to
retrieve the GNU keyring rather than using any keyservers.)

==================================================================

The checksums of the archive are:

$ sha1sum datamash-1.8.tar.gz
e77e15ed2c6b17b4045251fd87f16430c3bf2166  datamash-1.8.tar.gz

$ sha256sum datamash-1.8.tar.gz
94a4e11819ad259aa3745b7eca392e385e3a676d276e8cbb616269dbbb17fe6d  datamash-1.8.tar.gz

$ b2sum datamash-1.8.tar.gz
dfe4060ea65ea46a1796e01463fd9b0e55c2d633d06da153f585a3a569acf3e9211a14cb3905daf8ecae347358daa04db940d557b909f0ce5ebbba2f57d3a410  datamash-1.8.tar.gz

==================================================================

NEWS

  • Noteworthy changes in release 1.8 (2022-07-23) [stable]
    • Changes in Behavior

  Schedule -f/--full combined with non-linewise operations for deprecation.
  In a future release, -f/--full will only be usable with operations where
  it makes sense. For now, we print a warning to stderr when -f/--full is
  used with non-linewise operations, and such usage will no longer be
  supported.

  The bin operation now uses more intuitive bins. Previously, a command
  such as `datamash bin 1 <<< -0` would output -100; and -100 did not fall
  in its own bin. We now require all bins to take the form `[nx,(n+1)x)`
  with integer n and bin width x. We discard the sign on -0 and gate such
  inputs into the [0,x) bin.

  Operations taking more than one argument now provide more complete output
  with --header-out. Previously, an operation such as `pcov x:y` would
  produce an output header like `pcov(y)`, discarding the `x`. The new
  behavior will output header `pcov(x,y)`.

  datamash(1) no longer ignores --output-delimiter with the rmdup operation.

    • New Features

  New datamash option --sort-cmd argument to specify the program used
  by the -s option to sort input, plus enhancements to the security and
  portability of building sort command lines.

  New datamash option -c/--collapse-delimiter=X argument uses character
  X instead of comma between values in collapse and unique lists.

  New datamash operations: mean square (ms) and root mean square (rms).

  Decorate now supports sorting IP addresses of both versions 4 and 6
  together. IPv4 addresses are logically converted to IPv6 addresses,
  either as IPv4-Mapped (ipv6v4map) or IPv4-Compatible (ipv6v4comp)
  addresses.

  Add two command aliases:
    'echo' may now be used instead of 'cut'.
    'uniq' may now be used instead of 'unique'.

    • Improvements

  Updated the bash completion script to reflect recent additions.

    • Bug Fixes

  Datamash now passes the -z/--zero-terminated flag to the sort(1) child
  process when used with "--sort --zero-terminated". Additionally,
  if the system's sort(1) does not support -z, datamash reports the error
  and exits. Previously it would omit the "-z" when running sort(1),
  resulting in incorrect results.

  Documentation fixes and spelling corrections.

  Incorrect format in a decorate(1) error breaking compilation on some
  systems.

  datamash(1), decorate(1): Fix some minor memory leaks.

  datamash(1) no longer crashes when the unique or countunique operations
  are used with input data containing NUL bytes.  The problem was reported
  in https://lists.gnu.org/archive/html/bug-datamash/2020-11/msg00001.html
  by Catalin Patulea.

  datamash(1) no longer crashes when crosstab with --header-in is called
  by field name instead of index. I.e. `datamash --header-in ct x,y` now
  works as expected.

23 July, 2022 03:21AM by Tim Rice

July 21, 2022

gnuastro @ Savannah

Gnuastro 0.18 released

The 18th release of GNU Astronomy Utilities (Gnuastro) is now available. See the full announcement for all the new features in this release and the many bugs that have been found and fixed: https://lists.gnu.org/archive/html/info-gnuastro/2022-07/msg00001.html

21 July, 2022 11:17PM by Mohammad Akhlaghi

librejs @ Savannah

LibreJS 7.21.0 released

GNU LibreJS aims to address the JavaScript problem described in Richard
Stallman's article The JavaScript Trap. LibreJS is a free add-on for GNU IceCat and other Mozilla-based browsers. It blocks nonfree nontrivial JavaScript while allowing JavaScript that is free and/or trivial.

The user manual pages are at
<https://www.gnu.org/software/librejs/manual/>.

Source tarballs and signed xpis are available at
<https://ftp.gnu.org/gnu/librejs/>.

The binary can also be found at the Firefox Browser Addon website:
<https://addons.mozilla.org/en-US/firefox/addon/librejs/>.

GPG key ID: EF86DFD0
Fingerprint: 47F9 D050 1E11 8879 9040  4941 2126 7E93 EF86 DFD0

See also
<https://savannah.gnu.org/project/memberlist-gpgkeys.php?group=librejs>.

User-visible changes since 7.20.3

New in 7.21.0

21 July, 2022 01:52PM by Yuchen Pei

July 20, 2022

Andy Wingo

unintentional concurrency

Good evening, gentle hackfolk. Last time we talked about heuristics for when you might want to compact a heap. Compacting garbage collection is nice and tidy and appeals to our orderly instincts, and it enables heap shrinking and reallocation of pages to large object spaces and it can reduce fragmentation: all very good things. But evacuation is more expensive than just marking objects in place, and so a production garbage collector will usually just mark objects in place, and only compact or evacuate when needed.

Today's post is more details!

dedication

Just because it's been, oh, a couple decades, I would like to reintroduce a term I learned from Marnanel years ago on advogato, a nerdy group blog kind of a site. As I recall, there is a word that originates in the Oxbridge social environment, "narg", from "Not A Real Gentleman", and which therefore denotes things that not-real-gentlemen do: nerd out about anything that's not, like, fox-hunting or golf; or generally spending time on something not because it will advance you in conventional hierarchies but because you just can't help it, because you love it, because it is just your thing. Anyway, in the spirit of pursuits that are really not What One Does With One's Time, this post is dedicated to the word "nargery".

side note, bis: immix-style evacuation versus mark-compact

In my last post I described Immix-style evacuation, and noted that it might take a few cycles to fully compact the heap, and that it has a few pathologies: the heap might never reach full compaction, and that Immix might run out of free blocks in which to evacuate.

With these disadvantages, why bother? Why not just do a single mark-compact pass and be done? I implicitly asked this question last time but didn't really answer it.

For some people will be, yep, yebo, mark-compact is the right answer. And yet, there are a few reasons that one might choose to evacuate a fraction of the heap instead of compacting it all at once.

The first reason is object pinning. Mark-compact systems assume that all objects can be moved; you can't usefully relax this assumption. Most algorithms "slide" objects down to lower addresses, squeezing out the holes, and therefore every live object's address needs to be available to use when sliding down other objects with higher addresses. And yet, it would be nice sometimes to prevent an object from being moved. This is the case, for example, when you grant a foreign interface (e.g. a C function) access to a buffer: if garbage collection happens while in that foreign interface, it would be nice to be able to prevent garbage collection from moving the object out from under the C function's feet.

Another reason to want to pin an object is because of conservative root-finding. Guile currently uses the Boehm-Demers-Weiser collector, which conservatively scans the stack and data segments for anything that looks like a pointer to the heap. The garbage collector can't update such a global root in response to compaction, because you can't be sure that a given word is a pointer and not just an integer with an inconvenient value. In short, objects referenced by conservative roots need to be pinned. I would like to support precise roots at some point but part of my interest in Immix is to allow Guile to move to a better GC algorithm, without necessarily requiring precise enumeration of GC roots. Optimistic partial evacuation allows for the possibility that any given evacuation might fail, which makes it appropriate for conservative root-finding.

Finally, as moving objects has a cost, it's reasonable to want to only incur that cost for the part of the heap that needs it. In any given heap, there will likely be some data that stays live across a series of collections, and which, once compacted, can't be profitably moved for many cycles. Focussing evacuation on only the part of the heap with the lowest survival rates avoids wasting time on copies that don't result in additional compaction.

(I should admit one thing: sliding mark-compact compaction preserves allocation order, whereas evacuation does not. The memory layout of sliding compaction is more optimal than evacuation.)

multi-cycle evacuation

Say a mutator runs out of memory, and therefore invokes the collector. The collector decides for whatever reason that we should evacuate at least part of the heap instead of marking in place. How much of the heap can we evacuate? The answer depends primarily on how many free blocks you have reserved for evacuation. These are known-empty blocks that haven't been allocated into by the last cycle. If you don't have any, you can't evacuate! So probably you should keep some around, even when performing in-place collections. The Immix papers suggest 2% and that works for me too.

Then you evacuate some blocks. Hopefully the result is that after this collection cycle, you have more free blocks. But you haven't compacted the heap, at least probably not on the first try: not into 2% of total space. Therefore you tell the mutator to put any empty blocks it finds as a result of lazy sweeping during the next cycle onto the evacuation target list, and then the next cycle you have more blocks to evacuate into, and more and more and so on until after some number of cycles you fall below some overall heap fragmentation low-watermark target, at which point you can switch back to marking in place.

I don't know how this works in practice! In my test setups which triggers compaction at 10% fragmentation and continues until it drops below 5%, it's rare that it takes more than 3 cycles of evacuation until the heap drops to effectively 0% fragmentation. Of course I had to introduce fragmented allocation patterns into the microbenchmarks to even cause evacuation to happen at all. I look forward to some day soon testing with real applications.

concurrency

Just as a terminological note, in the world of garbage collectors, "parallel" refers to multiple threads being used by a garbage collector. Parallelism within a collector is essentially an implementation detail; when the world is stopped for collection, the mutator (the user program) generally doesn't care if the collector uses 1 thread or 15. On the other hand, "concurrent" means the collector and the mutator running at the same time.

Different parts of the collector can be concurrent with the mutator: for example, sweeping, marking, or evacuation. Concurrent sweeping is just a detail, because it just visits dead objects. Concurrent marking is interesting, because it can significantly reduce stop-the-world pauses by performing most of the computation while the mutator is running. It's tricky, as you might imagine; the collector traverses the object graph while the mutator is, you know, mutating it. But there are standard techniques to make this work. Concurrent evacuation is a nightmare. It's not that you can't implement it; you can. But it's very very hard to get an overall performance win from concurrent evacuation/copying.

So if you are looking for a good bargain in the marketplace of garbage collector algorithms, it would seem that you need to avoid concurrent copying/evacuation. It's an expensive product that would seem to not buy you very much.

All that is just a prelude to an observation that there is a funny source of concurrency even in some systems that don't see themselves as concurrent: mutator threads marking their own roots. To recall, when you stop the world for a garbage collection, all mutator threads have to somehow notice the request to stop, reach a safepoint, and then stop. Then the collector traces the roots from all mutators and everything they reference, transitively. Then you let the threads go again. Thing is, once you get more than a thread or four, stopping threads can take time. You'd be tempted to just have threads notice that they need to stop, then traverse their own stacks at their own safepoint to find their roots, then stop. But, this introduces concurrency between root-tracing and other mutators that might not have seen the request to stop. For marking, this concurrency can be fine: you are just setting mark bits, not mutating the roots. You might need to add an additional mark pattern that can be distinguished from marked-last-time and marked-the-time-before-but-dead-now, but that's a detail. Fine.

But if you instead start an evacuating collection, the gates of hell open wide and toothy maws and horns fill your vision. One thread could be stopping and evacuating the objects referenced by its roots, while another hasn't noticed the request to stop and is happily using the same objects: chaos! You are trying to make a minor optimization to move some work out of the stop-the-world phase but instead everything falls apart.

Anyway, this whole article was really to get here and note that you can't do ragged-stops with evacuation without supporting full concurrent evacuation. Otherwise, you need to postpone root traversal until all threads are stopped. Perhaps this is another argument that evacuation is expensive, relative to marking in place. In practice I haven't seen the ragged-stop effect making so much of a difference, but perhaps that is because evacuation is infrequent in my test cases.

Zokay? Zokay. Welp, this evening's nargery was indeed nargy. Happy hacking to all collectors out there, and until next time.

20 July, 2022 09:26PM by Andy Wingo

Parabola GNU/Linux-libre

elogind requires manual intervention

As part of the init scripts repackaging, elogind does no longer ship with its OpenRC init script. You have to manually install it when upgrading:

# pacman -Syu
# pacman -S elogind-openrc

20 July, 2022 03:29AM by David P.

July 19, 2022

[nonsystemd] NetworkManager, dbus and display managers require manual intervention

We have recently began a repackaging of [nonsystemd] packages (see #3290). The displaymanager-openrc package has been removed and specific init scripts have been added for their respective display manager (e.g. sddm-openrc for sddm, gdm-openrc for gdm and so on with lxdm, xdm and lightdm)

Regarding NetworkManager and dbus, their nonsystemd builds used to ship with their corresponding OpenRC init scripts, but now they were separated into networkmanager-openrc and dbus-openrc. Please install these when upgrading those packages.

19 July, 2022 03:06AM by David P.

July 18, 2022

health @ Savannah

GNU Health project to implement REUSE compliance

The REUSE initiative[1] is a Free Software Foundation Europe program that facilitates the documentation of licenses of Libre projects like GNU Health.

After several meetings with our friends from FSFE, we have decided to implement REUSE in all GNUHealth components, that is:

  • Hospital Management System
  • MyGNUHealth Personal Health Record
  • Thalamus
  • GH Federation Portal

We believe that for large projects like GNUHealth, with multiple files of different kinds (code, graphics, data, ..) REUSE will be a great companion.

1.- https://reuse.software/

18 July, 2022 08:15AM by Luis Falcon

July 17, 2022

a2ps @ Savannah

a2ps 4.14.90 released [alpha]

I’m delighted to announce a new alpha release of GNU a2ps. This release involves minimal changes to functionality, but involves a considerable update to the build system and code cleanup and simplification, as well as bug fixes. See below for more details.

I am particularly keen to hear from users and packagers about this release, as I plan to make a stable 4.15 release soon.

* Noteworthy changes in release 4.14.90 (2022-07-17) [alpha]
 * This is an alpha release, owing to the considerable changes since the
   last version.
 * New maintainer, Reuben Thomas.
 * Build:
   - Updated and fixed the build system, using gnulib and modern Autotools.
   - Remove OS/2 support.
   - Require libpaper.
 * Predefined delegations:
   - Remove support for defunct Netscape and proprietary Acrobat Reader.
   - Add lpr wrapper for automatic detection of different printing systems,
     including CUPS support.
 * Encodings:
   - Use libre fonts for KOI-8.
   - Composite fonts support.
 * Documentation:
   - Some English fixes.
   - Man page for fixnt.
 * Bug fixes:
   - Fixes for security bugs CVE-2001-1593, CVE-2015-8107 and CVE-2014-0466.
   - Minor bugs fixed.

17 July, 2022 07:19PM by Reuben Thomas

July 16, 2022

rush @ Savannah

GNU rush version 2.3

GNU rush version 2.3 is available for download.  This is a bug-fixing release.

16 July, 2022 05:20PM by Sergey Poznyakoff

July 14, 2022

Parabola GNU/Linux-libre

[From Arch]: wxgtk2 may require manual intervention

if pacman gives an error message like:

error: failed to prepare transaction (could not satisfy dependencies) :: removing wxgtk-common breaks dependency 'wxgtk-common' required by wxgtk2

you will need to uninstall 'wxgtk2' and it's dependents first (the only such parabola packages are 'freefilesync' and 'odamex')

# pacman -Rc wxgtk2

14 July, 2022 10:08PM by bill auger

Simon Josefsson

Towards pluggable GSS-API modules

GSS-API is a standardized framework that is used by applications to, primarily, support Kerberos V5 authentication. GSS-API is standardized by IETF and supported by protocols like SSH, SMTP, IMAP and HTTP, and implemented by software projects such as OpenSSH, Exim, Dovecot and Apache httpd (via mod_auth_gssapi). The implementations of Kerberos V5 and GSS-API that are packaged for common GNU/Linux distributions, such as Debian, include MIT Kerberos, Heimdal and (less popular) GNU Shishi/GSS.

When an application or library is packaged for a GNU/Linux distribution, a choice is made which GSS-API library to link with. I believe this leads to two problematic consequences: 1) it is difficult for end-users to chose between Kerberos implementation, and 2) dependency bloat for non-Kerberos users. Let’s discuss these separately.

  1. No system admin or end-user choice over the GSS-API/Kerberos implementation used

    There are differences in the bug/feature set of MIT Kerberos and that of Heimdal’s, and definitely that of GNU Shishi. This can lead to a situation where an application (say, Curl) is linked to MIT Kerberos, and someone discovers a Kerberos related problem that would have been working if Heimdal was used, or vice versa. Sometimes it is possible to locally rebuild a package using another set of dependencies. However doing so has a high maintenance cost to track security fixes in future releases. It is an unsatisfying solution for the distribution to flip flop between which library to link to, depending on which users complain the most. To resolve this, a package could be built in two variants: one for MIT Kerberos and one for Heimdal. Both can be shipped. This can help solve the problem, but the question of which variant to install by default leads to similar concerns, and will also eventually leads to dependency conflicts. Consider an application linked to libraries (possible in several steps) where one library only supports MIT Kerberos and one library only supports Heimdal.

    The fact remains that there will continue to be multiple Kerberos implementations. Distributions will continue to support them, and will be faced with the dilemma of which one to link to by default. Distributions and the people who package software will have little guidance on which implementation to chose from their upstream, since most upstream support both implementations. The result is that system administrators and end-users are not given a simple way to have flexibility about which implementation to use.
  2. Dependency bloat for non-Kerberos use-cases.

    Compared to the number of users of GNU/Linux systems out there, the number of Kerberos users on GNU/Linux systems is smaller. Here distributions face another dilemma. Should they enable GSS-API for all applications, to satisfy the Kerberos community, or should they be conservative with adding dependencies to reduce attacker surface for the non-Kerberos users? This is a dilemma with no clear answer, and one approach has been to ship two versions of a package: one with Kerberos support and one without. Another option here is for upstream to support loadable modules, for example Dovecot implement this and Debian ship with a separate ‘dovecot-gssapi’ package that extend the core Dovecot seamlessly. Few except some larger projects appear to be willing to carry that maintenance cost upstream, so most only support build-time linking of the GSS-API library.

    There are a number of real-world situations to consider, but perhaps the easiest one to understand for most GNU/Linux users is OpenSSH. The SSH protocol supports Kerberos via GSS-API, and OpenSSH implement this feature, and most GNU/Linux distributions ship a SSH client and SSH server linked to a GSS-API library. Someone made the choice of linking it to a GSS-API library, for the arguable smaller set of people interested in it, and also the choice which library to link to. Rebuilding OpenSSH locally without Kerberos support comes with a high maintenance cost. Many people will not need or use the Kerberos features of the SSH client or SSH server, and having it enabled by default comes with a security cost. Having a vulnerability in OpenSSH is critical for many systems, and therefor its dependencies are a reasonable concern. Wouldn’t it be nice if OpenSSH was built in a way that didn’t force you to install MIT Kerberos or Heimdal? While still making it easy for Kerberos users to use it, of course.

Hopefully I have made the problem statement clear above, and that I managed to convince you that the state of affairs is in need of improving. I learned of the problems from my personal experience with maintaining GNU SASL in Debian, and for many years I ignored this problem.

Let me introduce Libgssglue!

Matryoshka Dolls
Matryoshka Dolls – photo CC-4.0-BY-NC by PngAll

Libgssglue is a library written by Kevin W. Coffman based on historical GSS-API code, the initial release was in 2004 (using the name libgssapi) and the last release was in 2012. Libgssglue provides a minimal GSS-API library and header file, so that any application can link to it instead of directly to MIT Kerberos or Heimdal (or GNU GSS). The administrator or end-user can select during run-time which GSS-API library to use, through a global /etc/gssapi_mech.conf file or even a local GSSAPI_MECH_CONF environment variable. Libgssglue is written in C, has no external dependencies, and is BSD-style licensed. It was developed for the CITI NFSv4 project but libgssglue ended up not being used.

I have added support to build GNU SASL with libgssglue — the changes required were only ./configure.ac-related since GSS-API is a standardized framework. I have written a fairly involved CI/CD check that builds GNU SASL with MIT Kerberos, Heimdal, libgssglue and GNU GSS, sets ups a local Kerberos KDC and verify successful GSS-API and GS2-KRB5 authentications. The ‘gsasl’ command line tool connects to a local example SMTP server, also based on GNU SASL (linked to all variants of GSS-API libraries), and to a system-installed Dovecot IMAP server that use the MIT Kerberos GSS-API library. This is on Debian but I expect it to be easily adaptable to other GNU/Linux distributions. The check triggered some (expected) Shishi/GSS-related missing features, and triggered one problem related to authorization identities that may be a bug in GNU SASL. However, testing shows that it is possible to link GNU SASL with libgssglue and have it be operational with any choice of GSS-API library that is shipped with Debian. See GitLab CI/CD code and its CI/CD output.

This experiment worked so well that I contacted Kevin to learn that he didn’t have any future plans for the project. I have adopted libgssglue and put up a Libgssglue GitLab project page, and pushed out a libgssglue 0.5 release fixing only some minor build-related issues. There are still some missing newly introduced GSS-API interfaces that could be added, but I haven’t been able to find any critical issues with it. Amazing that an untouched 10 year old project works so well!

My current next steps are:

  • Release GNU SASL with support for Libgssglue and encourage its use in documentation.
  • Make GNU SASL link to Libgssglue in Debian, to avoid a hard dependency on MIT Kerberos, but still allowing a default out-of-the-box Kerberos experience with GNU SASL.
  • Maintain libgssglue upstream and implement self-checks, CI/CD testing, new GSS-API interfaces that have been defined, and generally fix bugs and improve the project. Help appreciated!
  • Maintain the libgssglue package in Debian.
  • Look into if there are applications in Debian that link to a GSS-API library that could instead be linked to libgssglue to allow flexibility for the end-user and reduce dependency bloat.

What do you think? Happy Hacking!

14 July, 2022 09:28AM by simon

July 13, 2022

Christine Lemmer-Webber

The Beginning of A Grueling Diet

Today I made a couple of posts on the social medias, I will repost them in their entirety:

FUCK eating "DELICIOUS" food. I'm DONE eating delicious food.

What has delicious food ever done for me? Only made me want to eat more of it.

From now on I am only eating BORING-ASS GRUELS that I can eat as much as I want of which will not be much because they are BORING -- fediverse post birdsite post

And then:

I am making a commitment: I will be eating nothing but boring GRUELS until the end of 2022.

Hold me to it. fediverse post birdsite post

I am hereby committing to "a grueling diet" for the rest of 2022, as an experiment. Here are the rules:

  • Gruel, Pottage, and soup, in unlimited quantities. But gruel is preferred.
  • Fresh, steamed, pickled, and roasted fruit, vegetables, and tofu may be eaten in unlimited quantity. Unsweetened baking chocolate, cottage cheese are also permitted.
  • Tea, seltzer water, milk (any kind), and coffee are fine to have.
  • Pottage (including gruel) may be adorned as deemed appropriate, but not too luxuriously. Jams and molasses may be had for a nice breakfast or dessert, but not too much. Generally, only adorn pottages at mealtime; pottages in-between mealtime should be eaten with as sparing of additions as possible.
  • Not meaning to be rude to visitors, guests, hosts, and other such good company, exceptions to the diet may be made when visiting, being visited, or going on dates.

I will provide followups to this post throughout the year, justifying the diet, describing how it has affected my health, putting it in historical context, providing recipes, etc.

In the meanwhile, to the rest of 2022: may that it be grueling indeed!

Edit (2022-07-15): Added tofu to the list of acceptible things to additionally eat. Clarified gruel/pottage adornment advice. Added roasting as an acceptable processing method for fruit/vegetables/tofu.

13 July, 2022 12:30PM by Christine Lemmer-Webber (cwebber@dustycloud.org)

July 09, 2022

GNUnet News

GNUnet 0.17.2

GNUnet 0.17.2

This is a bugfix release for gnunet 0.17.1.

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

Noteworthy changes in 0.17.1 (since 0.17.2)

  • DHT : Various bugfixes in the protocol.
  • RECLAIM : OpenID Connect plugin improvements.
  • ABE : Removed.

A detailed list of changes can be found in the ChangeLog and the bugtracker .

09 July, 2022 10:00PM

Christine Lemmer-Webber

Guile Steel: a proposal for a systems lisp

Before we get into this kind of stream-of-consciousness outline, I'd like to note that very topically to this, over at the Spritely Institute (where I'm CTO, did I mention on here yet that I'm the CTO of a nonprofit to improve networked communication on the internet on this blog? because I don't think I did) we published a Scheme Primer, and the feedback to it has been just lovely. This post isn't a Spritely Institute thing (at least, not yet, though if its ideas manifested it could be possible we might use some of the tech), but since it's about Scheme, I thought I'd mention that.

This blogpost outlines something I've had kicking around in my head for a while: the desire for a modern "systems lisp", you know, kind of like Rust, except hopefully much better than Rust, and in Lisp. (And, if it turns out to be for not other reason, it might simply be better by being written in a Lisp.) But let's be clear: I haven't written anything, this blogpost is a ramble, it's just kind of a set of feelings about what I'd like, what I think is possible.

Let's open by saying that there's no real definition of what a "systems language" is... but more or less what people mean is, "something like C". In other words, what people nowadays consider a low-level language, even though C used to be considered a high level language. And what people really mean is: it's fast, it's statically typed, and it's really for the bit-fiddling types of speed demons out there.

Actually, let's put down a few asides for a moment. People have conflated two different benefits fo "statically typed" languages because they've mostly been seen together:

  • Static typing for ahead-of-time more-correct programs
  • Static typing for faster or leaner programs (which subdivides in terms of memory and CPU benefits, more or less)

In the recent FOSS & Crafts episode What is Lisp? we talk a bit about how the assumptions that dynamically typed languages are "slow" is really due to lack of hardware support, and that lisp machines actually had hardware support directly (tagged memory architecture and hardware garbage collection) and even wrote low-level parts of their systems like the "graphics drivers" directly in lisp, and it was plenty fast, and that it would even be possible to have co-processors on which dynamic code (not just lisp) ran at "native speed" (this is what the MacIvory did), but this is all somewhat of an aside because that's not the world we live in. So as much as I, Christine, would love to have tagged architecture (co-)processors, they probably won't happen, except there's some RISC-V tagged architecture things but I don't think they've gotten very far and they seem mostly motivated by a security model that doesn't make any sense to me. But I'd love to be wrong on this! I would like tagged RISC-V to succeed! But still, there's the problem of memory management, and I don't think anyone's been working on a hardware garbage collector or if that would really be a better thing anyway.

The fact is, there's been a reinforcing effect over the last several decades since the death of the lisp machine: CPUs are optimized for C, and C is optimized for CPUs, and both of them try to optimize for each other. So "systems programming" really means "something like C" because that's what our CPUs like because that's what our languages like and these are pretty much re-inforcing.

And besides, C is basically the lingua franca of programming languages, right? If you want to make something widely portable, you target the C ABI, because pretty much all programming languages have some sort of C FFI toolkit thing or just make C bindings, and everyone is happy. Except, oh wait, C doesn't actually have an ABI! Well, okay, I guess not, but it doesn't matter because the C ABI triples, that's what the world works with.

Well also, you gotta target the web, right? And actually the story there is a bit nicer because WebAssembly is actually kinda awesome, and the hope and dream is that all programming languages in some way or another target WebAssembly, and then "you gotta write your thing in Javascript because it's the language of the web!!!" is no longer a thing I have to hear anymore. (Yes, all my friends who work on Javascript, I appreciate you for making it the one programming language which has mostly gotten better over time... hopefully it stays that way, and best of luck.) But the point is, any interesting programming language these days should be targeting Webassembly, and hopefully not just via Emscripten, but hopefully via actually targeting Webassembly directly.

So okay, we have at least two targets for our "system language": C, or something that is C-compatible, and Webassembly. And static type analysis in terms of preventing errors, that's also a useful thing, I won't deny it. (I think the division of "statically typed" and "dynamically typed" languages is probably more of a false one than we tend to think, but that's a future blogpost, to be written.) And these days, it's also how you get speed while also being maximally bit-twiddly fast, because that's how our machines (including the abstract one in Webassembly) are designed. So okay, grumbling about conflating two things aside, let's run with that.

So anyway, I promised to write about this "Guile Steel" thing I've been musing about, and we've gotten this far in the article, and I haven't yet. So, this is, more than a concrete proposal, a call to arms to implement just such a systems language for Guile. I might make a prototype at some point, but you, dear reader, are free to take the idea of "Guile Steel" and run with it. In fact, please do.

So anyway. First, about the name. It's probably pretty obvious based on the name that I'm suggesting this be a language for Guile Scheme. And "Guile" as a name itself is both a continuation of the kind of playfully mischevious names in the Scheme family and its predecessors, but also a pun on co-founder of the Scheme language, Guy L. Steele. So "Guile Steele" kinda brings that pun home, and "Steel" sounds low-level, close to the metal.

But also, Guile has a lovely compiler tower. It would be nice to put some more lovely things on it! Why not a systems language?

There's some precedent here. The lovely Scheme 48's lowest levels of code (including its garbage collector) are written in an interesting language called PreScheme (more on PreScheme), which is something that's kind of like Scheme, but not really. It doesn't do automatic garbage collection itself, and I think Rust has shown that this area could be improved for a more modern PreScheme system. But you can hack on it at the REPL, and then it can compile to C, and it also has an implementation on Common Lisp, so you can bootstrap it a few different ways. PreScheme uses a Hindley-Milner type system; I suspect we can do even better with a propagator approach but that's untested. Anyway, starting by porting PreScheme from Scheme48 to Guile directly would be a good way to get going.

Guile also has some pretty good reasons to want something like this. For one thing, if you're a Guile person, then by gosh you're probably a Guix person. And Rust, it's real popular these days, and for good reasons, we're all better of with less memory vulnerabilities in our lives, but you know... it's kind of a pain, packaging wise, I hear? Actually I've never tried packaging anything in Rust but Efraim certainly has and when your presentation starts with the slide "Packaging Rust crates in GNU Guix: How hard could it possibly be?" I guess the answer is going to be that it's a bit of a headache. So maybe it's not the end of the world, but I think it might be nice if on that ground we had our own alternative, but that's just a minor thing.

And I don't think there's anything wrong with Rust, but I'd love to see... can we do better? I feel like it could be hackable, accessible, and it also could, probably, be a lot of fun? That's a good reason, I know I'd like something like this myself, I'd like to play with it, I'd like to be able to use it.

But maybe also... well, let's not beat around the bush, a whole lot of Guile is written in C, and our dear wonderful Andy Wingo has done a lot of lovely things to make us less dependent on C, some half-straps and some baseline compilers and just rewriting a lot of stuff in Scheme and so on and so forth but it would be nice if we had something we could officially rally around as "hey this is the thing we're going to start rewriting things in", because you know, C really is kind of a hard world to trust, and I'd like the programming language environment I rely on to not be so heavily built on it.

And at this point in the article, I have to say that Xerz! pointed out that there is a thing called Carp which is indeed a lisp that compiles to C and you know what, I'm pretty embarassed for having not paid attention to it... I certainly saw it linked at one point but didn't pay enough attention, and... maybe it needs a closer look. Heck, it's written in Haskell, which is a pretty cool choice.

But hey, the Guile community still deserves a thing of its own, right? What do we have that compiler tower for if we're not going to add some cool things to it? And... gosh, I'd really like to get Guile in the browser, and there are some various paths, and Wingo gave a fun presentation on compiling to Webassembly last year, but wouldn't it be nice if just our whole language stack was written in something designed to compile to either something C-like or... something?

I might do some weekend fiddling towards this direction, but sadly this can't be my main project. As a call to arms, maybe it inspires someone to take it up as theirs though. I will say that if you work on it, I promise to spend some time using whatever you build and trying it out and sending patches. So that's it, that's my stream-of-consciousness post on Guile Steel: currently an idea... maybe eventually a reality?

09 July, 2022 11:26AM by Christine Lemmer-Webber (cwebber@dustycloud.org)

July 05, 2022

Site converted to Haunt

Lo and behold, I've converted the last of the sites I've been managing for ages to Haunt.

Haunt isn't well known. Apparently I am responsible for, er, many of the sites listed on awesome.haunt.page. But you know what? I've been making website things for a long time, and Haunt is honestly the only static site generator I've worked with (and I've worked with quite a few) that's actually truly customizable and programmable and pleasant to work with. And hackable!

This site has seen quite a few iterations... some custom code when I first launched it some time ago, then I used Zine, then I used PyBlosxom, and for quite a few years everything was running on Pelican. But I never liked hacking on any of those... I always kind of begrudgingly opened up the codebase and regretted having to change anything. But Haunt? Haunt's a dream, it's all there and ready for you, and I've even gotten some patches upstream. (Actually I owe Dave a few more, heh.)

Everything is Scheme in Haunt, which means, for instance, that this page needed an archive page for ages that actually worked and was sensible and I just didn't ever feel like doing it. But in Haunt, it's just delicious Guile flavored Scheme:

(define (archive-tmpl site posts)
  ;; build a map of (year -> posts)
  (define posts-by-year
    (let ((ht (make-hash-table)))      ; hash table we're building up
      (do ((posts posts (cdr posts)))  ; iterate over all posts
          ((null? posts) ht)           ; until we're out of posts
        (let* ((post (car posts))                   ; put this post in year bucket
               (year (date-year (post-date post)))
               (year-entries (hash-ref ht year '())))
          (hash-set! ht year (cons post year-entries))))))
  ;; sort all the years
  (define sorted-years
    (sort (hash-map->list (lambda (k v) k) posts-by-year) >))
  ;; rendering for one year
  (define (year-content year)
    `(div (@ (style "margin-bottom: 10px;"))
          (h3 ,year)
          (ul ,@(map post-content
                     (posts/reverse-chronological
                      (hash-ref posts-by-year year))))))
  ;; rendering for one post within a year
  (define (post-content post)
    `(li
      (a (@ (href ,(post-uri site post)))
         ,(post-ref post 'title))))
  ;; the whole page
  (define content
    `(div (@ (class "entry"))
          (h2 "Blog archive (by year)")
          (ul ,@(map year-content sorted-years))))
  ;; render within base template
  (base-tmpl site content))

Lambda, the ultimate static site generator!

At any rate, I expect some things are broken, to be fixed, etc. Let me know if you see 'em. Heck, you can browse the site contents should you be so curious!

But is there really anything more boring than a meta "updated my website code" post like this? Anyway, in the meanwhile I've corrected straggling instances of my deadname which were sitting around. The last post I made was me coming out as trans, and... well a lot has changed since then. So I guess I've got some more things to write. And also this whole theme... well I like some of it but I threw it together when I was but a wee web developer, back before CSS was actually nice to write, etc. So maybe I need to overhaul the look and feel too. And I always meant to put in that project directory, and ooh maybe an art gallery, and so on and so on...

But hey, I like updating my website again! So maybe I actually will!

05 July, 2022 01:20PM by Christine Lemmer-Webber (cwebber@dustycloud.org)

July 02, 2022

pspp @ Savannah

PSPP 1.6.2 has been released.

I'm very pleased to announce the release of a new version of GNU PSPP.  PSPP is a program for statistical analysis of sampled data.  It is a free replacement for the proprietary program SPSS.

Changes from 1.6.1 to 1.6.2:

  • Bug fixes.

Please send PSPP bug reports to bug-gnu-pspp@gnu.org.

02 July, 2022 04:08AM by Ben Pfaff

June 27, 2022

ghm @ Savannah

GHM 2022 confirmed

The next GNU Hackers' Meeting will take place in İzmir, Turkey, in October 2022.  Please see the event web page at https://gnu.org/ghm/2022 .

27 June, 2022 07:12PM by Luca Saiu

June 25, 2022

GNU Taler news

June 24, 2022

pspp @ Savannah

PSPP 1.6.1 has been released.

I'm very pleased to announce the release of a new version of GNU PSPP.  PSPP is a program for statistical analysis of sampled data.  It is a free replacement for the proprietary program SPSS.

Changes from 1.6.0 to 1.6.1:

  • The SET command now supports LEADZERO for controlling output of a leading zero in F, COMMA, and DOT format.
  • Bug fixes and translation updates.

Please send PSPP bug reports to bug-gnu-pspp@gnu.org.

24 June, 2022 04:57PM by Ben Pfaff

June 23, 2022

parallel @ Savannah

GNU Parallel 20220622 ('Bongbong') released

GNU Parallel 20220622 ('Bongbong') has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/

Quote of the month:

  Parallel has been (and still is) super useful and simple tool for speeding up all kinds of shell tasks during my career.
    -- ValtteriL@ycombinator

New in this release:

  • , can be used in --sshlogin if quoted as \, or ,,
  • --plus {/#regexp/str} replace ^regexp with str.
  • --plus {/%regexp/str} replace regexp$ with str.
  • --plus {//regexp/str} replace every regexp with str.
  • 'make install' installs bash+zsh completion files.
  • Bug fixes and man page updates.

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

23 June, 2022 08:29AM by Ole Tange

June 22, 2022

www @ Savannah

New Article by Richard Stallman

The GNU Education Team has published a new article by Richard Stallman on the threats of Big Tech in the field of education.

Many Governments Encourage Schools to Let Companies Snoop on Students

Human Rights Watch studied 164 software programs and web sites  recommended by various governments for schools to make students use.  It found that 146 of them gave data to advertising and tracking companies.

The researchers were thorough and checked for various snooping methods, including fingerprinting of devices to identify users. The targets of the investigation were not limited to programs and sites specifically “for education;” they included, for instance, Zoom and Microsoft Teams.

I expect that each program collected personal data for its developer. I'm not sure whether the results counted that, but they should. Once the developer company gets personal data, it can provide that data to advertising profilers, as well as to other companies and governments, and it can engage directly in manipulation of students and teachers.

The recommendations Human Rights Watch makes follow the usual approach of regulating the use of data once collected.  This is fundamentally inadequate; personal data, once collected, will surely be misused.

The only approach that makes it possible to end massive surveillance starts with demanding that the software be free. Then users will be able to modify the software to avoid giving real data to companies.

More at gnu/education...

22 June, 2022 01:42PM by Dora Scilipoti