Planet GNU

Aggregation of development blogs from the GNU Project

January 31, 2023

health @ Savannah

GNU Health Federation community server updated

Dear community

The GNU Health federation community server (federation.gnuhealth.org) has been updated! These are some of the main improvements:

- NGINX and WSGI: The Hospital Management component instance is now running behind uWSGI and NGINX.

- HMIS 4.2 Release Candidate 2: We are very close to HMIS 4.2 stable. This pre-release version will help us to test the upcoming features and find bugs before the release. :)

- Secure connection: You can test the demo web client using TLS. Please check the demo database section on the GNUHealth Wikibooks.

- Thalamus, the GNU Health Federation message and authentication server still runs on Gunicorn, also using https.

More info and resources:

Happy and healthy hacking!
Luis

31 January, 2023 11:44PM by Luis Falcon

a2ps @ Savannah

a2ps 4.14.94 released [alpha]

Another alpha release, some more tweaks and tidy-ups.

Here are the compressed sources and a GPG detached signature:
  https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.94.tar.gz
  https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.94.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

1c99e0200ed0d93119ad6ab54a4735692dbb6d26  a2ps-4.14.94.tar.gz
3+mUXOzeILDgtP08dCJjPI2BL5px92ndCH27qjW1RPI  a2ps-4.14.94.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.14.94.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.14.94.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5639-g80b225fe1e

NEWS

* Noteworthy changes in release 4.14.94 (2023-01-31) [alpha]
 * Features:
   - Replace the 'psmandup' utility with simpler 'lp2' to directly print
     documents to a simplex printer.
   - Remove the outdated 'psset' and 'fixnt', and simplify 'fixps' to
     always process its input with Ghostscript.
 * Documentation
   - Remove some obsolete explanations.
 * Build
   - Minor tidy up and removal of obsolete code.

31 January, 2023 07:13PM by Reuben Thomas

January 28, 2023

Applied Pokology

Applied Pokology - Interesting poke idiom: sparse tables

During tonight poke online office hours our friend hdzki came with an interesting use case. He is poking at some binary structures that are like sparse tables whose entries are distributed in the file in an arbitrary way. Each sparse table is characterized by an array of consecutive non-NULL pointers. Each pointer points to an entry in the table. The table entries can be anywhere in the IO space, and are not necessarily consecutive, nor be in order.

Full text...

28 January, 2023 12:00AM

January 27, 2023

Andy Wingo

three approaches to heap sizing

How much memory should a program get? Tonight, a quick note on sizing for garbage-collected heaps. There are a few possible answers, depending on what your goals are for the system.

you: doctor science

Sometimes you build a system and you want to study it: to identify its principal components and see how they work together, or to isolate the effect of altering a single component. In that case, what you want is a fixed heap size. You run your program a few times and determine a heap size that is sufficient for your problem, and then in future run the program with that new fixed heap size. This allows you to concentrate on the other components of the system.

A good approach to choosing the fixed heap size for a program is to determine the minimum heap size a program can have by bisection, then multiplying that size by a constant factor. Garbage collection is a space/time tradeoff: the factor you choose represents a point on the space/time tradeoff curve. I would choose 1.5 in general, but this is arbitrary; I'd go more with 3 or even 5 if memory isn't scarce and I'm really optimizing for throughput.

Note that a fixed-size heap is not generally what you want. It's not good user experience for running ./foo at the command line, for example. The reason for this is that program memory use is usually a function of the program's input, and only in some cases do you know what the input might look like, and until you run the program you don't know what the exact effect of input on memory is. Still, if you have a team of operations people that knows what input patterns look like and has experience with a GC-using server-side process, fixed heap sizes could be a good solution there too.

you: average josé/fina

On the other end of the spectrum is the average user. You just want to run your program. The program should have the memory it needs! Not too much of course; that would be wasteful. Not too little either; I can tell you, my house is less than 100m², and I spend way too much time shuffling things from one surface to another. If I had more space I could avoid this wasted effort, and in a similar way, you don't want to be too stingy with a program's heap. Do the right thing!

Of course, you probably have multiple programs running on a system that are making similar heap sizing choices at the same time, and the relative needs and importances of these programs could change over time, for example as you switch tabs in a web browser, so the right thing really refers to overall system performance, whereas what you are controlling is just one process' heap size; what is the Right Thing, anyway?

My corner of the GC discourse agrees that something like the right solution was outlined by Kirisame, Shenoy, and Panchekha in a 2022 OOPSLA paper, in which the optimum heap size depends on the allocation rate and the gc cost for a process, which you measure on an ongoing basis. Interestingly, their formulation of heap size calculation can be made by each process without coordination, but results in a whole-system optimum.

There are some details but you can imagine some instinctive results: for example, when a program stops allocating because it's waiting for some external event like user input, it doesn't need so much memory, so it can start shrinking its heap. After all, it might be quite a while before the program has new input. If the program starts allocating again, perhaps because there is new input, it can grow its heap rapidly, and might then shrink again later. The mechanism by which this happens is pleasantly simple, and I salute (again!) the authors for identifying the practical benefits that an abstract model brings to the problem domain.

you: a damaged, suspicious individual

Hoo, friends-- I don't know. I've seen some things. Not to exaggerate, I like to think I'm a well-balanced sort of fellow, but there's some suspicion too, right? So when I imagine a background thread determining that my web server hasn't gotten so much action in the last 100ms and that really what it needs to be doing is shrinking its heap, kicking off additional work to mark-compact it or whatever, when the whole point of the virtual machine is to run that web server and not much else, only to have to probably give it more heap 50ms later, I-- well, again, I exaggerate. The MemBalancer paper has a heartbeat period of 1 Hz and a smoothing function for the heap size, but it just smells like danger. Do I need danger? I mean, maybe? Probably in most cases? But maybe it would be better to avoid danger if I can. Heap growth is usually both necessary and cheap when it happens, but shrinkage is never necessary and is sometimes expensive because you have to shuffle around data.

So, I think there is probably a case for a third mode: not fixed, not adaptive like the MemBalancer approach, but just growable: grow the heap when and if its size is less than a configurable multiplier (e.g. 1.5) of live data. Never shrink the heap. If you ever notice that a process is taking too much memory, manually kill it and start over, or whatever. Default to adaptive, of course, but when you start to troubleshoot a high GC overhead in a long-lived proess, perhaps switch to growable to see its effect.

unavoidable badness

There is some heuristic badness that one cannot avoid: even with the adaptive MemBalancer approach, you have to choose a point on the space/time tradeoff curve. Regardless of what you do, your system will grow a hairy nest of knobs and dials, and if your system is successful there will be a lively aftermarket industry of tuning articles: "Are you experiencing poor object transit? One knob you must know"; "Four knobs to heaven"; "It's raining knobs"; "GC engineers DO NOT want you to grab this knob!!"; etc. (I hope that my British readers are enjoying this.)

These ad-hoc heuristics are just part of the domain. What I want to say though is that having a general framework for how you approach heap sizing can limit knob profusion, and can help you organize what you have into a structure of sorts.

At least, this is what I tell myself; inshallah. Now I have told you too. Until next time, happy hacking!

27 January, 2023 09:45PM by Andy Wingo

January 26, 2023

GNU Taler news

GNU Taler v0.9.1 released

We are happy to announce the release of GNU Taler v0.9.1.

26 January, 2023 11:00PM

a2ps @ Savannah

a2ps 4.14.93 released [alpha]

I am happy to announce another pre-release of what will eventually be the
first release of GNU a2ps since 2007.

I have had very little feedback about previous pre-releases, so I intend to
make a stable release soon. If you’re interested in GNU a2ps, please try
this pre-release! I hope that once I make a full release it will quickly be
packaged for distributions.

Here are the compressed sources and a GPG detached signature:
  https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.93.tar.gz
  https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.93.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

8eb28d7a8ca933a08918d706f231978a91e42d3f  a2ps-4.14.93.tar.gz
VoCuvBKrC1y5P/wZbx92C6O28jvtCfs9ZnskCjx/xmM  a2ps-4.14.93.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.14.93.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.14.93.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5639-g80b225fe1e

NEWS

* Noteworthy changes in release 4.14.93 (2023-01-26) [alpha]
 * Features:
   - Use libpaper's paper sizes. This includes user-defined paper sizes
     when using libpaper 2. It is still possible to define custom margins
     using "Medium:" specifications in the configuration file, and the
     one size defined by a2ps that libpaper does not know about, Quarto, is
     retained for backwards compatiblity, and as an example.
 * Bug fixes:
   - Avoid a crash when a medium is not specified; instead, use the default
     libpaper size (configured by the user or sysadmin, or the locale
     default).
   - Fix some other potential crashes and compiler warnings.
 * Documentation:
   - Reformat --help output consistently to 80 columns.
 * Build:
   - Require autoconf 2.71.
   - Require libpaper.

26 January, 2023 10:29PM by Reuben Thomas

January 25, 2023

FSF Blogs

Thank you and a very warm welcome to our new members

January 20, 2023 marked the end of our most recent fundraising campaign and associate member drive. We are proud to add 330 new associate members to our organization, and we have immense appreciation for the community that helped us get there. Please help us share our appreciation.

25 January, 2023 09:17PM

poke @ Savannah

GNU poke 3.0 released

I am happy to announce a new major release of GNU poke, version 3.0.

This release is the result of a year of development.  A lot of things have changed and improved with respect to the 2.x series; we have fixed many bugs and added quite a lot of new exciting and useful features.  See below for a description of many of them.

From now on, we intend to do not one but two major releases of poke every year.  What is moving us to change this is the realization that users have to wait for too long to enjoy new features, which are continuously being added in a project this young and active.

The tarball poke-3.0.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-3.0.tar.gz.

> GNU poke (http://www.jemarch.net/poke) is an interactive, extensible editor for binary data.  Not limited to editing basic entities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them.


Thanks to the people who contributed with code and/or documentation to this release.  In certain but no significant order they are:

   Mohammad-Reza Nabipoor
   Arsen Arsenović
   Luca Saiu
   Bruno Haible
   apache2
   Indu Bhagat
   Agathe Porte
   Alfred M. Szmidt
   Daiki Ueno
   Darshit Shah
   Jan Seeger
   Sergio Durigan Junior

   ... and yours truly

As always, thank you all!

But wait, this time we also have special thanks:

To Bruno Haible for his invaluable advise and his help in throughfully testing this new release in many different platforms and configurations.

To the Sourceware overseers, Mark Wielaard, Arsen Arsenović, and Sam James for their help in setting up the buildbots we are using for CI at sourceware.

What is new in this release:

User interface updates

  • A screen pager has been added to the poke application.  If enabled with the `.set pager yes' option, output will be paged one screenful at a time.
  • A tracer has been added to libpoke and the poke application. If enabled with the `.set tracer yes' option, subsequent loaded Poke types will be instrumentalized so calls to user-defined handlers are executed when certain events happen:
    • Every time a field gets mapped.
    • Every time a struct/union gets mapped.
    • Every time a field gets constructed.
    • Every time a struct/union gets constructed.
    • Every time an optional field is omitted when mapping or constructing.
  • A new command sdiff (for "structured diff") has been added to the poke application, that provides a way to generate patchable diffs of mapped structured Poke values.  This command is an interface to the structured diffs provided by the new diff.pk pickle.
  • When no name is passed to the .mem command, an unique name for the memory IOS with the form N will be used automatically, where N is a positive integer.
  • Auto-completion of 'attributes is now available in the poke application.
  • Constraint errors now contain details on the location (which field) where the constraint error happens, along with the particular expression that failed.
  • Inline assembler expressions and statements are now supported:

    ,----
    | asm (TEMPLATE [: OUTPUTS [: INPUTS]])
    | asm TYPE: (TEMPLATE [: INPUTS])
    `----

  • Both `printf' and `format' now support printing values of type `any'.
  • Both `printf' and `format' now support printing integral values interpreted as floating-point values encoded in IEEE 754.  Format tags %f, %g and %e are supported.  This feature, along with the new ieee754.pk pickle, eases dealing with floating-point data in binary data.
  • Pre-conditional optional fields are added to complement the currently supported post-conditional optional fields. A pre-conditional optional field like the following makes FNAME optional based on the evaluation of CONDITION.  But the field itself is not mapped if the condition evaluates to false:

    ,----
    | if (CONDITION)
    |   TYPE FNAME;
    `----

  • A new option `.set autoremap no' can be used in order to tell poke to not remap mapped values automatically.  This greatly speeds up things, but assumes that the contents of the IO space are not updated out of the control of the user.  See the manual for details.
  • The :to argument to the `extract' command is now optional, and defaults to the empty string.
  • ${XDG_CONFIG_HOME:-$HOME/.config} is now preferred to XDG_CONFIG_DIRS.

Poke Language updates

  • Array and struct constructors are now primaries in the Poke syntax. This means that it is no longer necessary to enclose them between parenthesis in constructions like:

    ,----
    | (Packet {}).field
    `----

    and this is now accepted:
    ,----
    | Packet {}.field
    `----

  • Bit-concatenation is now supported in l-values.  After executing the following code the value of `a' is 0x1N and the value of `b' is (uint<28>)0x2345678:

    ,----
    | var a = 0 as int<4>;
    | var b = 0 as uint<28>;
    |
    | a:::b = 0x12345678;
    `----

  • Arrays can now be indented by size, by specifying an offset as an index.  This is particularly useful for accessing structures such as string tables without having to explicitly iterate on the array's elements.
  • Union types can now be declared as "integral".  The same features of integral structs are now available for unions: integration, deintegration, the ability of being used in contexts where an integer is expected, etc.
  • Support for "computed fields" has been added to struct and union types.  Computed fields are accessed just like regular fields, but the semantics of referring to them and of assigning to them are specified by the user by the way of defining getter and setter methods.
  • This version introduces three new Poke attributes that work on values of type `any':

    ,----
    | VAL'elem (N)
    |    evaluates to the Nth element in VAL, as a value of type `any'.
    |
    | VAL'eoffset (N)
    |    evaluates to the offset of the Nth element in VAL.
    |
    | VAL'esize (N)
    |    evaluates to the size of the Nth element in VAL.
    |
    | VAL'ename (N)
    |    attribute evaluates to the name of the Nth element in VAL.
    `----

  • Two new operators have been introduced to facilitate operating Poke array as stacks in an efficient way: apush and apop.  Since these operators change the size of the involved arrays, they are only allowed in unbounded arrays.
  • Poke programs can now hook in the IO subsystem by installing functions that will be invoked when certain operations on IO spaces are being performed:

    ,----
    | ios_open_hook
    |   Functions in this hook are invoked once a new IO space has been
    |   opened.
    |
    | ios_set_hook
    |   Functions in this hook are invoked once the current IO space
    |   changes.
    |
    | ios_close_pre_hook
    | ios_close_hook
    |   Functions in these hooks are invoked before and after an IO space is
    |   closed, respectively.
    `----

  • The 'length attribute is now valid in values of type `any'.
  • Poke declarations can now be annotated as `immutable'.  It is not allowed to re-define immutable definitions.
  • A new compiler built-in `iolist' has been introduced, that returns an array with the IO space identifiers of currently open IOS.
  • We have changed the logic of the EXCOND operator ?!.  It now evaluates to 1 (true) if the execution of the first operand raises the specified exception, and to 0 (false) otherwise.  We profusedly apologize for the backwards incompatibility, but this is way better than the previous (reversed) logic.
  • The containing struct or union value can now be refered as SELF in the body of methods.  SELF is of type `any'.
  • Integer literal suffixes (B, H, U, etc) are case-insensitive. But until now little-case `b' wasn't being recognized as such.  Now `1B' is the same than `1b'.
  • Casting to union types now raise a compile-time error.
  • If no explicit message is specified in calls to `assert', a default one showing the source code of the failing condition is constructed and used instead.
  • An operator `remap' has been used in order to force a re-map of some mapped Poke value.
  • Signed integral types of one bit are not allowed.  How could they be, in two's complement?
  • The built-in function get_time has been renamed to gettime, to follow the usual naming of the corresponding standard C function.

Standard Poke Library updates

  • New standard functions:

    ,----
    | eoffset (V, N)
    |   Given a value of type `any' and a name, returns the offset of
    |   the element having that name.
    |
    | openset (HANDLER, [FLAGS])
    |   Open an IO space and make it the current IO space.
    |
    | with_temp_ios ([HANDLER], [FLAGS], [DO], [ENDIAN])
    |   Execute some code with a temporary IO space.
    |
    | with_cur_ios (IOS, [DO], [ENDIAN])
    |   Execute some code on some given IO space.
    `----

libpoke updates

  • New API function pk_struct_ref_set_field_value.
  • New API function pk_type_name.

Pickles updates

  • New pickles provided in the poke distribution:

    ,----
    | diff.pk
    |   Useful binary diffing utilities.  In particular, it implements
    |   the "structured diff" format as described in
    |   https://binary-tools.net/bindiff.pdf.
    |
    | io.pk
    |   Facilities to dump data to the terminal.
    |
    | pk-table.pk
    |   Convenient facilities to Poke programs to print tabulated data.
    |
    | openpgp.pk
    |   Pickle to poke at OpenPGP RFC 4880 data.
    |
    | sframe.pk
    | sframe-dump.pk
    |   Pickles for the SFrame unwinding format, and related dump
    |   utilities.
    |
    | search.pk
    |   Utility for searching data in IO spaces that conform to some
    |   given Poke type.
    |
    | riscv.pk
    |   Pickle to poke at instructions encoded in the RISC-V instruction
    |   set (RV32I).  It also provides methods to generate assembly
    |   language.
    |
    | coff.pk
    | coff-aarch64.pk
    | coff-i386.pk
    |   COFF object files.
    |
    | pe.pk
    | pe-amd64.pk
    | pe-arm.pk
    | pe-arm64.pk
    | pe-debug.pk
    | pe-i386.pk
    | pe-ia64.pk
    | pe-m32r.pk
    | pe-mips.pk
    | pe-ppc.pk
    | pe-riscv.pk
    | pe-sh3.pk
    |   PE/COFF object files.
    |
    | pcap.pk
    |   Capture file format.
    |
    | uuid.pk
    |   Universally Unique Identifier (UUID) as defined by RFC4122.
    |
    | redoxfs.pk
    |   RedoxFS files ystem of Redox OS.
    |
    | ieee754.pk
    |   IEEE Standard for Floating-Point Arithmetic.
    `----

  • The ELF pickle now provides functions implementing ELF hashing.

Build system updates

  • It is now supported to configure the poke sources with --disable-hserver.

Documentation updates

  • Documentation for the `format' language construction has been added to the poke manual.

Other updates

  • A new program poked, for "poke daemon", has been contributed to the poke distribution by Mohammad-Reza Nabipoor.  poked links with libpoke and uses Unix sockets to act as a broker to communicate with an instance of a Poke incremental compiler.  This is already used by several user interfaces to poke.
  • The machine-interface subsystem has been removed from poke, in favor of the poked approach.
  • The example GUI that was intended to be a test tool for the machine interface has been removed from the poke distribution.
  • Many bugs have been fixed.

--
Jose E. Marchesi
Frankfurt am Main
26 January 2023

25 January, 2023 07:08PM by Jose E. Marchesi

GNU Guile

GNU Guile 3.0.9 released

We are pleased to announce the release of GNU Guile 3.0.9! This release fixes a number of bugs and adds several new features, among which:

  • New bindings for POSIX functionality, including bindings for the at family of functions (openat, statat, etc.), a new spawn procedure that wraps posix_spawn and that system* now uses, and the ability to pass flags such as O_CLOEXEC to the pipe procedure.
  • A new bytevector-slice procedure.
  • Reduced memory consumption for the linker and assembler.

For full details, see the NEWS entry, and check out the download page.

Happy Guile hacking!

25 January, 2023 02:25PM by Ludovic Courtès (guile-devel@gnu.org)

January 24, 2023

FSF News

FSF board adopts updated by-laws to protect copyleft

BOSTON, Massachusetts, USA -- Tuesday, January 24, 2023 -- The board of the Free Software Foundation (FSF) today announced it has adopted updated bylaws for the nonprofit effective Feb. 1, 2023.

24 January, 2023 08:45PM

Andy Wingo

parallel ephemeron tracing

Hello all, and happy new year. Today's note continues the series on implementing ephemerons in a garbage collector.

In our last dispatch we looked at a serial algorithm to trace ephemerons. However, production garbage collectors are parallel: during collection, they trace the object graph using multiple worker threads. Our problem is to extend the ephemeron-tracing algorithm with support for multiple tracing threads, without introducing stalls or serial bottlenecks.

Recall that we ended up having to define a table of pending ephemerons:

struct gc_pending_ephemeron_table {
  struct gc_ephemeron *resolved;
  size_t nbuckets;
  struct gc_ephemeron *buckets[0];
};

This table holds pending ephemerons that have been visited by the graph tracer but whose keys haven't been found yet, as well as a singly-linked list of resolved ephemerons that are waiting to have their values traced. As a global data structure, the pending ephemeron table is a point of contention between tracing threads that we need to design around.

a confession

Allow me to confess my sins: things would be a bit simpler if I didn't allow tracing workers to race.

As background, if your GC supports marking in place instead of always evacuating, then there is a mark bit associated with each object. To reduce the overhead of contention, a common strategy is to actually use a whole byte for the mark bit, and to write to it using relaxed atomics (or even raw stores). This avoids the cost of a compare-and-swap, but at the cost that multiple marking threads might see that an object's mark was unset, go to mark the object, and think that they were the thread that marked the object. As far as the mark byte goes, that's OK because everybody is writing the same value. The object gets pushed on the to-be-traced grey object queues multiple times, but that's OK too because tracing should be idempotent.

This is a common optimization for parallel marking, and it doesn't have any significant impact on other parts of the GC--except ephemeron marking. For ephemerons, because the state transition isn't simply from unmarked to marked, we need more coordination.

high level

The parallel ephemeron marking algorithm modifies the serial algorithm in just a few ways:

  1. We have an atomically-updated state field in the ephemeron, used to know if e.g. an ephemeron is pending or resolved;

  2. We use separate fields for the pending and resolved links, to allow for concurrent readers across a state change;

  3. We introduce "traced" and "claimed" states to resolve races between parallel tracers on the same ephemeron, and track the "epoch" at which an ephemeron was last traced;

  4. We remove resolved ephemerons from the pending ephemeron hash table lazily, and use atomic swaps to pop from the resolved ephemerons list;

  5. We have to re-check key liveness after publishing an ephemeron to the pending ephemeron table.

Regarding the first point, there are four possible values for the ephemeron's state field:

enum {
  TRACED, CLAIMED, PENDING, RESOLVED
};

The state transition diagram looks like this:

  ,----->TRACED<-----.
 ,         | ^        .
,          v |         .
|        CLAIMED        |
|  ,-----/     \---.    |
|  v               v    |
PENDING--------->RESOLVED

With this information, we can start to flesh out the ephemeron object itself:

struct gc_ephemeron {
  uint8_t state;
  uint8_t is_dead;
  unsigned epoch;
  struct gc_ephemeron *pending;
  struct gc_ephemeron *resolved;
  void *key;
  void *value;
};

The state field holds one of the four state values; is_dead indicates if a live ephemeron was ever proven to have a dead key, or if the user explicitly killed the ephemeron; and epoch is the GC count at which the ephemeron was last traced. Ephemerons are born TRACED in the current GC epoch, and the collector is responsible for incrementing the current epoch before each collection.

algorithm: tracing ephemerons

When the collector first finds an ephemeron, it does a compare-and-swap (CAS) on the state from TRACED to CLAIMED. If that succeeds, we check the epoch; if it's current, we revert to the TRACED state: there's nothing to do.

(Without marking races, you wouldn't need either TRACED or CLAIMED states, or the epoch; it would be implicit in the fact that the ephemeron was being traced at all that you had a TRACED ephemeron with an old epoch.)

So now we have a CLAIMED ephemeron with an out-of-date epoch. We update the epoch and clear the pending and resolved fields, setting them to NULL. If, then, the ephemeron is_dead, we are done, and we go back to TRACED.

Otherwise we check if the key has already been traced. If so we forward it (if evacuating) and then trace the value edge as well, and transition to TRACED.

Otherwise we have a live E but we don't know about K; this ephemeron is pending. We transition E's state to PENDING and add it to the front of K's hash bucket in the pending ephemerons table, using CAS to avoid locks.

We then have to re-check if K is live, after publishing E, to account for other threads racing to mark to K while we mark E; if indeed K is live, then we transition to RESOLVED and push E on the global resolved ephemeron list, using CAS, via the resolved link.

So far, so good: either the ephemeron is fully traced, or it's pending and published, or (rarely) published-then-resolved and waiting to be traced.

algorithm: tracing objects

The annoying thing about tracing ephemerons is that it potentially impacts tracing of all objects: any object could be the key that resolves a pending ephemeron.

When we trace an object, we look it up in the pending ephemeron hash table. But, as we traverse the chains in a bucket, we also load each node's state. If we find a node that's not in the PENDING state, we atomically forward its predecessor to point to its successor. This is correct for concurrent readers because the end of the chain is always reachable: we only skip nodes that are not PENDING, nodes never become PENDING after they transition away from being PENDING, and we only add PENDING nodes to the front of the chain. We even leave the pending field in place, so that any concurrent reader of the chain can still find the tail, even when the ephemeron has gone on to be RESOLVED or even TRACED.

(I had thought I would need Tim Harris' atomic list implementation, but it turns out that since I only ever insert items at the head, having annotated links is not necessary.)

If we find a PENDING ephemeron that has K as its key, then we CAS its state from PENDING to RESOLVED. If this works, we CAS it onto the front of the resolved list. (Note that we also have to forward the key at this point, for a moving GC; this was a bug in my original implementation.)

algorithm: resolved ephemerons

Periodically a thread tracing the graph will run out of objects to trace (its mark stack is empty). That's a good time to check if there are resolved ephemerons to trace. We atomically exchange the global resolved list with NULL, and then if there were resolved ephemerons, then we trace their values and transition them to TRACED.

At the very end of the GC cycle, we sweep the pending ephemeron table, marking any ephemeron that's still there as is_dead, transitioning them back to TRACED, clearing the buckets of the pending ephemeron table as we go.

nits

So that's it. There are some drawbacks, for example that this solution takes at least three words per ephemeron. Oh well.

There is also an annoying point of serialization, which is related to the lazy ephemeron resolution optimization. Consider that checking the pending ephemeron table on every object visit is overhead; it would be nice to avoid this. So instead, we start in "lazy" mode, in which pending ephemerons are never resolved by marking; and then once the mark stack / grey object worklist fully empties, we sweep through the pending ephemeron table, checking each ephemeron's key to see if it was visited in the end, and resolving those ephemerons; we then switch to "eager" mode in which each object visit could potentially resolve ephemerons. In this way the cost of ephemeron tracing is avoided for that part of the graph that is strongly reachable. However, with parallel markers, would you switch to eager mode when any thread runs out of objects to mark, or when all threads run out of objects? You would get greatest parallelism with the former, but you run the risk of some workers prematurely running out of data, but when there is still a significant part of the strongly-reachable graph to traverse. If you wait for all threads to be done, you introduce a serialization point. There is a related question of when to pump the resolved ephemerons list. But these are engineering details.

Speaking of details, there are some gnarly pitfalls, particularly that you have to be very careful about pre-visit versus post-visit object addresses; for a semi-space collector, visiting an object will move it, so for example in the pending ephemeron table which by definition is keyed by pre-visit (fromspace) object addresses, you need to be sure to trace the ephemeron key for any transition to RESOLVED, and there are a few places this happens (the re-check after publish, sweeping the table after transitioning from lazy to eager, and when resolving eagerly).

implementation

If you've read this far, you may be interested in the implementation; it's only a few hundred lines long. It took me quite a while to whittle it down!

Ephemerons are challenging from a software engineering perspective, because they are logically a separate module, but they interact both with users of the GC and with the collector implementations. It's tricky to find the abstractions that work for all GC algorithms, whether they mark in place or move their objects, and whether they mark the heap precisely or if there are some conservative edges. But if this is the sort of thing that interests you, voilà the API for users and the API to and from collector implementations.

And, that's it! I am looking forward to climbing out of this GC hole, one blog at a time. There are just a few more features before I can seriously attack integrating this into Guile. Until the next time, happy hacking :)

24 January, 2023 10:48AM by Andy Wingo

January 23, 2023

texinfo @ Savannah

Texinfo 7.0.2 released

We have released version 7.0.2 of Texinfo, the GNU documentation format. This is a minor bug-fix release.

It's available via a mirror (xz is much smaller than gz, but gz is available too just in case):

http://ftpmirror.gnu.org/texinfo/texinfo-7.0.2.tar.xz
http://ftpmirror.gnu.org/texinfo/texinfo-7.0.2.tar.gz

Please send any comments to bug-texinfo@gnu.org.

Full announcement:

https://lists.gnu.org/archive/html/info-gnu/2023-01/msg00008.html

23 January, 2023 04:52PM by Gavin D. Smith

GNU Guix

Meet Guix at FOSDEM

GNU Guix will be present at FOSDEM next week, February 4th and 5th. This is the first time since the pandemic that FOSDEM takes place again “in the flesh” in Brussels, which is exciting to those of us lucky enough to get there! Everything will be live-streamed and recorded thanks to the amazing FOSDEM crew, so everyone can enjoy wherever they are; some of the talks this year will be “remote” too: pre-recorded videos followed by live Q&A sessions with the speaker.

Believe it or not, it’s the 9th year Guix is represented at FOSDEM, with more than 30 talks given in past editions! This year brings several talks that will let you learn more about different areas of the joyful Hydra Guix has become.

This all starts on Saturday, in particular with the amazing declarative and minimalistic computing track:

There are many other exciting talks in this track, some of which closely related to Guix and Guile; check it out!

You can also discover Guix in other tracks:

Guix Days logo

As was the case pre-pandemic, we are also organizing the Guix Days as a FOSDEM fringe event, a two-day Guix workshop where contributors and enthusiasts will meet. The workshop takes place on Thursday Feb. 2nd and Friday Feb. 3rd at the Institute of Cultural Affairs (ICAB) in Brussels.

Again this year there will be few talks; instead, the event will consist primarily of “unconference-style” sessions focused on specific hot topics about Guix, the Shepherd, continuous integration, and related tools and workflows.

Attendance to the workshop is free and open to everyone, though you are invited to register (there are few seats left!). Check out the workshop’s wiki page for registration and practical info. Hope to see you in Brussels!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64, and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

23 January, 2023 01:56PM by Ludovic Courtès

January 22, 2023

parallel @ Savannah

GNU Parallel 20230122 ('Bolsonaristas') released [stable]

GNU Parallel 20230122 ('Bolsanaristas') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  Colorful output
  parallel, with --color flag
  tasks more vibrant now
    -- ChatGPT

New in this release:

  • Bug fixes and man page updates.

News about GNU Parallel:

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

22 January, 2023 06:17PM by Ole Tange

Simon Josefsson

Understanding Trisquel

Ever wondered how Trisquel and Ubuntu differs and what’s behind the curtain from a developer perspective? I have. Sharing what I’ve learnt will allow you to increase knowledge and trust in Trisquel too.

Trisquel GNU/Linux logo

The scripts to convert an Ubuntu archive into a Trisquel archive are available in the ubuntu-purge repository. The easy to read purge-focal script lists the packages to remove from Ubuntu 20.04 Focal when it is imported into Trisquel 10.0 Nabia. The purge-jammy script provides the same for Ubuntu 22.04 Jammy and (the not yet released) Trisquel 11.0 Aramo. The list of packages is interesting, and by researching the reasons for each exclusion you can learn a lot about different attitudes towards free software and understand the desire to improve matters. I wish there were a wiki-page that for each removed package summarized relevant links to earlier discussions. At the end of the script there is a bunch of packages that are removed for branding purposes that are less interesting to review.

Trisquel adds a couple of Trisquel-specific packages. The source code for these packages are in the trisquel-packages repository, with sub-directories for each release: see 10.0/ for Nabia and 11.0/ for Aramo. These packages appears to be mostly for branding purposes.

Trisquel modify a set of packages, and here is starts to get interesting. Probably the most important package to modify is to use GNU Linux-libre instead of Linux as the kernel. The scripts to modify packages are in the package-helpers repository. The relevant scripts are in the helpers/ sub-directory. There is a branch for each Trisquel release, see helpers/ for Nabia and helpers/ for Aramo. To see how Linux is replaced with Linux-libre you can read the make-linux script.

This covers the basic of approaching Trisquel from a developers perspective. As a user, I have identified some areas that need more work to improve trust in Trisquel:

  • Auditing the Trisquel archive to confirm that the intended changes covered above are the only changes that are published.
  • Rebuild all packages that were added or modified by Trisquel and publish diffoscope output comparing them to what’s in the Trisquel archive. The goal would be to have reproducible builds of all Trisquel-related packages.
  • Publish an audit log of the Trisquel archive to allow auditing of what packages are published. This boils down to trust of the OpenPGP key used to sign the Trisquel archive.
  • Trisquel archive mirror auditing to confirm that they are publishing only what comes from the official archive, and that they do so timely.

I hope to publish more about my work into these areas. Hopefully this will inspire similar efforts in related distributions like PureOS and the upstream distributions Ubuntu and Debian.

Happy hacking!

22 January, 2023 11:10AM by simon

January 20, 2023

FSF Blogs

January 19, 2023

Associate members are invited: Nominate new candidates to the FSF board

19 January, 2023 11:05PM

FSF News

FSF now accepting board nominations from associate members

BOSTON, Massachusetts, USA -- Thursday, January 19, 2023 -- Associate members of the Free Software Foundation (FSF) now have the chance to nominate and evaluate candidates to serve on the board of directors for the first time since the nonprofit was founded thirty-seven years ago.

19 January, 2023 10:55PM

January 17, 2023

FSF Events

Free Software Directory meeting on IRC: Friday, January 27, starting at 12:00 EST (17:00 UTC)

Join the FSF and friends on Friday, January 27, from 12:00 to 15:00 EST (17:00 to 20:00 UTC) to help improve the Free Software Directory.

17 January, 2023 07:28PM

Free Software Directory meeting on IRC: Friday, January 20, starting at 12:00 EST (17:00 UTC)

Join the FSF and friends on Friday, January 20, from 12:00 to 15:00 EST (17:00 to 20:00 UTC) to help improve the Free Software Directory.

17 January, 2023 07:25PM

January 16, 2023

diffutils @ Savannah

diffutils-3.9 released [stable]

This is to announce diffutils-3.9, a stable release.

There have been 51 commits by 3 people in the 76 weeks since 3.8.

See the NEWS below for a brief summary.

Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Bruno Haible (1)
  Jim Meyering (14)
  Paul Eggert (36)

Jim [on behalf of the diffutils maintainers]
==================================================================

Here is the GNU diffutils home page:
    http://gnu.org/s/diffutils/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=diffutils.git;a=shortlog;h=v3.9
or run this command from a git-cloned diffutils directory:
  git shortlog v3.8..v3.9

To summarize the 931 gnulib-related changes, run these commands
from a git-cloned diffutils directory:
  git checkout v3.9
  git submodule summary v3.8

Here are the compressed sources and a GPG detached signature:
  https://ftp.gnu.org/gnu/diffutils/diffutils-3.9.tar.xz
  https://ftp.gnu.org/gnu/diffutils/diffutils-3.9.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://ftpmirror.gnu.org/diffutils/diffutils-3.9.tar.xz
  https://ftpmirror.gnu.org/diffutils/diffutils-3.9.tar.xz.sig

Here are the SHA1 and SHA256 checksums:

35905d7c3d1ce116e6794be7fe894cd25b2ded74  diffutils-3.9.tar.xz
2A076QogGGjeg9eNrTQTrYgWDMU7zDbrnq98INvwI/E  diffutils-3.9.tar.xz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify diffutils-3.9.tar.xz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096/0x7FD9FCCB000BEEEE 2010-06-14 [SCEA]
        Key fingerprint = 155D 3FC5 00C8 3448 6D1E  EA67 7FD9 FCCB 000B EEEE
  uid                   [ unknown] Jim Meyering <jim@meyering.net>
  uid                   [ unknown] Jim Meyering <meyering@fb.com>
  uid                   [ unknown] Jim Meyering <meyering@gnu.org>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key jim@meyering.net

  gpg --recv-keys 7FD9FCCB000BEEEE

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=diffutils&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify diffutils-3.9.tar.xz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.72a.65-d081
  Automake 1.16i
  Gnulib v0.1-5689-g83adc2f722

==================================================================
NEWS

* Noteworthy changes in release 3.9 (2023-01-15) [stable]

** Bug fixes

  diff -c and -u no longer output incorrect timezones in headers
  on platforms like Solaris where struct tm lacks tm_gmtoff.
  [bug#51228 introduced in 3.4]

16 January, 2023 12:16AM by Jim Meyering

January 14, 2023

GNU Health

Jérôme Lejeune Foundation adopts GNU Health

We start 2023 with exciting news for the medical and scientific community!

GNU Health has been adopted by he Jérôme Lejeune foundation, a leading organization in the research and management of trisomy 21 (Down Syndrome) and other intellectual disabilities of genetic origin.

Lejeune foundation has its headquarters in France, with offices in Argentina, the United States and Spain.

On December 2022, the faculty of engineering from the University of Entre Rios, represented by the dean Diego Campana and the head of the school of Public Health, Fernando Sassetti, formalized the agreement with the president of the Lejeune foundation in Argentina, Luz Morano.

The same month, I met in Madrid with the medical director and IT team of the Lejeune foundation Spain.

Luz Morano declared “[GNU Health] goes beyond the Foundation, providing the health professionals the specific features to manage a patient with trisomy 21. We are putting a project in the hands of humanity

[GNU Health] goes beyond the Foundation, providing the health professionals the specific features to manage a patient with trisomy 21. We are putting a project in the hands of humanity

Luz Morano, President of Lejeune Foundation, Argentina

Morano also stated: “GNU Health will pave the road for the medical management, and let us focus on our two other missions: Research and the defense of patient rights

The agreement is in the context of the GNU Health Alliance of Academic and Research Institutions that UNER has with GNU Solidario. In this sense, Fernando Sassetti explained “It provides tools for an integrative approach of those people with certain pathologies that due to the reduced number are not managed in the best way. This will benefit the organizations and health professionals, that today lack the means to do so in the best way and timely manner. It benefits the patients, in their right to have an integral health record.”

Research and Open Science

The adoption of GNUHealth by the Jérôme Lejeune Foundation opens new exciting avenues for the scientific community. In addition to the clinical management and medical history, GNU Health will enable scientists to dive into the fields of genomics, epigenetics and exposomics, gathering and processing information from multiple contexts and subjects, thanks to the distributed nature of the GNU Health Federation.

The GNU Health HMIS counts many packages and features, some of them of special interest for this project. In addition to the specific customizations for the foundation, the packages already present in GNUHealth, such as obstetrics, pediatrics, genomics, socioeconomics or lifestyle will provide a holistic approach to the person with trisomy 21 and other related conditions.

All of this will be done using exclusively Free/Libre software and open science.

People before Patients

Trisomy 21 poses challenges for the individual, their family, health professionals and the society. The scientific community needs to push the research to shed light on the etiology, physiopathology and associated clinical manifestations, such as heart defects, blood disorders or Alzheimer’s.

Most importantly, as part of the scientific community, we must put a stop to the discrimination and stigmatization. We must tear down the barriers and walls built on our societies that prevent the inclusion of individuals with trisomy 21.

As part of this effort, GNU Health provides the WHO International Classification on Functioning, disability and health (ICF). In other words, is not just the health condition or disorder we may have, but how the environmental factors and barriers influence the normal functioning and integration as individuals in the society. Many times, those physical, artificial barriers present in our daily lives are way more pernicious than the condition itself.

The strong focus of GNU Health in Social Medicine, and the way we perceive medicine as a social science will help improving the life of the person living with trisomy 21, and contribute to the much needed healing process in our societies. We need to work on the molecular basis of the health conditions, but little can be done if without empathetic, inclusive and supportive societies so people can live and enjoy life with dignity, no matter their health or socioeconomic status.

Projects like this represent the spirit of GNU Health and make me immensely proud to be part of this community.

Happy and healthy hacking!
Luis Falcon, MD
President, GNU Solidario

Links:

14 January, 2023 04:57PM by Luis Falcon

January 13, 2023

gcl @ Savannah

GCL 2.6.14 has been released

Greetings!  The GCL team is happy to announce the release of version 2.6.14, the latest achievement in the 'stable' (as opposed to 'development') series.  Please see http://www.gnu.org/software/gcl for downloading information.

This is a cleanup release with respect to 2.6.13, with the primary goal of supporting current gcc-12 signed integer tree-vrp optimizations, on by default at -O2 or higher.

There are a few portability fixes: X86_64_RELOC_SIGNED_1 support on macosx, centos readline/configure fixes, and R_RISCV_CALL_PLT support on riscv64.

A fix to decode-universal-time is included, which is backward
incompatible with a workaround in currently released maxima.  The
gcl_cleanup... and/or master maxima branches in git have been adjusted accordingly.

'si::help has been imported into the "USER" package.

13 January, 2023 04:20PM by Camm Maguire

January 12, 2023

FSF Events

Free Software Directory meeting on IRC: Friday, January 13, starting at 12:00 EST (17:00 UTC)

Join the FSF and friends on Friday, January 13, from 12:00 to 15:00 EST (17:00 to 20:00 UTC) to help improve the Free Software Directory.

12 January, 2023 10:11PM

FSF News

Software freedom in education advocate Erin Rose Glass to keynote FSF's LibrePlanet

BOSTON, Massachusetts, USA -- Thursday, January 12, 2023 -- The Free Software Foundation (FSF) today announced Erin Rose Glass as its first keynote speaker for LibrePlanet 2023, the fifteenth edition of the Free Software Foundation's conference on ethical technology and user freedom. The annual technology and social justice conference will be held March 18 and 19, 2023, online and in the Boston area, with the theme "Charting the Course."

12 January, 2023 08:40PM

January 09, 2023

health @ Savannah

GNU Health Hospital Management patchset 4.0.5 released

Dear community

GNUHealth 4.0.5 patchset has been released !

Priority: High

Table of Contents

  • About GNU Health Patchsets
  • Updating your system with the GNU Health control Center
  • Installation notes
  • List of other issues related to this patchset

About GNU Health Patchsets

We provide "patchsets" to stable releases. Patchsets allow applying bug fixes and updates on production systems. Always try to keep your production system up-to-date with the latest patches.

Patches and Patchsets maximize uptime for production systems, and keep your system updated, without the need to do a whole installation.

NOTE: Patchsets are applied on previously installed systems only. For new, fresh installations, download and install the whole tarball (ie, gnuhealth-4.0.5.tar.gz)

Updating your system with the GNU Health control Center

Starting GNU Health 3.x series, you can do automatic updates on the GNU Health HMIS kernel and modules using the GNU Health control center program.

Please refer to the administration manual section ( https://en.wikibooks.org/wiki/GNU_Health/Control_Center )

The GNU Health control center works on standard installations (those done following the installation manual on wikibooks). Don't use it if you use an alternative method or if your distribution does not follow the GNUHealth packaging guidelines.

Installation Notes

You must apply previous patchsets before installing this patchset. If your patchset level is 4.0.5, then just follow the general instructions. You can find the patchsets at GNU Health main download site at GNU.org (https://ftp.gnu.org/gnu/health/)

In most cases, GNU Health Control center (gnuhealth-control) takes care of applying the patches for you. 

Pre-requisites for upgrade to 4.0.5: None

Now follow the general instructions at

 

After applying the patches, make a full update of your GNU Health database as explained in the documentation.

When running "gnuhealth-control" for the first time, you will see the following message: "Please restart now the update with the new control center" Please do so. Restart the process and the update will continue.

  • Restart the GNU Health server

List of other issues and tasks related to this patchset

  • bug #63558: Fault: 'NoneType' object has no attribute 'name'
  • bug #63557: Missing view architecture for ('calendar.category', None, 'tree')
  • bug #63533: Model 'gnuhealth.pol' is missing a default access
  • bug #63532: health_caldav misses requiremnt vobject in setup.py
  • bug #63517: Fault: 'webdav' is not in list
  • bug #62777: The term health prof used for both initiating and signing professional in patient evaluation
  • bug #62634, Missing Spanish Translations

For detailed information about each issue, you can visit :
 https://savannah.gnu.org/bugs/?group=health

About each task, you can visit:
 https://savannah.gnu.org/task/?group=health

For detailed information you can read about Patches and Patchsets
 https://en.wikibooks.org/wiki/GNU_Health/Patches_and_Patchsets

09 January, 2023 06:31PM by Luis Falcon

January 07, 2023

mit-scheme @ Savannah

January 06, 2023

GNUnet News

GNUnet 0.19.2

06 January, 2023 11:00PM

FSF Blogs

Sharing knowledge about the GNU family of licenses

Copyright and licensing associate Craig Topham discusses the work done by the Licensing and Compliance Lab to answer licensing questions via articles, the FAQ, and email.

06 January, 2023 08:20PM

GNU Guix

The Filesystem Hierarchy Standard Comes to Guix Containers

GNU Guix is different from most other GNU/Linux distributions and perhaps nowhere is that more obvious than the organization of the filesystem: Guix does not conform to the Filesystem Hierarchy Standard (FHS). In practical terms, this means there is no global /lib containing libraries, /bin containing binaries,¹ and so on. This is very much at the core of how Guix works and some of the convenient features, like per-user installation of programs (different versions, for instance) and a declarative system configuration where the system is determined from a configuration file.

However, this also leads to a difference in how many pieces of software expect their world to look like, relying on finding a library in /lib or an external tool in /bin. When these are hard coded and not overcome with appropriate build options, we patch code to refer to absolute paths in the store, like /gnu/store/hrgqa7m498wfavq4awai3xz86ifkjxdr-grep-3.6/bin/grep, to keep everything consistently contained within the store.

It all works great and is thanks to the hard work of everyone that has contributed to Guix. But what if we need a more FHS-like environment for developing, testing, or running a piece of software?

To that end, we've recently added (available in Guix 1.4.0) a new option for guix shell (previously called guix environment): --emulate-fhs (or -F). This option is used in conjunction with the --container (or -C) option which creates an isolated, you guessed it, container. The new --emulate-fhs option will set up an environment in the container that follows FHS expectations, so that libraries are visible in /lib in the container, as an example.

Here is a very simple example:

$ guix shell --container --emulate-fhs coreutils -- ls /bin | head
[
b2sum
base32
base64
basename
basenc
cat
catchsegv
chcon
chgrp

and

$ guix shell --container --emulate-fhs coreutils -- ls /lib | head
Mcrt1.o
Scrt1.o
audit
crt1.o
crti.o
crtn.o
gconv
gcrt1.o
ld-2.33.so
ld-linux-x86-64.so.2

Contrast that with /bin on a Guix system:

$ ls /bin -l

total 4
lrwxrwxrwx 1 root root  61 Dec 12 09:57 sh -> \
    /gnu/store/d99ykvj3axzzidygsmdmzxah4lvxd6hw-bash-5.1.8/bin/sh*

and /lib

$ ls /lib
ls: cannot access '/lib': No such file or directory

Or, if you like to see it more in motion, here's a gif (courtesy of Ludovic Courtès): An animated gif showing the above 'guix shell' output.

Additionally, for the more technically-minded, the glibc used in this container will read from a global cache in /etc/ld.so.cache contrary to the behavior in Guix otherwise. This can help ensure that libraries are found when querying the ld cache or using the output of ldconfig -p, for example.

There are several uses that spring to mind for such a container in Guix. For developers, or those aspiring to hack on a project, this is a helpful tool when needing to emulate a different (non-Guix) environment. For example, one could use this to more easily follow build instructions meant for a general distribution, say when a Guix package is not (yet) available or easy to write immediately.

Another usage is to be able to use tools that don't really fit into Guix's model, like ones that use pre-built binaries. There are many reasons why this is not ideal and Guix strives to replace or supplement such tools, but practically speaking they can be hard to avoid entirely. The FHS container helps bridge this gap, providing an isolated and reproducible environment as needed.

Users not interested in development will also find the FHS container useful. For example, there may be software that is free and conforms to the Free System Distribution Guidelines (FSDG) Guix follows, yet is not feasible to be packaged by our standards. JavaScript and particularly Electron applications are not yet packaged for Guix due to the difficulties of a properly source-based and bootstrapable approach in this ecosystem.

As a more interesting example for this last point, let's dive right into a big one: the popular VSCodium editor. This is a freely licensed build of Microsoft's VS Code editor. This is based on Electron and pre-built AppImages are available. Downloading and making the AppImage executable (with a chmod +x), we can run it in a container with

guix shell --container --network --emulate-fhs \
    --development ungoogled-chromium gcc:lib \
    --preserve='^DISPLAY$' --preserve='^XAUTHORITY$' --expose=$XAUTHORITY \
    --preserve='^DBUS_' --expose=/var/run/dbus \
    --expose=/sys/dev --expose=/sys/devices --expose=/dev/dri \
    -- ./VSCodium-1.74.0.22342.glibc2.17-x86_64.AppImage --appimage-extract-and-run

The second line is a handy cheat to get lots of libraries often needed for graphical applications (development inputs of the package ungoogled-chromium) though it can be overkill if the AppImage does actually bundle everything (they don't!). The next line is for display on the host's X server, the one after for DBus communication, and lastly exposing some of the host hardware for rendering. This last part may be different on different hardware. That should do it, at least to see basic functionality of VSCodium. Note that we can't run an AppImage without the --appimage-extract-and-run option as it will want to use FUSE to mount the image which is not possible from the container.²

The FHS container is also useful to be able to run the exact same binary as anyone else, as you might want to for privacy reasons with the Tor Browser. While there is a long-standing set of patches to build the Tor Browser from source, with a container we can run the official build directly. After downloading, checking the signature, and unpacking, we can launch the Tor Browser from the root of the unpacked directory with:

guix shell --container --network --emulate-fhs \
    --preserve='^DISPLAY$' --preserve='^XAUTHORITY$' --expose=$XAUTHORITY \
    alsa-lib bash coreutils dbus-glib file gcc:lib \
    grep gtk+ libcxx pciutils sed \
    -- ./start-tor-browser.desktop -v

Here we've used a more minimal set of package inputs, rather than the ungoogled-chromium trick above. Usually this is found through some trial and error, looking at log output, maybe tracing, and sometimes from documentation. Though documentation of needed packages often has some assumptions on what is already available on typical systems. (Thanks to Jim Newsome for pointing out this example on the guix-devel mailing list.)

Another example is to get the latest nightly builds of Rust, via rustup.

$ mkdir ~/temphome

$ guix shell --network --container --emulate-fhs \
    bash coreutils curl grep nss-certs gcc:lib gcc-toolchain \
    pkg-config glib cairo atk pango@1.48.10 gdk-pixbuf gtk+ git \
    --share=$HOME/temphome=$HOME

~/temphome [env]$ curl --proto '=https' --tlsv1.2 -sSf <https://sh.rustup.rs> | sh

First we created a ~/temphome directory to use as $HOME in the container and then included a bunch of libraries in the container for the next example.

This will proceed without problem and we'll see

info: downloading installer

Welcome to Rust!

This will download and install the official compiler for the Rust
programming language, and its package manager, Cargo.

...

Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, run:
source "$HOME/.cargo/env"

After updating the shells environment as instructed, we can see it all worked

~/temphome [env]$ rustc --version
rustc 1.65.0 (897e37553 2022-11-02)

as Guix's current Rust is at 1.61.0 and we didn't even include Rust in the container, of course.

Finally, we can build a Rust project of desktop widgets, ElKowars wacky widgets (eww), following their directions. Ultimately this uses just the standard cargo build --release and builds after downloading all the needed libraries.

~/temphome/eww [env]$ git clone https://github.com/elkowar/eww
...
~/temphome/eww [env]$ cd eww

~/temphome/eww [env]$ cargo build --release
info: syncing channel updates for 'nightly-2022-08-27-x86_64-unknown-linux-gnu'
info: latest update on 2022-08-27, rust version 1.65.0-nightly (c07a8b4e0 2022-08-26)
...

Finished release [optimized] target(s) in 2m 06s

With this being a fresh container, you will need to make some directories that normally exist, like ~/.config and ~/.cache in this case. For basic display support, it is enough to add --preserve='^DISPLAY$' --preserve='^XAUTHORITY$' --expose=$XAUTHORITY to the container launch options and run the first example widget in the documentation.

As we can see, with containers more generally we have to provide the right inputs and options as the environment is completely specified at creation. Once you want to run something that needs hardware from the host or to access host files, the container becomes increasingly porous for more functionality. This is certainly a trade-off, but one which we have agency with a container we wouldn't get otherwise.

The FHS option provides another option to make a container in Guix to produce other environments, even those with a vastly different philosophy of the root filesystem! This is one more tool in the Guix toolbox for controlled and reproducible environments that also let's us do some things we couldn't (easily) do otherwise.

Notes

¹ Other than a symlink for sh from the bash package, for compatibility reasons.

² Actually, one can use flatpak-spawn from flatpak-xdg-utils to launch something on the host and get the AppImage to mount itself. However, it is not visible from the same container. Or, we can use a normal mounting process outside of the container to inspect the contents, but AppImages will have an offset. We can use the FHS container option to get this offset and then mount in one line with mount VSCodium-1.74.0.22342.glibc2.17-x86_64.AppImage <mountpoint> -o offset=$(guix shell --container --emulate-fhs zlib -- ./VSCodium-1.74.0.22342.glibc2.17-x86_64.AppImage --appimage-offset)

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64, and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

06 January, 2023 01:30PM by John Kehayias

January 04, 2023

FSF Blogs

GNU Guix

Dissecting Guix, Part 1: Derivations

To a new user, Guix's functional architecture can seem quite alien, and possibly offputting. With a combination of extensive #guix-querying, determined manual-reading, and plenty of source-perusing, they may eventually figure out how everything fits together by themselves, but this can be frustrating and often takes a fairly long time.

However, once you peel back the layers, the "Nix way" is actually rather elegant, if perhaps not as simple as the mutable, imperative style implemented by the likes of dpkg and pacman. This series of blog posts will cover basic Guix concepts, taking a "ground-up" approach by dealing with lower-level concepts first, and hopefully make those months of information-gathering unnecessary.

Before we dig in to Guix-specific concepts, we'll need to learn about one inherited from Nix, the original functional package manager and the inspiration for Guix; the idea of a derivation and its corresponding store items.

These concepts were originally described by Eelco Dolstra, the original author of Nix, in their PhD thesis; see ยง 2.1 The Nix store and ยง 2.4 Store Derivations.

Store Items

As you probably know, everything that Guix builds is stored in the store, which is almost always the /gnu/store directory. It's the job of the guix-daemon to manage the store and build things. If you run guix build PKG, PKG will be built or downloaded from a substitute server if available, and a path to an item in the store will be displayed.

$ guix build irssi
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3

This item contains the final result of building irssi. Let's peek inside:

$ ls $(guix build irssi)
bin/  etc/  include/  lib/  share/
$ ls $(guix build irssi)/bin
irssi*

irssi is quite a simple package. What about a more complex one, like glib?

$ guix build glib
/gnu/store/bx8qq76idlmjrlqf1faslsq6zjc6f426-glib-2.73.3-bin
/gnu/store/j65bhqwr7qq7l77nj0ahmk1f1ilnjr3a-glib-2.73.3-debug
/gnu/store/3pn4ll6qakgfvfpc4mw89qrrbsgj3jf3-glib-2.73.3-doc
/gnu/store/dvsk6x7d26nmwsqhnzws4iirb6dhhr1d-glib-2.73.3
/gnu/store/4c8ycz501n2d0xdi4blahvnbjhd5hpa8-glib-2.73.3-static

glib produces five /gnu/store items, because it's possible for a package to produce multiple outputs. Each output can be referred to separately, by prefixing a package's name with :OUTPUT where supported. For example, this guix install invocation will add glib's bin output to your profile:

$ guix install glib:bin

The default output is out, so when you pass glib by itself to that command, it will actually install glib:out to the profile.

guix build also provides the --source flag, which produces the store item corresponding to the given package's downloaded source code.

$ guix build --source irssi
/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz
$ guix build --source glib
/gnu/store/d22wzjq3xm3q8hwnhbgk2xd3ph7lb6ay-glib-2.73.3.tar.xz

But how does Guix know how to build these store outputs in the first place? That's where derivations come in.

.drv Files

You've probably seen these being printed by the Guix program now and again. Derivations, represented in the daemon's eyes by .drv files, contain instructions for building store items. We can retrieve the paths of these .drv files with the guix build --derivations command:

$ guix build --derivations irssi
/gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv

guix build can actually also accept derivation paths as an argument, in lieu of a package, like so:

$ guix build /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3

Let's look inside this derivation file.

Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")],[("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv",["out"]),("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv",["out"]),("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv",["out"]),("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv",["out"]),("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv",["out"]),("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv",["out"]),("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv",["out"]),("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv",["out"])],["/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],"x86_64-linux","/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","-C","/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],[("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3"),("preferLocalBuild","1")])

It's... not exactly human-readable. We could try to format it and break it down, but it'd still be pretty hard to understand, since .drv files contain no labels for the fields or any other human-readable indicators. Instead, we're going to explore derivations in a Guile REPL.

Exploring Guix Interactively

Before we continue, we'll want to start a REPL, so that we can try out the Guix Guile API interactively. To run a REPL in the terminal, simply call guix repl.

If you're using Emacs, you can instead install Geiser, which provides a comfortable Emacs UI for various Lisp REPLs, invoke guix repl --listen=tcp:37146 &, and type M-x geiser-connect RET RET RET to connect to the running Guile instance.

Your .guile file may contain code for enabling colours and readline bindings that Geiser will choke on. The default Guix System .guile contains code to suppress these features when INSIDE_EMACS is set, so you'll need to run guix repl like this:

INSIDE_EMACS=1 guix repl --listen=tcp:37146 &

There are a few Guix modules we'll need. Run this Scheme code to import them:

(use-modules (guix)
             (guix derivations)
             (guix gexp)
             (guix packages)
             (guix store)
             (gnu packages glib)
             (gnu packages irc))

We now have access to the store, G-expression, package, and derivation APIs, along with the irssi and glib <package> objects.

Creating a <derivation>

The Guix API for derivations revolves around the <derivation> record, which is the Scheme representation of that whole block of text surrounded by Derive(...). If we look in guix/derivations.scm, we can see that it's defined like this:

(define-immutable-record-type <derivation>
  (make-derivation outputs inputs sources system builder args env-vars
                   file-name)
  derivation?
  (outputs  derivation-outputs)      ; list of name/<derivation-output> pairs
  (inputs   derivation-inputs)       ; list of <derivation-input>
  (sources  derivation-sources)      ; list of store paths
  (system   derivation-system)       ; string
  (builder  derivation-builder)      ; store path
  (args     derivation-builder-arguments)         ; list of strings
  (env-vars derivation-builder-environment-vars)  ; list of name/value pairs
  (file-name derivation-file-name))               ; the .drv file name

With the exception of file-name, each of those fields corresponds to a field in the Derive(...) form. Before we can examine them, though, we need to figure out how to lower that irssi <package> object into a derivation.

guix repl provides the ,lower command to create derivations quickly, as shown in this sample REPL session:

scheme@(guile-user)> ,use (guix)
scheme@(guile-user)> ,use (gnu packages irc)
scheme@(guile-user)> irssi
$1 = #<package irssi@1.4.3 gnu/packages/irc.scm:153 7f3ff98e0c60>
scheme@(guile-user)> ,lower irssi
$2 = #<derivation /gnu/store/drjfddvlblpr635jazrg9kn5azd9hsbj-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7f3ff7782d70>
;; Below we use the $N variable automatically bound by the REPL.
scheme@(guile-user)> (derivation-system $2)
$3 = "x86_64-linux"

Since ,lower is a REPL command, however, we can't use it in proper Scheme code. It's quite useful for exploring specific derivations interactively, but since the purpose of this blog post is to explain how things work inside, we're going to use the pure-Scheme approach here.

The procedure we need to use to turn a high-level object like <package> into a derivation is called lower-object; more on that in a future post. However, this doesn't initially produce a derivation:

(pk (lower-object irssi))
;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)

pk is an abbreviation for the procedure peek, which takes the given object, writes a representation of it to the output, and returns it. It's especially handy when you want to view an intermediate value in a complex expression.

The returned object is a monadic value (more on those in the next post on monads) that needs to be evaluated in the context of a store connection. We do this by first using with-store to connect to the store and bind the connection to a name, then wrapping the lower-object call with run-with-store:

(define irssi-drv
  (pk (with-store %store
        (run-with-store %store
          (lower-object irssi)))))
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)

(define glib-drv
  (pk (with-store %store
        (run-with-store %store
          (lower-object glib)))))
;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)

And we have liftoff! Now we've got two <derivation> records to play with.

Exploring <derivation>

<derivation-output>

The first "argument" in the .drv file is outputs, which tells the Guix daemon about the outputs that this build can produce:

(define irssi-outputs
  (pk (derivation-outputs irssi-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))

(pk (assoc-ref irssi-outputs "out"))

(define glib-outputs
  (pk (derivation-outputs glib-drv)))
;;; ((("bin" . #<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>) ("debug" . #<<derivation-output> path: "/gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug" hash-algo: #f hash: #f recursive?: #f>) ("out" . #<<derivation-output> path: "/gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2" hash-algo: #f hash: #f recursive?: #f>) ("static" . #<<derivation-output> path: "/gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static" hash-algo: #f hash: #f recursive?: #f>)))

(pk (assoc-ref glib-outputs "bin"))
;;; (#<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>)

It's a simple association list mapping output names to <derivation-output> records, and it's equivalent to the first "argument" in the .drv file:

[ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
]

The hash-algo and hash fields are for storing the content hash and the algorithm used with that hash for what we term a fixed-output derivation, which is essentially a derivation where we know what the hash of the content will be in advance. For instance, origins produce fixed-output derivations:

(define irssi-src-drv
  (pk (with-store %store
        (run-with-store %store
          (lower-object (package-source irssi))))))
;;; (#<derivation /gnu/store/mcz3vzq7lwwaqjb8dy7cd69lvmi6d241-irssi-1.4.3.tar.xz.drv => /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz 7fe17b3c8d70>)

(define irssi-src-outputs
  (pk (derivation-outputs irssi-src-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)))
  
(pk (assoc-ref irssi-src-outputs "out"))
;;; (#<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)

Note how the hash and hash-algo now have values.

Perceptive readers may note that the <derivation-output> has four fields, whereas the tuple in the .drv file only has three (minus the label). The serialisation of recursive? is done by adding the prefix r: to the hash-algo field, though its actual purpose is difficult to explain, and is out of scope for this post.

<derivation-input>

The next field is inputs, which corresponds to the second field in the .drv file format:

[ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
  ("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv", ["out"]),
  ("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv", ["out"]),
  ("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv", ["out"]),
  ("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv", ["out"]),
  ("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv", ["out"]),
  ("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv", ["out"]),
  ("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv", ["out"])
]

Here, each tuple specifies a derivation that needs to be built before this derivation can be built, and the outputs of the derivation that the build process of this derivation uses. Let's grab us the Scheme equivalent:

(define irssi-inputs
  (pk (derivation-inputs irssi-drv)))
;;; [a fairly large amount of output]

(pk (car irssi-inputs))
;;; (#<<derivation-input> drv: #<derivation /gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv => /gnu/store/2jj2mxn6wfrcw7i85nywk71mmqbnhzps-glib-2.70.2 7fe1902b6640> sub-derivations: ("out")>)

Unlike derivation-outputs, derivation-inputs maps 1:1 to the .drv form; the drv field is a <derivation> to be built, and the sub-derivations field is a list of outputs.

Builder Configuration

The other fields are simpler; none of them involve new records. The third is derivation-sources, which contains a list of all store items used in the build which aren't themselves built using derivations, whereas derivation-inputs contains the dependencies which are.

This list usually just contains the path to the Guile build script that realises the store items when run, which we'll examine in a later post, and the path to a directory containing extra modules to add to the build script's %load-path, called /gnu/store/...-module-import.

The next field is derivation-system, which specifies the system type (such as x86_64-linux) we're building for. Then we have derivation-builder, pointing to the guile executable that runs the build script; and the second-to-last is derivation-builder-arguments, which is a list of arguments to pass to derivation-builder. Note how we use -L and -C to extend the Guile %load-path and %load-compiled-path to include the module-import and module-import-compiled directories:

(pk (derivation-system irssi-drv))
;;; ("x86_64-linux")

(pk (derivation-builder irrsi-drv))
;;; ("/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile")

(pk (derivation-builder-arguments irrsi-drv))
;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))

The final field contains a list of environment variables to set before we start the build process:

(pk (derivation-builder-environment-vars irssi-drv))
;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))

The last record field, derivation-file-name contains the path to the .drv file, and so isn't represented in a serialised derivation.

Utilising <derivation>

Speaking of serialisation, to convert between the .drv text format and the Scheme <derivation> record, you can use write-derivation, read-derivation, and read-derivation-from-file:

(define manual-drv
  (with-store %store
    (derivation %store "manual"
                "/bin/sh" '())))

(write-derivation manual-drv (current-output-port))
;;; -| Derive([("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example","","")],[],[],"x86_64-linux","/bin/sh",[],[("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example")])

(pk (read-derivation-from-file (derivation-file-name irssi-drv)))
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb3798788c0>)

(call-with-input-file (derivation-file-name irssi-drv)
  read-derivation)
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb37ad19e10>)

You can realise <derivation>s as store items using the build-derivations procedure:

(use-modules (ice-9 ftw))

(define irssi-drv-out
  (pk (derivation-output-path
       (assoc-ref (derivation-outputs irssi-drv) "out"))))
;;; ("/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3")

(pk (scandir irssi-drv-out))
;;; (#f)

(pk (with-store %store (build-derivations %store (list irssi-drv))))
;;; (#t)

(pk (scandir irssi-drv-out))
;;; (("." ".." "bin" "etc" "include" "lib" "share"))

Conclusion

Derivations are one of Guix's most important concepts, but are fairly easy to understand once you get past the obtuse .drv file format. They provide the Guix daemon with the initial instructions that it uses to build store items like packages, origins, and other file-likes such as computed-file and local-file, which will be discussed in a future post!

To recap, a derivation contains the following fields:

  1. derivation-outputs, describing the various output paths that the derivation builds
  2. derivation-inputs, describing the other derivations that need to be built before this one is
  3. derivation-sources, listing the non-derivation store items that the derivation depends on
  4. derivation-system, specifying the system type a derivation will be compiled for
  5. derivation-builder, the executable to run the build script with
  6. derivation-builder-arguments, arguments to pass to the builder
  7. derivation-builder-environment-vars, variables to set in the builder's environment

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

04 January, 2023 12:30PM by (

January 02, 2023

www-zh-cn @ Savannah

Summary of 2022

Dear CTT team:

Thank you very much for you contribution in 2022. There have been a few new comers joining our team, like Wind. Their work is appreciated.

The whole team have been doing a good job in 2022. Here is official letter from GNU.

Dear GNU translators!

This year was relatively quiet; the total number of new translations
was considerably lower than in 2021, especially in terms of size.

Almost two of every three translations were made in the "Simplified"
Chinese team; the Albanian and Turkish teams significantly reduced
the percentage of their outdated translations.

      General Statistics

In November, we reached new maximum values of translations per file
in important directories, 9.44 translations per file (0.12 more
than in 2021) and 8.85 translations weighted with size of articles
(0.46 more than in 2021), 50 Mi in 3300 files total.

Meanwhile, the percent of outdated translations was as high
as at the end of 2021 (about twice as high as the historical
minimum in 2014-2015).

The table below shows the number and size of newly translated
articles in important directories and typical number of outdated
GNUNified translations throughout the year.

+--team--+------new-------+--outdated--+
|   es   |  11 ( 70.9Ki)  |  1.9 (1%)  |
+--------+----------------+------------+
|   fa   |   4 ( 41.3Ki)  |  26 (90%)  |
+--------+----------------+------------+
|   fr   |   9 (103.2Ki)  | 0.8 (0.2%) |
+--------+----------------+------------+
|   it   |   0 (  0.0Ki)  |  32 (23%)  |
+--------+----------------+------------+
|   ja   |   0 (  0.0Ki)  |  32 (23%)  |
+--------+----------------+------------+
|   ml   |   0 (  0.0Ki)  |  31 (93%)  |
+--------+----------------+------------+
|   nl   |   0 (  0.0Ki)  |  52 (41%)  |
+--------+----------------+------------+
|   pl   |   1 (183.5Ki)  |  64 (43%)  |
+--------+----------------+------------+
|  pt-br |   0 (  0.0Ki)  | 114 (53%)  |
+--------+----------------+------------+
|   ru   |   4 ( 62.0Ki)  | 0.9 (0.3%) |
+--------+----------------+------------+
|   sq   |   7 ( 81.3Ki)  | 1.8 (2.4%) |
+--------+----------------+------------+
|   tr   |   1 (  5.6Ki)  | 2.3 (1.8%) |
+--------+----------------+------------+
|   uk   |   5 ( 64.0Ki)  |  78 (79%)  |
+--------+----------------+------------+
|  zh-cn |  66 (532.8Ki)  | 1.4 (0.9%) |
+--------+----------------+------------+
|  zh-tw |   0 (  0.0Ki)  |  45 (98%)  |
+--------+----------------+------------+
+--------+----------------+
| total  | 108 (1144.7Ki) |
+--------+----------------+

For the reference: 8 new articles were added, amounting to 72Ki,
and there were about 500 changes in about 150 English files
in the important directories, approximately two times less than
in 2021.

Happy Hacking
wxie

02 January, 2023 02:32AM by Wensheng XIE

December 30, 2022

Simon Josefsson

Preseeding Trisquel Virtual Machines Using “netinst” Images

I’m migrating some self-hosted virtual machines to Trisquel, and noticed that Trisquel does not offer cloud-images similar to the Debian Cloud and Ubuntu Cloud images. Thus my earlier approach based on virt-install --cloud-init and cloud-localds does not work with Trisquel. While I hope that Trisquel will eventually publish cloud-compatible images, I wanted to document an alternative approach for Trisquel based on preseeding. This is how I used to install Debian and Ubuntu in the old days, and the automated preseed method is best documented in the Debian installation manual. I was hoping to forget about the preseed format, but maybe it will become one of those legacy technologies that never really disappears? Like FAT16 and 8-bit microcontrollers.

Below I assume you have a virtual machine host server up that runs libvirt and has virt-install and similar tools; install them with the following command. I run a pre-release version of Trisquel 11 aramo on my VM-host, but I believe any recent dpkg-based distribution like Trisquel 9/10, PureOS 10, Debian 11 or Ubuntu 20.04/22.04 would work.

apt-get install libvirt-daemon-system virtinst genisoimage cloud-image-utils osinfo-db-tools

The approach can install Trisquel 9 (etiona), Trisquel 10 (nabia) and the pre-release of Trisquel 11. First download and verify the integrity of the netinst images that we will need. Unfortunately the Trisquel 11 netinst beta image does not have any checksum or signature available.

mkdir -p /root/iso
cd /root/iso
wget -q https://mirror.fsf.org/trisquel-images/trisquel-netinst_9.0.2_amd64.iso
wget -q https://mirror.fsf.org/trisquel-images/trisquel-netinst_9.0.2_amd64.iso.asc
wget -q https://mirror.fsf.org/trisquel-images/trisquel-netinst_9.0.2_amd64.iso.sha256
wget -q https://mirror.fsf.org/trisquel-images/trisquel-netinst_10.0.1_amd64.iso
wget -q https://mirror.fsf.org/trisquel-images/trisquel-netinst_10.0.1_amd64.iso.asc
wget -q https://mirror.fsf.org/trisquel-images/trisquel-netinst_10.0.1_amd64.iso.sha256
wget -q -O- https://archive.trisquel.info/trisquel/trisquel-archive-signkey.gpg | gpg --import
sha256sum -c trisquel-netinst_9.0.2_amd64.iso.sha256
gpg --verify trisquel-netinst_9.0.2_amd64.iso.asc
sha256sum -c trisquel-netinst_10.0.1_amd64.iso.sha256
gpg --verify trisquel-netinst_10.0.1_amd64.iso.asc
wget -q https://cdbuilds.trisquel.org/aramo/trisquel-netinst_11.0-20221225_amd64.iso
echo '179566639ca8f14f0c3d5658209c59a0916d9e3bf9c026660cc07b28f2311631  trisquel-netinst_11.0-20221225_amd64.iso' | sha256sum -c

I have developed the following fairly minimal preseed file that works with all three Trisquel releases. Compare it against the official Trisquel 11 preseed skeleton and the Debian 11 example preseed file. You should modify obvious things like SSH key, host/IP settings, partition layout and decide for yourself how to deal with passwords. While Ubuntu/Trisquel usually wants to setup a user account, I prefer to login as root hence setting ‘passwd/root-login‘ to true and ‘passwd/make-user‘ to false.


root@trana:~# cat>trisquel.preseed 
d-i debian-installer/locale select en_US
d-i keyboard-configuration/xkb-keymap select us

d-i netcfg/choose_interface select auto
d-i netcfg/disable_autoconfig boolean true

d-i netcfg/get_ipaddress string 192.168.10.201
d-i netcfg/get_netmask string 255.255.255.0
d-i netcfg/get_gateway string 192.168.10.46
d-i netcfg/get_nameservers string 192.168.10.46

d-i netcfg/get_hostname string trisquel
d-i netcfg/get_domain string sjd.se

d-i clock-setup/utc boolean true
d-i time/zone string UTC

d-i mirror/country string manual
d-i mirror/http/hostname string ftp.acc.umu.se
d-i mirror/http/directory string /mirror/trisquel/packages
d-i mirror/http/proxy string

d-i partman-auto/method string regular
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-basicfilesystems/no_swap boolean false
d-i partman-auto/expert_recipe string myroot :: 1000 50 -1 ext4 \
     $primary{ } $bootable{ } method{ format } \
     format{ } use_filesystem{ } filesystem{ ext4 } \
     mountpoint{ / } \
    .
d-i partman-auto/choose_recipe select myroot

d-i passwd/root-login boolean true
d-i user-setup/allow-password-weak boolean true
d-i passwd/root-password password r00tme
d-i passwd/root-password-again password r00tme
d-i passwd/make-user boolean false

tasksel tasksel/first multiselect
d-i pkgsel/include string openssh-server

popularity-contest popularity-contest/participate boolean false

d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i grub-installer/bootdev string default

d-i finish-install/reboot_in_progress note

d-i preseed/late_command string mkdir /target/root/.ssh ; echo ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILzCFcHHrKzVSPDDarZPYqn89H5TPaxwcORgRg+4DagE cardno:FFFE67252015 > /target/root/.ssh/authorized_keys
^D
root@trana:~# 

Use the file above as a skeleton for preparing a VM-specific preseed file as follows. The environment variables HOST and IPS will be used later on too.


root@trana:~# HOST=foo
root@trana:~# IP=192.168.10.197
root@trana:~# sed -e "s,get_ipaddress string.*,get_ipaddress string $IP," -e "s,get_hostname string.*,get_hostname string $HOST," < trisquel.preseed > vm-$HOST.preseed
root@trana:~# 

The following script is used to prepare the ISO images with the preseed file that we will need. This script is inspired by the Debian Wiki Preseed EditIso page and the Trisquel ISO customization wiki page. There are a couple of variations based on earlier works. Paths are updated to match the Trisquel netinst ISO layout, which differ slightly from Debian. We modify isolinux.cfg to boot the auto label without a timeout. On Trisquel 11 the auto boot label exists, but on Trisquel 9 and Trisquel 10 it does not exist so we add it in order to be able to start the automated preseed installation.


root@trana:~# cat gen-preseed-iso 
#!/bin/sh

# Copyright (C) 2018-2022 Simon Josefsson -- GPLv3+
# https://wiki.debian.org/DebianInstaller/Preseed/EditIso
# https://trisquel.info/en/wiki/customizing-trisquel-iso

set -e
set -x

ISO="$1"
PRESEED="$2"
OUTISO="$3"
LASTPWD="$PWD"

test -f "$ISO"
test -f "$PRESEED"
test ! -f "$OUTISO"

TMPDIR=$(mktemp -d)
mkdir "$TMPDIR/mnt"
mkdir "$TMPDIR/tmp"

cp "$PRESEED" "$TMPDIR"/preseed.cfg
cd "$TMPDIR"

mount "$ISO" mnt/
cp -rT mnt/ tmp/
umount mnt/

chmod +w -R tmp/
gunzip tmp/initrd.gz
echo preseed.cfg | cpio -H newc -o -A -F tmp/initrd
gzip tmp/initrd
chmod -w -R tmp/

sed -i "s/timeout 0/timeout 1/" tmp/isolinux.cfg
sed -i "s/default vesamenu.c32/default auto/" tmp/isolinux.cfg

if ! grep -q auto tmp/adtxt.cfg; then
    cat<<EOF >> tmp/adtxt.cfg
label auto
	menu label ^Automated install
	kernel linux
	append auto=true priority=critical vga=788 initrd=initrd.gz --- quiet
EOF
fi

cd tmp/
find -follow -type f | xargs md5sum  > md5sum.txt
cd ..

cd "$LASTPWD"

genisoimage -r -J -b isolinux.bin -c boot.cat \
            -no-emul-boot -boot-load-size 4 -boot-info-table \
            -o "$OUTISO" "$TMPDIR/tmp/"

rm -rf "$TMPDIR"

exit 0
^D
root@trana:~# chmod +x gen-preseed-iso 
root@trana:~# 

Next run the command on one of the downloaded ISO image and the generated preseed file.


root@trana:~# ./gen-preseed-iso /root/iso/trisquel-netinst_10.0.1_amd64.iso vm-$HOST.preseed vm-$HOST.iso
+ ISO=/root/iso/trisquel-netinst_10.0.1_amd64.iso
+ PRESEED=vm-foo.preseed
+ OUTISO=vm-foo.iso
+ LASTPWD=/root
+ test -f /root/iso/trisquel-netinst_10.0.1_amd64.iso
+ test -f vm-foo.preseed
+ test ! -f vm-foo.iso
+ mktemp -d
+ TMPDIR=/tmp/tmp.mNEprT4Tx9
+ mkdir /tmp/tmp.mNEprT4Tx9/mnt
+ mkdir /tmp/tmp.mNEprT4Tx9/tmp
+ cp vm-foo.preseed /tmp/tmp.mNEprT4Tx9/preseed.cfg
+ cd /tmp/tmp.mNEprT4Tx9
+ mount /root/iso/trisquel-netinst_10.0.1_amd64.iso mnt/
mount: /tmp/tmp.mNEprT4Tx9/mnt: WARNING: source write-protected, mounted read-only.
+ cp -rT mnt/ tmp/
+ umount mnt/
+ chmod +w -R tmp/
+ gunzip tmp/initrd.gz
+ echo preseed.cfg
+ cpio -H newc -o -A -F tmp/initrd
5 blocks
+ gzip tmp/initrd
+ chmod -w -R tmp/
+ sed -i s/timeout 0/timeout 1/ tmp/isolinux.cfg
+ sed -i s/default vesamenu.c32/default auto/ tmp/isolinux.cfg
+ grep -q auto tmp/adtxt.cfg
+ cat
+ cd tmp/
+ find -follow -type f
+ xargs md5sum
+ cd ..
+ cd /root
+ genisoimage -r -J -b isolinux.bin -c boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -o vm-foo.iso /tmp/tmp.mNEprT4Tx9/tmp/
I: -input-charset not specified, using utf-8 (detected in locale settings)
Using GCRY_000.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/gcry_sha512.mod (gcry_sha256.mod)
Using XNU_U000.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/xnu_uuid.mod (xnu_uuid_test.mod)
Using PASSW000.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/password_pbkdf2.mod (password.mod)
Using PART_000.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/part_sunpc.mod (part_sun.mod)
Using USBSE000.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/usbserial_pl2303.mod (usbserial_ftdi.mod)
Using USBSE001.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/usbserial_ftdi.mod (usbserial_usbdebug.mod)
Using VIDEO000.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/videotest.mod (videotest_checksum.mod)
Using GFXTE000.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/gfxterm_background.mod (gfxterm_menu.mod)
Using GCRY_001.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/gcry_sha256.mod (gcry_sha1.mod)
Using MULTI000.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/multiboot2.mod (multiboot.mod)
Using USBSE002.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/usbserial_usbdebug.mod (usbserial_common.mod)
Using MDRAI000.MOD;1 for  /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86_64-efi/mdraid09.mod (mdraid09_be.mod)
Size of boot image is 4 sectors -> No emulation
 22.89% done, estimate finish Thu Dec 29 23:36:18 2022
 45.70% done, estimate finish Thu Dec 29 23:36:18 2022
 68.56% done, estimate finish Thu Dec 29 23:36:18 2022
 91.45% done, estimate finish Thu Dec 29 23:36:18 2022
Total translation table size: 2048
Total rockridge attributes bytes: 24816
Total directory bytes: 40960
Path table size(bytes): 64
Max brk space used 46000
21885 extents written (42 MB)
+ rm -rf /tmp/tmp.mNEprT4Tx9
+ exit 0
root@trana:~#

Now the image is ready for installation, so invoke virt-install as follows. The machine will start directly, launching the preseed automatic installation. At this point, I usually click on the virtual machine in virt-manager to follow screen output until the installation has finished. If everything works OK the machines comes up and I can ssh into it.


root@trana:~# virt-install --name $HOST --disk vm-$HOST.img,size=5 --cdrom vm-$HOST.iso --osinfo linux2020 --autostart --noautoconsole --wait
Using linux2020 default --memory 4096

Starting install...
Allocating 'vm-foo.img'                                                                                                                                |    0 B  00:00:00 ... 
Creating domain...                                                                                                                                     |    0 B  00:00:00     

Domain is still running. Installation may be in progress.
Waiting for the installation to complete.
Domain has shutdown. Continuing.
Domain creation completed.
Restarting guest.
root@trana:~# 

There are some problems that I have noticed that would be nice to fix, but are easy to work around. The first is that at the end of the installation of Trisquel 9 and Trisquel 10, the VM hangs after displaying Sent SIGKILL to all processes followed by Requesting system reboot. I kill the VM manually using virsh destroy foo and start it up again using virsh start foo. For production use I expect to be running Trisquel 11, where the problem doesn’t happen, so this does not bother me enough to debug further. The remaining issue that once booted, a Trisquel 11 VM has lost its DNS nameserver configuration, presumably due to poor integration with systemd-resolved. Both Trisquel 9 and Trisquel 10 uses systemd-resolved where DNS works after first boot, so this appears to be a Trisquel 11 bug. You can work around it with rm -f /etc/resolv.conf && echo 'nameserver A.B.C.D' > /etc/resolv.conf or drink the systemd Kool-Aid. If you want to clean up and re-start the process, here is how you wipe out what you did. After this, you may run the sed, ./gen-preseed-iso and virt-install commands again. Remember, use virsh shutdown foo to gracefully shutdown a VM.


root@trana:~# virsh destroy foo
Domain 'foo' destroyed

root@trana:~# virsh undefine foo --remove-all-storage
Domain 'foo' has been undefined
Volume 'vda'(/root/vm-foo.img) removed.

root@trana:~# rm vm-foo.*
root@trana:~# 

Happy hacking on your virtal machines!

30 December, 2022 12:24AM by simon

December 28, 2022

GNUnet News

GNUnet 0.19.1

28 December, 2022 11:00PM

December 27, 2022

remotecontrol @ Savannah

The Christmas Electric Grid Emergency

https://www.wsj.com/articles/the-christmas-electric-grid-emergency-11672091317

Surrendering control of an HVAC thermostat to an electricity company enables them to change the temperature settings as they see fit.

Maintaining personal control of an HVAC thermostat is a wise choice.

27 December, 2022 03:09PM by Stephen H. Dawson DSL

December 24, 2022

Simon Josefsson

OpenPGP key on FST-01SZ

I use GnuPG to compute cryptographic signatures for my emails, git commits/tags, and software release artifacts (tarballs). Part of GnuPG is gpg-agent which talks to OpenSSH, which I login to remote servers and to clone git repositories. I dislike storing cryptographic keys on general-purpose machines, and have used hardware-backed OpenPGP keys since around 2006 when I got a FSFE Fellowship Card. GnuPG via gpg-agent handles this well, and the private key never leaves the hardware. The ZeitControl cards were (to my knowledge) proprietary hardware running some non-free operating system and OpenPGP implementation. By late 2012 the YubiKey NEO supported OpenPGP, and while the hardware and operating system on it was not free, at least it ran a free software OpenPGP implementation and eventually I setup my primary RSA key on it. This worked well for a couple of years, and when I in 2019 wished to migrate to a new key, the FST-01G device with open hardware running free software that supported Ed25519 had become available. I created a key and have been using the FST-01G on my main laptop since then. This little device has been working, the signature counter on it is around 14501 which means around 10 signatures/day since then!

Currently I am in the process of migrating towards a new laptop, and moving the FST-01G device between them is cumbersome, especially if I want to use both laptops in parallel. That’s why I need to setup a new hardware device to hold my OpenPGP key, which can go with my new laptop. This is a good time to re-visit alternatives. I quickly decided that I did not want to create a new key, only to import my current one to keep everything working. My requirements on the device to chose hasn’t changed since 2019, see my summary at the end of the earlier blog post. Unfortunately the FST-01G is out of stock and the newer FST-01SZ has also out of stock. While Tillitis looks promising (and I have one to play with), it does not support OpenPGP (yet). What to do? Fortunately, I found some FST-01SZ device in my drawer, and decided to use it pending a more satisfactory answer. Hopefully once I get around to generate a new OpenPGP key in a year or so, I will do a better survey of options that are available on the market then. What are your (freedom-respecting) OpenPGP hardware recommendations?

FST-01SZ circuit board

Similar to setting up the FST-01G, the FST-01SZ needs to be setup before use. I’m doing the following from Trisquel 11 but any GNU/Linux system would work. When the device is inserted at first time, some kernel messages are shown (see /var/log/syslog or use the dmesg command):


usb 3-3: new full-speed USB device number 39 using xhci_hcd
usb 3-3: New USB device found, idVendor=234b, idProduct=0004, bcdDevice= 2.00
usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 3-3: Product: Fraucheky
usb 3-3: Manufacturer: Free Software Initiative of Japan
usb 3-3: SerialNumber: FSIJ-0.0
usb-storage 3-3:1.0: USB Mass Storage device detected
scsi host1: usb-storage 3-3:1.0
scsi 1:0:0:0: Direct-Access     FSIJ     Fraucheky        1.0  PQ: 0 ANSI: 0
sd 1:0:0:0: Attached scsi generic sg2 type 0
sd 1:0:0:0: [sdc] 128 512-byte logical blocks: (65.5 kB/64.0 KiB)
sd 1:0:0:0: [sdc] Write Protect is off
sd 1:0:0:0: [sdc] Mode Sense: 03 00 00 00
sd 1:0:0:0: [sdc] No Caching mode page found
sd 1:0:0:0: [sdc] Assuming drive cache: write through
 sdc:
sd 1:0:0:0: [sdc] Attached SCSI removable disk

Interestingly, the NeuG software installed on the device I got appears to be version 1.0.9:


jas@kaka:~$ head /media/jas/Fraucheky/README
NeuG - a true random number generator implementation
						  Version 1.0.9
						     2018-11-20
					           Niibe Yutaka
			      Free Software Initiative of Japan
What's NeuG?
============
jas@kaka:~$ 

I could not find version 1.0.9 published anywhere, but the device came with a SD-card that contain a copy of the source, so I uploaded it until a more canonical place is located. Putting the device in the serial mode can be done using a sudo eject /dev/sdc command which results in the following syslog output.


usb 3-3: reset full-speed USB device number 39 using xhci_hcd
usb 3-3: device firmware changed
usb 3-3: USB disconnect, device number 39
sdc: detected capacity change from 128 to 0
usb 3-3: new full-speed USB device number 40 using xhci_hcd
usb 3-3: New USB device found, idVendor=234b, idProduct=0001, bcdDevice= 2.00
usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 3-3: Product: NeuG True RNG
usb 3-3: Manufacturer: Free Software Initiative of Japan
usb 3-3: SerialNumber: FSIJ-1.0.9-42315277
cdc_acm 3-3:1.0: ttyACM0: USB ACM device

Now download Gnuk, verify its integrity and build it. You may need some additional packages installed, try apt-get install gcc-arm-none-eabi openocd python3-usb. As you can see, I’m using the stable 1.2 branch of Gnuk, currently on version 1.2.20. The ./configure parameters deserve some explanation. The kdf_do=required sets up the device to require KDF usage. The --enable-factory-reset allows me to use the command factory-reset (with admin PIN) inside gpg --card-edit to completely wipe the card. Some may consider that too dangerous, but my view is that if someone has your admin PIN it is game over anyway. The --vidpid=234b:0000 is specifies the USB VID/PID to use, and --target=FST_01SZ is critical to set the platform (you’ll may brick the device if you pick the wrong --target setting).


jas@kaka:~/src$ rm -rf gnuk neug
jas@kaka:~/src$ git clone https://gitlab.com/jas/neug.git
Cloning into 'neug'...
remote: Enumerating objects: 2034, done.
remote: Counting objects: 100% (2034/2034), done.
remote: Compressing objects: 100% (603/603), done.
remote: Total 2034 (delta 1405), reused 2013 (delta 1405), pack-reused 0
Receiving objects: 100% (2034/2034), 910.34 KiB | 3.50 MiB/s, done.
Resolving deltas: 100% (1405/1405), done.
jas@kaka:~/src$ git clone https://salsa.debian.org/gnuk-team/gnuk/gnuk.git
Cloning into 'gnuk'...
remote: Enumerating objects: 13765, done.
remote: Counting objects: 100% (959/959), done.
remote: Compressing objects: 100% (337/337), done.
remote: Total 13765 (delta 629), reused 907 (delta 599), pack-reused 12806
Receiving objects: 100% (13765/13765), 12.59 MiB | 3.05 MiB/s, done.
Resolving deltas: 100% (10077/10077), done.
jas@kaka:~/src$ cd neug
jas@kaka:~/src/neug$ git describe 
release/1.0.9
jas@kaka:~/src/neug$ git tag -v `git describe`
object 5d51022a97a5b7358d0ea62bbbc00628c6cec06a
type commit
tag release/1.0.9
tagger NIIBE Yutaka <gniibe@fsij.org> 1542701768 +0900

Version 1.0.9.
gpg: Signature made Tue Nov 20 09:16:08 2018 CET
gpg:                using EDDSA key 249CB3771750745D5CDD323CE267B052364F028D
gpg:                issuer "gniibe@fsij.org"
gpg: Good signature from "NIIBE Yutaka <gniibe@fsij.org>" [unknown]
gpg:                 aka "NIIBE Yutaka <gniibe@debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 249C B377 1750 745D 5CDD  323C E267 B052 364F 028D
jas@kaka:~/src/neug$ cd ../gnuk/
jas@kaka:~/src/gnuk$ git checkout STABLE-BRANCH-1-2 
Branch 'STABLE-BRANCH-1-2' set up to track remote branch 'STABLE-BRANCH-1-2' from 'origin'.
Switched to a new branch 'STABLE-BRANCH-1-2'
jas@kaka:~/src/gnuk$ git describe
release/1.2.20
jas@kaka:~/src/gnuk$ git tag -v `git describe`
object 9d3c08bd2beb73ce942b016d4328f0a596096c02
type commit
tag release/1.2.20
tagger NIIBE Yutaka <gniibe@fsij.org> 1650594032 +0900

Gnuk: Version 1.2.20
gpg: Signature made Fri Apr 22 04:20:32 2022 CEST
gpg:                using EDDSA key 249CB3771750745D5CDD323CE267B052364F028D
gpg: Good signature from "NIIBE Yutaka <gniibe@fsij.org>" [unknown]
gpg:                 aka "NIIBE Yutaka <gniibe@debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 249C B377 1750 745D 5CDD  323C E267 B052 364F 028D
jas@kaka:~/src/gnuk/src$ git submodule update --init
Submodule 'chopstx' (https://salsa.debian.org/gnuk-team/chopstx/chopstx.git) registered for path '../chopstx'
Cloning into '/home/jas/src/gnuk/chopstx'...
Submodule path '../chopstx': checked out 'e12a7e0bb3f004c7bca41cfdb24c8b66daf3db89'
jas@kaka:~/src/gnuk$ cd chopstx
jas@kaka:~/src/gnuk/chopstx$ git describe
release/1.21
jas@kaka:~/src/gnuk/chopstx$ git tag -v `git describe`
object e12a7e0bb3f004c7bca41cfdb24c8b66daf3db89
type commit
tag release/1.21
tagger NIIBE Yutaka <gniibe@fsij.org> 1650593697 +0900

Chopstx: Version 1.21
gpg: Signature made Fri Apr 22 04:14:57 2022 CEST
gpg:                using EDDSA key 249CB3771750745D5CDD323CE267B052364F028D
gpg: Good signature from "NIIBE Yutaka <gniibe@fsij.org>" [unknown]
gpg:                 aka "NIIBE Yutaka <gniibe@debian.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 249C B377 1750 745D 5CDD  323C E267 B052 364F 028D
jas@kaka:~/src/gnuk/chopstx$ cd ../src
jas@kaka:~/src/gnuk/src$ kdf_do=required ./configure --enable-factory-reset --vidpid=234b:0000 --target=FST_01SZ
Header file is: board-fst-01sz.h
Debug option disabled
Configured for bare system (no-DFU)
PIN pad option disabled
CERT.3 Data Object is NOT supported
Card insert/removal by HID device is NOT supported
Life cycle management is supported
Acknowledge button is supported
KDF DO is required before key import/generation
jas@kaka:~/src/gnuk/src$ make | less
jas@kaka:~/src/gnuk/src$ cd ../regnual/
jas@kaka:~/src/gnuk/regnual$ make | less
jas@kaka:~/src/gnuk/regnual$ cd ../../
jas@kaka:~/src$ sudo python3 neug/tool/neug_upgrade.py -f gnuk/regnual/regnual.bin gnuk/src/build/gnuk.bin
gnuk/regnual/regnual.bin: 4608
gnuk/src/build/gnuk.bin: 109568
CRC32: b93ca829

Device: 
Configuration: 1
Interface: 1
20000e00:20005000
Downloading flash upgrade program...
start 20000e00
end   20002000
# 20002000: 32 : 4
Run flash upgrade program...
Wait 1 second...
Wait 1 second...
Device: 
08001000:08020000
Downloading the program
start 08001000
end   0801ac00
jas@kaka:~/src$ 

The kernel log will contain the following, and the card is ready to use as an OpenPGP card. You may unplug it and re-insert it as you wish.


usb 3-3: reset full-speed USB device number 41 using xhci_hcd
usb 3-3: device firmware changed
usb 3-3: USB disconnect, device number 41
usb 3-3: new full-speed USB device number 42 using xhci_hcd
usb 3-3: New USB device found, idVendor=234b, idProduct=0000, bcdDevice= 2.00
usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 3-3: Product: Gnuk Token
usb 3-3: Manufacturer: Free Software Initiative of Japan
usb 3-3: SerialNumber: FSIJ-1.2.20-42315277

Setting up the card is the next step, and there are many tutorials around for this, eventually I settled with the following sequence. Let’s start with setting the admin PIN. First make sure that pcscd nor scdaemon is running, which is good hygien since those processes cache some information and with a stale connection this easily leads to confusion. Cache invalidation… sigh.


jas@kaka:~$ gpg-connect-agent "SCD KILLSCD" "SCD BYE" /bye
jas@kaka:~$ ps auxww|grep -e pcsc -e scd
jas        30221  0.0  0.0   3468  1692 pts/3    R+   11:49   0:00 grep --color=auto -e pcsc -e scd
jas@kaka:~$ gpg --card-edit

Reader ...........: 234B:0000:FSIJ-1.2.20-42315277:0
Application ID ...: D276000124010200FFFE423152770000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 42315277
Name of cardholder: [not set]
Language prefs ...: [not set]
Salutation .......: 
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: off
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

gpg/card> admin
Admin commands are allowed

gpg/card> kdf-setup

gpg/card> passwd
gpg: OpenPGP card no. D276000124010200FFFE423152770000 detected

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 3
PIN changed.

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 

Now it would be natural to setup the PIN and reset code. However the Gnuk software is configured to not allow this until the keys are imported. You would get the following somewhat cryptical error messages if you try. This took me a while to understand, since this is device-specific, and some other OpenPGP implementations allows you to configure a PIN and reset code before key import.


Your selection? 4
Error setting the Reset Code: Card error

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 1
Error changing the PIN: Conditions of use not satisfied

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? q

Continue to configure the card and make it ready for key import. Some settings deserve comments. The lang field may be used to setup the language, but I have rarely seen it use, and I set it to ‘sv‘ (Swedish) mostly to be able to experiment if any software adhears to it. The URL is important to point to somewhere where your public key is stored, the fetch command of gpg --card-edit downloads it and sets up GnuPG with it when you are on a clean new laptop. The forcesig command changes the default so that a PIN code is not required for every digital signature operation, remember that I averaged 10 signatures per day for the past 2-3 years? Think of the wasted energy typing those PIN codes every time! Changing the cryptographic key type is required when I import 25519-based keys.


gpg/card> name
Cardholder's surname: Josefsson
Cardholder's given name: Simon

gpg/card> lang
Language preferences: sv

gpg/card> sex
Salutation (M = Mr., F = Ms., or space): m

gpg/card> login
Login data (account name): jas

gpg/card> url
URL to retrieve public key: https://josefsson.org/key-20190320.txt

gpg/card> forcesig

gpg/card> key-attr
Changing card key attribute for: Signature key
Please select what kind of key you want:
   (1) RSA
   (2) ECC
Your selection? 2
Please select which elliptic curve you want:
   (1) Curve 25519
   (4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519
Note: There is no guarantee that the card supports the requested size.
      If the key generation does not succeed, please check the
      documentation of your card to see what sizes are allowed.
Changing card key attribute for: Encryption key
Please select what kind of key you want:
   (1) RSA
   (2) ECC
Your selection? 2
Please select which elliptic curve you want:
   (1) Curve 25519
   (4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: cv25519
Changing card key attribute for: Authentication key
Please select what kind of key you want:
   (1) RSA
   (2) ECC
Your selection? 2
Please select which elliptic curve you want:
   (1) Curve 25519
   (4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519

gpg/card> 

Reader ...........: 234B:0000:FSIJ-1.2.20-42315277:0
Application ID ...: D276000124010200FFFE423152770000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 42315277
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: on
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

gpg/card> 

The device is now ready for key import! Bring out your offline laptop and boot it and use the keytocard command on the subkeys to import them. This assumes you saved a copy of the GnuPG home directory after generating the master and subkeys before, which I did in my own previous tutorial when I generated the keys. This may be a bit unusual, and there are simpler ways to do this (e.g., import a copy of the secret keys into a fresh GnuPG home directory).


$ cp -a gnupghome-backup-mastersubkeys gnupghome-import-fst01sz-42315277-2022-12-24
$ ps auxww|grep -e pcsc -e scd
$ gpg --homedir $PWD/gnupghome-import-fst01sz-42315277-2022-12-24 --edit-key B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE
...
Secret key is available.

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 1

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb* cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb* cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 1

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 2

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb* ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb* ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 2

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb  ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> key 3

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb* ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> keytocard
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1

sec  ed25519/D73CF638C53C06BE
     created: 2019-03-20  expired: 2019-10-22  usage: SC  
     trust: ultimate      validity: expired
ssb  cv25519/02923D7EE76EBD60
     created: 2019-03-20  expired: 2019-10-22  usage: E   
ssb  ed25519/80260EE8A9B92B2B
     created: 2019-03-20  expired: 2019-10-22  usage: A   
ssb* ed25519/51722B08FE4745A2
     created: 2019-03-20  expired: 2019-10-22  usage: S   
[ expired] (1). Simon Josefsson <simon@josefsson.org>

gpg> quit
Save changes? (y/N) y
$ 

Now insert it into your daily laptop and have GnuPG and learn about the new private keys and forget about any earlier locally available card bindings — this usually manifests itself by GnuPG asking you to insert a OpenPGP card with another serial number. Earlier I did rm -rf ~/.gnupg/private-keys-v1.d/ but the scd serialno followed by learn --force is nicer. I also sets up trust setting for my own key.


jas@kaka:~$ gpg-connect-agent "scd serialno" "learn --force" /bye
...
jas@kaka:~$ echo "B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE:6:" | gpg --import-ownertrust
jas@kaka:~$ gpg --card-status
Reader ...........: 234B:0000:FSIJ-1.2.20-42315277:0
Application ID ...: D276000124010200FFFE423152770000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 42315277
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 5 5 5
Signature counter : 3
KDF setting ......: on
Signature key ....: A3CC 9C87 0B9D 310A BAD4  CF2F 5172 2B08 FE47 45A2
      created ....: 2019-03-20 23:40:49
Encryption key....: A9EC 8F4D 7F1E 50ED 3DEF  49A9 0292 3D7E E76E BD60
      created ....: 2019-03-20 23:40:26
Authentication key: CA7E 3716 4342 DF31 33DF  3497 8026 0EE8 A9B9 2B2B
      created ....: 2019-03-20 23:40:37
General key info..: sub  ed25519/51722B08FE4745A2 2019-03-20 Simon Josefsson <simon@josefsson.org>
sec#  ed25519/D73CF638C53C06BE  created: 2019-03-20  expires: 2023-09-19
ssb>  ed25519/80260EE8A9B92B2B  created: 2019-03-20  expires: 2023-09-19
                                card-no: FFFE 42315277
ssb>  ed25519/51722B08FE4745A2  created: 2019-03-20  expires: 2023-09-19
                                card-no: FFFE 42315277
ssb>  cv25519/02923D7EE76EBD60  created: 2019-03-20  expires: 2023-09-19
                                card-no: FFFE 42315277
jas@kaka:~$ 

Verify that you can digitally sign and authenticate using the key and you are done!


jas@kaka:~$ echo foo|gpg -a --sign|gpg --verify
gpg: Signature made Sat Dec 24 13:49:59 2022 CET
gpg:                using EDDSA key A3CC9C870B9D310ABAD4CF2F51722B08FE4745A2
gpg: Good signature from "Simon Josefsson <simon@josefsson.org>" [ultimate]
jas@kaka:~$ ssh-add -L
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILzCFcHHrKzVSPDDarZPYqn89H5TPaxwcORgRg+4DagE cardno:FFFE42315277
jas@kaka:~$ 

So time to relax and celebrate christmas? Hold on… not so fast! Astute readers will have noticed that the output said ‘PIN retry counter: 5 5 5‘. That’s not the default PIN retry counter for Gnuk! How did that happen? Indeed, good catch and great question, my dear reader. I wanted to include how you can modify the Gnuk source code, re-build it and re-flash the Gnuk as well. This method is different than flashing Gnuk onto a device that is running NeuG so the commands I used to flash the firmware in the start of this blog post no longer works in a device running Gnuk. Fortunately modern Gnuk supports updating firmware by specifying the Admin PIN code only, and provides a simple script to achieve this as well. The PIN retry counter setting is hard coded in the openpgp-do.c file, and we run a a perl command to modify the file, rebuild Gnuk and upgrade the FST-01SZ. This of course wipes all your settings, so you will have the opportunity to practice all the commands earlier in this post once again!


jas@kaka:~/src/gnuk/src$ perl -pi -e 's/PASSWORD_ERRORS_MAX 3/PASSWORD_ERRORS_MAX 5/' openpgp-do.c
jas@kaka:~/src/gnuk/src$ make | less
jas@kaka:~/src/gnuk/src$ cd ../tool/
jas@kaka:~/src/gnuk/tool$ ./upgrade_by_passwd.py 
Admin password: 
Device: 
Configuration: 1
Interface: 0
../regnual/regnual.bin: 4608
../src/build/gnuk.bin: 110592
CRC32: b93ca829

Device: 
Configuration: 1
Interface: 0
20002a00:20005000
Downloading flash upgrade program...
start 20002a00
end   20003c00
Run flash upgrade program...
Waiting for device to appear:
  Wait 1 second...
  Wait 1 second...
Device: 
08001000:08020000
Downloading the program
start 08001000
end   0801b000
Protecting device
Finish flashing
Resetting device
Update procedure finished
jas@kaka:~/src/gnuk/tool$

Now finally, I wish you all a Merry Christmas and Happy Hacking!

24 December, 2022 01:36PM by simon

mit-scheme @ Savannah

Version 12.0.90 is released

This is the first testing release for version 12.

Please test it out and report any bugs, either on the bug-mit-scheme@gnu.org mailing list,
or the bug tracker https://savannah.gnu.org/bugs/?group=mit-scheme.

24 December, 2022 05:20AM by Chris Hanson

December 23, 2022

FSF News

FSF job opportunity: Operations assistant

The Free Software Foundation (FSF), a Massachusetts 501(c)(3) charity with a worldwide mission to protect and promote computer-user freedom, seeks a motivated and organized Boston-based individual to be our full-time operations assistant.

23 December, 2022 09:42PM

December 20, 2022

parallel @ Savannah

GNU Parallel 20221222 ('ChatGPT') released

GNU Parallel 20221222 ('ChatGPT') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  GNU Parallel absolutely rocks.
    -- Austin Mordahl@Stackoverflow

New in this release:

  • --results works on more file systems (e.g. fat)
  • Joblog gives the same exit code as bash.
  • Bug fixes and man page updates.

News about GNU Parallel:

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

20 December, 2022 10:36PM by Ole Tange

gcl @ Savannah

GCL 2.6.13 is released

Greetings!  The GCL team is happy to announce the release of version 2.6.13, the latest achievement in the 'stable' (as opposed to 'development') series.  Please see http://www.gnu.org/software/gcl for downloading information.

This release consolidates several years of work on GCL internals,
performance and ansi compliance.

Garbage collection has been overhauled and significantly accelerated.  Contiguous block handling is now as fast as or perhaps faster than relblock handling, leading to the now implemented promotion of relblock data to contiguous after a surviving a number of gc calls.  Relblock is only written once during gc.  Heap allocation is fully dynamic at runtime and controllable with environment variables without recompilation.  While SGC is supported, it is found in practice to be less useful with modern large memory cores and is off by default.

GCC on several platforms defaults to code which must lie within a
common 2Gb space, now an issue with heaps routinely larger than this.  Error protection for code address overflow is in place on most machines.  The variable si::*code-block-reserve* can be set to a static array of element type 'character to preallocate a code block early within an acceptable range.  On amd64, compile-file takes a :large-memory-model-p keyword (with compiler::*default-large-memory-model-p*) to compile somewhat slower code which can be loaded at an arbitrary address.

The COMMON-LISP package is fixed to the ansi standard.  A CLTL1-COMPAT package is defined to support earlier applications, and is used in non-ansi builds.

GCL can optionally manage a single heap load across multiple processes via the GCL_MULTIPROCESS_MEMORY_POOL environment variable. GCL can compile gprof profiling code in non-profiling images using the :prof-p keyword to compile, causing '(si::gprof-start)(...)(si::gprof-quit)' to only report calls to such code.  GCL supports riscv4, and 64bit cygwin on Windows in addition to the previous 21 architectures.  GCL has extensive support for hardware floating point exception handling via the #'si::break-on-floating-point-exceptions function, taking the floating point errors as keyword arguments.

Several ANSI compliance errors have been fixed, most particularly in pathnames and restarts. Hashtables have been accelerated, supporting caching, static allocation, and 'equalp tests.

Circle detection and handling has been greatly accelerated, using the gc marking algorithm for a copy-less implementation.

The compiler no longer writes data files reordering "package-operations", changing the data file format to one loadable on object file initialization.

Floating point reading and writing has been made more precise.  Inf/nan handling matches IEEE specifications.

20 December, 2022 06:05PM by Camm Maguire

FSF News

FSF job opportunity: Deputy director

The Free Software Foundation (FSF), a Massachusetts 501(c)(3) charity with a worldwide mission to protect computer user freedom, seeks a motivated and talented Boston-based individual to be our full-time deputy director.

20 December, 2022 01:45PM

December 19, 2022

GNU Guix

GNU Guix 1.4.0 released

We are pleased to announce the release of GNU Guix version 1.4.0!

The release comes with ISO-9660 installation images, a virtual machine image, and with tarballs to install the package manager on top of your GNU/Linux distro, either from source or from binaries—check out the download page. Guix users can update by running guix pull.

It’s been 18 months since the previous release. That’s a lot of time, reflecting both the fact that, as a rolling release, users continuously get new features and update by running guix pull; but let’s face it, it also shows an area where we could and should collectively improve our processes. During that time, Guix received about 29,000 commits by 453 people, which includes important new features as we’ll see; the project also changed maintainers, structured cooperation as teams, and celebrated its ten-year anniversary!

A happy frog sitting at a Guix-powered computer with an almighty benevolent Guix humanoid in its back.

Illustration by Luis Felipe, published under CC-BY-SA 4.0.

This post provides highlights for all the hard work that went into this release—and yes, these are probably the longest release notes in Guix’s history, so make yourself comfortable, relax, and enjoy.

Bonus! Here’s a chiptune (by Trevor Lentz, under CC-BY-SA 3.0) our illustrator Luis Felipe recommends that you listen to before going further.

Improved software environment management

One area where Guix shines is the management of software environments. The guix environment command was designed for that but it suffered from a clumsy interface. It is now superseded by guix shell, though we are committed to keeping guix environment until at least May 1st, 2023. guix shell is a tool that’s interesting to developers, but it’s also a useful tool when you’re willing to try out software without committing it to your profile with guix install. Let’s say you want to play SuperTuxKart but would rather not have it in your profile because, hey, it’s a professional laptop; here’s how you would launch it, fetching it first if necessary:

guix shell supertuxkart -- supertuxkart

In addition to providing a simpler interface, guix shell significantly improves performance through caching. It also simplifies developer workflows by automatically recognizing guix.scm and manifest.scm files present in a directory: drop one of these in your project and other developers can get started hacking just by running guix shell, without arguments. Speaking of which: --export-manifest will get you started by “converting� command-line arguments into a manifest. Read more about guix shell in the manual.

Another guix shell innovation is optional emulation of the filesystem hierarchy standard (FHS). The FHS specifies locations for different file categories—/bin for essential command binaries, /lib for libraries, and so on. Guix with its store does not adhere to the FHS, which prevents users from running programs that assume FHS adherence. The new --emulate-fhs (or -F) flag of guix shell, combined with --container (-C), instructs it to create a container environment that follows the FHS. This is best illustrated with this example, where the ls command of the coreutils package appears right under /bin, as if we were on an FHS system like Debian:

$ guix shell -CF coreutils -- /bin/ls -1p /
bin/
dev/
etc/
gnu/
home/
lib/
lib64
proc/
sbin/
sys/
tmp/
usr/

Another big new feature is Guix Home. In a nutshell, Home brings the declarative nature of Guix System to your home environment: it lets you declare all the aspects of your home environments—“dot files�, services, and packages—and can instantiate that environment, in your actual $HOME or in a container.

If you’re already maintaining your dot files under version control, or if you would like to keep things under control so you don’t have to spend days or weeks configuring again next time you switch laptops, this is the tool you need. Check out this excellent introduction that David Wilson gave at the Ten Years celebration, and read more about Guix Home in the manual.

Package transformation options give users fine control over the way packages are built. The new --tune option enables tuning of packages for a specific CPU micro-architecture; this enables the use of the newest single-instruction multiple-data (SIMD) instructions, such as AVX-512 on recent AMD/Intel CPUs, which can make a significant difference for some workloads such as linear algebra computations.

Since the 1.3.0 release, the project started maintaining an alternative build farm at https://bordeaux.guix.gnu.org. It’s independent from the build farm at ci.guix.gnu.org (donated and hosted by the Max Delbrück Center for Molecular Medicine in Berlin, Germany), which has two benefits: it lets us challenge substitutes produced by each system, and it provides redundancy should one of these two build farms go down. Guix is now configured by default to fetch substitutes from any of these two build farms. In addition, a bug was fixed, ensuring that Guix gracefully switches to another substitute provider when one goes down.

Those who’ve come to enjoy declarative deployment of entire fleets of machines will probably like the new --execute option of guix deploy.

Stronger distribution

The distribution itself has seen lots of changes. First, the Guix System installer received a number of bug fixes and it now includes a new mechanism that allows users to automatically report useful debugging information in case of a crash. This will help developers address bugs that occur with unusual configurations.

Application startup has been reduced thanks to a new per-application dynamic linker cache that drastically reduces the number of stat and open calls due to shared library lookup (we’re glad it inspired others).

Guix System is now using version 0.9 of the GNU Shepherd, which addresses shortcomings, improves logging, and adds features such as systemd-style service activation and inetd-style service startup. Speaking of services, the new guix system edit sub-command provides an additional way for users to inspect services, completing guix system search and guix system extension-graph.

There are 15 new system services to choose from, including Jami, Samba, fail2ban, and Gitile, to name a few.

A new interface is available to declare swap space in operating system configurations. This interface is more expressive and more flexible than what was available before.

Similarly, the interface to declare static networking configuration has been overhauled. On GNU/Linux, it lets you do roughly the same as the ip command, only in a declarative fashion and with static checks to prevent you from deploying obviously broken configurations.

Screenshot of GNOME 42.

More than 5,300 packages were added for a total of almost 22,000 packages, making Guix one of the top-ten biggest distros according to Repology. Among the many noteworthy package upgrades and addition, GNOME 42 is now available. KDE is not there yet but tens of KDE packages have been added so we’re getting closer; Qt 6 is also available. The distribution also comes with GCC 12.2.0, GNU libc 2.33, Xfce 4.16, Linux-libre 6.0.10, LibreOffice 7.4.3.2, and Emacs 28.2 (with just-in-time compilation support!).

In other news, motivated by the fact that Python 2 officially reached “end of life� in 2020, more than 500 Python 2 packages were removed—those whose name starts with python2-. This includes “big ones� like python2-numpy and python2-scipy. Those who still need these have two options: using guix time-machine to jump to an older commit that contains the packages they need, or using the Guix-Past channel to build some of those old packages in today’s environments—scientific computing is one area where this may come in handy.

On top of that, the Web site features a new package browser—at last! Among other things, the package browse provides stable package URLs like https://packages.guix.gnu.org/packages/PACKAGE.

The NEWS file lists additional noteworthy changes and bug fixes you may be interested in.

More documentation

As with past releases, we have worked on documentation to make Guix more approachable. “How-to� kind of sections have been written or improved, such as:

The Cookbook likewise keeps receiving how-to entries, check it out!

The Guix reference manual is fully translated into French and German; 70% is available in Spanish, and there are preliminary translations in Russian, Chinese, and other languages. Guix itself is fully translated in French, with almost complete translations in Brazilian Portuguese, German, Slovak, and Spanish, and partial translations in almost twenty other languages. Check out the manual on how to help or this guided tour by translator in chief Julien Lepiller!

Supporting long-term reproducibility

A salient feature of Guix is its support for reproducible software deployment. There are several aspects to that, one of which is being able to retrieve source code from the Software Heritage archive. While Guix was already able to fetch the source code of packages from Software Heritage as a fallback, with version 1.4.0 the source code of Guix channels is automatically fetched from Software Heritage if its original URL has become unreachable.

In addition, Guix is now able to retrieve and restore source code tarballs such as tar.gz files. Software Heritage archives the contents of tarballs, but not tarball themselves. This created an impedance mismatch for Guix, where the majority of package definitions refer to tarballs and expect to be able to verify the content hash of the tarball itself. To bridge this gap, Timothy Sample developed Disarchive, a tool that can (1) extract tarball metadata, and (2) assemble previously-extracted metadata and actual files to reconstruct a tarball, as shown in the diagram below.

Diagram showing Disarchive and Software Heritage.

The Guix project has set up a continuous integration job to build a Disarchive database, which is available at disarchive.gnu.org. The database includes metadata for all the tarballs packages refer to. When a source code tarball disappears, Guix now transparently retrieves tarball metadata from Disarchive database, fetches file contents from Software Heritage, and reconstructs the original tarball. As of the “Preservation of Guix Report� published in January 2022, almost 75% of the .tar.gz files packages refer to are now fully archived with Disarchive and Software Heritage. Running guix lint -c archival PKG will tell you about the archival status of PKG. You can read more in the annual report of Guix-HPC.

This is a significant step forward to provide, for the first time, a tool that can redeploy past software environments while maintaining the connection between source code and binaries.

Application bundles and system images

The guix pack command to create “application bundles�—standalone application images—has been extended: guix pack -f deb creates a standalone .deb package that can be installed on Debian and derivative distros; the new --symlink flag makes it create symlinks within the image.

At the system level, the new guix system image command supersedes previously existing guix system sub-commands, providing a single entry point to build images of all types: raw disk images, QCOW2 virtual machine images, ISO9660 CD/DVD images, Docker images, and even images for Microsoft’s Windows Subsystem for Linux (WSL2). This comes with a high-level interface that lets you declare the type of image you want: the storage format, partitions, and of course the operating system for that image. To facilitate its use, predefined image types are provided:

$ guix system image --list-image-types
The available image types are:

   - rock64-raw
   - pinebook-pro-raw
   - pine64-raw
   - novena-raw
   - hurd-qcow2
   - hurd-raw
   - raw-with-offset
   - iso9660
   - efi32-raw
   - wsl2
   - uncompressed-iso9660
   - efi-raw
   - docker
   - qcow2
   - tarball

That includes for example an image type for the Pine64 machines and for the GNU/Hurd operating system. For example, this is how you’d create an QCOW2 virtual machine image suitable for QEMU:

guix system image -t qcow2 my-operating-system.scm

… where my-operating-system.scm contains an operating system declaration.

Likewise, here’s how you’d create, on your x86_64 machine, an image for your Pine64 board, ready to be transferred to an SD card or similar storage device that the board will boot from:

guix system image -t pine64-raw my-operating-system.scm

The pine64-raw image type specifies that software in the image is actually cross-compiled to aarch64-linux-gnu—that is, GNU/Linux on an AArch64 CPU, with the appropriate U-Boot variant as its bootloader. Sky’s the limit!

Nicer packaging experience

A significant change that packagers will immediately notice is package simplification, introduced shortly after 1.3.0. The most visible effect is that package definitions now look clearer:

(package
  ;; …
  (inputs (list pkg-config guile-3.0)))  ;�

… instead of the old baroque style with “input labels�:

(package
  ;; …
  (inputs `(("pkg-config" ,pkg-config)   ;�
            ("guile" ,guile-3.0))))

The new guix style command can automatically convert from the “old� style to the “new� style of package inputs. It can also reformat whole Scheme files following the stylistic canons du jour, which is particularly handy when getting started with the language.

That’s just the tip of the iceberg: the new modify-inputs macro makes package input manipulation easier and clearer, and one can use G-expressions for instance in package phases. Read our earlier announcement for more info. On top of that, the new field sanitizer mechanism is used to validate some fields; for instance, the license field is now type-checked and the Texinfo syntax of description and synopsis is validated, all without any run-time overhead in common cases. We hope these changes will make it easier to get started with packaging.

The guix build command has new flags, --list-systems and --list-targets, to list supported system types (which may be passed to --system) and cross-compilation target triplets (for use with --target). Under the hood, the new (guix platform) module lets developers define “platforms�—a combination of CPU architecture and operating system—in an abstract way, unifying various bits of information previously scattered around.

In addition, packagers can now mark as “tunable� packages that would benefit from CPU micro-architecture optimizations, enabled with --tune.

Python packaging has seen important changes. First, the python package now honors the GUIX_PYTHONPATH environment variable rather than PYTHONPATH. That ensures that Python won’t unwillingly pick up packages not provided by Guix. Second, the new pyproject-build-system implements PEP 517. It complements the existing python-build-system, and both may eventually be merged together.

What’s great with packaging is when it comes for free. The guix import command gained support for several upstream package repositories: minetest (extensions of the Minetest game), elm (the Elm programming language), egg (for CHICKEN Scheme), and hexpm (for Erlang and Elixir packages). Existing importers have seen various improvements. The guix refresh command to automatically update package definitions has a new generic-git updater.

Try it!

There are several ways to get started using Guix:

  1. The installation script lets you quickly install Guix on top of another GNU/Linux distribution.

  2. The Guix System virtual machine image can be used with QEMU and is a simple way to discover Guix System without touching your system.

  3. You can install Guix System as a standalone distribution. The installer will guide you through the initial configuration steps.

To review all the installation options at your disposal, consult the download page and don't hesitate to get in touch with us.

Enjoy!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64, and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

19 December, 2022 03:30PM by Ludovic Courtès

December 17, 2022

lilypond @ Savannah

LilyPond 2.24.0 released!

We are proud to announce the release of GNU LilyPond 2.24.0. LilyPond
is a music engraving program devoted to producing the highest-quality
sheet music possible. It brings the aesthetics of traditionally
engraved music to computer printouts.

This version includes improvements and fixes since the branching of the
previous stable release in October 2020. A list of added features and
other user-visible changes can be found at
https://lilypond.org/doc/v2.24/Documentation/changes/
This release switches to Guile 2.2 and features a completely rewritten
infrastructure for creating the official packages, finally allowing us
to offer 64-bit binaries for macOS and Windows.

These pre-built binaries are linked from
https://lilypond.org/download.html and available from GitLab:
https://gitlab.com/lilypond/lilypond/-/releases/v2.24.0

LilyPond 2.24 is brought to you by

Main Developers:
Jean Abou Samra, Colin Campbell, Dan Eble, Jonas Hahnfeld, Phil Holmes,
David Kastrup, Werner Lemberg, Han-Wen Nienhuys, Francisco Vila

Core Contributors:
Erlend E. Aasland, Kevin Barry, Martín Rincón Botero, Tim Burgess,
Thibaut Cuvelier, Jefferson Felix, David Stephen Grant, Jordan
Henderson, Masamichi Hosoda, Nihal Jere, Martin Joerg, Michael Käppler,
Doug Kearns, Mark Knoop, Thomas Morley, Lukas-Fabian Moser, Martin
Neubauer, Knut Petersen, Valentin Petzel, Pete Siddall, Alen Šiljak,
Samuel Tam, Timofey, Nathan Whetsell

Font Contributors:
Johannes Feulner, David Stephen Grant, Owen Lamb

Documentation Writers:
Michael Käppler, Daniel Tobias Johansen Langhoff, Thomas Morley, John
Wheeler

Translators:
Federico Bruni, Walter Garcia-Fontes, Dénes Harmath, Masamichi Hosoda,
Guyutongxue, Chengrui Li, Jean-Charles Malahieude, Benkő Pál

and numerous other contributors.

17 December, 2022 03:47PM by Jean Abou Samra

December 15, 2022

Andy Wingo

ephemeral success

Good evening, patient hackers :) Today finishes off my series on implementing ephemerons in a garbage collector.

Last time, we had a working solution for ephemerons, but it involved recursively visiting any pending ephemerons from within the copy routine—the bit of a semi-space collector that is called when traversing the object graph and we see an object that we hadn't seen yet. This recursive visit could itself recurse, and so we could overflow the control stack.

The solution, of course, is "don't do that": instead of visiting recursively, enqueue the ephemeron for visiting later. Iterate, don't recurse. But here we run into a funny problem: how do we add an ephemeron to a queue or worklist? It's such a pedestrian question ("just... enqueue it?") but I think it illustrates some of the particular concerns of garbage collection hacking.

speak, memory

The issue is that we are in the land of "can't use my tools because I broke my tools with my tools". You can't make a standard List<T> because you can't allocate list nodes inside the tracing routine: if you had memory in which you could allocate, you wouldn't be calling the garbage collector.

If the collector needs a data structure whose size doesn't depend on the connectivity of the object graph, you can pre-allocate it in a reserved part of the heap. This adds memory overhead, of course; for a 1000 MB heap, say, you used to be able to make graphs 500 MB in size (for a semi-space collector), but now you can only do 475 MB because you have to reserve 50 MB (say) for your data structures. Another way to look at it is, if you have a 400 MB live set and then you allocate 2GB of garbage, if your heap limit is 500 MB you will collect 20 times, but if it's 475 MB you'll collect 26 times, which is more expensive. This is part of why GC algorithms are so primitive; implementors have to be stingy that we don't get to have nice things / data structures.

However in the case of ephemerons, we will potentially need one worklist entry per ephemeron in the object graph. There is no optimal fixed size for a worklist of ephemerons. Most object graphs will have no or few ephemerons. Some, though, will have practically the whole heap.

For data structure needs like this, the standard solution is to reserve the needed space for a GC-managed data structure in the object itself. For example, for concurrent copying collectors, the GC might reserve a word in the object for a forwarding pointer, instead of just clobbering the first word. If you needed a GC-managed binary tree for a specific kind of object, you'd reserve two words. Again there are strong pressures to minimize this overhead, but in the case of ephemerons it seems sensible to make them pay their way on a per-ephemeron basis.

so let's retake the thing

So sometimes we might need to put an ephemeron in a worklist. Let's add a member to the ephemeron structure:

struct gc_ephemeron {
  struct gc_obj header;
  int dead;
  struct gc_obj *key;
  struct gc_obj *value;
  struct gc_ephemeron *gc_link; // *
};

Incidentally this also solves the problem of how to represent the struct gc_pending_ephemeron_table; just reserve 0.5% of the heap or so as a bucket array for a buckets-and-chains hash table, and use the gc_link as the intrachain links.

struct gc_pending_ephemeron_table {
  struct gc_ephemeron *resolved;
  size_t nbuckets;
  struct gc_ephemeron buckets[0];
};

An ephemeron can end up in three states, then:

  1. Outside a collection: gc_link can be whatever.

  2. In a collection, the ephemeron is in the pending ephemeron table: gc_link is part of a hash table.

  3. In a collection, the ephemeron's key has been visited, and the ephemeron is on the to-visit worklist; gc_link is part of the resolved singly-linked list.

Instead of phrasing the interface to ephemerons in terms of visiting edges in the graph, the verb is to resolve ephemerons. Resolving an ephemeron adds it to a worklist instead of immediately visiting any edge.

struct gc_ephemeron **
pending_ephemeron_bucket(struct gc_pending_ephemeron_table *table,
                         struct gc_obj *key) {
  return &table->buckets[hash_pointer(obj) % table->nbuckets];
}

void add_pending_ephemeron(struct gc_pending_ephemeron_table *table,
                           struct gc_obj *key,
                           struct gc_ephemeron *ephemeron) {
  struct gc_ephemeron **bucket = pending_ephemeron_bucket(table, key);
  ephemeron->gc_link = *bucket;
  *bucket = ephemeron;
}

void resolve_pending_ephemerons(struct gc_pending_ephemeron_table *table,
                                struct gc_obj *obj) {
  struct gc_ephemeron **link = pending_ephemeron_bucket(table, obj);
  struct gc_ephemeron *ephemeron;
  while ((ephemeron = *link)) {
    if (ephemeron->key == obj) {
      *link = ephemeron->gc_link;
      add_resolved_ephemeron(table, ephemeron);
    } else {
      link = &ephemeron->gc_link;
    }
  }
}

Copying an object may add it to the set of pending ephemerons, if it is itself an ephemeron, and also may resolve other pending ephemerons.

void resolve_ephemerons(struct gc_heap *heap, struct gc_obj *obj) {
  resolve_pending_ephemerons(heap->pending_ephemerons, obj);

  struct gc_ephemeron *ephemeron;
  if ((ephemeron = as_ephemeron(forwarded(obj)))
      && !ephemeron->dead) {
    if (is_forwarded(ephemeron->key))
      add_resolved_ephemeron(heap->pending_ephemerons,
                             ephemeron);
    else
      add_pending_ephemeron(heap->pending_ephemerons,
                            ephemeron->key, ephemeron);
  }
}

struct gc_obj* copy(struct gc_heap *heap, struct gc_obj *obj) {
  ...
  resolve_ephemerons(heap, obj); // *
  return new_obj;
}

Finally, we need to add something to the core collector to scan resolved ephemerons:

int trace_some_ephemerons(struct gc_heap *heap) {
  struct gc_ephemeron *resolved = heap->pending_ephemerons->resolved;
  if (!resolved) return 0;
  heap->pending_ephemerons->resolved = NULL;
  while (resolved) {
    resolved->key = forwarded(resolved->key);
    visit_field(&resolved->value, heap);
    resolved = resolved->gc_link;
  }
  return 1;
}

void kill_pending_ephemerons(struct gc_heap *heap) {
  struct gc_ephemeron *ephemeron;
  struct gc_pending_ephemeron_table *table = heap->pending_ephemerons;
  for (size_t i = 0; i < table->nbuckets; i++) {
    for (struct gc_ephemeron *chain = table->buckets[i];
         chain;
         chain = chain->gc_link)
      chain->dead = 1;    
    table->buckets[i] = NULL;
  }
}

void collect(struct gc_heap *heap) {
  flip(heap);
  uintptr_t scan = heap->hp;
  trace_roots(heap, visit_field);
  do { // *
    while(scan < heap->hp) {
      struct gc_obj *obj = scan;
      scan += align_size(trace_heap_object(obj, heap, visit_field));
    }
  } while (trace_ephemerons(heap)); // *
  kill_pending_ephemerons(heap); // *
}

The result is... not so bad? It makes sense to make ephemerons pay their own way in terms of memory, having an internal field managed by the GC. In fact I must confess that in the implementation I have been woodshedding, I actually have three of these damn things; perhaps more on that in some other post. But the perturbation to the core algorithm is perhaps less than the original code. There are still some optimizations to make, notably postponing hash-table lookups until the whole strongly-reachable graph is discovered; but again, another day.

And with that, thanks for coming along with me for my journeys into ephemeron-space.

I would like to specifically thank Erik Corry and Steve Blackburn for their advice over the years, and patience with my ignorance; I can only imagine that it's quite amusing when you have experience in a domain to see someone new and eager come in and make many of the classic mistakes. They have both had a kind of generous parsimony in the sense of allowing me to make the necessary gaffes but also providing insight where it can be helpful.

I'm thinking of many occasions but I especially appreciate the advice to start with a semi-space collector when trying new things, be it benchmarks or test cases or API design or new functionality, as it's a simple algorithm, hard to get wrong on the implementation side, and perfect for bringing out any bugs in other parts of the system. In this case the difference between fromspace and tospace pointers has a material difference to how you structure the ephemeron implementation; it's not something you can do just in a trace_heap_object function, as you don't have the old pointers there, and the pending ephemeron table is indexed by old object addresses.

Well, until some other time, gentle hackfolk, do accept my sincerest waste disposal greetings. As always, yours in garbage, etc.,

15 December, 2022 09:21PM by Andy Wingo

December 14, 2022

health @ Savannah

GNU Health Hospital Management client 4.0.2 available

Dear GNUHealth community:

I am happy to announce the maintenance release 4.0.2 of the Hospital Management client (GTK).

Release 4.0.2 of the GNUHealth HMIS client includes bug fixes (see the Changelog[1]) and is REUSE compliant[2].

As usual, the source code can be downloaded from the official GNU ftp site[3]. You can also install it directly via pip[4].

You can join us at Mastodon for the latest news and events around GNUHealth! (https://mastodon.social/@gnuhealth)

Happy and healthy hacking!
Luis

1.- https://hg.savannah.gnu.org/hgweb/health-hmis-client/file/5d21a06fa998/Changelog
2.- https://reuse.software/
3.- https://ftp.gnu.org/gnu/health/
4.- https://pypi.org/project/gnuhealth-client/

14 December, 2022 05:17PM by Luis Falcon

December 13, 2022

Amin Bandali

Why I love participating in LibrePlanet

Also published on the Free Software Foundation's community blog:
Amin Bandali: Why it's fun to participate in LibrePlanet

I'm Amin Bandali, a free/libre software activist by passion, and a software developer/engineer and computing scientist by profession. I am a former intern and current volunteer with the Free Software Foundation (FSF), and a member of the GNU Project. One of the ways I volunteer with the FSF is through LibrePlanet. I've helped with various aspects of the conference's organization, currently mainly helping as a member of the LibrePlanet committee, which reviews all session proposals. In this blog post I'd like to give a quick background on how and why I got involved with LibrePlanet and how I contribute to it today. I will also share how you, too, could start helping with the organization of the conference in a number of different ways, if you're interested!

I first got involved with LibrePlanet as a volunteer a few years back. By that point, I'd enjoyed participating in the conference via IRC and watching the talks online for a few years, and I was looking for ways to get involved. As I couldn't make it to Boston to attend LibrePlanet in person, I volunteered online, with tasks such as helping watch over the conference IRC channels and answering questions as best as I could. I seemed to have done a decent job, since the FSF folks later asked if I could do the same for a few non-LibrePlanet online FSF events too, which I gladly accepted.

Having enjoyed both participating and volunteering for LibrePlanet, I thought it would be great if I could give a talk of my own, too. This only became possible for me after 2020 with the possibility of doing remote presentations. Since I sadly cannot attend the event in person currently, this was a welcome side-effect of the conference temporarily switching to an online-only format. So, I submitted a proposal to talk about "Jami and how it empowers users" for LibrePlanet 2021, which was accepted and became my first LibrePlanet talk. Though presenting, or even just submitting a talk at a large conference like LibrePlanet, may sometimes seem like an intimidating task, I had a great time presenting mine, thanks in no small part to the FSF staff and other volunteer organizers, as well as the audience members.

The FSF staff were supportive and encouraging throughout the entire process of preparing and presenting my talk, and the audience gave positive and/or constructive feedback after my presentation. Plus, I greatly enjoyed discussing various free software topics with them, which was not really surprising because the folks attending LibrePlanet tend to be free software enthusiasts or activists like myself who are often just as eager to watch and chat with others about free software. And, as my good GNU friend Jason Self puts it, LibrePlanet is a wonderful place for such enthusiasts to "recharge their free software batteries each year".

Back in 2020, I was invited to join the LibrePlanet committee, a diverse team of volunteers from different backgrounds and areas of expertise that review all sessions submitted, helping select session proposals in a way that provides an exciting lineup of talks for people of differing areas and levels of experience and interest. I humbly and happily accepted the invitation to join the committee, and I help with the reviews to date. (I of course don't review my own session proposals, nor the ones I recognize to be from people I know). If you are also interested in joining the LibrePlanet committee and helping review the wonderful session proposals the team receives for each conference, you can come by the #libreplanet or #fsf channels on the Libera.Chat IRC network and reach out to the FSF staff there, or send an email to campaigns@fsf.org.

Besides being part of the LibrePlanet committee and helping review session proposals, there are a number of other ways to contribute to the organization of the conference as well. Technical tasks include helping with the setup and/or the maintenance of some pieces of infrastructure for the conference, for example helping maintain the conference's self-hosted installation of LibreAdventure, which is the conference's online event space where people can have their avatars "bump" into each other to have a real-time videoconferencing chat, and they can explore sessions, the FSF office (digitized), virtual sponsor booths, and more. Non-technical tasks include helping with the moderation of the conference's IRC channels on the event days, and volunteering to introduce, caption, or transcribe talks. There are also other logistical tasks that need doing now that LibrePlanet is switching to a hybrid format with both online and in-person events (in Boston). If you are interested in getting involved and helping with any of these (or other) tasks, please email to resources@fsf.org.

The theme for LibrePlanet 2023 is "Charting the Course", which I find particularly apt and important. The free software movement has come a long way and thanks to the tireless efforts of people from projects and communities of varying sizes, today we can carry out a very wide range of computing tasks in total freedom. It is also crucially important to continue recognizing and making progress in the areas of digital life where avoiding nonfree software may not be currently possible or feasible. One such notorious area is online payments, where the GNU Taler folks have been hard at work making freedom-respecting, privacy-friendly online transactions possible. At LibrePlanet 2023, I hope to see talks on such areas of digital life. I look forward to talks presenting the state of available free software in a certain field and clarify to what extent we can participate in them in freedom, along with a wishlist for improvements and a roadmap for moving closer towards freedom in this specific field so that we will ultimately, hopefully, reach full digital freedom.

These, along with other factors — such as the FSF staff striving for LibrePlanet to be inclusive and accessible, as well as making it possible to participate online for those of us not able to attend the event in person — make LibrePlanet a free software event I'm most excited about and look forward to each year. I hope and expect that LibrePlanet 2023 will be a conference with a lineup of interesting, fun, educational, and thought-provoking user freedom themed talks and sessions, along with a chance to catch up and socialize with fellow free software hackers, activists, and/or enthusiasts from all over the world, just like it always has been — especially this time with its ever more relevant theme of "Charting the Course" to not only reflect and celebrate the path we've come so far, but to also look towards the future and chart the course to software user freedom for coming generations.

Take care, and I hope to see you around for LibrePlanet 2023!

Amin Bandali
LibrePlanet Committee Member and assistant GNUisance

Image of Amin Bandali

13 December, 2022 06:30PM

December 12, 2022

Andy Wingo

i'm throwing ephemeron party & you're invited

Good day, hackfolk. Today's note tries to extend our semi-space collector with support for ephemerons. Spoiler alert: we fail in a subtle and interesting way. See if you can spot it before the end :)

Recall that, as we concluded in an earlier article, a memory manager needs to incorporate ephemerons as a core part of the tracing algorithm. Ephemerons are not macro-expressible in terms of object trace functions.

Instead, to support ephemerons we need to augment our core trace routine. When we see an ephemeron E, we need to check if the key K is already visited (and therefore live); if so, we trace the value V directly, and we're done. Otherwise, we add E to a global table of pending ephemerons T, indexed under K. Finally whenever we trace a new object O, ephemerons included, we look up O in T, to trace any pending ephemerons for O.

So, taking our semi-space collector as a workbench, let's start by defining what an ephemeron is.

struct gc_ephemeron {
  struct gc_obj header;
  int dead;
  struct gc_obj *key;
  struct gc_obj *value;
};

enum gc_obj_kind { ..., EPHEMERON, ... };

static struct gc_ephemeron* as_ephemeron(struct gc_obj *obj) {
  uintptr_t ephemeron_tag = NOT_FORWARDED_BIT | (EPHEMERON << 1);
  if (obj->tag == ephemeron_tag)
    return (struct gc_ephemeron*)obj;
  return NULL;
}

First we need to allow the GC to know when an object is an ephemeron or not. This is somewhat annoying, as you would like to make this concern entirely the responsibility of the user, and let the GC be indifferent to the kinds of objects it's dealing with, but it seems to be unavoidable.

The heap will need some kind of data structure to track pending ephemerons:

struct gc_pending_ephemeron_table;

struct gc_heap {
  ...
  struct gc_pending_ephemeron_table *pending_ephemerons;
}

struct gc_ephemeron *
pop_pending_ephemeron(struct gc_pending_ephemeron_table*,
                      struct gc_obj*);
void
add_pending_ephemeron(struct gc_pending_ephemeron_table*,
                      struct gc_obj*, struct gc_ephemeron*);
struct gc_ephemeron *
pop_any_pending_ephemeron(struct gc_pending_ephemeron_table*);

Now let's define a function to handle ephemeron shenanigans:

void visit_ephemerons(struct gc_heap *heap, struct gc_obj *obj) {
  // We are visiting OBJ for the first time.
  // OBJ is the old address, but it is already forwarded.
  ASSERT(is_forwarded(obj));

  // First, visit any pending ephemeron for OBJ.
  struct gc_ephemeron *ephemeron;
  while ((ephemeron =
            pop_pending_ephemeron(heap->pending_ephemerons, obj))) {
    ASSERT(obj == ephemeron->key);
    ephemeron->key = forwarded(obj);
    visit_field(&ephemeron->value, heap);
  }

  // Then if OBJ is itself an ephemeron, trace it.
  if ((ephemeron = as_ephemeron(forwarded(obj)))
      && !ephemeron->dead) {
    if (is_forwarded(ephemeron->key)) {
      ephemeron->key = forwarded(ephemeron->key);
      visit_field(&ephemeron->value, heap);
    } else {
      add_pending_ephemeron(heap->pending_ephemerons,
                            ephemeron->key, ephemeron);
    }
  }
}

struct gc_obj* copy(struct gc_heap *heap, struct gc_obj *obj) {
  ...
  visit_ephemerons(heap, obj); // *
  return new_obj;
}

We wire it into the copy routine, as that's the bit of the collector that is called only once per object and which has access to the old address. We actually can't process ephemerons during the Cheney field scan, as there we don't have old object addresses.

Then at the end of collection, we kill any ephemeron whose key hasn't been traced:

void kill_pending_ephemerons(struct gc_heap *heap) {
  struct gc_ephemeron *ephemeron;
  while ((ephemeron =
            pop_any_pending_ephemeron(heap->pending_ephemerons)))
    ephemeron->dead = 1;    
}

void collect(struct gc_heap *heap) {
  // ...
  kill_pending_ephemerons(heap);
}

First observation: Gosh, this is quite a mess. It's more code than the core collector, and it's gnarly. There's a hash table, for goodness' sake. Goodbye, elegant algorithm!

Second observation: Well, at least it works.

Third observation: Oh. It works in the same way as tracing in the copy routine works: well enough for shallow graphs, but catastrophically for arbitrary graphs. Calling visit_field from within copy introduces unbounded recursion, as tracing one value can cause more ephemerons to resolve, ad infinitum.

Well. We seem to have reached a dead-end, for now. Will our hero wrest victory from the jaws of defeat? Tune in next time for find out: same garbage time (unpredictable), same garbage channel (my wordhoard). Happy hacking!

12 December, 2022 02:02PM by Andy Wingo

December 11, 2022

we iterate so that you can recurse

Sometimes when you see an elegant algorithm, you think "looks great, I just need it to also do X". Perhaps you are able to build X directly out of what the algorithm gives you; fantastic. Or, perhaps you can alter the algorithm a bit, and it works just as well while also doing X. Sometimes, though, you alter the algorithm and things go pear-shaped.

Tonight's little note builds on yesterday's semi-space collector article and discusses an worse alternative to the Cheney scanning algorithm.

To recall, we had this visit_field function that takes a edge in the object graph, as the address of a field in memory containing a struct gc_obj*. If the edge points to an object that was already copied, visit_field updates it to the forwarded address. Otherwise it copies the object, thus computing the new address, and then updates the field.

struct gc_obj* copy(struct gc_heap *heap, struct gc_obj *obj) {
  size_t size = heap_object_size(obj);
  struct gc_obj *new_obj = (struct gc_obj*)heap->hp;
  memcpy(new_obj, obj, size);
  forward(obj, new_obj);
  heap->hp += align_size(size);
  return new_obj;
}

void visit_field(struct gc_obj **field, struct gc_heap *heap) {
  struct gc_obj *from = *field;
  struct gc_obj *to =
    is_forwarded(from) ? forwarded(from) : copy(heap, from);
  *field = to;
}

Although a newly copied object is in tospace, all of its fields still point to fromspace. The Cheney scan algorithm later visits the fields in the newly copied object with visit_field, which both discovers new objects and updates the fields to point to tospace.

One disadvantage of this approach is that the order in which the objects are copied is a bit random. Given a hierarchical memory system, it's better if objects that are accessed together in time are close together in space. This is an impossible task without instrumenting the actual data access in a program and then assuming future accesses will be like the past. Instead, the generally-accepted solution is to ensure that objects that are allocated close together in time be adjacent in space. The bump-pointer allocator in a semi-space collector provides this property, but the evacuation algorithm above does not: it would need to preserve allocation order, but instead its order is driven by graph connectivity.

I say that the copying algorithm above is random but really it favors a breadth-first traversal; if you have a binary tree, first you will copy the left and the right nodes of the root, then the left and right children of the left, then the left and right children of the right, then grandchildren, and so on. Maybe it would be better to keep parent and child nodes together? After all they are probably allocated that way.

So, what if we change the algorithm:

struct gc_obj* copy(struct gc_heap *heap, struct gc_obj *obj) {
  size_t size = heap_object_size(obj);
  struct gc_obj *new_obj = (struct gc_obj*)heap->hp;
  memcpy(new_obj, obj, size);
  forward(obj, new_obj);
  heap->hp += align_size(size);
  trace_heap_object(new_obj, heap, visit_field); // *
  return new_obj;
}

void visit_field(struct gc_obj **field, struct gc_heap *heap) {
  struct gc_obj *from = *field;
  struct gc_obj *to =
    is_forwarded(from) ? forwarded(from) : copy(heap, from);
  *field = to;
}

void collect(struct gc_heap *heap) {
  flip(heap);
  trace_roots(heap, visit_field);
}

Here we favor a depth-first traversal: we eagerly call trace_heap_object within copy. No need for the Cheney scan algorithm; tracing does it all.

The thing is, this works! It might even have better performance for some workloads, depending on access patterns. And yet, nobody does this. Why?

Well, consider a linked list with a million nodes; you'll end up with a million recursive calls to copy, as visiting each link eagerly traverses the next. While I am all about unbounded recursion, an infinitely extensible stack is something that a language runtime has to provide to a user, and here we're deep into implementing-the-language-runtime territory. At some point a user's deep heap graph is going to cause a gnarly system failure via stack overflow.

Ultimately stack space needed by a GC algorithm counts towards collector memory overhead. In the case of a semi-space collector you already need twice the amount memory as your live object graph, and if you recursed instead of iterated this might balloon to 3x or more, depending on the heap graph shape.

Hey that's my note! All this has been context for some future article, so this will be on the final exam. Until then!

11 December, 2022 09:19PM by Andy Wingo

December 10, 2022

www @ Savannah

On Privacy at School

New article by Richard Stallman, On Privacy at School.

10 December, 2022 10:56PM by Dora Scilipoti

Simon Josefsson

How to complicate buying a laptop

I’m about to migrate to a new laptop, having done a brief pre-purchase review of options on Fosstodon and reaching a decision to buy the NovaCustom NV41. Given the rapid launch and decline of Mastodon instances, I thought I’d better summarize my process and conclusion on my self-hosted blog until the fediverse self-hosting situation improves.

Since 2010 my main portable computing device has been the Lenovo X201 that replaced the Dell Precision M65 that I bought in 2006. I have been incredibly happy with the X201, even to the point that in 2015 when I wanted to find a replacement, I couldn’t settle on a decision and eventually realized I couldn’t articulate what was wrong with the X201 and decided to just buy another X201 second-hand for my second office. There is still no deal-breaker with the X201, and I’m doing most of my computing on it including writing this post. However, today I can better articulate what is lacking with the X201 that I desire, and the state of the available options on the market has improved since my last attempt in 2015.

Briefly, my desired properties are:

  • Portable – weight under 1.5kg
  • Screen size 9-14″
  • ISO keyboard layout, preferably Swedish layout
  • Mouse trackpad, WiFi, USB and external screen connector
  • Decent market availability: I should be able to purchase it from Sweden and have consumer protection, warranty, and some hope of getting service parts for the device
  • Manufactured and sold by a vendor that is supportive of free software
  • Preferably RJ45 connector (for data center visits)
  • As little proprietary software as possible, inspired by FSF’s Respect Your Freedom
  • Able to run a free operating system

My workload for the machine is Emacs, Firefox, Nextcloud client, GNOME, Evolution (mail & calendar), LibreOffice Calc/Writer, compiling software and some podman/qemu for testing. I have used Debian as the main operating system for the entire life of this laptop, but have experimented with PureOS recently. My current X201 is useful enough for this, although support for 4K displays and a faster machine wouldn’t hurt.

Based on my experience in 2015 that led me to make no decision, I changed perspective. This is a judgement call and I will not be able to fulfil all criteria. I will have to decide on a balance and the final choice will include elements that I really dislike, but still it will hopefully be better than nothing. The conflict for me mainly center around these parts:

  • Non-free BIOS. This is software that runs on the main CPU and has full control of everything. I want this to run free software as much as possible. Coreboot is the main project in this area, although I prefer the more freedom-oriented Libreboot.
  • Proprietary and software-upgradeable parts of the main CPU. This includes CPU microcode that is not distributed as free software. The Intel Management Engine (AMD and other CPU vendors has similar technology) falls into this category as well, and is problematic because it is an entire non-free operating system running within the CPU, with many security and freedom problems. This aspect is explored in the Libreboot FAQ further. Even if these parts can be disabled (Intel ME) or not utilized (CPU microcode), I believe the mere presence of these components in the design of the CPU is a problem, and I would prefer a CPU without these properties.
  • Non-free software in other microprocessors in the laptop. Ultimately, I tend agree with the FSF’s “secondary processor” argument but when it is possible to chose between a secondary processor that runs free software and one that runs proprietary software, I would prefer as many secondary processors as possible to run free software. The libreboot binary blob reduction policy describes a move towards stronger requirements.
  • Non-free firmware that has to be loaded during runtime into CPU or secondary processors. Using Linux-libre solves this but can cause some hardware to be unusable.
  • WiFi, BlueTooth and physical network interface (NIC/RJ45). This is the most notable example of secondary processor problem with running non-free software and requiring non-free firmware. Sometimes these may even require non-free drivers, although in recent years this has usually been reduced into requiring non-free firmware.

A simple choice for me would be to buy one of the FSF RYF certified laptops. Right now that list only contains the 10+ year old Lenovo series, and I actually already have a X200 with libreboot that I bought earlier for comparison. The reason the X200 didn’t work out as a replacement for me was the lack of a mouse trackpad, concerns about non-free EC firmware, Intel ME uncertainty (is it really neutralized?) and non-free CPU microcode (what are the bugs that it fixes?), but primarily that for some reason that I can’t fully articulate it feels weird to use a laptop manufactured by Lenovo but modified by third parties to be useful. I believe in market forces to pressure manufacturers into Doing The Right Thing, and feel that there is no incentive for Lenovo to use libreboot in the future when this market niche is already fulfilled by re-sellers modifying Lenovo laptops. So I’d be happier buying a laptop from someone who is natively supportive of they way I’m computing. I’m sure this aspect could be discussed a lot more, and maybe I’ll come back to do that, and could even reconsider my thinking (the right-to-repair argument is compelling). I will definitely continue to monitor the list of RYF-certified laptops to see if future entries are more suitable options for me.

Eventually I decided to buy the NovaCustom NV41 laptop, and it arrived quickly and I’m in the process of setting it up. I hope to write a separate blog about it next.

10 December, 2022 11:26AM by simon

December 08, 2022

FSF Events

December 05, 2022

GNUnet News

GNUnet 0.19.0

GNUnet 0.19.0 released

We are pleased to announce the release of GNUnet 0.19.0.
GNUnet is an alternative network stack for building secure, decentralized and privacy-preserving distributed applications. Our goal is to replace the old insecure Internet protocol stack. Starting from an application for secure publication of files, it has grown to include all kinds of basic protocol components and applications towards the creation of a GNU internet.

This is a new major release. It breaks protocol compatibility with the 0.18.x versions. Please be aware that Git master is thus henceforth (and has been for a while) INCOMPATIBLE with the 0.18.x GNUnet network, and interactions between old and new peers will result in issues. 0.18.x peers will be able to communicate with Git master or 0.19.x peers, but some services will not be compatible.
In terms of usability, users should be aware that there are still a number of known open issues in particular with respect to ease of use, but also some critical privacy issues especially for mobile users. Also, the nascent network is tiny and thus unlikely to provide good anonymity or extensive amounts of interesting information. As a result, the 0.19.0 release is still only suitable for early adopters with some reasonable pain tolerance .

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links might be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

Noteworthy changes in 0.19.0 (since 0.18.2)

  • UTIL : Moved GNUNET_BIO_MetaData handling into FS .
  • BUILD : platform.h removed as it should not be used by third parties anyway. gnunet_config.h is renamed to gnunet_private_config.h and the new replacement gnunet_config.h is added to provide build information for components linking against/using GNUnet.
  • UTIL : Components part of gnunet_util_lib.h must now be included through gnunet_util_lib.h and through that header only .
  • NAMESTORE : gnunet-namestore can now parse a list of records into zones from stdin in new recordline format.
  • GTK : Added an identity selector to the search to accomodate for previously deprecated "default" identities for subsystems.
  • Other: Postgres plugins implementations modernized and previous regressions fixed.

A detailed list of changes can be found in the ChangeLog and the bug tracker .

Known Issues

  • There are known major design issues in the TRANSPORT, ATS and CORE subsystems which will need to be addressed in the future to achieve acceptable usability, performance and security.
  • There are known moderate implementation limitations in CADET that negatively impact performance.
  • There are known moderate design issues in FS that also impact usability and performance.
  • There are minor implementation limitations in SET that create unnecessary attack surface for availability.
  • The RPS subsystem remains experimental.
  • Some high-level tests in the test-suite fail non-deterministically due to the low-level TRANSPORT issues.

In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org which lists about 190 more specific issues.

Thanks

This release was the work of many people. The following people contributed code and were thus easily identified: Christian Grothoff, Tristan Schwieren, madmurphy, t3sserakt, TheJackiMonster and Martin Schanzenbach.

05 December, 2022 11:00PM

December 01, 2022

FSF Events

Free Software Directory meeting on IRC: Friday, December 30, starting at 12:00 EST (17:00 UTC)

Join the FSF and friends on Friday, December 30, from 12:00 to 15:00 EST (17:00 to 20:00 UTC) to help improve the Free Software Directory.

01 December, 2022 05:39PM

November 30, 2022

texinfo @ Savannah

Texinfo 7.0.1 released

We have released version 7.0.1 of Texinfo, the GNU documentation format. This is a minor bug-fix release.

It's available via a mirror (xz is much smaller than gz, but gz is available too just in case):
http://ftpmirror.gnu.org/texinfo/texinfo-7.0.1.tar.xz
http://ftpmirror.gnu.org/texinfo/texinfo-7.0.1.tar.gz

Please send any comments to bug-texinfo@gnu.org.

Full announcement:
https://lists.gnu.org/archive/html/bug-texinfo/2022-11/msg00237.html

30 November, 2022 06:38PM by Gavin D. Smith

November 24, 2022

hyperbole @ Savannah

Installing Hyperbole from GNU-devel ELPA Packages

Installing the latest development version of Hyperbole

The latest development version of Hyperbole can be installed directly from the GNU-devel ELPA Packages using built-in Emacs Package Manager.

The Elpa GNU-devel package repository provides a development version of Hyperbole. It pulls from the latest Hyperbole development branch to get the tip version and makes an installable package.  This is done on a daily basis. Installing this does not require any new package manager software.  Since Hyperbole is a mature package, this version is usually fine to use and is updated on a day-to-day basis.  But new features are tested on this branch and once in awhile it may break for a short time before a fix is pushed.

To download and install this version of the Hyperbole, you should add the following lines to your personal Emacs initialization file, typically "~/.emacs". (For further details, see info page "(emacs)Init File", or Init-File).

(when (< emacs-major-version 27)
  (error "Hyperbole requires Emacs 27 or above; you are running version %d" emacs-major-version))
(require 'package)
(add-to-list 'package-archives '("gnu-devel" . "https://elpa.gnu.org/devel/"))
(unless (package-installed-p 'hyperbole)
  (package-refresh-contents)
  (package-install 'hyperbole))
(hyperbole-mode 1)

Now save the file and restart Emacs.  Hyperbole will then be downloaded and compiled for use with your version of Emacs; give it a minute or two. You may see a bunch of compilation warnings but these can be safely ignored.

24 November, 2022 10:33PM by Mats Lidell

November 22, 2022

parallel @ Savannah

GNU Parallel 20221122 ('Херсо́н') released

GNU Parallel 20221122 ('Херсо́н') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  [GNU Parallel] is the most amazing tool ever invented for bioinformatics!
    -- Istvan Albert https://www.ialbert.me/

New in this release:

  • Support for IPv6 adresses and _ in hostnames in --sshlogin.
  • Use --total-jobs for --eta/--bar if generating jobs is slow.
  • A lot of bug fixed in --latest-line.
  • Better support for MSYS2.
  • Better Text::CSV error messages.
  • --bar supports UTF8.
  • GNU Parallel is now on Mastodon: @GNU_Parallel@hostux.social
  • Bug fixes and man page updates.

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

22 November, 2022 07:03PM by Ole Tange

November 21, 2022

Luca Saiu

Announcing make-gallery, a simple web image gallery generator

I wrote a script generating an image gallery suitable to be included in web pages. Since it can be generally useful I cleaned it up and published it, of course as free software (https://www.gnu.org/philosophy/free-sw.html); you are welcome to download a copy of ‘make-gallery’ from . The software is released under the GNU General Public Licence (https://www.gnu.org/licenses/gpl-3.0.html) version 3 or later; the generated code is in the public domain. I hate the web I have never made a mystery of my personal dislike for the web with its gratuitous ever-growing complexity, inefficiency, lack of expressivity, hostility to the developer and to ... [Read more]

21 November, 2022 12:33AM by Luca Saiu (positron@gnu.org)

November 20, 2022

hyperbole @ Savannah

GNU Hyperbole 8.0.0, the Epiphany release, is now available on GNU ELPA

========================================================================

  • Overview

========================================================================

GNU Hyperbole 8.0.0, the Epiphany release, is now available on GNU ELPA.
Hyperbole is a unique hypertextual information management Emacs package
that works across all Emacs modes, letting the computer do the hard work
while you benefit from its sophisticated context-sensitive linking and
navigation capabilities.  Hyperbole has always been one of the best
documented Emacs packages.  With Version 8 comes excellent test coverage:
over 200 automated tests to ensure quality. We hope you'll give it a try.

What's new in this release is described here:

  www.gnu.org/s/hyperbole/HY-NEWS.html

  Everything back until release 7.1.3 is new since the last major
  release announcement (over a year ago), so updates are extensive.

If you prefer video introductions, visit the videos linked to below; otherwise,
skip to the next section.

GNU Hyperbole Videos

========================================================================

  • Introduction

========================================================================

Hyperbole is like Markdown for hypertext.  Hyperbole automatically
recognizes dozens of common patterns in any buffer regardless of mode
and can instantly activate them as hyperbuttons with a single key:
email addresses, URLs, grep -n outputs, programming backtraces,
sequences of Emacs keys, programming identifiers, Texinfo and Info
cross-references, Org links, Markdown links and on and on.  All you do
is load Hyperbole and then your text comes to life with no extra
effort or complex formatting.

Hyperbole interlinks all your working information within Emacs for
fast access and editing, not just within special modes.  Every button
is automatically assigned a type and new types can be developed for
your own buttons with simple function definitions.  You can create
your own buttons by simply dragging between two buffers.

But Hyperbole is also a hub controller for your information supplying
built-in capabilities of contact management/hierarchical record
lookup, legal-numbered outlines with hyperlinkable views and a unique
window and frame manager.  It is even Org-compatible so you can use
all of Org's capabilities together with Hyperbole.

Hyperbole is unique, powerful, extensively documented, and free.  Like
Emacs, Org, Counsel and Helm, Hyperbole has many different uses all
based around the theme of reducing cognitive load and improving your
everyday information management.  It reduces cognitive load by using
a single Action Key, {M-RET}, across many different contexts
which automatically chooses the best action

Then as you grow with it across time, it helps you build new capabilities
that continue to speed your work.

========================================================================

  • Installing and Using Hyperbole

========================================================================

To install within GNU Emacs, use:

   {M-x package-install RET hyperbole RET}

   Hyperbole installs in less than a minute and can be uninstalled even
   faster if ever need be.  Give it a try.

Then to invoke its minibuffer menu, use:

   {C-h h} or {M-x hyperbole RET}

The best way to get a feel for many of its capabilities is to invoke the
all new, interactive DEMO and explore sections of interest:

   {C-h h d d}

To permanently activate Hyperbole in your Emacs initialization file, add
the line:

   (hyperbole-mode 1)

Hyperbole is a minor mode that may be disabled at any time with:

   {C-u 0 hyperbole-mode RET}

The Hyperbole home page with screenshots is here:

   www.gnu.org/s/hyperbole

For use cases, see:

   www.gnu.org/s/hyperbole/HY-WHY.html

For what users think about Hyperbole, see:

   www.gnu.org/s/hyperbole/hyperbole.html#user-quotes

Enjoy,

The Hyperbole Team

20 November, 2022 10:28PM by Mats Lidell

gnulib @ Savannah

Gnulib helps you get away from fork() + exec()

Spawning a new process has traditionally been coded by a fork() call, followed by an execv/execl/execlp/execvp call in the child process. This is often referred to as the fork + exec idiom.

In 90% of the cases, there is something better: the posix_spawn/posix_spawnp functions.

Why is that better?

First, it's faster. The glibc implementation of posix_spawn, on Linux, uses a specialized system call (clone3) with a custom child-process stack, that makes it outperform the fork + exec idiom already now. And another speedup of 30% is being considered, see https://lwn.net/Articles/908268/ .

Second, it's more portable. While most Unix-like operating systems nowadays have both fork and posix_spawn, there are platforms which don't have fork(), namely Windows (excluding Cygwin). Comes in Gnulib for portability: Gnulib provides a posix_spawn implementation not only for the Unix platforms which lack it (today, that's only HP-UX), but also for Windows. In fact, Gnulib's posix_spawn implementation is the world's first for Windows platforms; the mingw libraries don't have one.

Why only in 90% of the cases?

Typically, between the fork and exec part, the application code will set up or configure some things in the child process. Such as closing file descriptors (this is necessary when pipes are involved), changing the current directory, and things like that.

posix_spawn has a certain set of setup / configuration "actions" that are supported. Namely, searching for the program in $PATH, opening files, shuffling arounds or closing file descriptors, and setting the tty-related process group. If that's all that the application code needs, then posix_spawn fits the bill. That should be 90% of the cases in practice.

How to do the change?

Before you replace a bit of fork + exec code with posix_spawn, you need to understand the main difference: The setup / configuration "actions" are encoded as C system calls in the old approach. Whereas with posix_spawn they are specified declaratively, by constructing an "actions" object in memory.

When you have done this change, you would test it on a glibc system.

And finally, for portability, import the Gnulib modules corresponding to all the posix_spawn* functions that you need.

20 November, 2022 04:27PM by Bruno Haible