Planet GNU

Aggregation of development blogs from the GNU Project

May 28, 2023

unifont @ Savannah

Unifont 15.0.04 Released

28 May 2023 Unifont 15.0.04 is now available.  This is a minor release, and the final release to have TrueType fonts in its default build.  Future releases will only build their OpenType equivalents, although it will still be possible to build TrueType versions manually by invoking "make truetype" in the font directory.

Minor changes have been made to Greek and Coptic glyphs in the range U+0370..U+03FF, adjusting the tonos placement for capital vowels and ensuring that all baseline and x-height alignments are consistent.  Two heart icons were modified.  Glyphs for the ConScript Unicode Registry (CSUR) script Engsvanyáli (U+E100..U+E14F) have also been added.  Full details are in the ChangeLog file.

Download this release from GNU server mirrors at:

     https://ftpmirror.gnu.org/unifont/unifont-15.0.04/

or if that fails,

     https://ftp.gnu.org/gnu/unifont/unifont-15.0.04/

or, as a last resort,

     ftp://ftp.gnu.org/gnu/unifont/unifont-15.0.04/

These files are also available on the unifoundry.com website:

     https://unifoundry.com/pub/unifont/unifont-15.0.04/

Font files are in the subdirectory

     https://unifoundry.com/pub/unifont/unifont-15.0.04/font-builds/

A more detailed description of font changes is available at

      https://unifoundry.com/unifont/index.html

and of utility program changes at

      http://unifoundry.com/unifont/unifont-utilities.html

28 May, 2023 06:11PM by Paul Hardy

May 23, 2023

Gary Benson

Which uninstalled package provides a file?

$ apt-file find guestmount
guestmount: /usr/bin/guestmount
guestmount: /usr/share/bash-completion/completions/guestmount
guestmount: /usr/share/doc/guestmount/changelog.Debian.gz
guestmount: /usr/share/doc/guestmount/copyright
guestmount: /usr/share/man/ja/man1/guestmount.1.gz
guestmount: /usr/share/man/man1/guestmount.1.gz
guestmount: /usr/share/man/uk/man1/guestmount.1.gz

23 May, 2023 09:03AM by gbenson

May 20, 2023

Andy Wingo

approaching cps soup

Good evening, hackers. Today's missive is more of a massive, in the sense that it's another presentation transcript-alike; these things always translate to many vertical pixels.

In my defense, I hardly ever give a presentation twice, so not only do I miss out on the usual per-presentation cost amortization and on the incremental improvements of repetition, the more dire error is that whatever message I might have can only ever reach a subset of those that it might interest; here at least I can be more or less sure that if the presentation would interest someone, that they will find it.

So for the time being I will try to share presentations here, in the spirit of, well, why the hell not.

CPS Soup

A functional intermediate language

10 May 2023 – Spritely

Andy Wingo

Igalia, S.L.

Last week I gave a training talk to Spritely Institute collaborators on the intermediate representation used by Guile's compiler.

CPS Soup

Compiler: Front-end to Middle-end to Back-end

Middle-end spans gap between high-level source code (AST) and low-level machine code

Programs in middle-end expressed in intermediate language

CPS Soup is the language of Guile’s middle-end

An intermediate representation (IR) (or intermediate language, IL) is just another way to express a computer program. Specifically it's the kind of language that is appropriate for the middle-end of a compiler, and by "appropriate" I meant that an IR serves a purpose: there has to be a straightforward transformation to the IR from high-level abstract syntax trees (ASTs) from the front-end, and there has to be a straightforward translation from IR to machine code.

There are also usually a set of necessary source-to-source transformations on IR to "lower" it, meaning to make it closer to the back-end than to the front-end. There are usually a set of optional transformations to the IR to make the program run faster or allocate less memory or be more simple: these are the optimizations.

"CPS soup" is Guile's IR. This talk presents the essentials of CPS soup in the context of more traditional IRs.

How to lower?

High-level:

(+ 1 (if x 42 69))

Low-level:

  cmpi $x, #f
  je L1
  movi $t, 42
  j L2  
L1:
  movi $t, 69
L2:
  addi $t, 1

How to get from here to there?

Before we dive in, consider what we might call the dynamic range of an intermediate representation: we start with what is usually an algebraic formulation of a program and we need to get down to a specific sequence of instructions operating on registers (unlimited in number, at this stage; allocating to a fixed set of registers is a back-end concern), with explicit control flow between them. What kind of a language might be good for this? Let's attempt to answer the question by looking into what the standard solutions are for this problem domain.

1970s

Control-flow graph (CFG)

graph := array<block>
block := tuple<preds, succs, insts>
inst  := goto B
       | if x then BT else BF
       | z = const C
       | z = add x, y
       ...

BB0: if x then BB1 else BB2
BB1: t = const 42; goto BB3
BB2: t = const 69; goto BB3
BB3: t2 = addi t, 1; ret t2

Assignment, not definition

Of course in the early days, there was no intermediate language; compilers translated ASTs directly to machine code. It's been a while since I dove into all this but the milestone I have in my head is that it's the 70s when compiler middle-ends come into their own right, with Fran Allen's work on flow analysis and optimization.

In those days the intermediate representation for a compiler was a graph of basic blocks, but unlike today the paradigm was assignment to locations rather than definition of values. By that I mean that in our example program, we get t assigned to in two places (BB1 and BB2); the actual definition of t is implicit, as a storage location, and our graph consists of assignments to the set of storage locations in the program.

1980s

Static single assignment (SSA) CFG

graph := array<block>
block := tuple<preds, succs, phis, insts>
phi   := z := φ(x, y, ...)
inst  := z := const C
       | z := add x, y
       ...
BB0: if x then BB1 else BB2
BB1: v0 := const 42; goto BB3
BB2: v1 := const 69; goto BB3
BB3: v2 := φ(v0,v1); v3:=addi t,1; ret v3

Phi is phony function: v2 is v0 if coming from first predecessor, or v1 from second predecessor

These days we still live in Fran Allen's world, but with a twist: we no longer model programs as graphs of assignments, but rather graphs of definitions. The introduction in the mid-80s of so-called "static single-assignment" (SSA) form graphs mean that instead of having two assignments to t, we would define two different values v0 and v1. Then later instead of reading the value of the storage location associated with t, we define v2 to be either v0 or v1: the former if we reach the use of t in BB3 from BB1, the latter if we are coming from BB2.

If you think on the machine level, in terms of what the resulting machine code will be, this either function isn't a real operation; probably register allocation will put v0, v1, and v2 in the same place, say $rax. The function linking the definition of v2 to the inputs v0 and v1 is purely notational; in a way, you could say that it is phony, or not real. But when the creators of SSA went to submit this notation for publication they knew that they would need something that sounded more rigorous than "phony function", so they instead called it a "phi" (φ) function. Really.

2003: MLton

Refinement: phi variables are basic block args

graph := array<block>
block := tuple<preds, succs, args, insts>

Inputs of phis implicitly computed from preds

BB0(a0): if a0 then BB1() else BB2()
BB1(): v0 := const 42; BB3(v0)
BB2(): v1 := const 69; BB3(v1)
BB3(v2): v3 := addi v2, 1; ret v3

SSA is still where it's at, as a conventional solution to the IR problem. There have been some refinements, though. I learned of one of them from MLton; I don't know if they were first but they had the idea of interpreting phi variables as arguments to basic blocks. In this formulation, you don't have explicit phi instructions; rather the "v2 is either v1 or v0" property is expressed by v2 being a parameter of a block which is "called" with either v0 or v1 as an argument. It's the same semantics, but an interesting notational change.

Refinement: Control tail

Often nice to know how a block ends (e.g. to compute phi input vars)

graph := array<block>
block := tuple<preds, succs, args, insts,
               control>
control := if v then L1 else L2
         | L(v, ...)
         | switch(v, L1, L2, ...)
         | ret v

One other refinement to SSA is to note that basic blocks consist of some number of instructions that can define values or have side effects but which otherwise exhibit fall-through control flow, followed by a single instruction that transfers control to another block. We might as well store that control instruction separately; this would let us easily know how a block ends, and in the case of phi block arguments, easily say what values are the inputs of a phi variable. So let's do that.

Refinement: DRY

Block successors directly computable from control

Predecessors graph is inverse of successors graph

graph := array<block>
block := tuple<args, insts, control>

Can we simplify further?

At this point we notice that we are repeating ourselves; the successors of a block can be computed directly from the block's terminal control instruction. Let's drop those as a distinct part of a block, because when you transform a program it's unpleasant to have to needlessly update something in two places.

While we're doing that, we note that the predecessors array is also redundant, as it can be computed from the graph of block successors. Here we start to wonder: am I simpliying or am I removing something that is fundamental to the algorithmic complexity of the various graph transformations that I need to do? We press on, though, hoping we will get somewhere interesting.

Basic blocks are annoying

Ceremony about managing insts; array or doubly-linked list?

Nonuniformity: “local” vs ‘`global’' transformations

Optimizations transform graph A to graph B; mutability complicates this task

  • Desire to keep A in mind while making B
  • Bugs because of spooky action at a distance

Recall that the context for this meander is Guile's compiler, which is written in Scheme. Scheme doesn't have expandable arrays built-in. You can build them, of course, but it is annoying. Also, in Scheme-land, functions with side-effects are conventionally suffixed with an exclamation mark; after too many of them, both the writer and the reader get fatigued. I know it's a silly argument but it's one of the things that made me grumpy about basic blocks.

If you permit me to continue with this introspection, I find there is an uneasy relationship between instructions and locations in an IR that is structured around basic blocks. Do instructions live in a function-level array and a basic block is an array of instruction indices? How do you get from instruction to basic block? How would you hoist an instruction to another basic block, might you need to reallocate the block itself?

And when you go to transform a graph of blocks... well how do you do that? Is it in-place? That would be efficient; but what if you need to refer to the original program during the transformation? Might you risk reading a stale graph?

It seems to me that there are too many concepts, that in the same way that SSA itself moved away from assignment to a more declarative language, that perhaps there is something else here that might be more appropriate to the task of a middle-end.

Basic blocks, phi vars redundant

Blocks: label with args sufficient; “containing” multiple instructions is superfluous

Unify the two ways of naming values: every var is a phi

graph := array<block>
block := tuple<args, inst>
inst  := L(expr)
       | if v then L1() else L2()
       ...
expr  := const C
       | add x, y
       ...

I took a number of tacks here, but the one I ended up on was to declare that basic blocks themselves are redundant. Instead of containing an array of instructions with fallthrough control-flow, why not just make every instruction a control instruction? (Yes, there are arguments against this, but do come along for the ride, we get to a funny place.)

While you are doing that, you might as well unify the two ways in which values are named in a MLton-style compiler: instead of distinguishing between basic block arguments and values defined within a basic block, we might as well make all names into basic block arguments.

Arrays annoying

Array of blocks implicitly associates a label with each block

Optimizations add and remove blocks; annoying to have dead array entries

Keep labels as small integers, but use a map instead of an array

graph := map<label, block>

In the traditional SSA CFG IR, a graph transformation would often not touch the structure of the graph of blocks. But now having given each instruction its own basic block, we find that transformations of the program necessarily change the graph. Consider an instruction that we elide; before, we would just remove it from its basic block, or replace it with a no-op. Now, we have to find its predecessor(s), and forward them to the instruction's successor. It would be useful to have a more capable data structure to represent this graph. We might as well keep labels as being small integers, but allow for sparse maps and growth by using an integer-specialized map instead of an array.

This is CPS soup

graph := map<label, cont>
cont  := tuple<args, term>
term  := continue to L
           with values from expr
       | if v then L1() else L2()
       ...
expr  := const C
       | add x, y
       ...

SSA is CPS

This is exactly what CPS soup is! We came at it "from below", so to speak; instead of the heady fumes of the lambda calculus, we get here from down-to-earth basic blocks. (If you prefer the other way around, you might enjoy this article from a long time ago.) The remainder of this presentation goes deeper into what it is like to work with CPS soup in practice.

Scope and dominators

BB0(a0): if a0 then BB1() else BB2()
BB1(): v0 := const 42; BB3(v0)
BB2(): v1 := const 69; BB3(v1)
BB3(v2): v3 := addi v2, 1; ret v3

What vars are “in scope” at BB3? a0 and v2.

Not v0; not all paths from BB0 to BB3 define v0.

a0 always defined: its definition dominates all uses.

BB0 dominates BB3: All paths to BB3 go through BB0.

Before moving on, though, we should discuss what it means in an SSA-style IR that variables are defined rather than assigned. If you consider variables as locations to which values can be assigned and which initially hold garbage, you can read them at any point in your program. You might get garbage, though, if the variable wasn't assigned something sensible on the path that led to reading the location's value. It sounds bonkers but it is still the C and C++ semantic model.

If we switch instead to a definition-oriented IR, then a variable never has garbage; the single definition always precedes any uses of the variable. That is to say that all paths from the function entry to the use of a variable must pass through the variable's definition, or, in the jargon, that definitions dominate uses. This is an invariant of an SSA-style IR, that all variable uses be dominated by their associated definition.

You can flip the question around to ask what variables are available for use at a given program point, which might be read equivalently as which variables are in scope; the answer is, all definitions from all program points that dominate the use site. The "CPS" in "CPS soup" stands for continuation-passing style, a dialect of the lambda calculus, which has also has a history of use as a compiler intermediate representation. But it turns out that if we use the lambda calculus in its conventional form, we end up needing to maintain a lexical scope nesting at the same time that we maintain the control-flow graph, and the lexical scope tree can fail to reflect the dominator tree. I go into this topic in more detail in an old article, and if it interests you, please do go deep.

CPS soup in Guile

Compilation unit is intmap of label to cont

cont := $kargs names vars term
      | ...
term := $continue k src expr
      | ...
expr := $const C
      | $primcall ’add #f (a b)
      | ...

Conventionally, entry point is lowest-numbered label

Anyway! In Guile, the concrete form that CPS soup takes is that a program is an intmap of label to cont. A cont is the smallest labellable unit of code. You can call them blocks if that makes you feel better. One kind of cont, $kargs, binds incoming values to variables. It has a list of variables, vars, and also has an associated list of human-readable names, names, for debugging purposes.

A $kargs contains a term, which is like a control instruction. One kind of term is $continue, which passes control to a continuation k. Using our earlier language, this is just goto *k*, with values, as in MLton. (The src is a source location for the term.) The values come from the term's expr, of which there are a dozen kinds or so, for example $const which passes a literal constant, or $primcall, which invokes some kind of primitive operation, which above is add. The primcall may have an immediate operand, in this case #f, and some variables that it uses, in this case a and b. The number and type of the produced values is a property of the primcall; some are just for effect, some produce one value, some more.

CPS soup

term := $continue k src expr
      | $branch kf kt src op param args
      | $switch kf kt* src arg
      | $prompt k kh src escape? tag
      | $throw src op param args

Expressions can have effects, produce values

expr := $const val
      | $primcall name param args
      | $values args
      | $call proc args
      | ...

There are other kinds of terms besides $continue: there is $branch, which proceeds either to the false continuation kf or the true continuation kt depending on the result of performing op on the variables args, with immediate operand param. In our running example, we might have made the initial term via:

(build-term
  ($branch BB1 BB2 'false? #f (a0)))

The definition of build-term (and build-cont and build-exp) is in the (language cps) module.

There is also $switch, which takes an unboxed unsigned integer arg and performs an array dispatch to the continuations in the list kt, or kf otherwise.

There is $prompt which continues to its k, having pushed on a new continuation delimiter associated with the var tag; if code aborts to tag before the prompt exits via an unwind primcall, the stack will be unwound and control passed to the handler continuation kh. If escape? is true, the continuation is escape-only and aborting to the prompt doesn't need to capture the suspended continuation.

Finally there is $throw, which doesn't continue at all, because it causes a non-resumable exception to be thrown. And that's it; it's just a handful of kinds of term, determined by the different shapes of control-flow (how many continuations the term has).

When it comes to values, we have about a dozen expression kinds. We saw $const and $primcall, but I want to explicitly mention $values, which simply passes on some number of values. Often a $values expression corresponds to passing an input to a phi variable, though $kargs vars can get their definitions from any expression that produces the right number of values.

Kinds of continuations

Guile functions untyped, can multiple return values

Error if too few values, possibly truncate too many values, possibly cons as rest arg...

Calling convention: contract between val producer & consumer

  • both on call and return side

Continuation of $call unlike that of $const

When a $continue term continues to a $kargs with a $const 42 expression, there are a number of invariants that the compiler can ensure: that the $kargs continuation is always passed the expected number of values, that the vars that it binds can be allocated to specific locations (e.g. registers), and that because all predecessors of the $kargs are known, that those predecessors can place their values directly into the variable's storage locations. Effectively, the compiler determines a custom calling convention between each $kargs and its predecessors.

Consider the $call expression, though; in general you don't know what the callee will do to produce its values. You don't even generally know that it will produce the right number of values. Therefore $call can't (in general) continue to $kargs; instead it continues to $kreceive, which expects the return values in well-known places. $kreceive will check that it is getting the right number of values and then continue to a $kargs, shuffling those values into place. A standard calling convention defines how functions return values to callers.

The conts

cont := $kfun src meta self ktail kentry
      | $kclause arity kbody kalternate
      | $kargs names syms term
      | $kreceive arity kbody
      | $ktail

$kclause, $kreceive very similar

Continue to $ktail: return

$call and return (and $throw, $prompt) exit first-order flow graph

Of course, a $call expression could be a tail-call, in which case it would continue instead to $ktail, indicating an exit from the first-order function-local control-flow graph.

The calling convention also specifies how to pass arguments to callees, and likewise those continuations have a fixed calling convention; in Guile we start functions with $kfun, which has some metadata attached, and then proceed to $kclause which bridges the boundary between the standard calling convention and the specialized graph of $kargs continuations. (Many details of this could be tweaked, for example that the case-lambda dispatch built-in to $kclause could instead dispatch to distinct functions instead of to different places in the same function; historical accidents abound.)

As a detail, if a function is well-known, in that all its callers are known, then we can lighten the calling convention, moving the argument-count check to callees. In that case $kfun continues directly to $kargs. Similarly for return values, optimizations can make $call continue to $kargs, though there is still some value-shuffling to do.

High and low

CPS bridges AST (Tree-IL) and target code

High-level: vars in outer functions in scope

Closure conversion between high and low

Low-level: Explicit closure representations; access free vars through closure

CPS soup is the bridge between parsed Scheme and machine code. It starts out quite high-level, notably allowing for nested scope, in which expressions can directly refer to free variables. Variables are small integers, and for high-level CPS, variable indices have to be unique across all functions in a program. CPS gets lowered via closure conversion, which chooses specific representations for each closure that remains after optimization. After closure conversion, all variable access is local to the function; free variables are accessed via explicit loads from a function's closure.

Optimizations at all levels

Optimizations before and after lowering

Some exprs only present in one level

Some high-level optimizations can merge functions (higher-order to first-order)

Because of the broad remit of CPS, the language itself has two dialects, high and low. The high level dialect has cross-function variable references, first-class abstract functions (whose representation hasn't been chosen), and recursive function binding. The low-level dialect has only specific ways to refer to functions: labels and specific closure representations. It also includes calls to function labels instead of just function values. But these are minor variations; some optimization and transformation passes can work on either dialect.

Practicalities

Intmap, intset: Clojure-style persistent functional data structures

Program: intmap<label,cont>

Optimization: program→program

Identify functions: (program,label)→intset<label>

Edges: intmap<label,intset<label>>

Compute succs: (program,label)→edges

Compute preds: edges→edges

I mentioned that programs were intmaps, and specifically in Guile they are Clojure/Bagwell-style persistent functional data structures. By functional I mean that intmaps (and intsets) are values that can't be mutated in place (though we do have the transient optimization).

I find that immutability has the effect of deploying a sense of calm to the compiler hacker -- I don't need to worry about data structures changing out from under me; instead I just structure all the transformations that you need to do as functions. An optimization is just a function that takes an intmap and produces another intmap. An analysis associating some data with each program label is just a function that computes an intmap, given a program; that analysis will never be invalidated by subsequent transformations, because the program to which it applies will never be mutated.

This pervasive feeling of calm allows me to tackle problems that I wouldn't have otherwise been able to fit into my head. One example is the novel online CSE pass; one day I'll either wrap that up as a paper or just capitulate and blog it instead.

Flow analysis

A[k] = meet(A[p] for p in preds[k])
         - kill[k] + gen[k]

Compute available values at labels:

  • A: intmap<label,intset<val>>
  • meet: intmap-intersect<intset-intersect>
  • -, +: intset-subtract, intset-union
  • kill[k]: values invalidated by cont because of side effects
  • gen[k]: values defined at k

But to keep it concrete, let's take the example of flow analysis. For example, you might want to compute "available values" at a given label: these are the values that are candidates for common subexpression elimination. For example if a term is dominated by a car x primcall whose value is bound to v, and there is no path from the definition of V to a subsequent car x primcall, we can replace that second duplicate operation with $values (v) instead.

There is a standard solution for this problem, which is to solve the flow equation above. I wrote about this at length ages ago, but looking back on it, the thing that pleases me is how easy it is to decompose the task of flow analysis into manageable parts, and how the types tell you exactly what you need to do. It's easy to compute an initial analysis A, easy to define your meet function when your maps and sets have built-in intersect and union operators, easy to define what addition and subtraction mean over sets, and so on.

Persistent data structures FTW

  • meet: intmap-intersect<intset-intersect>
  • -, +: intset-subtract, intset-union

Naïve: O(nconts * nvals)

Structure-sharing: O(nconts * log(nvals))

Computing an analysis isn't free, but it is manageable in cost: the structure-sharing means that meet is usually trivial (for fallthrough control flow) and the cost of + and - is proportional to the log of the problem size.

CPS soup: strengths

Relatively uniform, orthogonal

Facilitates functional transformations and analyses, lowering mental load: “I just have to write a function from foo to bar; I can do that”

Encourages global optimizations

Some kinds of bugs prevented by construction (unintended shared mutable state)

We get the SSA optimization literature

Well, we're getting to the end here, and I want to take a step back. Guile has used CPS soup as its middle-end IR for about 8 years now, enough time to appreciate its fine points while also understanding its weaknesses.

On the plus side, it has what to me is a kind of low cognitive overhead, and I say that not just because I came up with it: Guile's development team is small and not particularly well-resourced, and we can't afford complicated things. The simplicity of CPS soup works well for our development process (flawed though that process may be!).

I also like how by having every variable be potentially a phi, that any optimization that we implement will be global (i.e. not local to a basic block) by default.

Perhaps best of all, we get these benefits while also being able to use the existing SSA transformation literature. Because CPS is SSA, the lessons learned in SSA (e.g. loop peeling) apply directly.

CPS soup: weaknesses

Pointer-chasing, indirection through intmaps

Heavier than basic blocks: more control-flow edges

Names bound at continuation only; phi predecessors share a name

Over-linearizes control, relative to sea-of-nodes

Overhead of re-computation of analyses

CPS soup is not without its drawbacks, though. It's not suitable for JIT compilers, because it imposes some significant constant-factor (and sometimes algorithmic) overheads. You are always indirecting through intmaps and intsets, and these data structures involve significant pointer-chasing.

Also, there are some forms of lightweight flow analysis that can be performed naturally on a graph of basic blocks without looking too much at the contents of the blocks; for example in our available variables analysis you could run it over blocks instead of individual instructions. In these cases, basic blocks themselves are an optimization, as they can reduce the size of the problem space, with corresponding reductions in time and memory use for analyses and transformations. Of course you could overlay a basic block graph on top of CPS soup, but it's not a well-worn path.

There is a little detail that not all phi predecessor values have names, since names are bound at successors (continuations). But this is a detail; if these names are important, little $values trampolines can be inserted.

Probably the main drawback as an IR is that the graph of conts in CPS soup over-linearizes the program. There are other intermediate representations that don't encode ordering constraints where there are none; perhaps it would be useful to marry CPS soup with sea-of-nodes, at least during some transformations.

Finally, CPS soup does not encourage a style of programming where an analysis is incrementally kept up to date as a program is transformed in small ways. The result is that we end up performing much redundant computation within each individual optimization pass.

Recap

CPS soup is SSA, distilled

Labels and vars are small integers

Programs map labels to conts

Conts are the smallest labellable unit of code

Conts can have terms that continue to other conts

Compilation simplifies and lowers programs

Wasm vs VM backend: a question for another day :)

But all in all, CPS soup has been good for Guile. It's just SSA by another name, in a simpler form, with a functional flavor. Or, it's just CPS, but first-order only, without lambda.

In the near future, I am interested in seeing what a new GC will do for CPS soup; will bump-pointer allocation palliate some of the costs of pointer-chasing? We'll see. A tricky thing about CPS soup is that I don't think that anyone else has tried it in other languages, so it's hard to objectively understand its characteristics independent of Guile itself.

Finally, it would be nice to engage in the academic conversation by publishing a paper somewhere; I would like to see interesting criticism, and blog posts don't really participate in the citation graph. But in the limited time available to me, faced with the choice between hacking on something and writing a paper, it's always been hacking, so far :)

Speaking of limited time, I probably need to hit publish on this one and move on. Happy hacking to all, and until next time.

20 May, 2023 07:10AM by Andy Wingo

May 19, 2023

FSF Blogs

Frans de Jonge tells us about KOReader in this AGPLv3 interview

This is the latest installment of our Licensing and Compliance Lab's series on free software developers who choose GNU licenses for their works.

19 May, 2023 05:02PM

May 18, 2023

GNU Health

ADAV Weimar and GNU Solidario join forces in Telemedicine with GNU Health in Afghanistan

ADAV-Weimar (Afghan-German Doctors Association) and GNU Solidario have formalized a agreement to implement GNU Health for local and remote physicians to improve the medical care of the people in Afghanistan.

ADAV-Weimar (Afghan-German Doctors Association) is a registered voluntary association in Germany, founded in 2004 and counts with the support of over 150 health professionals from around the world. The organization provides scientific and practical help with establishing medical facilities and efficient healthcare in Afghanistan. It supports with building small but efficient Special Clinics and practices knowledge transfer by training Afghani doctors and medical staff and providing telemedicine. Furthermore, ADAV Weimar in co-operation with German E-Learning specialist Lecturio and partner Universities in Afghanistan has established an E-Learning program for medical students that provides free access to content prepared by awarded lectures from world class. Thus, ADAV-Weimar has become an inherent part of the international relations of all medical faculties in Afghanistan.

Dr. Luis Falcon, president of GNU Solidario, and Dr. Azim Mosafer, chairman of ADAV-Weimar e.V, signed this past April an initial three-year agreement to setup GNU Health Hospital Management systems, where physicians both from Afghanistan and abroad can work together to improve the healthcare and lives of the Afghan women, men and children.

Dr. Azim Mosafer – an Afghan-born German spine surgeon – is the head of the German-Afghan Doctors Association and currently involved in a European-Afghani telemedicine project. In a recent interview, Dr. Mosafer, who travels once a year to Afghanistan stressed the importance of telemedicine, at a national level and also within the country, to also provide medical care to people living in rural areas.

Since 2005 I have traveled to Afghanistan once a year for two weeks, to provide medical care. However, as a result of the high security risks in Afghanistan, fewer and fewer of my colleagues were willing to accompany me. We also began to think about doing something more effective. Even within Afghanistan, people have to travel great distances and overcome geographic obstacles in order to obtain medical care. With telemedicine such geographic hurdles can by bypassed internationally, but also inside the country.

Dr. Azim Mosafer, interview on AO spine. Source ADAV Weimar

The project has already started and the initial GNU Health Hospital Management training provided to the ADAV Weimar personnel from Germany. Initially, the implementation will provide the GNU Health Hospital Management System with telemedicine support and functionality such as:

  • General Practice
  • Family Medicine
  • Surgery
  • Laboratory
  • Ophthalmology
  • Medical Imaging
  • Laboratory
  • Odontology
  • Gynecology and Obstetrics
  • Pediatrics
  • Health, Functioning and Disability
Source ADAV Weimar e.V.

In GNU Solidario, we are proud to cooperate with ADAV Weimar implementing the GNU Health ecosystem. We are excited to provide the latest technology in health informatics to the betterment of science and society. The European doctors will be able to cooperate with the local health professionals in providing the best clinical assessment and medical care possible to the Afghan children, women and men, specially in this difficult times. It is now when they need it most.

Dr. Luis Falcon

This is just the beginning of the journey. We are confident that other components from the Libre digital health ecosystem, such as the GNU Health Federation and MyGNUHealth Personal Health Record will further help health professionals in Afghanistan and Europe the best tools for cooperation, knowledge transfer and medical care to the people in Afghanistan.

Resources:

18 May, 2023 11:58AM by GNU Solidario

May 17, 2023

FSF Latin America

IRPF-Livre 2023 released

IRPF-Livre 2023 released

Governments come and go, but the oppression of imposed taxing software for taxation remains.

For a lot of people, using software they cannot control is like water for a fish: a part of the environment they're in. When the water is low on oxygen, they may even feel the discomfort, but they seldom trace it back to the root cause.

For us who love, live and breathe software freedom, any program that takes it away, that attempts to control our computing and ultimately ourselves, is painful like a sore toe in a tight shoe.

Uncomfortable and painful as the oxygen-deprived water and the tight shoe might be, being forced to breathe or wear them, prevented from seeking better waters or from taking the shoes out, is unbearable.

We struggle to correct an analogous injustice. We had a chance to relieve one case of imposed taxing software for taxation, so we took it, and held on to it:

Back in 2007, IRPF, the program that Brazilian taxpayers are required to run to prepare their income tax returns, was released without obfuscation, with debug information and surprisingly even under an acceptable license, which enabled us to reverse engineer it and from then on to update the rescued source code.

That relieved the primary oppression, but the government changes the software yearly, so every year brings a new threat to our freedom, and defending it requires duplicating the changes. That, too, is unjustly taxing!

Democratic governments ought to respect our freedom, not threaten it. The tax laws and regulations that the program implements are and must be public code. Nothing that the software is programmed to do should be a secret. The tax returns need to be and are verified after turning in. Nothing justifies making the program freedom depriving.

That it remains so is a manifestation of the bad habit of abusing power through software, of hijacking others' computers to serve one's purposes, without thinking much of it. Thus we draw the fish's attention to the toxic water, and to the root cause of its toxicity.

As we celebrate the 16th anniversary of the IRPF-Livre project, and take the too-tight shoes out by releasing its updates for 2023, we call upon the new Brazilian government, and indeed upon all democratic governments, to quit this bad habit, and to release, under freedom- and transparency-respecting terms, the source code for all government-mandated programs, so that they are not imposed taxing software.
https://www.fsfla.org/~lxoliva/fsfla/irpf-livre/2023/


About Imposed Taxing Software

Since 2006, we have been running a campaign against imposed taxing software: programs that are imposed in the sense that you cannot avoid them, and taxing in the sense that they burden you in a way that resembles a tax, but is exempt from social benefits and paid for with your freedom.

Nonfree programs are unjust and too onerous (even when they are nominally gratis), because they imply a loss of freedom, that is, of control over your digital life. When this burden (of suppressed freedom) is compounded with the imposition of use of such programs, they become profoundly oppressive: imposed taxing software.

Our initial focus was on oppressive software imposed by governments, such as mandatory tax-related programs and software required to interact with public banks.
https://www.fsfla.org/circular/2006-11#Editorial

While pressuring the government to liberate income tax software in Brazil, we have been updating and publishing a compatible and freedom-respecting version every year since 2007.
https://www.fsfla.org/anuncio/2012-10-Acesso-SoftImp
https://www.fsfla.org/~lxoliva/fsfla/irpf-livre/

In 2023, we extended the campaign to taxing software imposed by private providers: when freedom-depriving software is required to obtain or enjoy products or services.

To be clear, this campaign is not (solely) about software for taxation, but rather about software that is taxing (an unjust burden, because it taxes your freedom; the software is itself like a tax), and that, on top of that, is imposed, thus profoundly oppressive.


About IRPF-Livre

It's a software development project to prepare Natural Person's Income Tax returns compliant with the standards defined by the Brazilian Secretaria de Receita Federal (IRS), but without the technical and legal insecurity imposed by it.

IRPF-Livre is Free Software, that is, software that respects users' freedom to run it for any purpose, to study its source code and adapt it to their needs, and to distribute copies, modified or not.

The program is available both in source and Java object code forms:
http://www.fsfla.org/~lxoliva/fsfla/irpf-livre/


About FSFLA

Free Software Foundation Latin America joined in 2005 the international FSF network, previously formed by Free Software Foundations in the United States, in Europe and in India. These sister organizations work in their corresponding geographies towards promoting the same Free Software ideals and defending the same freedoms for software users and developers, working locally but cooperating globally.
https://www.fsfla.org/


Copyright 2023 FSFLA

Permission is granted to make and distribute verbatim copies of this entire document without royalty, provided the copyright notice, the document's official URL, and this permission notice are preserved.

Permission is also granted to make and distribute verbatim copies of individual sections of this document worldwide without royalty provided the copyright notice and the permission notice above are preserved, and the document's official URL is preserved or replaced by the individual section's official URL.

https://www.fsfla.org/anuncio/2023-05-IRPF-Livre-2023

17 May, 2023 10:00PM

May 16, 2023

FSF News

May 02, 2023

Announcing the FSF's board candidates

02 May, 2023 06:30PM

Andy Wingo

structure and interpretation of ark

Hello, dear readers! Today's article describes Ark, a new JavaScript-based mobile development platform. If you haven't read them yet, you might want to start by having a look at my past articles on Capacitor, React Native, NativeScript, and Flutter; having a common understanding of the design space will help us understand where Ark is similar and where it differs.

Ark, what it is

If I had to bet, I would guess that you have not heard of Ark. (I certainly hadn't either, when commissioned to do this research series.) To a first approximation, Ark—or rather, what I am calling Ark; I don't actually know the name for the whole architecture—is a loosely Flutter-like UI library implemented on top of a dialect of JavaScript, with build-time compilation to bytecode (like Hermes) but also with support for just-in-time and ahead-of-time compilation of bytecode to native code. It is made by Huawei.

At this point if you are already interested in this research series, I am sure this description raises more questions than it answers. Flutter-like? A dialect? Native compilation? Targetting what platforms? From Huawei? We'll get to all of these, but I think we need to start with the last question.

How did we get here?

In my last article on Flutter, I told a kind of just-so history of how Dart and Flutter came to their point in the design space. Thanks to corrections from a kind reader, it happened to also be more or less correct. In this article, though, I haven't talked with Ark developers at all; I don't have the benefit of a true claim on history. And yet, the only way I can understand Ark is by inventing a narrative, so here we go. It might even be true!

Recall that in 2018, Huawei was a dominant presence in the smartphone market. They were shipping excellent hardware at good prices both to the Chinese and to the global markets. Like most non-Apple, non-Google manufacturers, they shipped Android, and like most Android OEMs, they shipped Google's proprietary apps (mail, maps, etc.).

But then, over the next couple years, the US decided that allowing Huawei to continue on as before was, like, against national security interests or something. Huawei was barred from American markets, a number of suppliers were forbidden from selling hardware components to Huawei, and even Google was prohibited from shipping its mobile apps on Huawei devices. The effect on Huawei's market share for mobile devices was enormous: its revenue was cut in half over a period of a couple years.

In this position, as Huawei, what do you do? I can't even imagine, but specifically looking at smartphones, I think I would probably do about what they did. I'd fork Android, for starters, because that's what you already know and ship, and Android is mostly open source. I'd probably plan on continuing to use its lower-level operating system pieces indefinitely (kernel and so on) because that's not a value differentiator. I'd probably ship the same apps on top at first, because otherwise you slip all the release schedules and lose revenue entirely.

But, gosh, there is the risk that your product will be perceived as just a worse version of Android: that's not a good position to be in. You need to be different, and ideally better. That will take time. In the meantime, you claim that you're different, without actually being different yet. It's a somewhat ridiculous position to be in, but I can understand how you get here; Ars Technica published a scathing review poking fun at the situation. But, you are big enough to ride it out, knowing that somehow eventually you will be different.

Up to now, this part of the story is relatively well-known; the part that follows is more speculative on my part. Firstly, I would note that Huawei had been working for a while on a compiler and language run-time called Ark Compiler, with the goal of getting better performance out of Android applications. If I understand correctly, this compiler took the Java / Dalvik / Android Run Time bytecodes as its input, and outputted native binaries along with a new run-time implementation.

As I can attest from personal experience, having a compiler leads to hubris: you start to consider source languages like a hungry person looks at a restaurant menu. "Wouldn't it be nice to ingest that?" That's what we say at restaurants, right, fellow humans? So in 2019 and 2020 when the Android rug was pulled out from underneath Huawei, I think having in-house compiler expertise allowed them to consider whether they wanted to stick with Java at all, or whether it might be better to choose a more fashionable language.

Like black, JavaScript is always in fashion. What would it mean, then, to retool Huawei's operating system -- by then known by the name "HarmonyOS" -- to expose a JavaScript-based API as its primary app development framework? You could use your Ark compiler somehow to implement JavaScript (hubris!) and then you need a UI framework. Having ditched Java, it is now thinkable to ditch all the other Android standard libraries, including the UI toolkit: you start anew, in a way. So are you going to build a Capacitor, a React Native, a NativeScript, a Flutter? Surely not precisely any of these, but what will it be like, and how will it differ?

Incidentally, I don't know the origin story for the name Ark, but to me it brings to mind tragedy and rebuilding: in the midst of being cut off from your rich Android ecosystem, you launch a boat into the sea, holding a promise of a new future built differently. Hope and hubris in one vessel.

Two programming interfaces

In the end, Huawei builds two things: something web-like and something like Flutter. (I don't mean to suggest copying or degeneracy here; it's rather that I can only understand things in relation to other things, and these are my closest points of comparison for what they built.)

The web-like programming interface specifies UIs using an XML dialect, HML, and styles the resulting node tree with CSS. You augment these nodes with JavaScript behavior; the main app is a set of DOM-like event handlers. There is an API to dynamically create DOM nodes, but unlike the other systems we have examined, the HarmonyOS documentation doesn't really sell you on using a high-level framework like Angular.

If this were it, I think Ark would not be so compelling: the programming model is more like what was available back in the DHTML days. I wouldn't expect people to be able to make rich applications that delight users, given these primitives, though CSS animation and the HML loop and conditional rendering from the template system might be just expressive enough for simple applications.

The more interesting side is the so-called "declarative" UI programming model which exposes a Flutter/React-like interface. The programmer describes the "what" of the UI by providing a tree of UI nodes in its build function, and the framework takes care of calling build when necessary and of rendering that tree to the screen.

Here I need to show some example code, because it is... weird. Well, I find it weird, but it's not too far from SwiftUI in flavor. A small example from the fine manual:

@Entry
@Component
struct MyComponent {
  build() {
    Stack() {
        Image($rawfile('Tomato.png'))
        Text('Tomato')
            .fontSize(26)
            .fontWeight(500)
    }
  }
}

The @Entry decorator (*) marks this struct (**) as being the main entry point for the app. @Component marks it as being a component, like a React functional component. Components conform to an interface (***) which defines them as having a build method which takes no arguments and returns no values: it creates the tree in a somewhat imperative way.

But as you see the flavor is somewhat declarative, so how does that work? Also, build() { ... } looks syntactically a lot like Stack() { ... }; what's the deal, are they the same?

Before going on to answer this, note my asterisks above: these are concepts that aren't in JavaScript. Indeed, programs written for HarmonyOS's declarative framework aren't JavaScript; they are in a dialect of TypeScript that Huawei calls ArkTS. In this case, an interface is a TypeScript concept. Decorators would appear to correspond to an experimental TypeScript feature, looking at the source code.

But struct is an ArkTS-specific extension, and Huawei has actually extended the TypeScript compiler to specifically recognize the @Component decorator, such that when you "call" a struct, for example as above in Stack() { ... }, TypeScript will parse that as a new expression type EtsComponentExpression, which may optionally be followed by a block. When Stack() is invoked, its children (instances of Image and Text, in this case) will be populated via running the block.

Now, though TypeScript isn't everyone's bag, it's quite normalized in the JavaScript community and not a hard sell. Language extensions like the handling of @Component pose a more challenging problem. Still, Facebook managed to sell people on JSX, so perhaps Huawei can do the same for their dialect. More on that later.

Under the hood, it would seem that we have a similar architecture to Flutter: invoking the components creates a corresponding tree of elements (as with React Native's shadow tree), which then are lowered to render nodes, which draw themselves onto layers using Skia, in a multi-threaded rendering pipeline. Underneath, the UI code actually re-uses some parts of Flutter, though from what I can tell HarmonyOS developers are replacing those over time.

Restrictions and extensions

So we see that the source language for the declarative UI framework is TypeScript, but with some extensions. It also has its restrictions, and to explain these, we have to talk about implementation.

Of the JavaScript mobile application development frameworks we discussed, Capacitor and NativeScript used "normal" JS engines from web browsers, while React Native built their own Hermes implementation. Hermes is also restricted, in a way, but mostly inasmuch as it lags the browser JS implementations; it relies on source-to-source transpilers to get access to new language features. ArkTS—that's the name of HarmonyOS's "extended TypeScript" implementation—has more fundamental restrictions.

Recall that the Ark compiler was originally built for Android apps. There you don't really have the ability to load new Java or Kotlin source code at run-time; in Java you have class loaders, but those load bytecode. On an Android device, you don't have to deal with the Java source language. If we use a similar architecture for JavaScript, though, what do we do about eval?

ArkTS's answer is: don't. As in, eval is not supported on HarmonyOS. In this way the implementation of ArkTS can be divided into two parts, a frontend that produces bytecode and a runtime that runs the bytecode, and you never have to deal with the source language on the device where the runtime is running. Like Hermes, the developer produces bytecode when building the application and ships it to the device for the runtime to handle.

Incidentally, before we move on to discuss the runtime, there are actually two front-ends that generate ArkTS bytecode: one written in C++ that seems to only handle standard TypeScript and JavaScript, and one written in TypeScript that also handles "extended TypeScript". The former has a test262 runner with about 10k skipped tests, and the latter doesn't appear to have a test262 runner. Note, I haven't actually built either one of these (or any of the other frameworks, for that matter).

The ArkTS runtime is itself built on a non-language-specific common Ark runtime, and the set of supported instructions is the union of the core ISA and the JavaScript-specific instructions. Bytecode can be interpreted, JIT-compiled, or AOT-compiled.

On the side of design documentation, it's somewhat sparse. There are some core design docs; readers may be interested in the rationale to use a bytecode interface for Ark as a whole, or the optimization overview.

Indeed ArkTS as a whole has a surfeit of optimizations, to an extent that makes me wonder which ones are actually needed. There are source-to-source optimizations on bytecode, which I expect are useful if you are generating ArkTS bytecode from JavaScript, where you probably don't have a full compiler implementation. There is a completely separate optimizer in the eTS part of the run-time, based on what would appear to be a novel "circuit-based" IR that bears some similarity to sea-of-nodes. Finally the whole thing appears to bottom out in LLVM, which of course has its own optimizer. I can only assume that this situation is somewhat transitory. Also, ArkTS does appear to generate its own native code sometimes, notably for inline cache stubs.

Of course, when it comes to JavaScript, there are some fundamental language semantics and there is also a large and growing standard library. In the case of ArkTS, this standard library is part of the run-time, like the interpreter, compilers, and the garbage collector (generational concurrent mark-sweep with optional compaction).

All in all, when I step back from it, it's a huge undertaking. Implementing JavaScript is no joke. It appears that ArkTS has done the first 90% though; the proverbial second 90% should only take a few more years :)

Evaluation

If you told a younger me that a major smartphone vendor switched from Java to JavaScript for their UI, you would probably hear me react in terms of the relative virtues of the programming languages in question. At this point in my career, though, the only thing that comes to mind is what an expensive proposition it is to change everything about an application development framework. 200 people over 5 years would be my estimate, though of course teams are variable. So what is it that we can imagine that Huawei bought with a thousand person-years of investment? Towards what other local maximum are we heading?

Startup latency

I didn't mention it before, but it would seem that one of the goals of HarmonyOS is in the name: Huawei wants to harmonize development across the different range of deployment targets. To the extent possible, it would be nice to be able to write the same kinds of programs for IoT devices as you do for feature-rich smartphones and tablets and the like. In that regard one can see through all the source code how there is a culture of doing work ahead-of-time and preventing work at run-time; for example see the design doc for the interpreter, or for the file format, or indeed the lack of JavaScript eval.

Of course, this wide range of targets also means that the HarmonyOS platform bears the burden of a high degree of abstraction; not only can you change the kernel, but also the JavaScript engine (using JerryScript on "lite" targets).

I mention this background because sometimes in news articles and indeed official communication from recent years there would seem to be some confusion that HarmonyOS is just for IoT, or aimed to be super-small, or something. In this evaluation I am mostly focussed on the feature-rich side of things, and there my understanding is that the developer will generate bytecode ahead-of-time. When an app is installed on-device, the AOT compiler will turn it into a single ELF image. This should generally lead to fast start-up.

However it would seem that the rendering library that paints UI nodes into layers and then composits those layers uses Skia in the way that Flutter did pre-Impeller, which to be fair is a quite recent change to Flutter. I expect therefore that Ark (ArkTS + ArkUI) applications also experience shader compilation jank at startup, and that they may be well-served by tesellating their shapes into primitives like Impeller does so that they can precompile a fixed, smaller set of shaders.

Jank

Maybe it's just that apparently I think Flutter is great, but ArkUI's fundamental architectural similarity to Flutter makes me think that jank will not be a big issue. There is a render thread that is separate from the ArkTS thread, so like with Flutter, async communication with main-thread interfaces is the main jank worry. And on the ArkTS side, ArkTS even has a number of extensions to be able to share objects between threads without copying, should that be needed. I am not sure how well-developed and well-supported these extensions are, though.

I am hedging my words, of course, because I am missing a bit of social proof; HarmonyOS is still in infant days, and it doesn't have much in the way of a user base outside China, from what I can tell, and my ability to read Chinese is limited to what Google Translate can do for me :) Unlike other frameworks, therefore, I haven't been as able to catch a feel of the pulse of the ArkUI user community: what people are happy about, what the pain points are.

It's also interesting that unlike iOS or Android, HarmonyOS is only exposing these "web-like" and "declarative" UI frameworks for app development. This makes it so that the same organization is responsible for the software from top to bottom, which can lead to interesting cross-cutting optimizations: functional reactive programming isn't just a developer-experience narrative, but it can directly affect the shape of the rendering pipeline. If there is jank, someone in the building is responsible for it and should be able to fix it, whether it is in the GPU driver, the kernel, the ArkTS compiler, or the application itself.

Peak performance

I don't know how to evaluate ArkTS for peak performance. Although there is a JIT compiler, I don't have the feeling that it is as tuned for adaptive optimization as V8 is.

At the same time, I find it interesting that HarmonyOS has chosen to modify JavaScript. While it is doing that, could they switch to a sound type system, to allow the kinds of AOT optimizations that Dart can do? It would be an interesting experiment.

As it is, though, if I had to guess, I would say that ArkTS is well-positioned for predictably good performance with AOT compilation, although I would be interested in seeing the results of actually running it.

Aside: On the importance of storytelling

In this series I have tried to be charitable towards the frameworks that I review, to give credit to what they are trying to do, even while noting where they aren't currently there yet. That's part of why I need a plausible narrative for how the frameworks got where they are, because that lets me have an idea of where they are going.

In that sense I think that Ark is at an interesting inflection point. When I started reading documentation about ArkUI and HarmonyOS and all that, I bounced out—there were too many architectural box diagrams, too many generic descriptions of components, too many promises with buzzwords. It felt to me like the project was trying to justify itself to a kind of clueless management chain. Was there actually anything here?

But now when I see the adoption of a modern rendering architecture and a bold new implementation of JavaScript, along with the willingness to experiment with the language, I think that there is an interesting story to be told, but this time not to management but to app developers.

Of course you wouldn't want to market to app developers when your system is still a mess because you haven't finished rebuilding an MVP yet. Retaking my charitable approach, then, I can only think that all the architectural box diagrams were a clever blind to avoid piquing outside interest while the app development kit wasn't ready yet :) As and when the system starts working well, presumably over the next year or so, I would expect HarmonyOS to invest much more heavily in marketing and developer advocacy; the story is interesting, but you have to actually tell it.

Aside: O platform, my platform

All of the previous app development frameworks that we looked at were cross-platform; Ark is not. It could be, of course: it does appear to be thoroughly open source. But HarmonyOS devices are the main target. What implications does this have?

A similar question arises in perhaps a more concrete way if we start with the mature Flutter framework: what would it mean to make a Flutter phone?

The first thought that comes to mind is that having a Flutter OS would allow for the potential for more cross-cutting optimizations that cross abstraction layers. But then I think, what does Flutter really need? It has the GPU drivers, and we aren't going to re-implement those. It has the bridge to the platform-native SDK, which is not such a large and important part of the app. You get input from the platform, but that's also not so specific. So maybe optimization is not the answer.

On the other hand, a Flutter OS would not have to solve the make-it-look-native problem; because there would be no other "native" toolkit, your apps won't look out of place. That's nice. It's not something that would make the platform compelling, though.

HarmonyOS does have this embryonic concept of app mobility, where like you could put an app from your phone on your fridge, or something. Clearly I am not doing it justice here, but let's assume it's a compelling use case. In that situation it would be nice for all devices to present similar abstractions, so you could somehow install the same app on two different kinds of devices, and they could communicate to transfer data. As you can see here though, I am straying far from my domain of expertise.

One reasonable way to "move" an app is to have it stay running on your phone and the phone just communicates pixels with your fridge (or whatever); this is the low-level solution. I think HarmonyOS appears to be going for the higher-level solution where the app actually runs logic on the device. In that case it would make sense to ship UI assets and JavaScript / extended TypeScript bytecode to the device, which would run the app with an interpreter (for low-powered devices) or use JIT/AOT compilation. The Ark runtime itself would live on all devices, specialized to their capabilities.

In a way this is the Apple WatchOS solution (as I understand it); developers publish their apps as LLVM bitcode, and Apple compiles it for the specific devices. A FlutterOS with a Flutter run-time on all devices could do something similar. As with WatchOS you wouldn't have to ship the framework itself in the app bundle; it would be on the device already.

Finally, publishing apps as some kind of intermediate representation also has security benefits: as the OS developer, you can ensure some invariants via the toolchain that you control. Of course, you would have to ensure that the Flutter API is sufficiently expressive for high-performance applications, while also not having arbitrary machine code execution vulnerabilities; there is a question of language and framework design as well as toolchain and runtime quality of implementation. HarmonyOS could be headed in this direction.

Conclusion

Ark is a fascinating effort that holds much promise. It's also still in motion; where will it be when it anneals to its own local optimum? It would appear that the system is approaching usability, but I expect a degree of churn in the near-term as Ark designers decide which language abstractions work for them and how to, well, harmonize them with the rest of JavaScript.

For me, the biggest open question is whether developers will love Ark in the way they love, say, React. In a market where Huawei is still a dominant vendor, I think the material conditions are there for a good developer experience: people tend to like Flutter and React, and Ark is similar. Huawei "just" needs to explain their framework well (and where it's hard to explain, to go back and change it so that it is explainable).

But in a more heterogeneous market, to succeed Ark would need to make a cross-platform runtime like the one Flutter has and engage in some serious marketing efforts, so that developers don't have to limit themselves to targetting the currently-marginal HarmonyOS. Selling extensions to JavaScript will be much more difficult in a context where the competition is already established, but perhaps Ark will be able to productively engage with TypeScript maintainers to move the language so it captures some of the benefits of Dart that facilitate ahead-of-time compilation.

Well, that's it for my review round-up; hope you have enjoyed the series. I have one more pending article, speculating about some future technologies. Until then, happy hacking, and see you next time.

02 May, 2023 09:13AM by Andy Wingo

May 01, 2023

FSF Blogs

FSF Events

LibrePlanet talk - June 5 - WorldVistA EHR version of the Department of Veterans Affairs Electronic Health Record

01 May, 2023 03:57PM

Free Software Directory meeting on IRC: Friday, May 26, starting at 12:00 EDT (16:00 UTC)

Join the FSF and friends on Friday, May 26, from 12:00 to 15:00 EDT (16:00 to 19:00 UTC) to help improve the Free Software Directory.

01 May, 2023 03:04PM

Free Software Directory meeting on IRC: Friday, May 19, starting at 12:00 EDT (16:00 UTC)

Join the FSF and friends on Friday, May 19, from 12:00 to 15:00 EDT (16:00 to 19:00 UTC) to help improve the Free Software Directory.

01 May, 2023 02:59PM

Free Software Directory meeting on IRC: Friday, May 12, starting at 12:00 EDT (16:00 UTC)

Join the FSF and friends on Friday, May 12, from 12:00 to 15:00 EDT (16:00 to 19:00 UTC) to help improve the Free Software Directory.

01 May, 2023 02:54PM

April 30, 2023

FSF Latin America

Bankrupt

Bankrupt

Banking institutions have sought to automate customer service through websites and, more recently, through TRApps.
https://www.fsfla.org/anuncio/2023-03-TRApps

What these banks are saving in offices and staff, we customers are paying for with security and freedom. They are morally bankrupt.

Genuine security never depends on secret software. On the contrary, transparency strengthens security.

Nevertheless, these banks impose on us, in the name of security (their own, not ours), various harmful behaviors:

  • the use of software that we cannot control and whose actions on our computers are hidden from us;

  • the use of too-short passwords; and

  • the use of devices and operating systems designed to run under someone else's control, and to collect and exfiltrate our data.

Running software controlled by others always implies a loss of freedom, and a threat to security and privacy.

The requirement to use these programs has become so common and persistent that it seems unavoidable. Thus, we have decided to expand our campaign against imposed taxing software beyond state-controlled institutions to also include private services and goods whose providers converge on such impositions.
https://www.fsfla.org/anuncio/2023-04-bancarrota#softimp

We share our board member Alexandre Oliva's recent account of his over 20 years of struggle against technological abuse by banks in his country. We highlight his recent legal victory: online banking services must be restored without requiring the installation of programs other than a standard browser. Read more:
https://www.fsfla.org/texto/bancarrota


About Imposed Taxing Software

Since 2006, we have been running a campaign against imposed taxing software: programs that are imposed in the sense that you cannot avoid them, and taxing in the sense that they burden you in a way that resembles a tax, but is exempt from social benefits and paid for with your freedom.

Nonfree programs are unjust and too onerous (even when they are nominally gratis), because they imply a loss of freedom, that is, of control over your digital life. When this burden (of suppressed freedom) is compounded with the imposition of use of such programs, they become profoundly oppressive: imposed taxing software.

Our initial focus was on oppressive software imposed by governments, such as mandatory tax-related programs and software required to interact with public banks.
https://www.fsfla.org/circular/2006-11#Editorial

While pressuring the government to liberate income tax software in Brazil, we have been updating and publishing a compatible and freedom-respecting version every year since 2007.
https://www.fsfla.org/anuncio/2012-10-Acesso-SoftImp
https://www.fsfla.org/~lxoliva/fsfla/irpf-livre/

In 2023, we extended the campaign to taxing software imposed by private providers: when freedom-depriving software is required to obtain or enjoy products or services.

To be clear, this campaign is not (solely) about software for taxation, but rather about software that is taxing (an unjust burden, because it taxes your freedom; the software is itself like a tax), and that, on top of that, is imposed, thus profoundly oppressive.


About FSFLA

Free Software Foundation Latin America joined in 2005 the international FSF network, previously formed by Free Software Foundations in the United States, in Europe and in India. These sister organizations work in their corresponding geographies towards promoting the same Free Software ideals and defending the same freedoms for software users and developers, working locally but cooperating globally.
https://www.fsfla.org/


Copyright 2023 FSFLA

Permission is granted to make and distribute verbatim copies of this entire document without royalty, provided the copyright notice, the document's official URL, and this permission notice are preserved.

Permission is also granted to make and distribute verbatim copies of individual sections of this document worldwide without royalty provided the copyright notice and the permission notice above are preserved, and the document's official URL is preserved or replaced by the individual section's official URL.

https://www.fsfla.org/anuncio/2023-04-bancarrota

30 April, 2023 09:59PM

April 29, 2023

Simon Josefsson

How To Trust A Machine

Let’s reflect on some of my recent work that started with understanding Trisquel GNU/Linux, improving transparency into apt-archives, working on reproducible builds of Trisquel, strengthening verification of apt-archives with Sigstore, and finally thinking about security device threat models. A theme in all this is improving methods to have trust in machines, or generally any external entity. While I believe that everything starts by trusting something, usually something familiar and well-known, we need to deal with misuse of that trust that leads to failure to deliver what is desired and expected from the trusted entity. How can an entity behave to invite trust? Let’s argue for some properties that can be quantitatively measured, with a focus on computer software and hardware:

  • Deterministic Behavior – given a set of circumstances, it should behave the same.
  • Verifiability and Transparency – the method (the source code) should be accessible for understanding (compare scientific method) and its binaries verifiable, i.e., it should be possible to verify that the entity actually follows the intended deterministic method (implying efforts like reproducible builds and bootstrappable builds).
  • Accountable – the entity should behave the same for everyone, and deviation should be possible prove in a way that is hard to deny, implying efforts such as Certificate Transparency and more generic checksum logs like Sigstore and Sigsum.
  • Liberating – the tools and documentation should be available as free software to enable you to replace the trusted entity if so desired. An entity that wants to restrict you from being able to replace the trusted entity is vulnerable to corruption and may stop acting trustworthy. This point of view reinforces that open source misses the point; it has become too common to use trademark laws to restrict re-use of open source software (e.g., firefox, chrome, rust).

Essentially, this boils down to: Trust, Verify and Hold Accountable. To put this dogma in perspective, it helps to understand that this approach may be harmful to human relationships (which could explain the social awkwardness of hackers), but it remains useful as a method to improve the design of computer systems, and a useful method to evaluate safety of computer systems. When a system fails some of the criteria above, we know we have more work to do to improve it.

How far have we come on this journey? Through earlier efforts, we are in a fairly good situation. Richard Stallman through GNU/FSF made us aware of the importance of free software, the Reproducible/Bootstrappable build projects made us aware of the importance of verifiability, and Certificate Transparency highlighted the need for accountable signature logs leading to efforts like Sigstore for software. None of these efforts would have seen the light of day unless people wrote free software and packaged them into distributions that we can use, and built hardware that we can run it on. While there certainly exists more work to be done on the software side, with the recent amazing full-source build of Guix based on a 357-byte hand-written seed, I believe that we are closing that loop on the software engineering side.

So what remains? Some inspiration for further work:

  • Accountable binary software distribution remains unresolved in practice, although we have some software components around (e.g., apt-sigstore and guix git authenticate). What is missing is using them for verification by default and/or to improve the signature process to use trustworthy hardware devices, and committing the signatures to transparency logs.
  • Trustworthy hardware to run trustworthy software on remains a challenge, and we owe FSF’s Respect Your Freedom credit for raising awareness of this. Many modern devices requires non-free software to work which fails most of the criteria above and are thus inherently untrustworthy.
  • Verifying rebuilds of currently published binaries on trustworthy hardware is unresolved.
  • Completing a full-source rebuild from a small seed on trustworthy hardware remains, preferably on a platform wildly different than X86 such as Raptor’s Talos II.
  • We need improved security hardware devices and improved established practices on how to use them. For example, while Gnuk on the FST enable a trustworthy software and hardware solution, the best process for using it that I can think of generate the cryptographic keys on a more complex device. Efforts like Tillitis are inspiring here.

Onwards and upwards, happy hacking!

Update 2023-05-03: Added the “Liberating” property regarding free software, instead of having it be part of the “Verifiability and Transparency”.

29 April, 2023 11:45AM by simon

April 27, 2023

FSF Blogs

April 26, 2023

GNU Guix

The Full-Source Bootstrap: Building from source all the way down

We are delighted and somewhat relieved to announce that the third reduction of the Guix bootstrap binaries has now been merged in the main branch of Guix! If you run guix pull today, you get a package graph of more than 22,000 nodes rooted in a 357-byte program—something that had never been achieved, to our knowledge, since the birth of Unix.

We refer to this as the Full-Source Bootstrap. In this post, we explain what this means concretely. This is a major milestone—if not the major milestone—in our quest for building everything from source, all the way down.

How did we get there, and why? In two previous blog posts, we elaborated on why this reduction and bootstrappability in general is so important.

One reason is to properly address supply chain security concerns. The Bitcoin community was one of the first to recognize its importance well enough to put the idea into practice. At the Breaking Bitcoin conference 2020, Carl Dong gave a fun and remarkably gentle introduction. At the end of the talk, Carl states:

The holy grail for bootstrappability will be connecting hex0 to mes.

Two years ago, at FOSDEM 2021, I (Janneke) gave a short talk about how we were planning to continue this quest.

If you think one should always be able to build software from source, then it follows that the “trusting trust” attack is only a symptom of an incomplete or missing bootstrap story.

The Road to Full-Source Bootstrap

Three years ago, the bootstrap binaries were reduced to just GNU Mes and MesCC-Tools (and the driver to build Guix packages: a static build of GNU Guile 2.0.9).

The new Full-Source Bootstrap, merged in Guix master yesterday, removes the binaries for Mes and MesCC-Tools and replaces them by bootstrap-seeds. For x86-linux (which is also used by the x86_64-linux build), this means this program hex0-seed, with ASCII-equivalent hex0_x86.hex0. Hex0 is self-hosting and its source looks like this:

    ; Where the ELF Header is going to hit  
    ; Simply jump to _start  
    ; Our main function  
    # :_start ; (0x8048054)  
    58 # POP_EAX ; Get the number of arguments  

you can spot two types of line-comment: hex0 (;) and assembly (#). The only program-code in this snippet is 58: two hexidecimal digits that are taken as two nibbles and compiled into the corresponding byte with binary value 58.

Starting from this 357-byte hex0-seed binary provided by the bootstrap-seeds, the stage0-posix package created by Jeremiah Orians first builds hex0 and then all the way up: hex1, catm, hex2, M0, cc_x86, M1, M2, get_machine (that's all of MesCC-Tools), and finally M2-Planet.

The new GNU Mes v0.24 release can be built with M2-Planet. This time with only a remarkably small change, the bottom of the package graph now looks like this (woohoo!):

                              gcc-mesboot (4.9.4)
                                      ^
                                      |
                                    (...)
                                      ^
                                      |
               binutils-mesboot (2.20.1a), glibc-mesboot (2.2.5),
                          gcc-core-mesboot (2.95.3)
                                      ^
                                      |
                             patch-mesboot (2.5.9)
                                      ^
                                      |
                       bootstrappable-tcc (0.9.26+31 patches)
                                      ^
                                      |
                            gnu-make-mesboot0 (3.80)
                                      ^
                                      |
                              gzip-mesboot (1.2.4)
                                      ^
                                      |
                               tcc-boot (0.9.27)
                                      ^
                                      |
                                 mes-boot (0.24.2)
                                      ^
                                      |
                         stage0-posix (hex0..M2-Planet)
                                      ^
                                      |
                          gash-boot, gash-utils-boot
                                      ^
                                      |
                                      *
                     bootstrap-seeds (357-bytes for x86)
                                     ~~~
                   [bootstrap-guile-2.0.9 driver (~25 MiB)]

full graph

We are excited that the NLnet Foundation has been sponsoring this work!

However, we aren't done yet; far from it.

Lost Paths

The idea of reproducible builds and bootstrappable software is not very new. Much of that was implemented for the GNU tools in the early 1990s. Working to recreate it in present time shows us much of that practice was forgotten.

Most bootstrap problems or loops are not so easy to solve and sometimes there are no obvious answers, for example:

While these examples make for a delightful puzzle from a bootstrappability perspective, we would love to see the maintainers of GNU packages consider bootstrappability and start taking more responsibility for the bootstrap story of their packages.

Next Steps

Despite this major achievement, there is still work ahead.

First, while the package graph is rooted in a 357-byte program, the set of binaries from which packages are built includes a 25 MiB statically-linked Guile, guile-bootstrap, that Guix uses as its driver to build the initial packages. 25 MiB is a tenth of what the initial bootstrap binaries use to weigh, but it is a lot compared to those 357 bytes. Can we get rid of this driver, and how?

A development effort with Timothy Sample addresses the dependency on guile-bootstrap of Gash and Gash-Utils, the pure-Scheme POSIX shell implementation central to our second milestone. On the one hand, Mes is gaining a higher level of Guile compatibility: hash table interface, record interface, variables and variable-lookup, and Guile (source) module loading support. On the other hand, Gash and Gash-Utils are getting Mes compatibility for features that Mes is lacking (notably syntax-case macros). If we pull this off, guile-bootstrap will only be used as a dependency of bootar and as the driver for Guix.

Second, the full-source bootstrap that just landed in Guix master is limited to x86_64-linux and i686-linux, but ARM and RISC-V will be joining soon. We are most grateful and excited that the NLnet Foundation has decided to continue sponsoring this work!

Some time ago, Wladimir van der Laan contributed initial RISC-V support for Mes but a major obstacle for the RISC-V bootstrap is that the “vintage” GCC-2.95.3 that was such a helpful stepping stone does not support RISC-V. Worse, the RISC-V port of GCC was introduced only in GCC 7.5.0—a version that requires C++ and cannot be bootstrapped! To this end, we have been improving MesCC, the C compiler that comes with Mes, so it is able to build GCC 4.6.5; meanwhile, Ekaitz Zarraga backported RISC-V support to GCC 4.6.5, and backported RISC-V support from the latest tcc to our bootstrappable-tcc.

Outlook

The full-source bootstrap was once deemed impossible. Yet, here we are, building the foundations of a GNU/Linux distro entirely from source, a long way towards the ideal that the Guix project has been aiming for from the start.

There are still some daunting tasks ahead. For example, what about the Linux kernel? The good news is that the bootstrappable community has grown a lot, from two people six years ago there are now around 100 people in the #bootstrappable IRC channel. Interesting times ahead!

About Bootstrappable Builds and GNU Mes

Software is bootstrappable when it does not depend on a binary seed that cannot be built from source. Software that is not bootstrappable---even if it is free software---is a serious security risk (supply chain security) for a variety of reasons. The Bootstrappable Builds project aims to reduce the number and size of binary seeds to a bare minimum.

GNU Mes is closely related to the Bootstrappable Builds project. Mes is used in the full-source bootstrap path for the Guix System.

Currently, Mes consists of a mutual self-hosting scheme interpreter and C compiler. It also implements a C library. Mes, the scheme interpreter, is written in about 5,000 lines of code of simple C and can be built with M2-Planet. MesCC, the C compiler, is written in scheme. Together, Mes and MesCC can compile bootstrappable TinyCC that is self-hosting. Using this TinyCC and the Mes C library, the entire Guix System for i686-linux and x86_64-linux is bootstrapped.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

26 April, 2023 04:00PM by Janneke Nieuwenhuizen, Ludovic Courtès

Andy Wingo

structure and interpretation of flutter

Good day, gentle hackfolk. Like an old-time fiddler I would appear to be deep in the groove, playing endless variations on a theme, in this case mobile application frameworks. But one can only recognize novelty in relation to the familiar, and today's note is a departure: we are going to look at Flutter, a UI toolkit based not on JavaScript but on the Dart language.

The present, from the past

Where to start, even? The problem is big enough that I'll approach it from three different angles: from the past, from the top, and from the bottom.

With the other frameworks we looked at, we didn't have to say much about their use of JavaScript. JavaScript is an obvious choice, in 2023 at least: it is ubiquitous, has high quality implementations, and as a language it is quite OK and progressively getting better. Up to now, "always bet on JS" has had an uninterrupted winning streak.

But winning is not the same as unanimity, and Flutter and Dart represent an interesting pole of contestation. To understand how we got here, we have to go back in time. Ten years ago, JavaScript just wasn't a great language: there were no modules, no async functions, no destructuring, no classes, no extensible iteration, no optional arguments to functions. In addition it was hobbled with a significant degree of what can only be called accidental sloppiness: with which can dynamically alter a lexical scope, direct eval that can define new local variables, Function.caller, and so on. Finally, larger teams were starting to feel the need for more rigorous language tooling that could use types to prohibit some classes of invalid programs.

All of these problems in JavaScript have been addressed over the last decade, mostly successfully. But in 2010 or so if you were a virtual machine engineer, you might look at JavaScript and think that in some other world, things could be a lot better. That's effectively what happened: the team that originally built V8 broke off and started to work on what became Dart.

Initially, Dart was targetted for inclusion in the Chrome web browser as an alternate "native" browser language. This didn't work, for various reasons, but since then Dart grew the Flutter UI toolkit, which has breathed new life into the language. And this is a review of Flutter, not a review of Dart, not really anyway; to my eyes, Dart is spiritually another JavaScript, different but in the same family. Dart's implementation has many interesting aspects as well that we'll get into later on, but all of these differences are incidental: they could just as well be implemented on top of JavaScript, TypeScript, or another source language in that family. Even if Flutter isn't strictly part of the JavaScript-based mobile application development frameworks that we are comparing, it is valuable to the extent that it shows what is possible, and in that regard there is much to say.

Flutter, from the top

At its most basic, Flutter is a UI toolkit for Dart. In many ways it is like React. Like React, its interface follows the functional-reactive paradigm: programmers describe the "what", and Flutter takes care of the "how". Also, like the phenomenon in which new developers can learn React without really knowing JavaScript, Flutter is the killer app for Dart: Flutter developers mostly learn Dart at the same time that they pick up Flutter.

In some other ways, Flutter is the logical progression of React, going in the same direction but farther along. Whereas React-on-the-web takes the user's declarative specifications of what the UI should look like and lowers them into DOM trees, and React Native lowers them to platform-native UI widgets, Flutter has its own built-in layout, rasterization, and compositing engine: Flutter draws all the pixels.

This has the predictable challenge that Flutter has to make significant investments so that its applications don't feel out-of-place on their platform, but on the other hand it opens up a huge space for experimentation and potential optimization: Flutter has the potential to beat native at its own game. Recall that with React Native, the result of the render-commit-mount process is a tree of native widgets. The native platform will surely then perform a kind of layout on those widgets, divide them into layers that correspond to GPU textures, paint those layers, then composite them to the screen -- basically, what a web engine will do.

What if we could instead skip the native tree and go directly to the lower GPU layer? That is the promise of Flutter. Flutter has the potential to create much more smooth and complex animations than the other application development frameworks we have mentioned, with lower overhead and energy consumption.

In practice... that's always the question, isn't it? Again, please accept my usual caveat that I am a compilers guy moonlighting in the user interface domain, but my understanding is that Flutter mostly lives up to its promise, but with one significant qualification which we'll get to in a minute. But before that, let's traverse Flutter from the other direction, coming up from Dart.

Dart, from the bottom

To explain some aspects of Dart I'm going to tell a just-so story that may or may not be true. I know and like many of the Dart developers, and we have similar instincts, so it's probably not too far from the truth.

Let's say you are the team that originally developed V8, and you decide to create a new language. You write a new virtual machine that looks like V8, taking Dart source code as input and applying advanced adaptive compilation techniques to get good performance. You can even be faster than JS because your language is just a bit more rigid than JavaScript is: you have traded off expressivity for performance. (Recall from our discussion of NativeScript that expressivity isn't a value judgment: there can be reasons to pay for more "mathematically appealing operational equivalences", in Felleisen's language, in exchange for applying more constraints on a language.)

But, you fail to ship the VM in a browser; what do you do? The project could die; that would be annoying, but you work for Google, so it happens all the time. However, a few interesting things happen around the same time that will cause you to pivot. One is a concurrent experiment by Chrome developers to pare the web platform down to its foundations and rebuild it. This effort will eventually become Flutter; while it was originally based on JS, eventually they will choose to switch to Dart.

The second thing that happens is that recently-invented smart phones become ubiquitous. Most people have one, and the two platforms are iOS and Android. Flutter wants to target them. You are looking for your niche, and you see that mobile application development might be it. As the Flutter people continue to experiment, you start to think about what it would mean to target mobile devices with Dart.

The initial Dart VM was made to JIT, but as we know, Apple doesn't let people do this on iOS. So instead you look to write a quick-and-dirty ahead-of-time compiler, based on your JIT compiler that takes your program as input, parses and baseline-compiles it, and generates an image that can be loaded at runtime. It ships on iOS. Funnily enough, it ships on Android too, because AOT compilation allows you to avoid some startup costs; forced to choose between peak performance via JIT and fast startup via AOT, you choose fast startup.

It's a success, you hit your product objectives, and you start to look further to a proper ahead-of-time compiler native code that can stand alone without the full Dart run-time. After all, if you have to compile at build-time, you might as well take the time to do some proper optimizations. You actually change the language to have a sound typing system so that the compiler can make program transformations that are valid as long as it can rely on the program's types.

Fun fact: I am told that the shift to a sound type system actually started before Flutter and thus before AOT, because of a Dart-to-JavaScript compiler that you inherited from the time in which you thought the web would be the main target. The Dart-to-JS compiler used to be a whole-program compiler; this enabled it to do flow-sensitive type inference, resulting in faster and smaller emitted JavaScript. But whole-program compilation doesn't scale well in terms of compilation time, so Dart-to-JS switched to separate per-module compilation. But then you lose lots of types! The way to recover the fast-and-small-emitted-JS property was through a stronger, sound type system for Dart.

At this point, you still have your virtual machine, plus your ahead-of-time compiler, plus your Dart-to-JS compiler. Such riches, such bounty! It is not a bad situation to be in, in 2023: you can offer a good development experience via the just-in-time compiled virtual machine. Apparently you can even use the JIT on iOS in developer mode, because attaching ptrace to a binary allows for native code generation. Then when you go to deploy, you make a native binary that includes everything.

For the web, you also have your nice story, even nicer than with JavaScript in some ways: because the type checker and ahead-of-time compiler are integrated in Dart, you don't have to worry about WebPack or Vite or minifiers or uglifiers or TypeScript or JSX or Babel or any of the other things that JavaScript people are used to. Granted, the tradeoff is that innovation is mostly centralized with the Dart maintainers, but currently Google seems to be investing enough so that's OK.

Stepping back, this story is not unique to Dart; many of its scenes also played out in the world of JavaScript over the last 5 or 10 years as well. Hermes (and QuickJS, for that matter) does ahead-of-time compilation, albeit only to bytecode, and V8's snapshot facility is a form of native AOT compilation. But the tooling in the JavaScript world is more diffuse than with Dart. With the perspective of developing a new JavaScript-based mobile operating system in mind, the advantages that Dart (and thus Flutter) has won over the years are also on the table for JavaScript to win. Perhaps even TypeScript could eventually migrate to have a sound type system, over time; it would take a significant investment but the JS ecosystem does evolve, if slowly.

(Full disclosure: while the other articles in this series were written without input from the authors of the frameworks under review, through what I can only think was good URL guesswork, a draft copy of this article leaked to Flutter developers. Dart hacker Slava Egorov kindly sent me a mail correcting a number of misconceptions I had about Dart's history. Fair play on whoever guessed the URL, and many thanks to Slava for the corrections; any remaining errors are wholly mine, of course!)

Evaluation

So how do we expect Flutter applications to perform? If we were writing a new mobile OS based on JavaScript, what would it mean in terms of performance to adopt a Flutter-like architecture?

Startup latency

Flutter applications are well-positioned to start fast, with ahead-of-time compilation. However they have had problems realizing this potential, with many users seeing a big stutter when they launch a Flutter app.

To explain this situation, consider the structure of a typical low-end Android mobile device: you have a small number of not-terribly-powerful CPU cores, but attached to the same memory you also have a decent GPU with many cores. For example, the SoC in the low-end Moto E7 Plus has 8 CPU cores and 128 GPU cores (texture shader units). You could paint widget pixels into memory from either the CPU or the GPU, but you'd rather do it in the GPU because it has so many more cores: in the time it takes to compute the color of a single pixel on the CPU, on the GPU you could do, like, 128 times as many, given that the comparison is often between multi-threaded rasterization on the GPU versus single-threaded rasterization on the CPU.

Flutter has always tried to paint on the GPU. Historically it has done so via a GPU back-end to the Skia graphics library, notably used by Chrome among other projects. But, Skia's API is a drawing API, not a GPU API; Skia is the one responsible for configuring the GPU to draw what we want. And here's the problem: configuring the GPU takes time. Skia generates shader code at run-time for rasterizing the specific widgets used by the Flutter programmer. That shader code then needs to be compiled to the language the GPU driver wants, which looks more like Vulkan or Metal. The process of compilation and linking takes time, potentially seconds, even.

The solution to "too much startup shader compilation" is much like the solution to "too much startup JavaScript compilation": move this phase to build time. The new Impeller rendering library does just that. However to do that, it had to change the way that Flutter renders: instead of having Skia generate specialized shaders at run-time, Impeller instead lowers the shapes that it draws to a fixed set of primitives, and then renders those primitives using a smaller, fixed set of shaders. These primitive shaders are pre-compiled at build time and included in the binary. By switching to this new renderer, Flutter should be able to avoid startup jank.

Jank

Of all the application development frameworks we have considered, to my mind Flutter is the best positioned to avoid jank. It has the React-like asynchronous functional layout model, but "closer to the metal"; by skipping the tree of native UI widgets, it can potentially spend less time for each frame render.

When you start up a Flutter app on iOS, the shell of the application is actually written in Objective C++. On Android it's the same, except that it's Java. That shell then creates a FlutterView widget and spawns a new thread to actually run Flutter (and the user's Dart code). Mostly, Flutter runs on its own, rendering frames to the GPU resources backing the FlutterView directly.

If a Flutter app needs to communicate with the platform, it passes messages across an asynchronous channel back to the main thread. Although these messages are asynchronous, this is probably the largest potential source of jank in a Flutter app, outside the initial frame paint: any graphical update which depends on the answer to an asynchronous call may lag.

Peak performance

Dart's type system and ahead-of-time compiler optimize for predictable good performance rather than the more variable but potentially higher peak performance that could be provided by just-in-time compilation.

This story should probably serve as a lesson to any future platform. The people that developed the original Dart virtual machine had a built-in bias towards just-in-time compilation, because it allows the VM to generate code that is specialized not just to the program but also to the problem at hand. A given system with ahead-of-time compilation can always be made to perform better via the addition of a just-in-time compiler, so the initial focus was on JIT compilation. On iOS of course this was not possible, but on Android and other platforms where this was available it was the default deployment model.

However, even Android switched to ahead-of-time compilation instead of the JIT model in order to reduce startup latency: doing any machine code generation at all at program startup was more work than was needed to get to the first frame. One could add JIT back again on top of AOT but it does not appear to be a high priority.

I would expect that Capacitor could beat Dart in some raw throughput benchmarks, given that Capacitor's JavaScript implementation can take advantage of the platform's native JIT capability. Does it matter, though, as long as you are hitting your frame budget? I do not know.

Aside: An escape hatch to the platform

What happens if you want to embed a web view into a Flutter app?

If you think on the problem for a moment I suspect you will arrive at the unsatisfactory answer, which is that for better or for worse, at this point it is too expensive even for Google to make a new web engine. Therefore Flutter will have to embed the native WebView. However Flutter runs on its own threads; the native WebView has its own process and threads but its interface to the app is tied to the main UI thread.

Therefore either you need to make the native WebView (or indeed any other native widget) render itself to (a region of) Flutter's GPU backing buffer, or you need to copy the native widget's pixels into their own texture and then composite them in Flutter-land. It's not so nice! The Android and iOS platform view documentation discuss some of the tradeoffs and mitigations.

Aside: For want of a canvas

There is a very funny situation in the React Native world in which, if the application programmer wants to draw to a canvas, they have to embed a whole WebView into the React Native app and then proxy the canvas calls into the WebView. Flutter is happily able to avoid this problem, because it includes its own drawing library with a canvas-like API. Of course, Flutter also has the luxury of defining its own set of standard libraries instead of necessarily inheriting them from the web, so when and if they want to provide equivalent but differently-shaped interfaces, they can do so.

Flutter manages to be more expressive than React Native in this case, without losing much in the way of understandability. Few people will have to reach to the canvas layer, but it is nice to know it is there.

Conclusion

Dart and Flutter are terribly attractive from an engineering perspective. They offer a delightful API and a high-performance, flexible runtime with a built-in toolchain. Could this experience be brought to a new mobile operating system as its primary programming interface, based on JavaScript? React Native is giving it a try, but I think there may be room to take things further to own the application from the program all the way down to the pixels.

Well, that's all from me on Flutter and Dart for the time being. Next up, a mystery guest; see you then!

26 April, 2023 01:50PM by Andy Wingo

April 25, 2023

FSF Blogs

April 24, 2023

Andy Wingo

structure and interpretation of nativescript

Greetings, hackers tall and hackers small!

We're only a few articles in to this series on mobile application development frameworks, but I feel like we are already well into our journey. We started our trip through the design space with a look at Ionic / Capacitor, which defines its user interface in terms of the web platform, and only calls out to iOS or Android native features as needed. We proceeded on to React Native, which moves closer to native by rendering to platform-provided UI widgets, layering a cross-platform development interface on top.

Today's article takes an in-depth look at NativeScript, whose point in the design space is further on the road towards the platform, unabashedly embracing the specificities of the API available on iOS and Android, exposing these interfaces directly to the application programmer.

In practice what this looks like is that a NativeScript app is a native app which simply happens to call JavaScript on the main UI thread. That JavaScript has access to all native APIs, directly, without the mediation of serialization or message-passing over a bridge or message queue.

The first time I heard this I thought that it couldn't actually be all native APIs. After all, new versions of iOS and Android come out quite frequently, and surely it would take some effort on the part of NativeScript developers to expose the new APIs to JavaScript. But no, it really includes all of the various native APIs: the NativeScript developers wrote a build-time inspector that uses the platform's native reflection capabilities to grovel through all available APIs and to automatically generate JavaScript bindings, with associated TypeScript type definitions so that the developer knows what is available.

Some of these generated files are checked into source, so you can get an idea of the range of interfaces that are accessible to programmers; for example, see the iOS type definitions for x86-64. There are bindings for, like, everything.

Given access to all the native APIs, how do you go about making an app? You could write the same kind of programs that you would in Swift or Kotlin, but in JavaScript. But this would require more than just the ability to access native capabilities when needed: it needs a thorough knowledge of the platform interfaces, plus NativeScript itself on top. Most people don't have this knowledge, and those that do are probably programming directly in Swift or Kotlin already.

On one level, NativeScript's approach is to take refuge in that most ecumenical of adjectives, "unopinionated". Whereas Ionic / Capacitor encourages use of web platform interfaces, and React Native only really supports React as a programming paradigm, NativeScript provides a low-level platform onto which you can layer a number of different high-level frameworks.

Now, most high-level JavaScript application development frameworks are oriented to targetting the web: they take descriptions of user interfaces and translate them to the DOM. When targetting NativeScript, you could make it so that they target native UI widgets instead. However given the baked-in assumptions of how widgets should be laid out (notably via CSS), there is some impedance-matching to do between DOM-like APIs and native toolkits.

NativeScript's answer to this problem is a middle layer: a cross-platform UI library that provides DOM-like abstractions and CSS layout in a way that bridges the gap between web-like and native. You can even define parts of the UI using a NativeScript-specific XML vocabulary, which NativeScript compiles to native UI widget calls at run-time. Of course, there is no CSS engine in UIKit or Android's UI toolkit, so NativeScript includes its own, implemented in JavaScript of course.

You could program directly to this middle layer, but I suspect that its real purpose is in enabling Angular, Vue, Svelte, or the like. The pitch would be that NativeScript lets app developers use pleasant high-level abstractions, but while remaining close to the native APIs; you can always drop down for more power and expressiveness if needed.

Diving back down to the low level, as we mentioned all of the interactions between JavaScript and the native platform APIs happen on the main application UI thread. NativeScript does also allow programmers to create background threads, using an implementation of the Web Worker API. One could even in theory run a React-based UI in a worker thread and proxy native UI updates to the main thread; as an unopinionated platform, NativeScript can support many different frameworks and paradigms.

Finally, there is the question of how NativeScript runs the JavaScript in an application. Recall that Ionic / Capacitor uses the native JS engine, by virtue of using the native WebView, and that React Native used to use JavaScriptCore on both platforms but now uses its own Hermes implementation. NativeScript is another point in the design space, using V8 on both platforms. (They used to use JavaScriptCore on iOS but switched to V8 once V8 was able to run on iOS in "jitless" mode.) Besides the reduced maintenance burden of using a single implementation on all platforms, this also has the advantage of being able to use V8 snapshots to move JavaScript parse-and-compile work to build-time, even on iOS.

Evaluation

NativeScript is fundamentally simple: it's V8 running in an application's main UI thread, with access to all platform native APIs. So how do we expect it to perform?

Startup latency

In theory, applications with a NativeScript-like architecture should have no problem with startup time, because they can pre-compile all of their JavaScript into V8 snapshots. Snapshots are cheap to load up because they are already in a format that V8 is ready to consume.

In practice, it would seem that V8 snapshots do not perform as expected for NativeScript. There are a number of aspects about this situation that I don't understand, which I suspect relate to the state of the tooling around V8 rather than to the fundamental approach of ahead-of-time compilation. V8 is really made for Chrome, and it could be that not enough maintenance resources have been devoted to this snapshot facility.

In the meantime, NativeScript instead uses V8's code cache feature, which caches the result of parsing and compiling JavaScript files on the device. In this way the first time an app is installed or updated, it might start up slowly, but subsequent runs are faster. If you were designing a new operating system, you'd probably want to move this work to app install-time.

As we mentioned above, NativeScript apps have access to all native APIs. That is a lot of APIs, and only some of those interfaces will actually be used by any given app. In an ideal world, we would expect the build process to only include JavaScript code for those APIs that are needed by the application. However in the presence of eval and dynamic property lookup, pruning the native API surface to the precise minimum is a hard problem for a bundler to perform on its own. The solution for the time being is to manually allow and deny subsets of the platform native API. It's not an automatic process though, so it can be error-prone.

Besides the work that the JavaScript engine has to do to load an application's code, the other startup overhead involves whatever work that JavaScript might need to perform before the first frame is shown. In the case of NativeScript, more work is done before the initial layout than one would think: the main UI XML file is parsed by an XML parser written in JavaScript, any needed CSS files are parsed and loaded (again by JavaScript), and the tree of XML elements is translated to a tree of UI elements. The layout of the items in the view tree is then computed (in JavaScript, but calling into native code to measure text and so on), and then the app is ready.

At this point, I am again going to wave my "I am just a compiler engineer" flag: I am not a UI specialist, much less a NativeScript specialist. As in compilers, performance measurement and monitoring are key to UI development, but I suspect that also as in compilers there is a role for gut instinct. Incremental improvements are best driven by metrics, but qualitative leaps are often the result of somewhat ineffable hunches or even guesswork. In that spirit I can only surmise that React Native has an advantage over NativeScript in time-to-first-frame, because its layout is performed in C++ and because its element tree is computed directly from JavaScript instead of having JavaScript interpret XML and CSS files. In any case, I look forward to the forthcoming part 2 of the NativeScript and React Native performance investigations that were started in November 2022.

If I were NativeScript and using NativeScript's UI framework, and if startup latency proves to actually be a problem, I would lean into something in the shape of Angular's ahead-of-time compilation mode, but for the middle NativeScript UI layer.

Jank

On the face of it, NativeScript is the most jank-prone of the three frameworks we have examined, because it runs JavaScript on the main application UI thread, interleaved with UI event handling and painting and all of that. If an app's JavaScript takes too long to run, the app might miss frames or fail to promptly handle an event.

On the other hand, relative to React Native, the user's code is much closer to the application's behavior. There's no asynchrony between the application's logic and its main loop: in NativeScript it is easy to identify the code causing jank and eventually fix it.

The other classic JavaScript-on-the-main-thread worry relates to garbage collection pauses. V8's garbage collector does try to minimize the stop-the-world phase by tracing the heap concurrently and leveraging parallelism during pauses. Also, the user interface of a mobile app runs in an event loop, and typically spends most of its time idle; V8 exposes some API that can take advantage of this idle time to perform housekeeping tasks instead of needing to do them when handling high-priority events.

That said, having looked into the code of both the iOS and Android run-times, NativeScript does not currently take advantage of this facility. I dug deeper and it would seem that V8 itself is in flux, as the IdleNotificationDeadline API is on its way out; is the thought that concurrent tracing is largely sufficient? I would expect that if conservative stack scanning lands, we will see a re-introduction of this kind of API, as it does make sense to synchronize with the event loop when scanning the main thread stack.

Peak performance

As we have seen in our previous evaluations, this question boils down to "is the JavaScript engine state-of-the-art, and can it perform just-in-time compilation". In the case of NativeScript, the answers are yes and maybe, respectively: V8 is state-of-the-art, and it can JIT on Android, but not on iOS.

Perhaps the mitigation here is that the hardware that iOS runs on tends to be significantly more powerful than median Android devices; if you had to pick a subset of users to penalize with an interpreter-only run-time, people with iPhones are the obvious choice, because they can afford it.

Aside: Are markets wise?

Recall that our perspective in this series is that of the designer of a new JavaScript-based mobile development platform. We are trying to answer the question of what would it look like if a new platform offered a NativeScript-like experience. In this regard, only the structure of NativeScript is of interest, and notably its "market success" is not relevant, except perhaps in some Hayekian conception of the world in which markets are necessarily smarter than, well, me, or you, or any one of us.

It must be said, though, that React Native is the 800-pound gorilla of JavaScript mobile application development. The 2022 State of JS survey shows that among survey respondents, more people are aware of React Native than any other mobile framework, and people are generally more positive about React Native than other frameworks. Does NativeScript's mitigated market share indicate something about its architecture, or does it speak speak more to the size of Facebook's budget, both on the developer experience side and on marketing?

Aside: On the expressive power of application framworks

Oddly, I think the answer to the market wisdom question might be found in a 35-year-old computer science paper, "On the expressive power of programming languages" (PDF).

In this paper, Matthias Felleisen considers the notion of what it means for one programming language to be more expressive than another. For example, is a language with just for less expressive than a language with both for and while? Intuitively we would say no, these are similar things; you can make a simple local transformation of while (x) {...} to for (;x;) {...} and you have exactly the same program semantics. On the other hand a language with just for is less expressive than one which also has goto; there is no simple local rewrite that can turn goto into for.

In the same way, we can consider the question of what it would mean for one library to be more expressive than another. After all, the API of a library exposes a language in which its user can write programs; we should be able to reason about these languages. So between React Native and NativeScript, which one is more expressive?

By Felleisen's definitions, NativeScript is clearly the more expressive language: there is no simple local transformation that can turn imperative operations on native UI widgets into equivalent functional-reactive programs. Yes, with enough glue code React Native can reach directly to native APIs in a similar way as NativeScript, but everything that touches the native UI tree is expressly under React Native's control: there is no sanctioned escape hatch.

You might think that "more expressive" is always better, but Felleisen's take is more nuanced than that. Yes, he says, more expressive languages do allow programmers to make more concise programs, because they allow programmers to define abstractions that encapsulate patterns, and this is a good thing. However he also concludes that "an increase in expressive power is related to a decrease of the set of 'natural' (mathematically appealing) operational equivalences." Less expressive programming languages are easier to reason about, in general, and indeed that is one of the recognized strengths of React's programming model: it is easy to compose components and have confidence that the result will work.

Summary

A NativeScript-like architecture offers the possibility of performance: the developer has all the capabilities needed for writing pleasant-to-use applications that blend in with the platform-native experience. It is up to the developers to choose how to use the power at their disposal. In the wild, I expect that the low-level layer of NativeScript's API is used mainly by expert developers, who know how to assemble well-functioning machines from the parts on offer.

As a primary programming interface for a new JavaScript-based mobile platform, though, just providing a low-level API would seem to be not enough. NativeScript rightly promotes the use of more well-known high-level frameworks on top: Angular, Vue, Svelte, and such. Less experienced developers should use an opinionated high-level UI framework; these developers don't have good opinions yet and the API should lead them in the right direction.

That's it for today. Thanks for reading these articles, by the way; I have enjoyed diving into this space. Next up, we'll take a look beyond JavaScript, to Flutter and Dart. Until then, happy hacking!

24 April, 2023 09:10AM by Andy Wingo

April 21, 2023

structure and interpretation of react native

Hey hey! Today's missive continues exploring the space of JavaScript and mobile application development.

Yesterday we looked into Ionic / Capacitor, giving a brief structural overview of what Capacitor apps look like under the hood and how this translates to three aspects of performance: startup latency, jank, and peak performance. Today we'll apply that same approach to another popular development framework, React Native.

Background: React

I don't know about you, but I find that there is so much marketing smoke and lights around the whole phenomenon that is React and React Native that sometimes it's hard to see what's actually there. This is compounded by the fact that the programming paradigm espoused by React (and its "native" cousin that we are looking at here) is so effective at enabling JavaScript UI programmers to focus on the "what" and not the "how" that the machinery supporting React recedes into the background.

At its most basic, React is what they call a functional reactive programming model. It is functional in the sense that the user interface elements render as a function of the global application state. The reactive comes into how user input is handled, but I'm not going to focus on that here.

React's rendering process starts with a root element tree, describing the root node of the user interface. An element is a JavaScript object with a type property. To render an element tree, if the value of the type property is a string, then the element is terminal and doesn't need further lowering, though React will visit any node in the children property of the element to render them as needed.

Otherwise if the type property of an element is a function, then the element node is functional. In that case React invokes the node's render function (the type property), passing the JavaScript element object as the argument. React will then recursively re-render the element tree produced as a result of rendering the component until all nodes are terminal. (Functional element nodes can instead have a class as their type property, but the concerns are pretty much the same.)

(In the language of React Native, a terminal node is a React Host Component, and a functional node is a React Composite Component, and both are React Elements. There are many imprecisely-used terms in React and I will continue this tradition by using the terms I mention above.)

The rendering phase of a React application is thus a function from an element tree to a terminal element tree. Nodes of element trees can be either functional or terminal. Terminal element trees are composed only of terminal elements. Rendering lowers all functional nodes to terminal nodes. This description applies both to React (targetting the web) and React Native (which we are reviewing here).

It's probably useful to go deeper into what React does with a terminal element tree, before building to the more complex pipeline used in React Native, so here we go. The basic idea is that React-on-the-web does impedance matching between the functional description of what the UI should have, as described by a terminal element tree, and the stateful tree of DOM nodes that a web browser uses to actually paint and display the UI. When rendering yields a new terminal element tree, React will compute the difference between the new and old trees. From that difference React then computes the set of imperative actions needed to mutate the DOM tree to correspond to what the new terminal element tree describes, and finally applies those changes.

In this way, small changes to the leaves of a React element tree should correspond to small changes in the DOM. Additionally, since rendering is a pure function of the global application state, we can avoid rendering at all when the application state hasn't changed. We'll dive into performance more deeply later on in the article.

React Native doesn't use a WebView

React Native is similar to React-on-the-web in intent but different in structure. Instead of using a WebView on native platforms, as Ionic / Capacitor does, React Native renders the terminal element tree to platform-native UI widgets.

When a React Native functional element renders to a terminal element, it will create not just a JS object for the terminal node as React-on-the-web does, but also a corresponding C++ shadow object. The fully lowered tree of terminal elements will thus have a corresponding tree of C++ shadow objects. React Native will then calculate the layout for each node in the shadow tree, and then commit the shadow tree: as on the web, React Native computes the set of imperative actions needed to change the current UI so that it corresponds to what the shadow tree describes. These changes are then applied on the main thread of the application.

The twisty path that leads one to implement JavaScript

The description above of React Native's rendering pipeline applies to the so-called "new architecture", which has been in the works for some years and is only now (April 2023) starting to be deployed. The key development that has allowed React Native to move over to this architecture is tighter integration and control over its JavaScript implementation. Instead of using the platform's JavaScript engine (JavaScriptCore on iOS or V8 on Android), Facebook went and made their own whole new JavaScript implementation, Hermes. Let's step back a bit to see if we can imagine why anyone in their right mind would make a new JS implementation.

In the last article, I mentioned that the only way to get peak JS performance on iOS is to use the platform's WkWebView, which enables JIT compilation of JavaScript code. React Native doesn't want a WebView, though. I guess you could create an invisible WebView and just run your JavaScript in it, but the real issue is that the interface to the JavaScript engine is so narrow as to be insufficiently expressive. You can't cheaply synchronously create a shadow tree of layout objects, for example, because every interaction with JavaScript has to cross a process boundary.

So, it may be that JIT is just not worth paying for, if it means having to keep JavaScript at arm's distance from other parts of the application. How do you do JavaScript without a browser on mobile, though? Either you use the platform's JavaScript engine, or you ship your own. It would be nice to use the same engine on iOS and Android, though. When React Native was first made, V8 wasn't able to operate in a mode that didn't JIT, so React Native went with JavaScriptCore on both platforms.

Bundling your own JavaScript engine has the nice effect that you can easily augment it with native extensions, for example to talk to the Swift or Java app that actually runs the main UI. That's what I describe above with the creation of the shadow tree, but that's not quite what the original React Native did; I can only speculate but I suspect that there was a fear that JavaScript rendering work (or garbage collection!) could be heavy enough to cause the main UI to drop frames. Phones were less powerful in 2016, and JavaScript engines were less good. So the original React Native instead ran JavaScript in a separate thread. When a render would complete, the resulting terminal element tree would be serialized as JSON and shipped over to the "native" side of the application, which would actually apply the changes.

This arrangement did work, but it ran into problems whenever the system needed synchronous communication between native and JavaScript subsystems. As I understand it, this was notably the case when React layout would need the dimensions of a native UI widget; to avoid a stall, React would assume something about the dimensions of the native UI, and then asynchronously re-layout once the actual dimensions were known. This was particularly gnarly with regards to text measurements, which depend on low-level platform-specific rendering details.

To recap: React Native had to interpret its JS on iOS and was using a "foreign" JS engine on Android, so they weren't gaining anything by using a platform JS interpreter. They would sometimes have some annoying layout jank when measuring native components. And what's more, React Native apps would still experience the same problem as Ionic / Capacitor apps, in that application startup time was dominated by parsing and compiling the JavaScript source files.

The solution to this problem was partly to switch to the so-called "new architecture", which doesn't serialize and parse so much data in the course of rendering. But the other side of it was to find a way to move parsing and compiling JavaScript to the build phase, instead of having to parse and compile JS every time the app was run. On V8, you would do this by generating a snapshot. On JavaScriptCore, which React Native used, there was no such facility. Faced with this problem and armed with Facebook's bank account, the React Native developers decided that the best solution would be to make a new JavaScript implementation optimized for ahead-of-time compilation.

The result is Hermes. If you are familiar with JavaScript engines, it is what you might expect: a JavaScript parser, originally built to match the behavior of Esprima; an SSA-based intermediate representation; a set of basic optimizations; a custom bytecode format; an interpreter to run that bytecode; a GC to manage JS objects; and so on. Of course, given the presence of eval, Hermes needs to include the parser and compiler as part of the virtual machine, but the hope is that most user code will be parsed and compiled ahead-of-time.

If this were it, I would say that Hermes seems to me to be a dead end. V8 is complete; Hermes is not. For example, Hermes doesn't have with, async function implementation has been lagging, and so on. Why Hermes when you can V8 (with snapshots), now that V8 doesn't require JIT code generation?

I thought about this for a while and in the end, given that V8's main target isn't as an embedded library in a mobile app, perhaps the binary size question is the one differentiating factor (in theory) for Hermes. By focussing on lowering distribution size, perhaps Hermes will be a compelling JS engine in its own right. In any case, Facebook can afford to keep Hermes running for a while, regardless of whether it has a competitive advantage or not.

It sounds like I'm criticising Hermes here but that's not really the point. If you can afford it, it's good to have code you control. For example one benefit that I see React Native getting from Hermes is that they control the threading model; they can mostly execute JS in its own thread, but interrupt that thread and switch to synchronous main-thread execution in response to high-priority events coming from the user. You might be able to do that with V8 at some point but the mobile-apps-with-JS domain is still in flux, so it's nice to have a sandbox that React Native developers can use to explore the system design space.

Evaluation

With that long overview out of the way, let's take a look to what kinds of performance we can expect out of a React Native system.

Startup latency

Because React Native apps have their JavaScript code pre-compiled to Hermes bytecode, we can expect that the latency imposed by JavaScript during application startup is lower than is the case with Ionic / Capacitor, which needs to parse and compile the JavaScript at run-time.

However, it must be said that as a framework, React tends to result in large application sizes and incurs significant work at startup time. One of React's strengths is that it allows development teams inside an organization to compose well: because rendering is a pure function, it's easy to break down the task of making an app into subtasks to be handled by separate groups of people. Could this strength lead to a kind of weakness, in that there is less of a need for overall coordination on the project management level, such that in the end nobody feels responsible for overall application performance? I don't know. I think the concrete differences between React Native and React (the C++ shadow object tree, the multithreading design, precompilation) could mean that React Native is closer to an optimum in the design space than React. It does seem to me though that whether a platform's primary development toolkit shold be React-like remains an open question.

Jank

In theory React Native is well-positioned to avoid jank. JavaScript execution is mostly off the main UI thread. The threading model changes to allow JavaScript rendering to be pre-empted onto the main thread do make me wonder, though: what if that work takes too much time, or what if there is a GC pause during that pre-emption? I would not be surprised to see an article in the next year or two from the Hermes team about efforts to avoid GC during high-priority event processing.

Another question I would have about jank relates to interactivity. Say the user is dragging around a UI element on the screen, and the UI needs to re-layout itself. If rendering is slow, then we might expect to see a lag between UI updates and the dragging motion; the app technically isn't dropping frames, but the render can't complete in the 16 milliseconds needed for a 60 frames-per-second update frequency.

Peak perf

But why might rendering be slow? On the one side, there is the fact that Hermes is not a high-performance JavaScript implementation. It uses a simple bytecode interpreter, and will never be able to meet the performance of V8 with JIT compilation.

However the other side of this is the design of the application framework. In the limit, React suffers from the O(n) problem: any change to the application state requires the whole element tree to be recomputed. Rendering and layout work is proportional to the size of the application, which may have thousands of nodes.

Of course, React tries to minimize this work, by detecting subtrees whose layout does not change, by avoiding re-renders when state doesn't change, by minimizing the set of mutations to the native widget tree. But the native widgets aren't the problem: the programming model is, or it can be anyway.

Aside: As good as native?

Again in theory, React Native can used to write apps that are as good as if they were written directly against platform-native APIs in Kotlin or Swift, because it uses the same platform UI toolkits as native applications. React Native can also do this at the same time as being cross-platform, targetting iOS and Android with the same code. In practice, besides the challenge of designing suitable cross-platform abstractions, React Native has to grapple with potential performance and memory use overheads of JavaScript, but the result has the potential to be quite satisfactory.

Aside: Haven't I seen that rendering model somewhere?

As I mentioned in the last article, I am a compiler engineer, not a UI specialist. In the course of my work I do interact with a number of colleagues working on graphics and user interfaces, notably in the context of browser engines. I was struck when reading about React Native's rendering pipeline about how much it resembled what a browser itself will do as part of the layout, paint, and render pipeline: translate a tree of objects to a tree of immutable layout objects, clip those to the viewport, paint the ones that are dirty, and composite the resulting textures to the screen.

It's funny to think about how many levels we have here: the element tree, the recursively expanded terminal element tree, the shadow object tree, the platform-native widget tree, surely a corresponding platform-native layout tree, and then the GPU backing buffers that are eventually composited together for the user to see. Could we do better? I could certainly imagine any of these mobile application development frameworks switching to their own Metal/Vulkan-based rendering architecture at some point, to flatten out these layers.

Summary

By all accounts, React Native is a real delight to program for; it makes developers happy. The challenge is to make it perform well for users. With its new rendering architecture based on Hermes, React Native may well be on the path to addressing many of these problems. Bytecode pre-compilation should go a long way towards solving startup latency, provided that React's expands-to-fit-all-available-space tendency is kept in check.

If you were designing a new mobile operating system from the ground up, though, I am not sure that you would necessarily end up with React Native as it is. At the very least, you would include Hermes and the base run-time as part of your standard library, so that every app doesn't have to incur the space costs of shipping the run-time. Also, in the same way that Android can ahead-of-time and just-in-time compile its bytecode, I would expect that a mobile operating system based on React Native would extend its compiler with on-device post-install compilation and possibly JIT compilation as well. And at that point, why not switch back to V8?

Well, that's food for thought. Next up, NativeScript. Until then, happy hacking!

21 April, 2023 08:20AM by Andy Wingo

April 20, 2023

Simon Josefsson

Sigstore for Apt Archives: apt-cosign

As suggested in my initial announcement of apt-sigstore my plan was to look into stronger uses of Sigstore than rekor, and I’m now happy to announce that the apt-cosign plugin has been added to apt-sigstore and the operational project debdistcanary is publishing cosign-statements about the InRelease file published by the following distributions: Trisquel GNU/Linux, PureOS, Gnuinos, Ubuntu, Debian and Devuan.

Summarizing the commands that you need to run as root to experience the great new world:

# run everything as root: su / sudo -i / doas -s
apt-get install -y apt gpg bsdutils wget
wget -nv -O/usr/local/bin/apt-verify-gpgv https://gitlab.com/debdistutils/apt-verify/-/raw/main/apt-verify-gpgv
chmod +x /usr/local/bin/apt-verify-gpgv
mkdir -p /etc/apt/verify.d
ln -s /usr/bin/gpgv /etc/apt/verify.d
echo 'APT::Key::gpgvcommand "apt-verify-gpgv";' > /etc/apt/apt.conf.d/75verify
wget -O/usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.0.1/cosign-linux-amd64
echo 924754b2e62f25683e3e74f90aa5e166944a0f0cf75b4196ee76cb2f487dd980  /usr/local/bin/cosign | sha256sum -c
chmod +x /usr/local/bin/cosign
wget -nv -O/etc/apt/verify.d/apt-cosign https://gitlab.com/debdistutils/apt-sigstore/-/raw/main/apt-cosign
chmod +x /etc/apt/verify.d/apt-cosign
mkdir -p /etc/apt/trusted.cosign.d
dist=$(lsb_release --short --id | tr A-Z a-z)
wget -O/etc/apt/trusted.cosign.d/cosign-public-key-$dist.txt "https://gitlab.com/debdistutils/debdistcanary/-/raw/main/cosign/cosign-public-key-$dist.txt"
echo "Cosign::Base-URL \"https://gitlab.com/debdistutils/canary/$dist/-/raw/main/cosign\";" > /etc/apt/apt.conf.d/77cosign

Then run your usual apt-get update and look in the syslog to debug things.

This is the kind of work that gets done while waiting for the build machines to attempt to reproducibly build PureOS. Unfortunately, the results is that a meager 16% of the 765 added/modifed packages are reproducible by me. There is some infrastructure work to be done to improve things: we should use sbuild for example. The build infrastructure should produce signed statements for each package it builds: One statement saying that it attempted to reproducible build a particular binary package (thus generated some build logs and diffoscope-output for auditing), and one statements saying that it actually was able to reproduce a package. Verifying such claims during apt-get install or possibly dpkg -i is a logical next step.

There is some code cleanups and release work to be done now. Which distribution will be the first apt-based distribution that includes native support for Sigstore? Let’s see.

Sigstore is not the only relevant transparency log around, and I’ve been trying to learn a bit about Sigsum to be able to support it as well. The more improved confidence about system security, the merrier!

20 April, 2023 05:04PM by simon

April 19, 2023

GNU Guix

Dissecting Guix, Part 3: G-Expressions

Welcome back to Dissecting Guix! Last time, we discussed monads, the functional programming idiom used by Guix to thread a store connection through a series of store-related operations.

Today, we'll be talking about a concept rather more specific to Guix: g-expressions. Being an implementation of the Scheme language, Guile is built around s-expressions, which can represent, as the saying goes, code as data, thanks to the simple structure of Scheme forms.

As Guix's package recipes are written in Scheme, it naturally needs some way to represent code that is to be run only when the package is built. Additionally, there needs to be some way to reference dependencies and retrieve output paths; otherwise, you wouldn't be able to, for instance, create a phase to install a file in the output directory.

So, how do we implement this "deferred" code? Well, initially Guix used plain old s-expressions for this purpose.

Once Upon a Time

Let's say we want to create a store item that's just a symlink to the bin/irssi file of the irssi package. How would we do that with an s-expression? Well, the s-expression itself, which we call the builder, is fairly simple:

(define sexp-builder
  `(let* ((out (assoc-ref %outputs "out"))
          (irssi (assoc-ref %build-inputs "irssi"))
          (bin/irssi (string-append irssi "/bin/irssi")))
     (symlink bin/irssi out)))

If you aren't familliar with the "quoting" syntax used to create s-expressions, I strongly recommend that you read the excellent Scheme Primer; specifically, section 7, Lists and "cons" and section 11, On the extensibility of Scheme (and Lisps in general)

The %outputs and %build-inputs variables are bound within builder scripts to association lists, which are lists of pairs that act like key/value stores, for instance:

'(("foo" . "bar")
  ("floob" . "blarb")
  ("fvoolag" . "bvarlag"))

To retrieve values from association lists, which are often referred to as alists, we use the assoc-ref procedure:

(assoc-ref '(("boing" . "bouncy")
             ("floing" . "flouncy"))
           "boing")
⇒ "bouncy"

%outputs, as the name might suggest, maps derivation output names to the paths of their respective store items, the default output being out, and %build-inputs maps inputs labels to their store items.

The builder is the easy part; we now need to turn it into a derivation and tell it what "irssi" actually refers to. For this, we use the build-expression->derivation procedure from (guix derivations):

(use-modules (guix derivations)
             (guix packages)
             (guix store)
             (gnu packages guile)
             (gnu packages irc))

(with-store store
  (let ((guile-3.0-drv (package-derivation store guile-3.0))
        (irssi-drv (package-derivation store irssi)))
    (build-expression->derivation store "irssi-symlink" sexp-builder
      #:guile-for-build guile-3.0-drv
      #:inputs `(("irssi" ,irssi-drv)))))
⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>

There are several things to note here:

  • The inputs must all be derivations, so we need to first convert the packages using package-derivation.
  • We need to explicitly set #:guile-for-build; there's no default value.
  • The build-expression->derivation and package-derivation procedures are not monadic, so we need to explicitly pass them the store connection.

The shortcomings of using s-expressions in this way are numerous: we have to convert everything to a derivation before using it, and inputs are not an inherent aspect of the builder. G-expressions were designed to overcome these issues.

Premortem Examination

A g-expression is fundamentally a record of type <gexp>, which is, naturally, defined in (guix gexp). The two most important fields of this record type, out of a total of five, are proc and references; the former is a procedure that returns the equivalent s-expression, the latter a list containing everything from the "outside world" that's used by the g-expression.

When we want to turn the g-expression into something that we can actually run as code, we combine these two fields by first building any g-expression inputs that can become derivations (leaving alone those that cannot), and then passing the built references as the arguments of proc.

Here's an example g-expression that is essentially equivalent to our sexp-builder:

(use-modules (guix gexp))

(define gexp-builder
  #~(symlink #$(file-append irssi "/bin/irssi")
             #$output))

gexp-builder is far more concise than sexp-builder; let's examine the syntax and the <gexp> object we've created. To make a g-expression, we use the #~ syntax, equivalent to the gexp macro, rather than the quasiquote backtick used to create s-expressions.

When we want to embed values from outside as references, we use #$, or ungexp, which is, in appearance if not function, equivalent to unquote (,). ungexp can accept any of four reference types:

  • S-expressions (strings, lists, etc), which will be embedded literally.
  • Other g-expressions, embedded literally.
  • Expressions returning any sort of object that can be lowered into a derivation, such as <package>, embedding that object's out store item; if the expression is specifically a symbol bound to a buildable object, you can optionally follow it with a colon and an alternative output name, so package:lib is permitted, but (get-package):lib isn't.
  • The symbol output, embedding an output path. Like symbols bound to buildable objects, this can be followed by a colon and the output name that should be used rather than the default out.

All these reference types will be represented by <gexp-input> records in the references field, except for the last kind, which will become <gexp-output> records. To give an example of each type of reference (with the return value output formatted for easier reading):

(use-modules (gnu packages glib))

#~(list #$"foobar"                         ;s-expression
        #$#~(string-append "foo" "bar")    ;g-expression
        #$(file-append irssi "/bin/irssi") ;buildable object (expression)
        #$glib:bin                         ;buildable object (symbol)
        #$output:out)                      ;output
⇒ #<gexp (list #<gexp-input "foobar":out>
               #<gexp-input #<gexp (string-append "foo" "bar") …>:out>
               #<gexp-input #<file-append #<package irssi@1.4.3 …> "/bin/irssi">:out>
               #<gexp-input #<package glib@2.70.2 …>:bin>
               #<gexp-output out>) …>

Note the use of file-append in both the previous example and gexp-builder; this procedure produces a <file-append> object that builds its first argument and is embedded as the concatenation of the first argument's output path and the second argument, which should be a string. For instance, (file-append irssi "/bin/irssi") builds irssi and expands to /gnu/store/…-irssi/bin/irssi, rather than the /gnu/store/…-irssi that the package alone would be embedded as.

So, now that we have a g-expression, how do we turn it into a derivation? This process is known as lowering; it entails the use of the aptly-named lower-gexp monadic procedure to combine proc and references and produce a <lowered-gexp> record, which acts as a sort of intermediate representation between g-expressions and derivations. We can piece apart this lowered form to get a sense of what the final derivation's builder script would look like:

(define lowered-gexp-builder
  (with-store store
    (run-with-store store
      (lower-gexp gexp-builder))))

(lowered-gexp-sexp lowered-gexp-builder)
⇒ (symlink
   "/gnu/store/…-irssi-1.4.3/bin/irssi"
   ((@ (guile) getenv) "out"))

And there you have it: a s-expression compiled from a g-expression, ready to be written into a builder script file in the store. So, how exactly do you turn this into said derivation?

Well, it turns out that there isn't an interface for turning lowered g-expressions into derivations, only one for turning regular g-expressions into derivations that first uses lower-gexp, then implements the aforementioned conversion internally, rather than outsourcing it to some other procedure, so that's what we'll use.

Unsurprisingly, that procedure is called gexp->derivation, and unlike its s-expression equivalent, it's monadic. (build-expression->derivation and other deprecated procedures were in Guix since before the monads system existed.)

(with-store store
  (run-with-store store
    (gexp->derivation "irssi-symlink" gexp-builder)))
⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>

Finally, we have a g-expression-based equivalent to the derivation we earlier created with build-expression->derivation! Here's the code we used for the s-expression version in full:

(define sexp-builder
  `(let* ((out (assoc-ref %outputs "out"))
          (irssi (assoc-ref %build-inputs "irssi"))
          (bin/irssi (string-append irssi "/bin/irssi")))
     (symlink bin/irssi out)))

(with-store store
  (let ((guile-3.0-drv (package-derivation store guile-3.0))
        (irssi-drv (package-derivation store irssi)))
    (build-expression->derivation store "irssi-symlink" sexp-builder
      #:guile-for-build guile-3.0-drv
      #:inputs `(("irssi" ,irssi-drv)))))

And here's the g-expression equivalent:

(define gexp-builder
  #~(symlink #$(file-append irssi "/bin/irssi")
             #$output))

(with-store store
  (run-with-store store
    (gexp->derivation "irssi-symlink" gexp-builder)))

That's a lot of complexity abstracted away! For more complex packages and services, especially, g-expressions are a lifesaver; you can refer to the output paths of inputs just as easily as you would a string constant. You do, however, have to watch out for situations where ungexp-native, written as #+, would be preferable over regular ungexp, and that's something we'll discuss later.

A brief digression before we continue: if you'd like to look inside a <gexp> record, but you'd rather not build anything, you can use the gexp->approximate-sexp procedure, which replaces all references with dummy values:

(gexp->approximate-sexp gexp-builder)
⇒ (symlink (*approximate*) (*approximate*))

The Lowerable-Object Hardware Shop

We've seen two examples already of records we can turn into derivations, which are generally referred to as lowerable objects or file-like objects:

  • <package>, a Guix package.
  • <file-append>, which wraps another lowerable object and appends a string to the embedded output path when ungexped.

There are many more available to us. Recall from the previous post, The Store Monad, that Guix provides the two monadic procedures text-file and interned-file, which can be used, respectively, to put arbitrary text or files from the filesystem in the store, returning the path to the created item.

This doesn't work so well with g-expressions, though; you'd have to wrap each ungexped use of either of them with (with-store store (run-with-store store …)), which would be quite tedious. Thankfully, (guix gexp) provides the plain-file and local-file procedures, which return equivalent lowerable objects. This code example builds a directory containing symlinks to files greeting the world:

(use-modules (guix monads)
             (ice-9 ftw)
             (ice-9 textual-ports))

(define (build-derivation monadic-drv)
  (with-store store
    (run-with-store store
      (mlet* %store-monad ((drv monadic-drv))
        (mbegin %store-monad
          ;; BUILT-DERIVATIONS is the monadic version of BUILD-DERIVATIONS.
          (built-derivations (list drv))
          (return (derivation-output-path
                   (assoc-ref (derivation-outputs drv) "out"))))))))
                   
(define world-greeting-output
  (build-derivation
   (gexp->derivation "world-greeting"
     #~(begin
         (mkdir #$output)
         (symlink #$(plain-file "hi-world"
                      "Hi, world!")
                  (string-append #$output "/hi"))
         (symlink #$(plain-file "hello-world"
                      "Hello, world!")
                  (string-append #$output "/hello"))
         (symlink #$(plain-file "greetings-world"
                      "Greetings, world!")
                  (string-append #$output "/greetings"))))))

;; We turn the list into multiple values using (APPLY VALUES …).
(apply values
       (map (lambda (file-path)
              (let* ((path (string-append world-greeting-output "/" file-path))
                     (contents (call-with-input-file path get-string-all)))
                (list path contents)))
            ;; SCANDIR from (ICE-9 FTW) returns the list of all files in a
            ;; directory (including ``.'' and ``..'', so we remove them with the
            ;; second argument, SELECT?, which specifies a predicate).
            (scandir world-greeting-output
                     (lambda (path)
                       (not (or (string=? path ".")
                                (string=? path "..")))))))
⇒ ("/gnu/store/…-world-greeting/greetings" "Greetings, world!")
⇒ ("/gnu/store/…-world-greeting/hello" "Hello, world!")
⇒ ("/gnu/store/…-world-greeting/hi" "Hi, world!")

Note that we define a procedure for building the output; we will need to build more derivations in a very similar fashion later, so it helps to have this to reuse instead of copying the code in world-greeting-output.

There are many other useful lowerable objects available as part of the gexp library. These include computed-file, which accepts a gexp that builds the output file, program-file, which creates an executable Scheme script in the store using a g-expression, and mixed-text-file, which allows you to, well, mix text and lowerable objects; it creates a file from the concatenation of a sequence of strings and file-likes. The G-Expressions manual page has more details.

So, you may be wondering, at this point: there's so many lowerable objects included with the g-expression library, surely there must be a way to define more? Naturally, there is; this is Scheme, after all! We simply need to acquaint ourselves with the define-gexp-compiler macro.

The most basic usage of define-gexp-compiler essentially creates a procedure that takes as arguments a record to lower, the host system, and the target system, and returns a derivation or store item as a monadic value in %store-monad.

Let's try implementing a lowerable object representing a file that greets the world. First, we'll define the record type:

(use-modules (srfi srfi-9))

(define-record-type <greeting-file>
  (greeting-file greeting)
  greeting?
  (greeting greeting-file-greeting))

Now we use define-gexp-compiler like so; note how we can use lower-object to compile down any sort of lowerable object into the equivalent store item or derivation; essentially, lower-object is just the procedure for applying the right gexp-compiler to an object:

(use-modules (ice-9 i18n))

(define-gexp-compiler (greeting-file-compiler
                       (greeting-file <greeting-file>)
                       system target)
  (lower-object
   (let ((greeting (greeting-file-greeting greeting-file)))
     (plain-file (string-append greeting "-greeting")
       (string-append (string-locale-titlecase greeting) ", world!")))))

Let's try it out now. Here's how we could rewrite our greetings directory example from before using <greeting-file>:

(define world-greeting-2-output
  (build-derivation
   (gexp->derivation "world-greeting-2"
     #~(begin
         (mkdir #$output)
         (symlink #$(greeting-file "hi")
                  (string-append #$output "/hi"))
         (symlink #$(greeting-file "hello")
                  (string-append #$output "/hello"))
         (symlink #$(greeting-file "greetings")
                  (string-append #$output "/greetings"))))))

(apply values
       (map (lambda (file-path)
              (let* ((path (string-append world-greeting-2-output
                                          "/" file-path))
                     (contents (call-with-input-file path get-string-all)))
                (list path contents)))
            (scandir world-greeting-2-output
                     (lambda (path)
                       (not (or (string=? path ".")
                                (string=? path "..")))))))
⇒ ("/gnu/store/…-world-greeting-2/greetings" "Greetings, world!")
⇒ ("/gnu/store/…-world-greeting-2/hello" "Hello, world!")
⇒ ("/gnu/store/…-world-greeting-2/hi" "Hi, world!")

Now, this is probably not worth a whole new gexp-compiler. How about something a bit more complex? Sharp-eyed readers who are trying all this in the REPL may have noticed the following output when they used define-gexp-compiler (formatted for ease of reading):

⇒ #<<gexp-compiler>
    type: #<record-type <greeting-file>>
    lower: #<procedure … (greeting-file system target)>
    expand: #<procedure default-expander (thing obj output)>>

Now, the purpose of type and lower is self-explanatory, but what's this expand procedure here? Well, if you recall file-append, you may realise that the text produced by a gexp-compiler for embedding into a g-expression doesn't necessarily have to be the exact output path of the produced derivation.

There turns out to be another way to write a define-gexp-compiler form that allows you to specify both the lowering procedure, which produces the derivation or store item, and the expanding procedure, which produces the text.

Let's try making another new lowerable object; this one will let us build a Guile package and expand to the path to its module directory. Here's our record:

(define-record-type <module-directory>
  (module-directory package)
  module-directory?
  (package module-directory-package))

Here's how we define both a compiler and expander for our new record:

(use-modules (gnu packages guile)
             (guix utils))

(define lookup-expander (@@ (guix gexp) lookup-expander))

(define-gexp-compiler module-directory-compiler <module-directory>
  compiler => (lambda (obj system target)
                (let ((package (module-directory-package obj)))
                  (lower-object package system #:target target)))
  expander => (lambda (obj drv output)
                (let* ((package (module-directory-package obj))
                       (expander (or (lookup-expander package)
                                     (lookup-expander drv)))
                       (out (expander package drv output))
                       (guile (or (lookup-package-input package "guile")
                                  guile-3.0))
                       (version (version-major+minor
                                 (package-version guile))))
                  (string-append out "/share/guile/site/" version))))

Let's try this out now:

(use-modules (gnu packages guile-xyz))

(define module-directory-output/guile-webutils
  (build-derivation
   (gexp->derivation "module-directory-output"
     #~(symlink #$(module-directory guile-webutils) #$output))))

(readlink module-directory-output/guile-webutils)
⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/3.0"

(scandir module-directory-output/guile-webutils)
⇒ ("." ".." "webutils")

(define module-directory-output/guile2.2-webutils
  (build-derivation
   (gexp->derivation "module-directory-output"
     #~(symlink #$(module-directory guile2.2-webutils) #$output))))

(readlink module-directory-output/guile2.2-webutils)
⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/2.2"

(scandir module-directory-output/guile2.2-webutils)
⇒ ("." ".." "webutils")

Who knows why you'd want to do this, but it certainly works! We've looked at why we need g-expressions, how they work, and how to extend them, and we've now only got two more advanced features to cover: cross-build support, and modules.

Importing External Modules

Let's try using one of the helpful procedures from the (guix build utils) module in a g-expression.

(define simple-directory-output
  (build-derivation
   (gexp->derivation "simple-directory"
     #~(begin
         (use-modules (guix build utils))
         (mkdir-p (string-append #$output "/a/rather/simple/directory"))))))

Looks fine, right? We've even got a use-modules in th--

ERROR:
  1. &store-protocol-error:
      message: "build of `/gnu/store/…-simple-directory.drv' failed"
      status: 100

OUTRAGEOUS. Fortunately, there's an explanation to be found in the Guix build log directory, /var/log/guix/drvs; locate the file using the first two characters of the store hash as the subdirectory, and the rest as the file name, and remember to use zcat or zless, as the logs are gzipped:

Backtrace:
           9 (primitive-load "/gnu/store/…")
In ice-9/eval.scm:
   721:20  8 (primitive-eval (begin (use-modules (guix build #)) (?)))
In ice-9/psyntax.scm:
  1230:36  7 (expand-top-sequence ((begin (use-modules (guix ?)) #)) ?)
  1090:25  6 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
  1222:19  5 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
   259:10  4 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) ?)
In ice-9/boot-9.scm:
  3927:20  3 (process-use-modules _)
   222:17  2 (map1 (((guix build utils))))
  3928:31  1 (_ ((guix build utils)))
   3329:6  0 (resolve-interface (guix build utils) #:select _ #:hide ?)

ice-9/boot-9.scm:3329:6: In procedure resolve-interface:
no code for module (guix build utils)

It turns out use-modules can't actually find (guix build utils) at all. There's no typo; it's just that to ensure the build is isolated, Guix builds module-import and module-importe-compiled directories, and sets the Guile module path within the build environment to contain said directories, along with those containing the Guile standard library modules.

So, what to do? Turns out one of the fields in <gexp> is modules, which, funnily enough, contains the names of the modules which will be used to build the aforementioned directories. To add to this field, we use the with-imported-modules macro. (gexp->derivation does provide a modules parameter, but with-imported-modules lets you add the required modules directly to the g-expression value, rather than later on.)

(define simple-directory-output
  (build-derivation
   (gexp->derivation "simple-directory"
     (with-imported-modules '((guix build utils))
       #~(begin
           (use-modules (guix build utils))
           (mkdir-p (string-append #$output "/a/rather/simple/directory")))))))
           
simple-directory-output
⇒ "/gnu/store/…-simple-directory"

It works, yay. It's worth noting that while passing just the list of modules to with-imported-modules works in this case, this is only because (guix build utils) has no dependencies on other Guix modules. Were we to try adding, say, (guix build emacs-build-system), we'd need to use the source-module-closure procedure to add its dependencies to the list:

(use-modules (guix modules))

(source-module-closure '((guix build emacs-build-system)))
⇒ ((guix build emacs-build-system)
   (guix build gnu-build-system)
   (guix build utils)
   (guix build gremlin)
   (guix elf)
   (guix build emacs-utils))

Here's another scenario: what if we want to use a module not from Guix or Guile but a third-party library? In this example, we'll use guile-json , a library for converting between S-expressions and JavaScript Object Notation.

We can't just with-imported-modules its modules, since it's not part of Guix, so <gexp> provides another field for this purpose: extensions. Each of these extensions is a lowerable object that produces a Guile package directory; so usually a package. Let's try it out using the guile-json-4 package to produce a JSON file from a Scheme value within a g-expression.

(define helpful-guide-output
  (build-derivation
   (gexp->derivation "json-file"
     (with-extensions (list guile-json-4)
       #~(begin
           (use-modules (json))
           (mkdir #$output)
           (call-with-output-file (string-append #$output "/helpful-guide.json")
             (lambda (port)
               (scm->json '((truth . "Guix is the best!")
                            (lies . "Guix isn't the best!"))
                          port))))))))

(call-with-input-file
    (string-append helpful-guide-output "/helpful-guide.json")
  get-string-all)
⇒ "{\"truth\":\"Guix is the best!\",\"lies\":\"Guix isn't the best!\"}"

Amen to that, helpful-guide.json. Before we continue on to cross-compilation, there's one last feature of with-imported-modules you should note. We can add modules to a g-expression by name, but we can also create entirely new ones using lowerable objects, such as in this pattern, which is used in several places in the Guix source code to make an appropriately-configured (guix config) module available:

(with-imported-modules `(((guix config) => ,(make-config.scm))
                         …)
  …)

In case you're wondering, make-config.scm is found in (guix self) and returns a lowerable object that compiles to a version of the (guix config) module, which contains constants usually substituted into the source code at compile time.

Native ungexp

There is another piece of syntax we can use with g-expressions, and it's called ungexp-native. This helps us distinguish between native inputs and regular host-built inputs in cross-compilation situations. We'll cover cross-compilation in detail at a later date, but the gist of it is that it allows you to compile a derivation for one architecture X, the target, using a machine of architecture Y, the host, and Guix has excellent support for it.

If we cross-compile a g-expression G that non-natively ungexps L1, a lowerable object, from architecture Y to architecture X, both G and L1 will be compiled for architecture X. However, if G natively ungexps L1, G will be compiled for X and L1 for Y.

Essentially, we use ungexp-native in situations where there would be no difference between compiling on different architectures (for instance, if L1 were a plain-file), or where using L1 built for X would actually break G (for instance, if L1 corresponds to a compiled executable that needs to be run during the build; the executable would fail to run on Y if it was built for X.)

The ungexp-native macro naturally has a corresponding reader syntax, #+, and there's also ungexp-native-splicing, which is written as #+@. These two pieces of syntax are used in the same way as their regular counterparts.

Conclusion

What have we learned in this post? To summarise:

  • G-expressions are essentially abstractions on top of s-expressions used in Guix to stage code, often for execution within a build environment or a Shepherd service script.
  • Much like you can unquote external values within a quasiquote form, you can ungexp external values to access them within a gexp form. The key difference is that you may use not only s-expressions with ungexp, but other g-expressions and lowerable objects too.
  • When a lowerable object is used with ungexp, the g-expression ultimately receives the path to the object's store item (or whatever string the lowerable object's expander produces), rather than the object itself.
  • A lowerable object is any record that has a "g-expression compiler" defined for it using the define-gexp-compiler macro. G-expression compilers always contain a compiler procedure, which converts an appropriate record into a derivation, and sometimes an expander procedure, which produces the string that is to be expanded to within g-expressions when the object is ungexped.
  • G-expressions record the list of modules available in their environment, which you may expand using with-imported-modules to add Guix modules, and with-extensions to add modules from third-party Guile packages.
  • ungexp-native may be used within g-expressions to compile lowerable objects for the host rather than the target system in cross-compilation scenarios.

Mastering g-expressions is essential to understanding Guix's inner workings, so the aim of this blog post is to be as thorough as possible. However, if you still find yourself with questions, please don't hesitate to stop by at the IRC channel #guix:libera.chat and mailing list help-guix@gnu.org; we'll be glad to assist you!

Also note that due to the centrality of g-expressions to Guix, there exist a plethora of alternative resources on this topic; here are some which you may find useful:

  • Arun Isaac's post on using g-expressions with guix deploy.
  • Marius Bakke's "Guix Drops" post which explains g-expressions in a more "top-down" way.
  • This 2020 FOSDEM talk by Christopher Marusich on the uses of g-expressions.
  • And, of course, the one and only original g-expression paper by Ludovic Courtès, the original author of Guix.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

19 April, 2023 12:30PM by (

April 18, 2023

FSF Blogs

Watch the LibrePlanet: Charting the Course videos on GNU MediaGoblin and PeerTube

March 18 & 19 was a weekend packed full of events and get-togethers, where people gathered for the Free Software Foundation's (FSF) fifteenth edition of its annual LibrePlanet conference. Relive LibrePlanet: Charting the Course and catch up on the talks you missed with video and audio versions.

18 April, 2023 08:55PM

April 15, 2023

GNU Taler news

GNU Taler v0.9.2 released

We are happy to announce the release of GNU Taler v0.9.2.

15 April, 2023 10:00PM

Simon Josefsson

Sigstore protects Apt archives: apt-verify & apt-sigstore

Do you want your apt-get update to only ever use files whose hash checksum have been recorded in the globally immutable tamper-resistance ledger rekor provided by the Sigstore project? Well I thought you’d never ask, but now you can, thanks to my new projects apt-verify and apt-sigstore. I have not done proper stable releases yet, so this is work in progress. To try it out, adapt to the modern era of running random stuff from the Internet as root, and run the following commands. Use a container or virtual machine if you have trust issues.

apt-get install -y apt gpg bsdutils wget
wget -nv -O/usr/local/bin/rekor-cli 'https://github.com/sigstore/rekor/releases/download/v1.1.0/rekor-cli-linux-amd64'
echo afde22f01d9b6f091a7829a6f5d759d185dc0a8f3fd21de22c6ae9463352cf7d  /usr/local/bin/rekor-cli | sha256sum -c
chmod +x /usr/local/bin/rekor-cli
wget -nv -O/usr/local/bin/apt-verify-gpgv https://gitlab.com/debdistutils/apt-verify/-/raw/main/apt-verify-gpgv
chmod +x /usr/local/bin/apt-verify-gpgv
mkdir -p /etc/apt/verify.d
ln -s /usr/bin/gpgv /etc/apt/verify.d
echo 'APT::Key::gpgvcommand "apt-verify-gpgv";' > /etc/apt/apt.conf.d/75verify
wget -nv -O/etc/apt/verify.d/apt-rekor https://gitlab.com/debdistutils/apt-sigstore/-/raw/main/apt-rekor
chmod +x /etc/apt/verify.d/apt-rekor
apt-get update
less /var/log/syslog

If the stars are aligned (and the puppet projects’ of debdistget and debdistcanary have ran their GitLab CI/CD pipeline recently enough) you will see a successful output from apt-get update and your syslog will contain debug logs showing the entries from the rekor log for the release index files that you downloaded. See sample outputs in the README.

If you get tired of it, disabling is easy:

chmod -x /etc/apt/verify.d/apt-rekor

Our project currently supports Trisquel GNU/Linux 10 (nabia) & 11 (aramo), PureOS 10 (byzantium), Gnuinos chimaera, Ubuntu 20.04 (focal) & 22.04 (jammy), Debian 10 (buster) & 11 (bullseye), and Devuan GNU+Linux 4.0 (chimaera). Others can be supported to, please open an issue about it, although my focus is on FSDG-compliant distributions and their upstreams.

This is a continuation of my previous work on apt-canary. I have realized that it was better to separate out the generic part of apt-canary into my new project apt-verify that offers a plugin-based method, and then rewrote apt-canary to be one such plugin. Then apt-sigstore‘s apt-rekor was my second plugin for apt-verify.

Due to the design of things, and some current limitations, Ubuntu is the least stable since they push out new signed InRelease files frequently (mostly due to their use of Phased-Update-Percentage) and debdistget and debdistcanary CI/CD runs have a hard time keeping up. If you have insight on how to improve this, please comment me in the issue tracking the race condition.

There are limitations of what additional safety a rekor-based solution actually provides, but I expect that to improve as I get a cosign-based approach up and running. Currently apt-rekor mostly make targeted attacks less deniable. With a cosign-based approach, we could design things such that your machine only downloads updates when they have been publicly archived in an immutable fashion, or submitted for validation by a third-party such as my reproducible build setup for Trisquel GNU/Linux aramo.

What do you think? Happy Hacking!

15 April, 2023 08:33AM by simon

April 12, 2023

Gary Benson

Scripting DNS lookups

Are you writing a script and some command doesn’t accept hostnames and you don’t want to inline the IP address? dig +short is your friend!

$ dig +short gbenson.net
69.163.152.201

12 April, 2023 01:18PM by gbenson

April 11, 2023

GNU MediaGoblin

MediaGoblin 0.12.1

We're pleased to announce the release of MediaGoblin 0.12.1. See the release notes for full details and upgrading instructions.

This patch release fixes a number of Python dependency issues, allows us to support newer autoconf versions, fixes a few small bugs and improves the documentation. Support for Debian Bookwork, Ubuntu 22.04 and Fedora 36 is notably missing from this release, but will be addressed in the upcoming version 0.13.0.

Thanks go to Olivier Mehani, Elisei Roca, Jgart, Dan Helfman and Peter Horvath for their contributions in this release. Since our last release, long-time MediaGoblin user and contributor Olivier has joined me as co-maintainer on the project. Thanks for all your help Olivier!

To join us and help improve MediaGoblin, please visit our getting involved page.

11 April, 2023 05:00AM by Ben Sturmfels

April 10, 2023

parted @ Savannah

parted-3.6 released [stable]

I have released parted 3.6

Here are the compressed sources and a GPG detached signature[*]:
  http://ftp.gnu.org/gnu/parted/parted-3.6.tar.xz
  http://ftp.gnu.org/gnu/parted/parted-3.6.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA256 checksums:

3b43dbe33cca0f9a18601ebab56b7852b128ec1a3df3a9b30ccde5e73359e612  ./parted-3.6.tar.xz
cdc0e7fcf5056e7f3f45d43bb980bd6d835b09a5c762ecd2b65c47742a0e583e  ./parted-3.6.tar.xz.sig

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify parted-3.6.tar.xz.sig

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to update
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key bcl@redhat.com

  gpg --recv-keys 117E8C168EFE3A7F

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=parted&download=1' | gpg --import -

This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gettext 0.21
  Gnulib v0.1-5949-g480a59ba60
  Gperf 3.1

NEWS

  • Noteworthy changes in release 3.6 (2023-04-10) [stable]


  Promoting alpha release to stable release 3.6

  • Noteworthy changes in release 3.5.28 (2023-03-24) [alpha]



** New Features

  Support GPT partition attribute bit 63 as no_automount flag.

  Add type commands to set type-id on MS-DOS and type-uuid on GPT.

  Add swap flag support to the dasd disklabel

  Add display of GPT disk and partition UUIDs in JSON output


** Bug Fixes

  Fix use of enums in flag limits by switching to using #define

  Fix ending sector location when using kibi IEC suffix

10 April, 2023 11:49PM by Brian C. Lane

Simon Josefsson

Trisquel is 42% Reproducible!

The absolute number may not be impressive, but what I hope is at least a useful contribution is that there actually is a number on how much of Trisquel is reproducible. Hopefully this will inspire others to help improve the actual metric.

tl;dr: go to reproduce-trisquel.

When I set about to understand how Trisquel worked, I identified a number of things that would improve my confidence in it. The lowest hanging fruit for me was to manually audit the package archive, and I wrote a tool called debdistdiff to automate this for me. That led me to think about apt archive transparency more in general. I have made some further work in that area (hint: apt-verify) that deserve its own blog post eventually. Most of apt archive transparency is futile if we don’t trust the intended packages that are in the archive. One way to measurable increase trust in the package are to provide reproducible builds of the packages, which should by now be an established best practice. Code review is still important, but since it will never provide positive guarantees we need other processes that can identify sub-optimal situations automatically. The way reproducible builds easily identify negative results is what I believe has driven much of its success: its results are tangible and measurable. The field of software engineering is in need of more such practices.

The design of my setup to build Trisquel reproducible are as follows.

  • The project debdistget is responsible for downloading Release/Packages files (which are the most relevant files from dists/) from apt archives, and works by commiting them into GitLab-hosted git-repositories. I maintain several such repositories for popular apt-archives, including for Trisquel and its upstream Ubuntu. GitLab invokes a schedule pipeline to do the downloading, and there is some race conditions here.
  • The project debdistdiff is used to produce the list of added and modified packages, which are the input to actually being able to know what packages to reproduce. It publishes human readable summary of difference for several distributions, including Trisquel vs Ubuntu. Early on I decided that rebuilding all of the upstream Ubuntu packages is out of scope for me: my personal trust in the official Debian/Ubuntu apt archives are greater than my trust of the added/modified packages in Trisquel.
  • The final project reproduce-trisquel puts the pieces together briefly as follows, everything being driven from its .gitlab-ci.yml file.
    • There is a (manually triggered) job generate-build-image to create a build image to speed up CI/CD runs, using a simple Dockerfile.
    • There is a (manually triggered) job generate-package-lists that uses debdistdiff to generate and store package lists and puts its output in lists/. The reason this is manually triggered right now is due to a race condition.
    • There is a (scheduled) job that does two things: from the package lists, the script generate-ci-packages.sh builds a GitLab CI/CD instruction file ci-packages.yml that describes jobs for each package to build. The second part is generate-readme.sh that re-generate the project’s README.md based on the build logs and diffoscope outputs that stored in the git repository.
    • Through the ci-packages.yml file, there is a large number of jobs that are dynamically defined, which currently are manually triggered to not overload the build servers. The script build-package.sh is invoked and attempts to rebuild a package, and stores build log and diffoscope output in the git project itself.

I did not expect to be able to use the GitLab shared runners to do the building, however they turned out to work quite well and I postponed setting up my own runner. There is a manually curated lists/disabled-aramo.txt with some packages that all required too much disk space or took over two hours to build. Today I finally took the time to setup a GitLab runner using podman running Trisquel aramo, and I expect to complete builds of the remaining packages soon — one of my Dell R630 server with 256GB RAM and dual 2680v4 CPUs should deliver sufficient performance.

Current limitations and ideas on further work (most are filed as project issues) include:

  • We don’t support *.buildinfo files. As far as I am aware, Trisquel does not publish them for their builds. Improving this would be a first step forward, anyone able to help? Compare buildinfo.debian.net. For example, many packages differ only in their NT_GNU_BUILD_ID symbol inside the ELF binary, see example diffoscope output for libgpg-error. By poking around in jenkins.trisquel.org I managed to discover that Trisquel built initramfs-utils in the randomized path /build/initramfs-tools-bzRLUp and hard-coding that path allowed me to reproduce that package. I expect the same to hold for many other packages. Unfortunately, this failure turned into success with that package moved the needle from 42% reproducibility to 43% however I didn’t let that stand in the way of a good headline.
  • The mechanism to download the Release/Package-files from dists/ is not fool-proof: we may not capture all ever published such files. While this is less of a concern for reproducibility, it is more of a concern for apt transparency. Still, having Trisquel provide a service similar to snapshot.debian.org would help.
  • Having at least one other CPU architecture would be nice.
  • Due to lack of time and mental focus, handling incremental updates of new versions of packages is not yet working. This means we only ever build one version of a package, and never discover any newly published versions of the same package. Now that Trisquel aramo is released, the expected rate of new versions should be low, but still happens due to security or backports.
  • Porting this to test supposedly FSDG-compliant distributions such as PureOS and Gnuinos should be relatively easy. I’m also looking at Devuan because of Gnuinos.
  • The elephant in the room is how reproducible Ubuntu is in the first place.

Happy Easter Hacking!

Update 2023-04-17: The original project “reproduce-trisquel” that was announced here has been archived and replaced with two projects, one generic “debdistreproduce” and one with results for Trisquel: “reproduce/trisquel“.

10 April, 2023 05:36PM by simon

April 07, 2023

health @ Savannah

GNU Health Hospital Management patchset 4.2.1 released

Dear community

GNU Health 4.2.1 patchset has been released !

Priority: High

Table of Contents


  • About GNU Health Patchsets
  • Updating your system with the GNUHealth control Center
  • Installation notes
  • List of other issues related to this patchset


About GNU Health Patchsets


We provide "patchsets" to stable releases. Patchsets allow applying bug fixes and updates on production systems. Always try to keep your production system up-to-date with the latest patches.

Patches and Patchsets maximize uptime for production systems, and keep your system updated, without the need to do a whole installation.

NOTE: Patchsets are applied on previously installed systems only. For new, fresh installations, download and install the whole tarball (ie, gnuhealth-4.2.1.tar.gz)

Updating your system with the GNU Health control Center


Starting GNU Health 3.x series, you can do automatic updates on the GNU Health HMIS kernel and modules using the GNU Health control center program.

Please refer to the administration manual section ( https://en.wikibooks.org/wiki/GNU_Health/Control_Center )

The GNU Health control center works on standard installations (those done following the installation manual on wikibooks). Don't use it if you use an alternative method or if your distribution does not follow the GNU Health packaging guidelines.

Installation Notes


You must apply previous patchsets before installing this patchset. If your patchset level is 4.2.1, then just follow the general instructions. You can find the patchsets at GNU Health main download site at GNU.org (https://ftp.gnu.org/gnu/health/)

In most cases, GNU Health Control center (gnuhealth-control) takes care of applying the patches for you. 

Pre-requisites for upgrade to 4.2.1: None

Now follow the general instructions at
 https://en.wikibooks.org/wiki/GNU_Health/Control_Center

 
After applying the patches, make a full update of your GNU Health database as explained in the documentation.

When running "gnuhealth-control" for the first time, you will see the following message: "Please restart now the update with the new control center" Please do so. Restart the process and the update will continue.
 

  • Restart the GNU Health server


List of other issues and tasks related to this patchset


  • bug #64014: Update gender identity in patient evaluations and reports
  • bug #64009: Include signing health professional and avoid scrolling in patient evaluation
  • bug #64007: Summary report is not using FreeFonts family
  • bug #63993: Python-sql error on patient evaluation report



Update gender identity in patient evaluations and reports


For detailed information about each issue, you can visit :
 https://savannah.gnu.org/bugs/?group=health
 
About each task, you can visit:
 https://savannah.gnu.org/task/?group=health

For detailed information you can read about Patches and Patchsets

07 April, 2023 10:39AM by Luis Falcon

April 06, 2023

FSF News

Free Software Gigabit Mini VPN Router (TPE-R1400) from ThinkPenguin, Inc. now FSF-certified to Respect Your Freedom

BOSTON, Massachusetts, USA -- Thursday, April 6, 2023 -- The Free Software Foundation (FSF) awarded Respects Your Freedom (RYF) certification to the Free Software Gigabit Mini VPN Router (TPE-R1400) from ThinkPenguin, Inc. The RYF certification mark means that this product meets the FSF's standards in regard to users' freedom, control over the product, and privacy.

06 April, 2023 04:25PM

April 01, 2023

GNU Guix

Reinstating an iconic error message

Software development is a social process. What might be a “bug” for someone might well be a “feature” for someone else. The Guix project rediscovered it the hard way when, after “fixing a bug” that had been present in Guix System for years, it was confronted with an uproar in its user base.

In this post we look at why developers considered the initial behavior a “bug”, why users on the contrary had come to rely on it, and why developers remained blind to it. A patch to reinstate the initial behavior is being reviewed. This post is also an opportunity for us Guix developers to extend our apologies to our users whose workflow was disrupted.

The crux of the matter

Anyone who’s used Guix System in the past has seen this message on the console during the boot process:

error in finalization thread: Success

The following picture shows a typical boot screen (with additional messages in the same vein):

Picture of a monitor showing the error/success boot message.

If you have never seen it before, it may look surprising to you. Guix System users lived with it literally for years; the message became a hint that the boot process was, indeed, successful.

A few months ago, a contributor sought to satisfy their curiosity by finding the origin of the message. It did look like a spurious error message, after all, and perhaps the right course of action would be to address the problem at its root—or so they thought.

As it turns out, the message originated in Guile—check out the Guile manual if you’re curious about finalization. Investigation revealed two things: first, that this perror call in Guile was presumably reporting the wrong error code—this was fixed.

The second error—the core of the problem—lied in Guix System itself. Remember that, in its quest of memory safety™, statelessness, and fun, Guix System does it all in Guile Scheme—well, except for the kernel (for now). As soon as Linux has booted, Guix System spawns Guile to run boot code that’s in its initial RAM disk (“initrd”). Right before executing shepherd, its service manager, as PID 1, the initrd code would carelessly close all the file descriptors above 2 to make sure they do not leak into PID 1. The problem—you guessed it—is that one of them was the now-famous file descriptor of the finalization thread’s pipe; the finalization thread would quickly notice and boom!

error in finalization thread: Success

Our intrepid developers thought: “hey, we found it! Let’s fix it!”. And so they did.

Breaking user workflows

This could have been the end of the story, but there’s more to it than software. As Xkcd famously captured, this was bound to break someone’s workflow. Indeed, had developers paid more attention to what users had to say, they would have known that the status quo was preferable.

For some time now, users had shown that they held the error/success message deep in their heart. The message was seen on the blackboard at the Ten Years of Guix celebration, as a motto, as a rallying cry, spontaneously put on display:

Picture of a blackboard with the famous message (by Julien Lepiller, under CC0).

What’s more, a fellow NixOS hacker and Guix enthusiast, beguiled by this powerful message, designed stickers and brought them to FOSDEM in February 2023:

Picture of error/success stickers (under CC0).

The sticker design builds upon the “test pilot” graphics made by Luis Felipe for the 1.3.0 release. The test pilot has a bug on its helmet. In a way, the drawing and error message both represent, metaphorically, a core tenet of Guix as a project; just like Haskell is avoiding success at all costs, Guix seems trapped in an error/success quantum state.

Had it gone too far? Was calling it a “bug” the demonstration of the arrogance of developers detached from the reality of the community?

Fixing our mistakes

Those who installed Guix System starting from version 1.4.0 have been missing out on the error/success boot message. The patch submitted today finally reinstates that message. The review process will determine whether consensus is to enable it by default—as part of %base-service—or whether to make it optional—after all, we also need to accommodate the needs of new users who never saw this message. This will allow users to restore their workflow, while also ensuring that those freshly printed stickers remain relevant.

This incident had broader consequences in the project. It led some to suggest that we, finally, set up a request-for-comment (RFC) kind of process that would give all the community a say on important topics—a process most large free software projects have developed in one form or another. Such a process could have prevented this incident: instead of arrogantly labeling it as a “bug”, developers would have proposed an RFC to remove the message; the discussion period, most likely, would have made it clear that removal was not a desirable outcome and we would all have moved on.

This incident made many users uncomfortable, but we are glad that it is now being addressed. The lessons learned will be beneficial to the project for the years to come.

Picture of a metal bird holding an error/success sticker (under CC0).

Credits

Test pilot by Luis Felipe distributed under the terms of CC-BY-SA 4.0; sticker design distributed under CC-BY-SA 4.0 as well. Blackboard picture by Julien Lepiller under CC0; sticker pictures under CC0.

Many thanks to the anonymous sticker provider!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

01 April, 2023 11:00AM by Ludovic Courtès

March 31, 2023

GNUnet News

GNUnet 0.19.4

GNUnet 0.19.4

This is a bugfix release for gnunet 0.19.3. Special thanks goes out to ulfvonbelow who provided an array of patches. This is not an April Fool's joke.

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links may be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

A detailed list of changes can be found in the git log , the NEWS and the bug tracker .

31 March, 2023 10:00PM

March 28, 2023

parted @ Savannah

parted-3.5.28 released [alpha]

I have released an alpha version of parted-3.5.28

Here are the compressed sources and a GPG detached signature[*]:
  http://alpha.gnu.org/gnu/parted/parted-3.5.28.tar.xz
  http://alpha.gnu.org/gnu/parted/parted-3.5.28.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA256 checksums:

af8a880df2e7b577c99ed9ee27a38e3f645896de8354dbfc05d8e81179a6d6dc  parted-3.5.28.tar.xz
49e8c4fc8aae92d8922f39aaae1fcdb0c8be3f3a80d34e006916e93a4a4852fc  parted-3.5.28.tar.xz.sig

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify parted-3.5.28.tar.xz.sig

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to update
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key bcl@redhat.com

  gpg --recv-keys 117E8C168EFE3A7F

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=parted&download=1' | gpg --import -

This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gettext 0.21
  Gnulib v0.1-5949-g480a59ba60
  Gperf 3.1

NEWS

  • Noteworthy changes in release 3.5.28 (2023-03-24) [alpha]


** New Features

  Support GPT partition attribute bit 63 as no_automount flag.

  Add type commands to set type-id on MS-DOS and type-uuid on GPT.

  Add swap flag support to the dasd disklabel

  Add display of GPT disk and partition UUIDs in JSON output


** Bug Fixes

  Fix use of enums in flag limits by switching to using #define

  Fix ending sector location when using kibi IEC suffix

28 March, 2023 06:32PM by Brian C. Lane

March 26, 2023

a2ps @ Savannah

a2ps 4.15.3 released [stable]


GNU a2ps is an Any to PostScript filter.  Of course it processes plain
text files, but also pretty prints quite a few popular languages.

For more information, see https://www.gnu.org/software/a2ps/

This release is a minor bug-fix release; no pressing need to update unless
you’re affected by a bug it fixes (see the end of this message for details).


Here are the compressed sources and a GPG detached signature:
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.3.tar.gz
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.3.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

b2ae4016b789a198c50a2f1dc0fefc11bda18ebe  a2ps-4.15.3.tar.gz
0A6B4OtNy/LUlj2J4d8rtm9x5m1ztBUsQ8+YOOaq98c  a2ps-4.15.3.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.15.3.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.15.3.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5892-g83006fa8c9

NEWS

* Noteworthy changes in release 4.15.3 (2023-03-26) [stable]
 * Bug fixes:
   - Fix fixps to use GhostScript’s ps2write device instead of defunct
     pswrite.
 * Build:
   - Fix a problem building PDF version of manual.


26 March, 2023 08:39PM by Reuben Thomas

texinfo @ Savannah

Texinfo 7.0.3 released

We have released version 7.0.3 of Texinfo, the GNU documentation format. This is a minor bug-fix release.

It's available via a mirror (xz is much smaller than gz, but gz is available too just in case):

http://ftpmirror.gnu.org/texinfo/texinfo-7.0.3.tar.xz
http://ftpmirror.gnu.org/texinfo/texinfo-7.0.3.tar.gz

Please send any comments to bug-texinfo@gnu.org.

Full announcement:

https://lists.gnu.org/archive/html/bug-texinfo/2023-03/msg00087.html

26 March, 2023 12:49PM by Gavin D. Smith

March 25, 2023

Greg Casamento

Swift->ObjC interop

Some interesting notes. I will update this posting as i find more: * https://dart.dev/guides/libraries/objective-c-interop

25 March, 2023 10:27AM by Unknown (noreply@blogger.com)

Compatibility project almost complete

As the much villified theme for star trek enterprise says "its been a long road getting from there to here" i am almost done with all of the work that needed to be done to get us to Catalina compatibility in GNUstep. The reason this is still significant is because Apple hasn't made many changes to either the Foundation or AppKit APIs since then. I have been workinf hard over the last three years. All of the new classes are fully tested. Once this effort is completed I am going to focus on printing, which has always been a problem in GS. And possibly a "reference" distribution.

25 March, 2023 10:13AM by Unknown (noreply@blogger.com)

March 23, 2023

grep @ Savannah

grep-3.10 released [stable]


This is to announce grep-3.10, a stable release,
fixing a bug with -P and \d. TL;DR, grep-3.9 would do this:

  $ LC_ALL=en_US.UTF-8 grep -P '\d' <<< ٠١٢٣٤٥٦٧٨٩
  ٠١٢٣٤٥٦٧٨٩

It should print nothing, like it has always done.
For more detail, see https://lists.gnu.org/r/bug-grep/2023-03/msg00005.html

Thanks to Paul Eggert for catching the \D variant and to Bruno Haible
for assiduously tending gnulib and for testing grep on so many
different systems.

There have been 12 commits by 2 people in the 17 days since 3.9.

See the NEWS below for a brief summary.

Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Jim Meyering (8)
  Paul Eggert (4)

Jim
 [on behalf of the grep maintainers]
==================================================================

Here is the GNU grep home page:
    http://gnu.org/s/grep/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=grep.git;a=shortlog;h=v3.10
or run this command from a git-cloned grep directory:
  git shortlog v3.9..v3.10

Here are the compressed sources:
  https://ftp.gnu.org/gnu/grep/grep-3.10.tar.gz   (2.7MB)
  https://ftp.gnu.org/gnu/grep/grep-3.10.tar.xz   (1.7MB)

Here are the GPG detached signatures:
  https://ftp.gnu.org/gnu/grep/grep-3.10.tar.gz.sig
  https://ftp.gnu.org/gnu/grep/grep-3.10.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

  7d3d830703183532f0b66619f0b148827e86eda7  grep-3.10.tar.gz
  3nsh2OM0jqZWnG/Vc06QoxFp72JCnqPc5Ipvwd2F0mA=  grep-3.10.tar.gz
  b8413017681fcd6249e0d0fb9c78225944074f23  grep-3.10.tar.xz
  JO+ltZX7WnEAh5tRuIaKC7h6ccGD0CxMYCYzuIr2hVs=  grep-3.10.tar.xz

Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify grep-3.10.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096/0x7FD9FCCB000BEEEE 2010-06-14 [SCEA]
        Key fingerprint = 155D 3FC5 00C8 3448 6D1E  EA67 7FD9 FCCB 000B EEEE
  uid                   [ unknown] Jim Meyering <jim@meyering.net>
  uid                   [ unknown] Jim Meyering <meyering@fb.com>
  uid                   [ unknown] Jim Meyering <meyering@gnu.org>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key jim@meyering.net

  gpg --recv-keys 7FD9FCCB000BEEEE

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=grep&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify grep-3.10.tar.gz.sig

This release was bootstrapped with the following tools:
  Autoconf 2.72a.92-8db0
  Automake 1.16i
  Gnulib v0.1-5916-gf61570c0ef

NEWS

* Noteworthy changes in release 3.10 (2023-03-22) [stable]

** Bug fixes

  With -P, \d now matches only ASCII digits, regardless of PCRE
  options/modes. The changes in grep-3.9 to make \b and \w work
  properly had the undesirable side effect of making \d also match
  e.g., the Arabic digits: ٠١٢٣٤٥٦٧٨٩.  With grep-3.9, -P '\d+'
  would match that ten-digit (20-byte) string. Now, to match such
  a digit, you would use \p{Nd}. Similarly, \D is now mapped to [^0-9].
  [bug introduced in grep 3.9]


23 March, 2023 01:31AM by Jim Meyering

parallel @ Savannah

GNU Parallel 20230322 ('Arrest Warrant') released

GNU Parallel 20230322 ('Arrest Warrant') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  GNU parallel is magic, half of my work uses it, to the point where they're referenced and thanked in my thesis
    -- Best Catboy Key Grip @alamogordoglass@twitter

New in this release:

  • Better support for wide characters in --latest-line.
  • Support for rsync 3.2.7.
  • Bug fixes and man page updates.

News about GNU Parallel:

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel


GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:


About GNU SQL


GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients.

So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.


About GNU Niceload


GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

23 March, 2023 12:12AM by Ole Tange

March 20, 2023

coreutils @ Savannah

coreutils-9.2 released [stable]


This is to announce coreutils-9.2, a stable release.
See the NEWS below for a brief summary.

Thanks to everyone who has contributed!
There have been 209 commits by 14 people in the 48 weeks since 9.1.


Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Arsen Arsenović (1)     Jim Meyering (7)
  Bernhard Voelker (3)    Paul Eggert (90)
  Bruno Haible (1)        Pierre Marsais (1)
  Carl Edquist (2)        Pádraig Brady (98)
  ChuanGang Jiang (2)     Rasmus Villemoes (1)
  Dennis Williamson (1)   Stefan Kangas (1)
  Ivan Radić (1)          Álvar Ibeas (1)


Pádraig [on behalf of the coreutils maintainers]

==================================================================

Here is the GNU coreutils home page:
    http://gnu.org/s/coreutils/

For a summary of changes and contributors, see:
    http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=shortlog;h=v9.2
or run this command from a git-cloned coreutils directory:
    git shortlog v9.1..v9.2

To summarize the 665 gnulib-related changes, run these commands
from a git-cloned coreutils directory:
     git checkout v9.2
     git submodule summary v9.1

==================================================================

Here are the compressed sources:
  https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.gz   (14MB)
  https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.xz   (5.6MB)

Here are the GPG detached signatures:
  https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.gz.sig
  https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

  6afa9ce3729afc82965a33d02ad585d1571cdeef  coreutils-9.2.tar.gz
  ebWNqhmcY84g95GRF3NLISOUnJLReVZPkI4yiQFZzUg=  coreutils-9.2.tar.gz
  3769071b357890dc36d820c597c1c626a1073fcb  coreutils-9.2.tar.xz
  aIX/R7nNshHeR9NowXhT9Abar5ixSKrs3xDeKcwEsLM=  coreutils-9.2.tar.xz

Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify coreutils-9.2.tar.xz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096 2011-09-23 [SC]
        6C37 DC12 121A 5006 BC1D  B804 DF6F D971 3060 37D9
  uid           [ unknown] Pádraig Brady <P@draigBrady.com>
  uid           [ unknown] Pádraig Brady <pixelbeat@gnu.org>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key P@draigBrady.com

  gpg --recv-keys DF6FD971306037D9

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=coreutils&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify coreutils-9.2.tar.gz.sig

This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5857-gf17d397771
  Bison 3.8.2

==================================================================

NEWS

* Noteworthy changes in release 9.2 (2023-03-20) [stable]

** Bug fixes

  'comm --output-delimiter="" --total' now delimits columns in the total
  line with the NUL character, consistent with NUL column delimiters in
  the rest of the output.  Previously no delimiters were used for the
  total line in this case.
  [bug introduced with the --total option in coreutils-8.26]

  'cp -p' no longer has a security hole when cloning into a dangling
  symbolic link on macOS 10.12 and later.
  [bug introduced in coreutils-9.1]

  'cp -rx / /mnt' no longer complains "cannot create directory /mnt/".
  [bug introduced in coreutils-9.1]

  cp, mv, and install avoid allocating too much memory, and possibly
  triggering "memory exhausted" failures, on file systems like ZFS,
  which can return varied file system I/O block size values for files.
  [bug introduced in coreutils-6.0]

  cp, mv, and install now immediately acknowledge transient errors
  when creating copy-on-write or cloned reflink files, on supporting
  file systems like XFS, BTRFS, APFS, etc.
  Previously they would have tried again with other copy methods
  which may have resulted in data corruption.
  [bug introduced in coreutils-7.5 and enabled by default in coreutils-9.0]

  cp, mv, and install now handle ENOENT failures across CIFS file systems,
  falling back from copy_file_range to a better supported standard copy.
  [issue introduced in coreutils-9.0]

  'mv --backup=simple f d/' no longer mistakenly backs up d/f to f~.
  [bug introduced in coreutils-9.1]

  rm now fails gracefully when memory is exhausted.
  Previously it may have aborted with a failed assertion in some cases.
  [This bug was present in "the beginning".]

  rm -d (--dir) now properly handles unreadable empty directories.
  E.g., before, this would fail to remove d: mkdir -m0 d; src/rm -d d
  [bug introduced in v8.19 with the addition of this option]

  runcon --compute no longer looks up the specified command in the $PATH
  so that there is no mismatch between the inspected and executed file.
  [bug introduced when runcon was introduced in coreutils-6.9.90]

  'sort -g' no longer infloops when given multiple NaNs on platforms
  like x86_64 where 'long double' has padding bits in memory.
  Although the fix alters sort -g's NaN ordering, that ordering has
  long been documented to be platform-dependent.
  [bug introduced 1999-05-02 and only partly fixed in coreutils-8.14]

  stty ispeed and ospeed options no longer accept and silently ignore
  invalid speed arguments, or give false warnings for valid speeds.
  Now they're validated against both the general accepted set,
  and the system supported set of valid speeds.
  [This bug was present in "the beginning".]

  stty now wraps output appropriately for the terminal width.
  Previously it may have output 1 character too wide for certain widths.
  [bug introduced in coreutils-5.3]

  tail --follow=name works again with non seekable files.  Previously it
  exited with an "Illegal seek" error when such a file was replaced.
  [bug introduced in fileutils-4.1.6]

  'wc -c' will again efficiently determine the size of large files
  on all systems.  It no longer redundantly reads data from certain
  sized files larger than SIZE_MAX.
  [bug introduced in coreutils-8.24]

** Changes in behavior

  Programs now support the new Ronna (R), and Quetta (Q) SI prefixes,
  corresponding to 10^27 and 10^30 respectively,
  along with their binary counterparts Ri (2^90) and Qi (2^100).
  In some cases (e.g., 'sort -h') these new prefixes simply work;
  in others, where they exceed integer width limits, they now elicit
  the same integer overflow diagnostics as other large prefixes.

  'cp --reflink=always A B' no longer leaves behind a newly created
  empty file B merely because copy-on-write clones are not supported.

  'cp -n' and 'mv -n' now exit with nonzero status if they skip their
  action because the destination exists, and likewise for 'cp -i',
  'ln -i', and 'mv -i' when the user declines.  (POSIX specifies this
  for 'cp -i' and 'mv -i'.)

  cp, mv, and install again read in multiples of the reported block size,
  to support unusual devices that may have this constraint.
  [behavior inadvertently changed in coreutils-7.2]

  du --apparent now counts apparent sizes only of regular files and
  symbolic links.  POSIX does not specify the meaning of apparent
  sizes (i.e., st_size) for other file types, and counting those sizes
  could cause confusing and unwanted size mismatches.

  'ls -v' and 'sort -V' go back to sorting ".0" before ".A",
  reverting to the behavior in coreutils-9.0 and earlier.
  This behavior is now documented.

  ls --color now matches a file extension case sensitively
  if there are different sequences defined for separate cases.

  printf unicode \uNNNN, \UNNNNNNNN syntax, now supports all valid
  unicode code points.  Previously is was restricted to the C
  universal character subset, which restricted most points <= 0x9F.

  runcon now exits with status 125 for internal errors.  Previously upon
  internal errors it would exit with status 1, which was less distinguishable
  from errors from the invoked command.

  'split -n N' now splits more evenly when the input size is not a
  multiple of N, by creating N output files whose sizes differ by at
  most 1 byte.  Formerly, it did this only when the input size was
  less than N.

  'stat -c %s' now prints sizes as unsigned, consistent with 'ls'.

** New Features

  cksum now accepts the --base64 (-b) option to print base64-encoded
  checksums.  It also accepts/checks such checksums.

  cksum now accepts the --raw option to output a raw binary checksum.
  No file name or other information is output in this mode.

  cp, mv, and install now accept the --debug option to
  print details on how a file is being copied.

  factor now accepts the --exponents (-h) option to print factors
  in the form p^e, rather than repeating the prime p, e times.

  ls now supports the --time=modification option, to explicitly
  select the default mtime timestamp for display and sorting.

  mv now supports the --no-copy option, which causes it to fail when
  asked to move a file to a different file system.

  split now accepts options like '-n SIZE' that exceed machine integer
  range, when they can be implemented as if they were infinity.

  split -n now accepts piped input even when not in round-robin mode,
  by first copying input to a temporary file to determine its size.

  wc now accepts the --total={auto,never,always,only} option
  to give explicit control over when the total is output.

** Improvements

  cp --sparse=auto (the default), mv, and install,
  will use the copy_file_range syscall now also with sparse files.
  This may be more efficient, by avoiding user space copies,
  and possibly employing copy offloading or reflinking,
  for the non sparse portion of such sparse files.

  On macOS, cp creates a copy-on-write clone in more cases.
  Previously cp would only do this when preserving mode and timestamps.

  date --debug now diagnoses if multiple --date or --set options are
  specified, as only the last specified is significant in that case.

  rm outputs more accurate diagnostics in the presence of errors
  when removing directories.  For example EIO will be faithfully
  diagnosed, rather than being conflated with ENOTEMPTY.

  tail --follow=name now works with single non regular files even
  when their modification time doesn't change when new data is available.
  Previously tail would not show any new data in this case.

  tee -p detects when all remaining outputs have become broken pipes, and
  exits, rather than waiting for more input to induce an exit when written.

  tee now handles non blocking outputs, which can be seen for example with
  telnet or mpirun piping through tee to a terminal.
  Previously tee could truncate data written to such an output and fail,
  and also potentially output a "Resource temporarily unavailable" error.


20 March, 2023 03:53PM by Pádraig Brady

Amin Bandali

LibrePlanet 2023: What's new in Jami

Update: Jami has won this year's Award for Project of Social Benefit, presented by the Free Software Foundation "to a project or team responsible for applying free software, or the ideas of the free software movement, to intentionally and significantly benefit society. This award stresses the use of free software in service to humanity."

Today I gave a talk at LibrePlanet 2023 on what's new in and about Jami since my Jami and how it empowers users talk for LibrePlanet 2021.

Here is the abstract for my talk, also available on the LibrePlanet 2023's speakers page:

Jami is free/libre software for universal communication that respects the freedoms and privacy of its users. An official GNU package, Jami is an end-to-end encrypted secure and distributed communication tool for calling, conferencing, messaging, and file transfer. Jami has end-user applications across multiple operating systems and platforms, as well as multiple APIs and a plugin system for building upon and extending Jami as a framework for secure and private communication.

This talk gives an update on what's new in and about Jami since bandali's "Jami and how it empowers users" talk at LibrePlanet 2021.

Presentation slides: pdf (with notes, only notes) | bib
LaTeX sources: tar.gz | zip
Video: coming soon

I will add the presentation video once the conference recordings have been processed and published by the Free Software Foundation.

LibrePlanet is a conference about software freedom, happening on March 19-20, 2023. The event is hosted by the Free Software Foundation, and brings together software developers, law and policy experts, activists, students, and computer users to learn skills, celebrate free software accomplishments, and face upcoming challenges. Newcomers are always welcome, and LibrePlanet 2023 will feature programming for all ages and experience levels.

20 March, 2023 01:16AM

March 19, 2023

a2ps @ Savannah

a2ps 4.15.2 released [stable]


GNU a2ps is an Any to PostScript filter. Of course it processes plain text
files, but also pretty prints quite a few popular languages.

More detailed web pages about GNU a2ps is available at
https://savannah.gnu.org/projects/a2ps/.

This release is a minor bug-fix release. It fixes a long-standing but rare
crash, makes a minor fix to the build system, and finally puts the manual
online; see:

https://gnu.org/software/a2ps/manual/

Here are the compressed sources and a GPG detached signature:
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.2.tar.gz
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.2.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

b02c9f4066ebb2899f7615b93b354fb77192377c  a2ps-4.15.2.tar.gz
7FKQSp+sEmQWsyrJokBfPWF92pfWUpnhpBVVT+Az0iU  a2ps-4.15.2.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.15.2.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.15.2.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5853-ge0aefd96b6

NEWS

* Noteworthy changes in release 4.15.2 (2023-03-19) [stable]
 * Bug fixes:
   - Fix old crash when using --stdin="".
 * Build
   - Make configure stop if libpaper is not found.
   - Enable building the manual for gnu.org.


19 March, 2023 07:06PM by Reuben Thomas

Trisquel GNU/Linux

Trisquel 11.0 "Aramo" release announcement

Our most ambitious release to date, Trisquel 11.0 Aramo is out! After extensive work and thorough testing, we are proud to declare Aramo to be production-ready. This release brings many improvements and covers more grounds both in terms of machines supported and in installation options. Here are some highlights of the main accomplishments included in this release:

Major achievements

  • New supported architectures. Following the addition of 32bit ARM support in Trisquel 10, we now introduce support for 64bit ARM and POWER architectures, to expand the options for hardware compatibility.
  • d-i/Netinstall (also called "debian-installer") is the text-mode installer for Trisquel, which allows for advanced and custom installations, often used for servers. After Ubuntu dropped support for this installation method, we stepped in to add any removed pieces and make it functional again, often from upstream Debian sources.
  • Browser packaging: as Ubuntu continues to shift towards snap packaging, we took on the task of continuing to package key components such as Abrowser (our improved Firefox derivative) as a standard .deb package. Abrowser continues to bring a fully free browser that balances privacy and usability.
  • Compatibility with AMD/ATI graphics cards. We made a specific effort to improve the support for these cards without requiring non-free firmware blobs. The result is a default configuration that should make most of these types of cards work at a basic level, without 2D/3D acceleration. Please report if you test it with one of those cards!

Aramo is based on Ubuntu 22.04LTS and will receive support until 2027. Users of Trisquel 10 Nabia can upgrade directly using the update-manager or do-release-upgrade commands at a console terminal.

Editions

  • Trisquel. We keep MATE (version 1.26 on this release) as the default desktop environment due to its great accessibility support, simple user interface and no dependency on 3D acceleration.
  • Triskel. Our KDE (v5.92) based edition now as mature as our MATE release is excellent for customizing the design and functionality in fine detail.
  • Trisquel Mini. Running LXDE (v0.99.2), the Mini edition is a lightweight desktop perfect for netbooks, old computers and users with minimal resource usage needs.
  • Trisquel Sugar or Trisquel On A Sugar Toast (TOAST): Based on the Sugar learning platform (v0.119), TOAST comes with dozens of educational activities for children.
  • Network installer image: To deploy with a command-line install interface, it is ideal for servers and advanced users who want to explore custom designed environments. GNOME users will be happy to find that Trisquel GNOME environment has been added to the tasksel step on the d-i installer.

In addition, this release had put some additional work on Budgie and Kylin alternative desktop environments for users that would like them to test them out, help us with some feedback.

Upcoming changes

One pending task that will get into motion soon is the main website redesign and improvement on l10n support and as we transition to the trisquel.org domain.

During Aramo's lifespan we'll continue to improve installation support for ARM and POWERPC whose base rootfs are available at our cdimage archive.

With the continuous support form our community and contributors we will keep providing one of the best fully free operating systems, and tackle the big technical challenges ahead. Trisquel is a non-profit project, you can help sustain it by becoming a member, donating or buying from our store.

We can't finish this great release without thanking for all the donors that keep the project going, and to the hackers Amin Bandali, bill-auger, David L, David Philipe Gil, Denis "GNUtoo" Carikli, "dragestil", Francis Meetze, Jason Self, Joshua Aspinall , Keno Goertz, "knife", "Lappi", Legimet, Mason Hock, Pablo Correa, "Parodper", Simon Josefsson, and many others for all the code, patches, bug reports, translations, and advice. Special thanks to Luis "Ark74" Guzmán, for his very prolific contributions, and to the wonderful community that keeps the project going.

19 March, 2023 06:58PM by quidam

FSF Latin America

The TRApp Trap

The TRApp Trap

Mobile phone apps, that our board member Alexandre Oliva calls TRApps in his new article, have replaced, not very spontaneously, web sites that adhered to international standards and were compatible with free systems, TRApping people in a duopoly of proprietary and invasive systems.

When private businesses do so, it's bad; but when governments impose on citizens the use of proprietary operating systems and programs, to get public services or to comply with legal obligations, we denounce them as imposed taxing software.

They're "imposed" in the sense that you can't avoid them, and "taxing" in that they charge you and take from you your most valuable good: your freedom.

We call for consumers, citizens and users at large to resist these impositions and insist that public and private services be available through sites that will work properly when accessed with a standard browser on a free operating system, without installing freedom-depriving programs, not even those that even standard browsers themselves would install and run automatically from visited sites. And, when it's necessary to run software on the service recipient's computer, the software ought to be free.

Read the full article on our site, without TRApps or proprietary JavaScript.
https://www.fsfla.org/texto/TRApps


About FSFLA

Free Software Foundation Latin America joined in 2005 the international FSF network, previously formed by Free Software Foundations in the United States, in Europe and in India. These sister organizations work in their corresponding geographies towards promoting the same Free Software ideals and defending the same freedoms for software users and developers, working locally but cooperating globally.
https://www.fsfla.org/


Copyright 2023 FSFLA

Permission is granted to make and distribute verbatim copies of this entire document without royalty, provided the copyright notice, the document's official URL, and this permission notice are preserved.

Permission is also granted to make and distribute verbatim copies of individual sections of this document worldwide without royalty provided the copyright notice and the permission notice above are preserved, and the document's official URL is preserved or replaced by the individual section's official URL.

https://www.fsfla.org/anuncio/2023-03-TRApps

19 March, 2023 03:12PM

FSF News

Free Software Awards winners announced: Eli Zaretskii, Tad (SkewedZeppelin), GNU Jami

BOSTON, Massachusetts, USA -- Saturday, March 18, 2023 -- The Free Software Foundation (FSF) today announced the recipients of the 2022 Free Software Awards, which are given annually at the FSF's LibrePlanet conference to groups and individuals in the free software community who have made significant contributions to the cause for software freedom. This year's recipients of the awards are Eli Zaretskii, Tad (SkewedZeppelin), and GNU Jami. As LibrePlanet 2023 is a hybrid in-person and online conference this year, the ceremony was conducted both in person and virtually.

19 March, 2023 01:05AM

March 18, 2023

FSF Latin America

Linux-libre turns 15!

Linux-libre turns 15!

It was February 2008 when Jeff Moe announced Linux-libre, a project to share the efforts that freedom-respecting distros had to undertake to drop the nonfree bits distributed as part of the kernel Linux.
https://web.archive.org/web/1/lists.autistici.org/message/20080221.002845.467ba592.en.html

"For fifteen years, the Linux-libre project has remained dedicated to providing a kernel that respects everyone's freedom and has become an essential part of the free software movement. Linux-libre is widely used by those who value their freedom to use, study, change, and share software without restrictions or limitations. These freedoms are essential to creating a just society."
-- Jason Self

Since around 1996, Linux has carried sourceless firmware encoded as sequences of numbers disguised as source code. UTUTO and gNewSense pioneered the efforts of removing them. Cleaning Linux up is a substantial amount of work, so the existence of Linux-libre has alleviated one of the main difficulties in maintaining GNU+Linux distros that abide by the GNU Free Software Distribution Guidelines. The Linux-libre compiled kernel distributions maintained by Jason Self, Freesh (.deb), liberRTy (low-latency .deb) and RPMFreedom (.rpm), make it easy for users of other GNU+Linux distros to take a step towards freedom when their hardware is not too user-hostile.

"Thanks to Linux-libre, we have entirely libre GNU+Linux distros. Thanks to Linux-libre, people like me who are not kernel hackers can install one of those distros and have a computer which never runs a nonfree program on the CPU. (Provided we use LibreJS as well to reject nonfree Javascript programs that web sites send us.)"
-- Richard Stallman

Early pieces of firmware in Linux ran peripheral devices, but some of the blobs loaded by Linux nowadays reconfigure the primary central processing units and others contain an entire operating system for the peripherals' CPUs, including a copy of the kernel Linux itself and several other freedom-depriving programs!

After years of our denouncing the social, technical, and legal risks out of Linux's misbehavior, most of the blobs got moved to separate files, still part of the kernel Linux, and then to separate packages, which mitigates some of the legal risks, but the problem keeps growing: more and more devices depend on nonfree firmware and thus remain under exclusive and proprietary control by their suppliers.

Challenge

For 27 years, the nonfree versions of Linux have shown that tolerating blobs and making it easy for users to install and accept them makes users increasingly dependent on user-hostile, blob-requiring devices for their computing. Refusing to give these devices' suppliers what they wish, namely your money and control over your computing, is more likely to succeed at changing their practices if more users refuse.

If you're the kind of software freedom supporter who demands respect for your freedom, keep on enjoying the instant gratification that GNU Linux-libre affords you, and supporting (or being!) those who refurbish old computers and build new ones to respect our autonomy.

However, if you're of the kind for whom last-generation computers are hard to resist, even though you'd prefer if they were more respectful of your freedom, you may wish to consider a delayed gratification challenge: if you and your friends resist hostile computers now, you may get more respectful ones later, for yourselves and for all of us; if you don't, the next generations will likely be even more hostile. Are you up for the challenge?
https://en.wikipedia.org/wiki/Delayed_gratification

Present and Future

GNU Linux-libre releases are currently prepared with scripts that automate the cleaning-up and part of the verification. For each upstream major and stable release, we run the scripts, updating them as needed, and publish them, along with the cleaning-up logs and the cleaned-up sources, in a git repository. Each source release is an independent tag, as in, there are no branches for cleaned-up sources. This is so we can quickly retract releases if freedom bugs are found.

We have plans to change the cleaning-up process and the repository structure in the future: we're (slowly) preparing to move to a rewritten git repository, in which, for each commit in upstream Linux main and stable repositories, there will be a corresponding cleaned-up commit in ours. Undesirable bits are going to be cleaned up at the commit corresponding to the one in which upstream introduced or modified them, and other modifications will be checked and integrated unchanged, mirroring the upstream commit graph, with "git replace" mappings for individual commits and, perhaps, also for cleaned-up files.

This is expected to enable us to track upstream development very closely, to get stable and major releases out nearly instantly and often automatically and to enable Linux developers to clone our freed repository instead of our upstream to write and test their changes. The same techniques used to create the cleaned-up repository can be used to fix freedom bugs in it.

Artwork

Jason Self has made several beautiful pictures of his version of Freedo, our light-blue penguin mascot, and we've used them for our recent releases.

Marking the beginning of the week in which we celebrate 15 years of Linux-libre, we had the pleasure of publishing a major release, 6.2-gnu, codenamed "la quinceañera", with a picture of Freedo dressed up for the occasion.
https://www.fsfla.org/pipermail/linux-libre/2023-February/003502.html

But there's more! He also made a commemorative black-and-white wallpaper with classic Freedo, also dressed up for the occasion. Check them out, and feel free to tune the colors to your liking!
https://linux-libre.fsfla.org/#news

He also modeled a 3D Freedo in Blender, and we're looking for someone who could 3D-print it and get it to the FSF office in time for the LibrePlanet conference. Rumor has it that Richard Stallman is going to auction it off to raise funds for the FSF! Can you help?
https://libreplanet.org/2023/


About GNU Linux-libre

GNU Linux-libre is a GNU package maintained by Alexandre Oliva, on behalf of FSFLA, and by Jason Self. It releases cleaned-up versions of Linux, suitable for use in distributions that comply with the Free Software Distribution Guidelines published by the GNU project, and by users who wish to run Free versions of Linux on their GNU systems. The project offers cleaning-up scripts, Free sources, binaries for some GNU+Linux distributions, and artwork with GNU and the Linux-libre mascot: Freedo, the clean, Free and user-friendly light-blue penguin. Visit our web site and Be Free!
https://linux-libre.fsfla.org/
https://www.gnu.org/distros/

About the GNU Operating System and Linux

Richard Stallman announced in September 1983 the plan to develop a Free Software Unix-like operating system called GNU. GNU is the only operating system developed specifically for the sake of users' freedom.
https://www.gnu.org/
https://www.gnu.org/gnu/the-gnu-project.html

In 1992, the essential components of GNU were complete, except for one, the kernel. When in 1992 the kernel Linux was re-released under the GNU GPL, making it Free Software, the combination of GNU and Linux formed a complete Free operating system, which made it possible for the first time to run a PC without non-Free Software. This combination is the GNU+Linux system.
https://www.gnu.org/gnu/gnu-linux-faq.html

About FSFLA

Free Software Foundation Latin America joined in 2005 the international FSF network, previously formed by Free Software Foundations in the United States, in Europe and in India. These sister organizations work in their corresponding geographies towards promoting the same Free Software ideals and defending the same freedoms for software users and developers, working locally but cooperating globally.
https://www.fsfla.org/


Copyright 2023 FSFLA

Permission is granted to make and distribute verbatim copies of this entire document without royalty, provided the copyright notice, the document's official URL, and this permission notice are preserved.

Permission is also granted to make and distribute verbatim copies of individual sections of this document worldwide without royalty provided the copyright notice and the permission notice above are preserved, and the document's official URL is preserved or replaced by the individual section's official URL.

https://www.fsfla.org/anuncio/2023-02-Linux-libre-15

18 March, 2023 06:13PM

GNU Health

Leading Public Mental Health Hospital in Argentina embraces GNU Health

The World Health Organization defines health as a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity.

Unfortunately, this definition is far from being a reality in our societies. Instead of embracing the system of health, we live in the system of disease, ruled by a reactive, reductionist and unsustainable model of healthcare. The beautiful noble art and science of medicine is ill. Financial institutions and giant technological corporations are removing the human factor from medicine, transforming people and patients into clients. They are reducing the non-negotiable human right to healthcare to a privilege of a few.

Coming back to the formal definition of health, in the current system of disease very little is taken into account from the social and mental well-being . Today, many people with mental health conditions not only have to deal with the physiopathological aspects of the disorder, but also with the stigma, exclusion and invisibilization from the society.

But there is hope. Medicine is a social science, and GNUHealth is a social project with some technology behind. That feeling of optimism and hope has been reinforced in last week trip to Argentina and their people. In the end, medicine is about people interacting and taking care of people. Is about people before patients. I know them well, because I did my medical career in Argentina.

Group picture with health professionals from HESM, UNER, Government officials and GNU Solidario at the entrance of the leading Public Mental Health Hospital in Entre Ríos, Argentina

The Mental Health Hospital has chosen GNUHealth to improve the management of the institution resources, as well as to provide the best medical care for their community, both in outpatient and inpatient settings. Being able to properly identify every person who needs attention, and knowing the socio-sanitary, medical and clinical history in real time will make a big difference in the care of the individual.

The implementation of GNUHealth in this health institution will be lead by Prof. Dr. Fernando Sassetti and the department of public health studies of the University of Entre Ríos in the context of the GNU Health Alliance of Academic and Research Institutions agreement signed with GNU Solidario.

Health is an equilibrium of the inseparable and interconnected physical, social, mental and spiritual domains. Medicine is about taking into consideration and maintaining this body-mind-spirit-environment balance. This holistic approach to medicine is encoded in the genome of every nurse, psychologist, social worker and doctor from the Mental Health Hospital and the Primary care centers I got to know during these years in Entre Ríos, Argentina.

Links / References

Un software Libre para mejorar las políticas de salud: https://www.eldiario.com.ar/253548-un-software-para-mejorar-las-politicas-de-salud/

Hospital Escuela de Salud Mental : http://www.hesm.gob.ar/

Audiovisual institucional Hospital Escuela de Salud Mental: https://www.youtube.com/watch?v=Jx08WyfKRIE&t=12s

GNU Health: https://www.gnuhealth.org

18 March, 2023 12:53AM by Luis Falcon

March 16, 2023

FSF News

Gary Benson

Which APT repository did a package come from?

$ apt policy wget
wget:
  Installed: 1.21.2-2ubuntu1
  Candidate: 1.21.2-2ubuntu1
  Version table:
 *** 1.21.2-2ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

16 March, 2023 01:18PM by gbenson

Python debugger

By the way, if you’ve never used Python’s debugger, it really is as simple as adding a call to the built-in function breakpoint() at the point you want it to stop.

16 March, 2023 10:09AM by gbenson

March 15, 2023

GNU Guix

Building Toolchains with Guix

In order to deploy embedded software using Guix we first need to teach Guix how to cross-compile it. Since Guix builds everything from source, this means we must teach Guix how to build our cross-compilation toolchain.

The Zephyr Project uses its own fork of GCC with custom configs for the architectures supported by the project. In this article, we describe the cross-compilation toolchain we defined for Zephyr; it is implemented as a Guix channel.

About Zephyr

Zephyr is a real-time operating system from the Linux Foundation. It aims to provide a common environment which can target even the most resource constrained devices.

Zephyr introduces a module system which allows third parties to share code in a uniform way. Zephyr uses CMake to perform physical component composition of these modules. It searches the filesystem and generates scripts which the toolchain will use to successfully combine those components into a firmware image.

The fact that Zephyr provides this mechanism is one reason I chose to target it in the first place.

This separation of modules in an embedded context is a really great thing. It brings many of the advantages that it brings to the Linux world such as code re-use, smaller binaries, more efficient cache/RAM usage, etc. It also allows us to work as independent groups and compose contributions from many teams.

It also brings all of the complexity. Suddenly most of the problems that plague traditional deployment now apply to our embedded system. The fact that the libraries are statically linked at compile time instead of dynamically at runtime is simply an implementation detail. I say most because everything is statically linked so there is no runtime component discovery that needs to be accounted for.

Anatomy of a Toolchain

Toolchains are responsible for taking high level descriptions of programs and lowering them down to a series of equivalent machine instructions. This process involves more than just a compiler. The compiler uses the GNU Binutils to manipulate its internal representation down to a given architecture. It also needs the use of the C standard library as well as a few other libraries needed for some compiler optimizations.

The C library provides the interface to the underlying kernel. System calls like write and read are provided by GNU C Library (glibc) on most distributions.

In embedded systems, smaller implementations like RedHat's newlib and newlib-nano are used.

Bootstrapping a Toolchain

In order to compile GCC we need a C library that's been compiled for our target architecture. How can we cross compile our C library if we need our C library to build a cross compiler? The solution is to build a simpler compiler that doesn't require the C library to function. It will not be capable of as many optimizations and it will be very slow, however it will be able to build the C libraries as well as the complete version of GCC.

In order to build the simpler compiler we need to compile the Binutils to work with our target architecture. Binutils can be bootstrapped with our host GCC and have no target dependencies. More information is available in this article.

Doesn't sound so bad right? It isn't... in theory. However internet forums since time immemorial have been littered with the laments of those who came before. From incorrect versions of ISL to the wrong C library being linked or the host linker being used, etc. The one commonality between all of these issues is the environment. Building GCC is difficult because isolating build environments is hard.

In fact as of v0.14.2, the Zephyr “software development kit” (SDK) repository took down the build instructions and posted a sign that read "Building this is too complicated, don't worry about it." (I'm paraphrasing, but not by much.)

We will neatly sidestep all of these problems and not risk destroying or polluting our host system with garbage by using Guix to manage our environments for us.

Our toolchain only requires the first pass compiler because newlib(-nano) is statically linked and introduced to the toolchain by normal package composition.

Defining the Packages

All of the base packages are defined in zephyr/packages/zephyr.scm. Zephyr modules (coming soon!) are defined in zephyr/packages/zephyr-xyz.scm, following the pattern of other module systems implemented by Guix.

Binutils

First thing we need to build is the arm-zephyr-eabi binutils. This is very easy in Guix.

(define-public arm-zephyr-eabi-binutils
  (let ((xbinutils (cross-binutils "arm-zephyr-eabi")))
    (package
      (inherit xbinutils)
      (name "arm-zephyr-eabi-binutils")
      (version "2.38")
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://github.com/zephyrproject-rtos/binutils-gdb")
                      (commit "6a1be1a6a571957fea8b130e4ca2dcc65e753469")))
                (file-name (git-file-name name version))
                (sha256 (base32 "0ylnl48jj5jk3jrmvfx5zf8byvwg7g7my7jwwyqw3a95qcyh0isr"))))
      (arguments
       `(#:tests? #f
         ,@(substitute-keyword-arguments (package-arguments xbinutils)
             ((#:configure-flags flags)
          `(cons "--program-prefix=arm-zephyr-eabi-" ,flags)))))
      (native-inputs
       (modify-inputs (package-native-inputs xbinutils)
         (prepend texinfo bison flex gmp dejagnu)))
      (home-page "https://zephyrproject.org")
      (synopsis "Binutils for the Zephyr RTOS"))))

The function cross-binutils returns a package which has been configured for the given GNU triplet. We simply inherit that package and replace the source. The Zephyr build system expects the binutils to be prefixed with arm-zephyr-eabi- which is accomplished by adding another flag to the #:configure-flags argument.

We can test our package definition using the -L flag with guix build to add our packages.

$ guix build -L guix-zephyr zephyr-binutils

/gnu/store/...-zephyr-binutils-2.38

This directory contains the results of make install.

GCC sans libc

This one is a bit more involved. Don't be afraid! This version of GCC wants ISL version 0.15. It's easy enough to make that happen. Inherit the current version of ISL and swap out the source and update the version. For most packages the build process doesn't change that much between versions.

(define-public isl-0.15
  (package
    (inherit isl)
    (version "0.15")
    (source (origin
              (method url-fetch)
              (uri (list (string-append "mirror://sourceforge/libisl/isl-"
                            version ".tar.gz")))
              (sha256
               (base32
                "11vrpznpdh7w8jp4wm4i8zqhzq2h7nix71xfdddp8xnzhz26gyq2"))))))

Like the binutils, there is a cross-gcc function for creating cross-GCC packages. This one accepts keywords specifying which binutils and libc to use. If libc isn't given (like here), gcc is configured with many options disabled to facilitate being built without libc. Therefore we need to add the extra options we want (I got them from the SDK configuration scripts in the sdk-ng Git repository as well as the commits to use for each of the tools).

(define-public gcc-arm-zephyr-eabi-12
  (let ((xgcc (cross-gcc "arm-zephyr-eabi"
                         #:xbinutils zephyr-binutils)))
    (package
      (inherit xgcc)
      (version "12.1.0")
      (source (origin
                (method git-fetch)
                (uri (git-reference
                      (url "https://github.com/zephyrproject-rtos/gcc")
                      (commit "0218469df050c33479a1d5be3e5239ac0eb351bf")))
                (file-name (git-file-name (package-name xgcc) version))
                (sha256
                 (base32
                  "1s409qmidlvzaw1ns6jaanigh3azcxisjplzwn7j2n3s33b76zjk"))
                (patches (search-patches
                          "gcc-12-cross-environment-variables.patch"
                          "gcc-cross-gxx-include-dir.patch"))))
      (native-inputs (modify-inputs (package-native-inputs xgcc)
                       ;; Get rid of stock ISL
                       (delete "isl")
                       ;; Add additional dependencies that xgcc doesn't have
                       ;; including our special ISL
                       (prepend flex
                                isl-0.15)))
      (arguments
       (substitute-keyword-arguments (package-arguments xgcc)
         ((#:phases phases)
          `(modify-phases ,phases
             (add-after 'unpack 'fix-genmultilib
               (lambda _
                 (patch-shebang "gcc/genmultilib")))

             (add-after 'set-paths 'augment-CPLUS_INCLUDE_PATH
               (lambda* (#:key inputs #:allow-other-keys)
                 (let ((gcc (assoc-ref inputs "gcc")))
                   ;; Remove the default compiler from CPLUS_INCLUDE_PATH to
                   ;; prevent header conflict with the GCC from native-inputs.
                   (setenv "CPLUS_INCLUDE_PATH"
                           (string-join (delete (string-append gcc
                                                               "/include/c++")
                                                (string-split (getenv
                                                               "CPLUS_INCLUDE_PATH")
                                                              #\:)) ":"))
                   (format #t
                    "environment variable `CPLUS_INCLUDE_PATH' changed to `a`%"
                    (getenv "CPLUS_INCLUDE_PATH")))))))

         ((#:configure-flags flags)
          ;; The configure flags are largely identical to the flags used by the
          ;; "GCC ARM embedded" project.
          `(append (list
                    "--enable-multilib"
                    "--with-newlib"
                    "--with-multilib-list=rmprofile"
                    "--with-host-libstdcxx=-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm"
                    "--enable-plugins"
                    "--disable-decimal-float"
                    "--disable-libffi"
                    "--disable-libgomp"
                    "--disable-libmudflap"
                    "--disable-libquadmath"
                    "--disable-libssp"
                    "--disable-libstdcxx-pch"
                    "--disable-nls"
                    "--disable-shared"
                    "--disable-threads"
                    "--disable-tls"
                    "--with-gnu-ld"
                    "--with-gnu-as"
                    "--enable-initfini-array")
                   (delete "--disable-multilib"
                           ,flags)))))
      (native-search-paths
       (list (search-path-specification
              (variable "CROSS_C_INCLUDE_PATH")
              (files '("arm-zephyr-eabi/include")))
             (search-path-specification
              (variable "CROSS_CPLUS_INCLUDE_PATH")
              (files '("arm-zephyr-eabi/include" "arm-zephyr-eabi/c++"
                       "arm-zephyr-eabi/c++/arm-zephyr-eabi")))
             (search-path-specification
              (variable "CROSS_LIBRARY_PATH")
              (files '("arm-zephyr-eabi/lib")))))
      (home-page "https://zephyrproject.org")
      (synopsis "GCC for the Zephyr RTOS"))))

This GCC can be built like so.

$ guix build -L guix-zephyr gcc-cross-sans-libc-arm-zephyr-eabi

/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0-lib
/gnu/store/...-gcc-cross-sans-libc-arm-zephyr-eabi-12.1.0

Great! We now have our stage-1 compiler.

Newlib(-nano)

The newlib package package is quite straight forward (relatively). It is mostly adding in the relevent configuration flags and patching the files the patch-shebangs phase missed.

(define-public zephyr-newlib
  (package
    (name "zephyr-newlib")
    (version "3.3")
    (source (origin
          (method git-fetch)
          (uri (git-reference
            (url "https://github.com/zephyrproject-rtos/newlib-cygwin")
            (commit "4e150303bcc1e44f4d90f3489a4417433980d5ff")))
          (sha256
           (base32 "08qwjpj5jhpc3p7a5mbl7n6z7rav5yqlydqanm6nny42qpa8kxij"))))
    (build-system gnu-build-system)
    (arguments
     `(#:out-of-source? #t
       #:configure-flags '("--target=arm-zephyr-eabi"
               "--enable-newlib-io-long-long"
               "--enable-newlib-io-float"
               "--enable-newlib-io-c99-formats"
               "--enable-newlib-retargetable-locking"
               "--enable-newlib-lite-exit"
               "--enable-newlib-multithread"
               "--enable-newlib-register-fini"
               "--enable-newlib-extra-sections"
               "--disable-newlib-wide-orient"
               "--disable-newlib-fseek-optimization"
               "--disable-newlib-supplied-syscalls"
               "--disable-newlib-target-optspace"
               "--disable-nls")
       #:phases
       (modify-phases %standard-phases
     (add-after 'unpack 'fix-references-to-/bin/sh
       (lambda _
         (substitute# '("libgloss/arm/cpu-init/Makefile.in"
                "libgloss/arm/Makefile.in"
                "libgloss/libnosys/Makefile.in"
                "libgloss/Makefile.in")
           (("/bin/sh") (which "sh")))
         #t)))))
    (native-inputs
     `(("xbinutils" ,zephyr-binutils)
       ("xgcc" ,gcc-arm-zephyr-eabi-12)
       ("texinfo" ,texinfo)))
    (home-page "https://www.sourceware.org/newlib/")
    (synopsis "C library for use on embedded systems")
    (description "Newlib is a C library intended for use on embedded
systems.  It is a conglomeration of several library parts that are easily
usable on embedded products.")
    (license (license:non-copyleft
          "https://www.sourceware.org/newlib/COPYING.NEWLIB"))))

And the build.

$ guix build -L guix-zephyr zephyr-newlib

/gnu/store/...-zephyr-newlib-3.3

Complete Toolchain

Mostly complete. libstdc++ does not build because arm-zephyr-eabi is not arm-none-eabi so a dynamic link check is performed/failed. I cannot figure out how crosstool-ng handles this.

Now that we've got the individual tools it's time to create our complete toolchain. For this we need to do some package transformations. Because these transformations are going to have to be done for every combination of binutils/gcc/newlib it is best to create a function which we can reuse for every version of the SDK.

(define (arm-zephyr-eabi-toolchain xgcc newlib version)
  "Produce a cross-compiler zephyr toolchain package with the compiler XGCC and the C\n  library variant NEWLIB."
  (let ((newlib-with-xgcc
         (package
           (inherit newlib)
           (native-inputs
            (modify-inputs (package-native-inputs newlib)
              (replace "xgcc" xgcc))))))
    (package
      (name (string-append "arm-zephyr-eabi"
                           (if (string=? (package-name newlib-with-xgcc)
                                         "newlib-nano")
                               "-nano"
                               "")
                           "-toolchain"))
      (version version)
      (source #f)
      (build-system trivial-build-system)
      (arguments
       '(#:modules ((guix build union)
                    (guix build utils))
         #:builder (begin
                     (use-modules (ice-9 match)
                                  (guix build union)
                                  (guix build utils))
                     (let ((out (assoc-ref %outputs "out")))
                       (mkdir-p out)
                       (match %build-inputs
                         (((names . directories) ...)
                          (union-build (string-append out "/arm-zephyr-eabi")
                                       directories)))))))
      (inputs `(("binutils" ,zephyr-binutils)
                ("gcc" ,xgcc)
                ("newlib" ,newlib-with-xgcc)))
      (synopsis "Complete GCC tool chain for ARM zephyrRTOS development")
      (description
       "This package provides a complete GCC tool chain for ARM
  bare metal development with zephyr rtos.  This includes the GCC arm-zephyr-eabi cross compiler
  and newlib (or newlib-nano) as the C library.  The supported programming
  language is C.")
      (home-page (package-home-page xgcc))
      (license (package-license xgcc)))))

This function creates a special package which consists of the toolchain in a special directory hierarchy, i.e arm-zephyr-eabi/. Our complete toolchain definition looks like this.

(define-public arm-zephyr-eabi-toolchain-0.15.0
  (arm-zephyr-eabi-toolchain gcc-arm-zephyr-eabi-12 zephyr-newlib
                             "0.15.0"))

To build:

$ guix build -L guix-zephyr arm-zephyr-eabi-toolchain
/gnu/store/...-arm-zephyr-eabi-toolchain-0.15.0

Note: Guix now includes a mechanism to describe platforms at a high level, and which the --system and --target build options build upon. It is not used here but could be a way to better integrate Zephyr support in the future.

Integrating with Zephyr Build System

Zephyr uses CMake as its build system. It contains numerous CMake files in both the so-called ZEPHYR_BASE, the zephyr source code repository, as well as a handful in the SDK which help select the correct toolchain for a given board.

There are standard locations the build system will look for the SDK. We are not using any of them. Our SDK lives in the store, immutable forever. According to the Zephyr documentation, the variable ZEPHYR_SDK_INSTALL_DIR needs to point to our custom spot.

We also need to grab the CMake files from the repository and create a file, sdk_version, which contains the version string ZEPHYR_BASE uses to find a compatible SDK.

Along with the SDK proper we need to include a number of python packages required by the build system.

(define-public zephyr-sdk
  (package
    (name "zephyr-sdk")
    (version "0.15.0")
    (home-page "https://zephyrproject.org")
    (source (origin
              (method git-fetch)
              (uri (git-reference
                    (url "https://github.com/zephyrproject-rtos/sdk-ng")
                    (commit "v0.15.0")))
              (file-name (git-file-name name version))
              (sha256
               (base32
                "04gsvh20y820dkv5lrwppbj7w3wdqvd8hcanm8hl4wi907lwlmwi"))))
    (build-system trivial-build-system)
    (arguments
     `(#:modules ((guix build union)
                  (guix build utils))
       #:builder (begin
                   (use-modules (guix build union)
                                (ice-9 match)
                                (guix build utils))
                   (let ((out (assoc-ref %outputs "out"))
                         (cmake-scripts (string-append (assoc-ref
                                                        %build-inputs
                                                        "source")
                                                       "/cmake"))
                         (sdk-out (string-append out "/zephyr-sdk-0.15.0")))
                     (mkdir-p out)

                     (match (assoc-remove! %build-inputs "source")
                       (((names . directories) ...)
                        (union-build sdk-out directories)))

                     (copy-recursively cmake-scripts
                                       (string-append sdk-out "/cmake"))

                     (with-directory-excursion sdk-out
                       (call-with-output-file "sdk_version"
                         (lambda (p)
                           (format p "0.15.0"))))))))
    (propagated-inputs (list arm-zephyr-eabi-toolchain-0.15.0
                             zephyr-binutils
                             dtc
                             python-3
                             python-pyelftools
                             python-pykwalify
                             python-pyyaml
                             python-packaging))
    (native-search-paths
     (list (search-path-specification
            (variable "ZEPHYR_SDK_INSTALL_DIR")
            (separator #f)
            (files '("")))))
    (synopsis "Zephyr SDK")
    (description
     "zephyr-sdk contains bundles a complete gcc toolchain as well
as host tools like dtc, openocd, qemu, and required python packages.")
    (license license:apsl2)))

Testing

In order to test we will need an environment with the SDK installed. We can take advantage of guix shell to avoid installing test packages into our home environment. This way if it causes problems we can just exit the shell and try again.

guix shell -L guix-zephyr zephyr-sdk cmake ninja git

ZEPHYR_BASE can be cloned into a temporary workspace to test our toolchain functionality. (For now. Eventually we will need to create a package for zephyr-base that our Guix zephyr-build-system can use.)

mkdir /tmp/zephyr-project
cd /tmp/zephyr-project
git clone https://github.com/zephyrproject-rtos/zephyr
export ZEPHYR_BASE=/tmp/zephyr-project/zephyr

In order to build for the test board (k64f in this case) we need to get a hold of the vendor Hardware Abstraction Layers and CMSIS. (These will also need to become Guix packages to allow the build system to compose modules).

git clone https://github.com/zephyrproject-rtos/hal_nxp && \
git clone https://github.com/zephyrproject-rtos/cmsis

To inform the build system about this module we pass it in with -DZEPHYR_MODULES= which is a semicolon separated list of paths containing a module.yml file.

To build the hello world sample we use the following incantation.

cmake -Bbuild $ZEPHYR_BASE/samples/hello_world \
    -GNinja \
    -DBOARD=frdm_k64f \
    -DBUILD_VERSION=3.1.0 \
    -DZEPHYR_MODULES="/tmp/zephyr-project/hal_nxp;/tmp/zephyr-project/cmsis" \
      && ninja -Cbuild

If everything is set up correctly we will end up with a ./build directory with all our build artifacts. The SDK is correctly installed!

Conclusion

A customized cross toolchain is one of the most difficult pieces of software to build. Using Guix, we do not need to be afraid of the complexity! We can fiddle with settings, swap out components, and do the most brain dead things to our environments without a care in the world. Just exit the environment and it's like it never happened at all.

It highlights one of my favorite aspects of Guix, every package is a working reference design for you to modify and learn from.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

15 March, 2023 04:00PM by Mitchell Schmeisser

March 12, 2023

a2ps @ Savannah

a2ps 4.15.1 released [stable]


GNU a2ps is a filter which generates PostScript from various formats,
with pretty-printing features, strong support for many alphabets, and
customizable layout.

See https://www.gnu.org/software/a2ps/ for more information.

This is a bug-fix release. Users of 4.15 should upgrade. See below for more
details.


Here are the compressed sources and a GPG detached signature:
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.1.tar.gz
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.1.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

8674b90626d6d1505af8b2ae392f2495b589a052  a2ps-4.15.1.tar.gz
l5dwi6AoBa/DtbkeBsuOrJe4WEOpDmbP3mp8Y8oEKyo  a2ps-4.15.1.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.15.1.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.15.1.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5853-ge0aefd96b6

NEWS

* Noteworthy changes in release 4.15.1 (2023-03-12) [stable]
 * Bug fixes:
   - Use “grep -F” rather than obsolete fgrep.
   - Fix broken a2ps-lpr-wrapper script, and translate to sh for
     portability.


12 March, 2023 03:06PM by Reuben Thomas

March 07, 2023

a2ps 4.15 released [stable]


I am delighted to announce the first stable release of GNU a2ps since 2007!

This release contains few user-visible changes. It does however contain a
lot of changes “under the hood”: code clean-up, etc. Therefore, it’s likely
that there are new bugs. Do report them to Savannah[1], or the mailing list
please!

A big thank-you to all those who tested pre-releases, and especially to
Bruno Haible’s tireless work to promote portability: he both tested a2ps on
many systems and found lots of minor portability problems, and advised on
their solution (often, gnulib code that he wrote). Remaining problems are of
course mine!

[1] https://savannah.gnu.org/projects/a2ps


Here are the compressed sources and a GPG detached signature:
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.tar.gz
  https://ftpmirror.gnu.org/a2ps/a2ps-4.15.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

807667f838c29bde73bb91fae60ef98826bd460e  a2ps-4.15.tar.gz
pa3FqSIvmESKV8a162lItydD6vmjDGehNN8ILpnHZlI  a2ps-4.15.tar.gz

The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify a2ps-4.15.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa2048 2013-12-11 [SC]
        2409 3F01 6FFE 8602 EF44  9BB8 4C8E F3DA 3FD3 7230
  uid   Reuben Thomas <rrt@sc3d.org>
  uid   keybase.io/rrt <rrt@keybase.io>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key rrt@sc3d.org

  gpg --recv-keys 4C8EF3DA3FD37230

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=a2ps&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify a2ps-4.15.tar.gz.sig


This release was bootstrapped with the following tools:
  Autoconf 2.71
  Automake 1.16.5
  Gnulib v0.1-5857-gf17d397771

NEWS

* Noteworthy changes in release 4.15 (2023-03-07) [stable]
 * New maintainer, Reuben Thomas.
 * Features:
   - Replace the 'psmandup' utility with simpler 'lp2' to directly print
     documents to a simplex printer.
   - Remove the outdated 'psset' and 'fixnt', and simplify 'fixps' to
     always process its input with Ghostscript.
   - Use libpaper's paper sizes. This includes user-defined paper sizes
     when using libpaper 2. It is still possible to define custom margins
     using "Medium:" specifications in the configuration file, and the
     one size defined by a2ps that libpaper does not know about, Quarto, is
     retained for backwards compatiblity, and as an example.
 * Documentation
   - Remove some obsolete explanations.
   - Reformat --help output consistently to 80 columns.
   - Some English fixes.
 * Bug fixes:
   - Avoid a crash when a medium is not specified; instead, use the default
     libpaper size (configured by the user or sysadmin, or the locale
     default).
   - Fix some other potential crashes and compiler warnings.
   - Fixes for security bugs CVE-2001-1593, CVE-2015-8107 and CVE-2014-0466.
   - Minor bugs fixed.
 * Predefined delegations:
   - Remove support for defunct Netscape and proprietary Acrobat Reader.
   - Add lpr wrapper for automatic detection of different printing systems,
     including CUPS support.
 * Encodings:
   - Use libre fonts for KOI-8.
   - Composite fonts support.
 * Build
   - Update build system to more recent autotools and gettext versions.
   - Build man pages in a simpler and more robust way.
   - Document runtime dependencies.
   - Minor code quality improvements.
   - Minor tidy up and removal of obsolete code.
   - Require libpaper.
   - Remove OS/2 support.


07 March, 2023 06:20PM by Reuben Thomas

March 05, 2023

grep @ Savannah

grep-3.9 released [stable]


This is to announce grep-3.9, a stable release.

The NEWS below describes the two main bug fixes since 3.8.

There have been 38 commits by 4 people in the 26 weeks since 3.8.

Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Bruno Haible (2)
  Carlo Marcelo Arenas Belón (2)
  Jim Meyering (11)
  Paul Eggert (23)

Jim
 [on behalf of the grep maintainers]
==================================================================

Here is the GNU grep home page:
    http://gnu.org/s/grep/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=grep.git;a=shortlog;h=v3.9
or run this command from a git-cloned grep directory:
  git shortlog v3.8..v3.9

Here are the compressed sources:
  https://ftp.gnu.org/gnu/grep/grep-3.9.tar.gz   (2.7MB)
  https://ftp.gnu.org/gnu/grep/grep-3.9.tar.xz   (1.7MB)

Here are the GPG detached signatures:
  https://ftp.gnu.org/gnu/grep/grep-3.9.tar.gz.sig
  https://ftp.gnu.org/gnu/grep/grep-3.9.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

  f84afbfc8d6e38e422f1f2fc458b0ccdbfaeb392  grep-3.9.tar.gz
  7ZF6C+5DtxJS9cpR1IwLjQ7/kAfSpJCCbEJb9wmfWT8=  grep-3.9.tar.gz
  bcaa3f0c4b81ae4192c8d0a2be3571a14ea27383  grep-3.9.tar.xz
  q80RQJ7iPUyvNf60IuU7ushnAUz+7TE7tfSIrKFwtZk=  grep-3.9.tar.xz

Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify grep-3.9.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096/0x7FD9FCCB000BEEEE 2010-06-14 [SCEA]
        Key fingerprint = 155D 3FC5 00C8 3448 6D1E  EA67 7FD9 FCCB 000B EEEE
  uid                   [ unknown] Jim Meyering <jim@meyering.net>
  uid                   [ unknown] Jim Meyering <meyering@fb.com>
  uid                   [ unknown] Jim Meyering <meyering@gnu.org>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key jim@meyering.net

  gpg --recv-keys 7FD9FCCB000BEEEE

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=grep&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify grep-3.9.tar.gz.sig

This release was bootstrapped with the following tools:
  Autoconf 2.72a.65-d081
  Automake 1.16i
  Gnulib v0.1-5861-g2ba7c75ed1

NEWS

* Noteworthy changes in release 3.9 (2023-03-05) [stable]

** Bug fixes

  With -P, some non-ASCII UTF8 characters were not recognized as
  word-constituent due to our omission of the PCRE2_UCP flag. E.g.,
  given f(){ echo Perú|LC_ALL=en_US.UTF-8 grep -Po "$1"; } and
  this command, echo $(f 'r\w'):$(f '.\b'), before it would print ":r".
  After the fix, it prints the correct results: "rú:ú".

  When given multiple patterns the last of which has a back-reference,
  grep no longer sometimes mistakenly matches lines in some cases.
  [Bug#36148#13 introduced in grep 3.4]


05 March, 2023 04:09PM by Jim Meyering

March 01, 2023

guile-cv @ Savannah

Guile-CV version 0.4.0

Guile-CV version 0.4.0 is released! (February 2023)

This is a maintenance release, which introduces new interfaces.

Changes since the previous version

For a list of changes since the previous version, visit the NEWS file. For a complete description, consult the git summary and git log

01 March, 2023 04:26AM by David Pirotte

February 26, 2023

make @ Savannah

GNU Make 4.4.1 Released!

The next stable version of GNU Make, version 4.4.1, has been released and is available for download from https://ftp.gnu.org/gnu/make/?C=M;O=D

Please see the NEWS file that comes with the GNU Make distribution for details on user-visible changes.

26 February, 2023 08:13PM by Paul D. Smith