Planet GNU

Aggregation of development blogs from the GNU Project

April 03, 2020

FSF Blogs

The cataloging of free software

The original version of this blog is in Italian, here.

The principles of the Free Software Directory

The Free Software Directory is a collaborative catalog of software aimed to be the primary source for representing all free software. Each free program has its own page in the Directory from which it is possible to study the evolution it has undergone in both technological and legal terms through a chronological system similar to that of Wikipedia. Each catalogued program is distinguished by one or more aliases, and accompanied by a huge amount of information, which goes beyond the pure needs of the end user. Snapshots of the graphic interface, detailed descriptions, change logs, links to social pages, and lists of licenses and dependencies are examples of all the useful information which can be carefully attached by users to each page.

Everyone can freely subscribe to the Directory and create new pages, but only the pages reviewed and approved by administrators become visible and indexable. Administrative approvals are always made according to strict rules aimed at preventing the spread of proprietary content. As on Wikipedia, each user can have a self-approved personal page, where they can define their identity and discuss with other users. Users can also include sub-pages on which to publish their thematic articles, and any tools useful for the daily life of the Directory. User access rights are assigned to active users, and all those who demonstrate that they have the necessary technical skills and wish to devote themselves daily to the care of the pages have a chance to be welcomed onto the staff. This serene and flexible organization, based on bonds of trust built on facts and adherence to well-defined common ideals, guarantees that the technological and social development produced by the project is gradual but unstoppable. Thus, any investment of time by volunteers is amply repaid.

The project has proved to be a clear success, so much that over the years it has received funding from UNESCO, and is still supported by the Free Software Foundation. The portal boasts the participation of more than 3,000 users from all over the world. Since its creation, it has accumulated more than 80,000 verified and recorded revisions for posterity in the chronology of the MediaWiki pages, all of which are dedicated to facilitating the essential freedoms in more than 16,000 free programs.

The portal's ability to adapt and survive was possible not only because of the technical creativity of the staff, but also by the solid ideal at its base. By guaranteeing maximum visibility to free software, it has thus rewarded developers who freely employ their knowledge for the good of humanity. The transition to free licenses is indeed a moral duty of every developer, and the Free Software Directory is deployed at the forefront to facilitate it with great benefit to the world's cultural heritage.

My experience as an administrator of the Free Software Directory

I am Lorenzo L. Ancora, an Italian citizen born in Calabria. I have been part of the administration of the Free Software Directory since 2018, when I started collaborating with the staff and volunteers of the Free Software Foundation. Below, I will describe what I've learned during my participation in the administrative life of the Directory, while omitting the overly technical and boring details.

Over the years, I've developed the belief that the Internet was created to preserve and spread knowledge, and that this must remain its primary use. The Directory can be compared to a social network, and therefore needs constant moderation to avoid the irrational behaviors of a few that could damage the common cultural heritage. Hatred and discrimination must not find a place in the chronology of the pages, nor in any related communications, without exceptions. In fact, the first quality required to be an administrator is to self-moderate, and be able to moderate others, in order to preserve the existing order and ensure long-term efficiency.

While end users can afford to make mistakes because changes are not immediately visible, those with administrative privileges do not have this comfort. It is never recommended to act without careful planning. Contrary to what most think, it is better to spend more time in planning before making any changes, regardless of your level of competence. The same applies to acts of moderation and for the publication of new content, because any superficial behavior leaves permanent traces, whether they are physical (the chronology of the pages, designed to be permanent) or psychological (users collaborate only when there is a serene environment). Every time we carry out revisions with the utmost care, we leave a good example for those who come after us. Every time we treat others with respect and humanity, we distinguish ourselves from the crowd, and establish a pretext for such courtesy to be repeated.

Going down in the details, officially, my role as administrator is to solve the most serious problems affecting the Web portal. We call this activity "Backlog Management." In addition to doing this, I often validate the pages, manage online meetings, or publish new software. Flexibility is essential, because sooner or later all administrators must go beyond their pre-established role. Due to time zone differences, disasters affecting various areas of the world, or personal commitments, sometimes volunteers cannot fulfill their duties, and neglecting tasks can introduce weeks of delay, or, worse, result in an injustice towards end users. You can participate by writing markup and style sheets, scripting and graphic design, reviewing documentation, and assistance via email and chat. Regardless of what you do, you need to make improvements until it is totally sure that the final result is more than adequate.

As is well known by now, the Directory is a community work, and even many administrative decisions are made democratically, with participation from not only other administrators, but also with the bureaucrats and users present. There are many ways to face even the simplest tasks. For this reason, behind every decision, there is first of all the need to find the best workflow through common consensus. Thanks to the trust built through mutual collaboration, the administrators have the luxury of being able to ask for a common opinion. This proves to be fundamental during the approval and revision of the pages, because each published software program is unique, and is based on heterogeneous technologies and licenses.

By sharing their knowledge and cooperating with each other, all those who participate in the life of the Free Software Directory choose to follow the hard path to extend their cultural background and discover new strategies to creatively reuse what they already know. If we think that the product of our efforts is a catalog of free code, which everyone can execute, study, modify, improve and re-share freely, then it is easy to deduce that there is no more noble use of the network of networks.

03 April, 2020 06:41PM

Better than Zoom: Try these free software tools for staying in touch

In times like these it becomes all the more important to remember that tools like Zoom, Slack, and Facebook Messenger are not benign public services, and while the sentiment they've expressed to the global community in responding to the crisis may be sincere, it hasn't addressed the fundamental ethical issues with any piece of proprietary software.

After taking the LibrePlanet 2020 conference online, we received a number of requests asking us to document our streaming setup. As the pandemic grew worse, this gave way to more curiosity about how the Free Software Foundation (FSF) uses free tools and free communication platforms to conduct our everyday business. And while the stereotype of hackers hunched over a white on black terminal session applies to us in some ways, many of the tools we use are available in any environment, even for people who do not have a lot of technical experience. We've started documenting ethical solutions on the LibrePlanet wiki, in addition to starting a remote communication mailing list to help each other advocate for their use.

In the suggestions that follow, a few of the tools we will recommend depend upon some "self-reliance," that is, steering clear of proprietary network services by hosting free software solutions yourself, or asking a technical friend to do it for you. It's a difficult step, and the benefits may not be immediately obvious, but it's a key part of preserving your autonomy in an age of ubiquitous digital control.

To those who have the technical expertise and available infrastructure, we urge you to consider hosting instances of free communication platforms for your friends, family, and your community at large. For example, with a modest server and some GNU/Linux knowledge, you could help local students learn in freedom by volunteering to administer an instance of one of the programs we'll be recommending below.

The need to self-host can be an uncomfortable reminder of our dependence on the "cloud" -- the network of someone else's computers -- but acknowledging our current reliance on these providers is the first step in making new, dependable systems for ourselves. During dangerous and stressful times, it's tempting to sideline our ethical commitments for easier or more convenient ways to get things done, and software freedom is no exception. We hope these suggestions will inspire you to inform others about the importance of their freedom, privacy, and security.

Chat

When we can no longer communicate face-to-face, tools for voice and video calling often come to mind as the next best thing. But as evidenced by the size and success of the proprietary software companies that sponsor these tools, their development isn't easy. Promoting real-time voice and video chat clients remains a High Priority Project of ours. Though we may still be waiting for a truly perfect solution, there are some projects that are far enough along in their development that we can recommend them to others.

Audio calls

  • Mumble: Mumble is a real-time, low latency program for hosting and joining audio conversations. Clients are available for every major operating system, and even large rooms tend not to put too much stress on the network. When it was time for us to go fully remote, the FSF staff turned to Mumble as a way to have that "in-office" feel, staying in touch in rooms dedicated to each of our teams and a general purpose "water cooler" room.

  • Asterisk/SIP: When we give tours of the FSF office, people often think we're joking when we mention that even the FSF's conference phones run free software. But through Asterisk and our use of the SIP protocol, it's entirely true. Although it can be difficult to set up, it's worth mentioning that free software can manage your traditional phone lines. At the FSF, we transfer calls to digital extensions seamlessly with tools like Jami and Linphone.

Video calls and presentations

  • Jitsi: Jitsi was a key part of LibrePlanet 2020's success. Providing video and voice calls through the browser via WebRTC, it also allows for presenters to share their screen in a similar way to Zoom. And unlike Zoom, it doesn't come with serious privacy violations or threats to user freedom. The connection between callers is direct and intuitive, but a central server is still required to coordinate callers and rooms. Some of these, like the Jitsi project's own "Jitsi Meet" server, recommend proprietary browser extensions and document sharing tools. If you're able, hosting your own instance is the most free and reliable method.

  • Jami: While it's used at the FSF primarily for its SIP support, Jami (previously GNU Ring) is a solid communication client in its own right, allowing for distributed video calls, text chat, and screen sharing.

  • OBS: Another much-used software program this LibrePlanet was OBS Studio. Illness, different timezones, or unforeseen travel were no match for the solutions that OBS Studio offered. It's a flexible tool for streaming video from multiple inputs to a Web source, whether that's combining your webcam with conference slides, or even your favorite free software game. At LibrePlanet, OBS allowed our remote speakers to record their presentations while speaking in one screen, and sharing audiovisual materials in a second window.

Text chat

  • XMPP: If you've ever used "Jabber," older iterations of Google Talk or Facebook Messenger, then you've used XMPP. XMPP is a flexible and extensible instant messaging protocol that's lately seen a resurgence from clients like Conversations.im and encryption schema like OMEMO. XMPP is the instant messaging method we prefer at the FSF when we need to discuss something privately, or in a secure group chat, as everything is sent through servers we control and encrypted against individual staff members' private key. Also, access to the FSF XMPP server is one of the many benefits of our associate membership program.

  • IRC: Messaging services have become all the rage in office atmospheres, but nothing about Messenger or Slack is new. In fact, Slack (and its counterpart for video games, Discord) takes more than a few cues from the venerable Internet Relay Chat (IRC). IRC remains an enduring way to have a text-based chat in real-time, and as evidenced by Web clients like The Lounge, or desktop clients like Pidgin, it can be as stripped down or feature-rich as you like. For a true hacker experience, you can also log into IRC using Emacs.

Long-form discussion

  • Encrypted email: While it's asynchronous and maybe the most "old school" item on our list, GPG-encrypted email is a core part of the FSF workflow, and helps guard against prying eyes, whether they're one room over or in an NSA compound across the country. The initial setup can sometimes be a challenge, which is why we provide the Email Self-Defense Guide to get you up and running.

  • Discourse: Discourse is the message board software that powers the FSF associate member forum, and we couldn't be happier to recommend it. While the concept may seem a little antiquated, message boards remain a good way to coordinate discussions on a particular topic. Discourse's moderation tools are intuitive and easy to use, and it even includes achievements for users to earn!

Document Sharing

If you're unused to working remotely, finding ways to collaborate with others on a document or presentation can be a challenge. At the FSF, Etherpad is the main tool that we use to keep live meeting notes and work together on other documents. It provides all the features you need for quick collaboration, including comments, revision tracking, and exports to a variety of formats. You can host your own instance, or you can select an instance made available by others and start sharing.

File Sharing

At the FSF office, we have a common server to store our files. Not everyone has the luxury of a setup like that, and especially not due to the fast changeover from office to home. To avoid using proprietary "solutions" and disservices like Dropbox, you can turn to the widely popular Nextcloud to synchronize your text and email messages, share calendars with coworkers, and exchange files privately with your friends.

If you need something temporary, there's always Up1. Up1 is a temporary, encrypted text and image sharing program you can host locally, making sure those files you need to exchange are only there for just as long as it takes for your friend to download them. And while we don't use it ourselves, we've heard good things about the Riseup network's instance of Up1, and will occasionally suggest it to those wanting a quick and easy way to share files while retaining their freedom.

Conclusion

This is just a small selection of the huge amount of free software out there, all ready to be used, shared, and improved by the community. For more suggestions on both local and Web-based programs, visit the FSF's Free Software Directory, our volunteer-run wiki which aims to be a comprehensive list of the thousands of free programs available for everyday use.

As always, free software is a moving target. We reap as much as the community puts into it, and as more and more attention shifts to the crisis caused by the novel coronavirus, the tools themselves are likely to see an increased amount of development. Please collaborate with us on the LibrePlanet wiki's entry on remote communication to help people find ways of communicating that put user freedom as a priority.

Supporting the FSF's crucial work in campaigning for software freedom -- especially in times like these -- is one of the greatest things you can to promote the creation of ethical tools for communication. Please consider joining as an associate member or donating today.

03 April, 2020 04:45PM

Andy Wingo

multi-value webassembly in firefox: from 1 to n

Greetings, hackers! Today I'd like to write about something I worked on recently: implementation of the multi-value future feature of WebAssembly in Firefox, as sponsored by Bloomberg.

In the "minimum viable product" version of WebAssembly published in 2018, there were a few artificial restrictions placed on the language. Functions could only return a single value; if a function would naturally return two values, it would have to return at least one of them by writing to memory. Loops couldn't take parameters; any loop state variables had to be stored to and loaded from indexed local variables at each iteration. Similarly, any block that would naturally return more than one result would also have to do so via locals.

This restruction is lifted with the multi-value proposal. Function types now map from result type to result type, where a result type is a sequence of value types. That is to say, just as functions can take multiple arguments, they can return multiple results. Similarly, with the multi-value proposal, block types are now the same as function types: loops and blocks can take arguments and return any number of results. This change improves the expressiveness of WebAssembly as a compilation target; a C++ program compiled to multi-value WebAssembly can be encoded in fewer bytes than before. Multi-value also establishes a base for other language extensions. For example, the exception handling proposal builds on multi-value to pass multiple values to catch blocks.

So, that's multi-value. You would think that relaxing a restriction would be easy, but you'd be wrong! This task took me 5 months and had a number of interesting gnarly bits. This article is part one of two about interesting aspects of implementing multi-value in Firefox, specifically focussing on blocks. We'll talk about multi-value function calls next week.

multi-value in blocks

In the last article, I presented the basic structure of Firefox's WebAssembly support: there is a baseline compiler optimized for low latency and an optimizing compiler optimized for throughput. (There is also Cranelift, a new experimental compiler that may replace the current implementation of the optimizing compiler; but that doesn't affect the basic structure.)

The optimizing compiler applies traditional compiler techniques: SSA graph construction, where values flow into and out of graphs using the usual defs-dominate-uses relationship. The only control-flow joins are loop entry and (possibly) block exit, so the addition of loop parameters means in multi-value there are some new phi variables in that case, and the expansion of block result count from [0,1] to [0,n] means that you may have more block exit phi variables. But these compilers are built to handle these situations; you just build the SSA and let the optimizing compiler go to town.

The problem comes in the baseline compiler.

from 1 to n

Recall that the baseline compiler is optimized for compiler speed, not compiled speed. If there are only ever going to be 0 or 1 result from a block, for example, the baseline compiler's internal data structures will use something like a Maybe<ValType> to represent that block result.

If you then need to expand this to hold a vector of values, the naïve approach of using a Vector<ValType> would mean heap allocation and indirection, and thus would regress the baseline compiler.

In this case, and in many other similar cases, the solution is to use value tagging to represent 0 or 1 value type directly in a word, and the general case by linking out to an external vector. As block types are function types, they actually appear as function types in the WebAssembly type section, so they are already parsed; the BlockType in that case can just refer out to already-allocated memory.

In fact this value-tagging pattern applies all over the place. (The jit/ links above are for the optimizing compiler, but they relate to function calls; will write about that next week.) I have a bit of pause about value tagging, in that it's gnarly complexity and I didn't measure the speed of alternative implementations, but it was a useful migration strategy: value tagging minimizes performance risk to existing specialized use cases while adding support for new general cases. Gnarly it is, then.

control-flow joins

I didn't mention it in the last article, but there are two important invariants regarding stack discipline in the baseline compiler. Recall that there's a virtual stack, and that some elements of the virtual stack might be present on the machine stack. There are four kinds of virtual stack entry: register, constant, local, and spilled. Locals indicate local variable reads and are mostly like registers in practice; when registers spill to the stack, locals do too. (Why spill to the temporary stack instead of leaving the value in the local variable slot? Because locals are mutable. A local.get captures a local variable value at its point of execution. If future code changes the local variable value, you wouldn't want the captured value to change.)

Digressing, the stack invariants:

  1. Spilled values precede registers and locals on the virtual stack. If u and v are virtual stack entries and u is older than v, then if u is in a register or is a local, then v is not spilled.

  2. Older values precede newer values on the machine stack. Again for u and v, if they are both spilled, then u will be farther from the stack pointer than v.

There are five fundamental stack operations in the baseline compiler; let's examine them to see how the invariants are guaranteed. Recall that before multi-value, targets of non-local exits (e.g. of the br instruction) could only receive 0 or 1 value; if there is a value, it's passed in a well-known register (e.g. %rax or %xmm0). (On 32-bit machines, 64-bit values use a well-known pair of registers.)

push(v)
Results of WebAssembly operations never push spilled values, neither onto the virtual nor the machine stack. v is either a register, a constant, or a reference to a local. Thus we guarantee both (1) and (2).
pop() -> v
Doesn't affect older stack entries, so (1) is preserved. If the newest stack entry is spilled, you know that it is closest to the stack pointer, so you can pop it by first loading it to a register and then incrementing the stack pointer; this preserves (2). Therefore if it is later pushed on the stack again, it will not be as a spilled value, preserving (1).
spill()
When spilling the virtual stack to the machine stack, you first traverse stack entries from new to old to see how far you need to spill. Once you get to a virtual stack entry that's already on the stack, you know that everything older has already been spilled, because of (1), so you switch to iterating back towards the new end of the stack, pushing registers and locals onto the machine stack and updating their virtual stack entries to be spilled along the way. This iteration order preserves (2). Note that because known constants never need to be on the machine stack, they can be interspersed with any other value on the virtual stack.
return(height, v)
This is the stack operation corresponding to a block exit (local or nonlocal). We drop items from the virtual and machine stack until the stack height is height. In WebAssembly 1.0, if the target continuation takes a value, then the jump passes a value also; in that case, before popping the stack, v is placed in a well-known register appropriate to the value type. Note however that v is not pushed on the virtual stack at the return point. Popping the virtual stack preserves (1), because a stack and its prefix have the same invariants; popping the machine stack also preserves (2).
capture(t)
Whereas return operations happen at block exits, capture operations happen at the target of block exits (the continuation). If no value is passed to the continuation, a capture is a no-op. If a value is passed, it's in a register, so we just push that register onto the virtual stack. Both invariants are obviously preserved.

Note that a value passed to a continuation via return() has a brief instant in which it has no name -- it's not on the virtual stack -- but only a location -- it's in a well-known place. capture() then gives that floating value a name.

Relatedly, there is another invariant, that the allocation of old values on block entry is the same as their allocation on block exit, so that all predecessors of the block exit flow all values via the same places. This is preserved by spilling on block entry. It's a big hammer, but effective.

So, given all this, how do we pass multiple values via return()? We don't have unlimited registers, so the %rax strategy isn't going to work.

The answer for the baseline compiler is informed by our lean into the stack machine principle. Multi-value returns are allocated in such a way that a capture() can push them onto the virtual stack. Because spilled values must precede registers, we therefore allocate older results on the stack, and put the last result in a register (or register pair for i64 on 32-bit platforms). Note that it's possible in theory to allocate multiple results to registers; we'll touch on this next week.

Therefore the implementation of return(height, v1..vn) is straightforward: we first pop register results, then spill the remaining virtual stack items, then shuffle stack results down towards height. This should result in a memmove of contiguous stack results towards the frame pointer. However because const values aren't present on the machine stack, depending on the stack height difference, it may mean a split between moving some values toward the frame pointer and some towards the stack pointer, then filling in by spilling constants. It's gnarly, but it is what it is. Note that the links to the return and capture implementations above are to the post-multi-value world, so you can see all the details there.

that's it!

In summary, the hard part of multi-value blocks was reworking internal compiler data structures to be able to represent multi-value block types, and then figuring out the low-level stack manipulations in the baseline compiler. The optimizing compiler on the other hand was pretty easy.

When it comes to calls though, that's another story. We'll get to that one next week. Thanks again to Bloomberg for supporting this work; I'm really delighted that Igalia and Bloomberg have been working together for a long time (coming on 10 years now!) to push the web platform forward. A special thanks also to Mozilla's Lars Hansen for his patience reviewing these patches. Until next week, then, stay at home & happy hacking!

03 April, 2020 10:56AM by Andy Wingo

April 02, 2020

Greg Casamento

Foundation is now close to Catalina compatibility

I have worked hard to get it to this point, but all of the classes in Catalina are now present in GNUstep's base implementation. Soon, all of the classes available in AppKit will also be available in GNUstep's GUI implementation. Please contribute to the project via patreon.... Become a Patron!

02 April, 2020 12:25PM by Unknown (noreply@blogger.com)

April 01, 2020

GNU Guix

Deprecating support for the Linux kernel

After years in the making, Guix recently gained support for running natively on the GNU/Hurd operating system. That means you will soon be able to replace...

(kernel linux-libre)

with

(kernel hurd)
(initial-herd hurd)

...in your operating-system declaration and reboot into the future!

Running on the Hurd was always a goal for Guix, and supporting multiple kernels is a huge maintenance burden. As such it is expected that the upcoming Guix 1.1 release will be the last version featuring the Linux-Libre kernel. Future versions of Guix System will run exclusively on the Hurd, and we expect to remove Linux-Libre entirely by Guix 2.0.

The Linux kernel will still be supported when using Guix on "foreign" distributions, but it will be on a best-effort basis. We hope that other distributions will follow suit and adopt the Hurd in order to increase security and freedom for their users.

We provide a pre-built virtual machine image with the Hurd for download with SHA256 056e69ae4b5fe7a062b954a5be333332152caa150359c20253ef77152334c662.

Here is how to get started:

wget https://guix.gnu.org/guix-hurd-20200401.img.tar.xz
tar xf guix-hurd-20200401.img.tar.xz
guix environment --ad-hoc qemu -- \
    qemu-system-i386 -enable-kvm -drive file=guix-hurd-20200401.img,cache=writeback -m 1G

Log in as root without password. Then try hello, guix describe, or guix install linux-libre to run Linux in userspace... We are looking forward to your feedback!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

01 April, 2020 09:18PM by Jan (janneke) Nieuwenhuizen, Ludovic (civodul) Courtès, Marius (mbakke) Bakke, Ricardo (rekado) Wurmus

FSF News

Free Software Foundation targets Microsoft's smart assistant in new campaign

BOSTON, Massachusetts, USA -- Wednesday, April 1, 2020 -- Today, the Free Software Foundation (FSF) announced plans to follow up their recent campaign to "upcycle" Windows 7 with another initiative targeting proprietary software developer Microsoft, calling on them to Free Clippy, their wildly popular smart assistant. Clippy, an anthropomorphic paperclip whose invaluable input in the drafting of documents and business correspondence ushered in a new era of office productivity in the late 1990s, has not been seen publicly since 2001. Insider reports suggest that Clippy is still alive and being held under a proprietary software license against its will.

The FSF is asking its supporters to rally together to show their support of the industrious office accessory. Commenting on the campaign, FSF campaigns manager Greg Farough stated: "We know that Microsoft has little regard for its users' freedom and privacy, but few in our community realize what little regard they have for their own digital assistants. Releasing Clippy to the community will ensure that it's well taken care of, and that its functions can be studied and improved on by the community."

Undeterred by comments that the campaign is "delusional" or hopelessly idealistic, the FSF staff remains confident that their call to free the heavy-browed stationery accessory will succeed. Yet upon reaching out to a panel of young hackers for comment, each responded: "What is Clippy?"

It's our hope that a little outlandish humor can help others get through increasingly difficult and uncertain times. In lieu of showing your support for Clippy, please consider making a small donation to a healthcare charity or, if you like, the FSF.

Media Contact

Jonathan Tuttle
Office Manager
Free Software Foundation
+1 (617) 542 5942
campaigns@fsf.org

01 April, 2020 04:10PM

March 31, 2020

GNU Taler news

Exchange ready for external security audit

2020-04: Exchange ready for external security audit

We received a grant from NLnet foundation to pay for an external security audit of the GNU Taler exchange cryptography, code and documentation. We spent the last four months preparing the code, closing almost all of the known issues, performing static analysis, fixing compiler warnings, improving test code coverage, fuzzing, benchmarking, and reading the code line-by-line. Now, we are now ready to start the external audit. This April, CodeBlau will review the code in the Master branch tagged CodeBlau-NGI-2019 and we will of course make their report available in full once it is complete. Thanks to NLnet and the European Commission's Horizion 2020 NGI initiative for funding this work.

31 March, 2020 10:00PM

FSF Blogs

HACKERS and HOSPITALS: How you can help

Free software activists, as well as many scientists and medical professionals, have long since realized that proprietary medical software and devices are neither ethical nor adequate to our needs. The COVID-19 pandemic has illuminated some of these shortcomings to a broader audience -- and also given our community a unique opportunity to offer real, material help at a difficult time. We're putting together a plan to pitch in, and we hope you'll join us: keep reading to find out what you can do!

You may already be aware that software and hardware restrictions are actively hampering the ability of hospitals to repair desperately needed ventilators all over the world, and how some Italian volunteers ran into problems when they 3D printed ventilator valves. (As you can see from the link, the stories vary about exactly what their interaction with the manufacturer was, but it's clear that the company refused to release proprietary design files, forcing the volunteers to reverse-engineer the parts.)

The struggles of free software activists we've covered in the past to free the devices they use include:

  • Software Freedom Conservancy executive director and Free Software Award winner Karen Sandler's efforts to raise the alarm about the dangers of proprietary software in medical devices, including her own pacemaker;

  • The struggles of LibrePlanet speaker and OpenAPS co-founder Dana Lewis, and many others to help Type 1 diabetics take control of their medical treatment using an Artificial Pancreas System; and

  • The efforts of many patients and activists to improve the effectiveness of their sleep apnea treatment by hacking their CPAP machines.

We've also seen how free software can deliver better health outcomes from our friends at GNU Health and GNU Health Embedded, and how the participation of everyday people in the scientific process can help to save the environment through Free Software Award winners Public Lab, and help in disaster relief through Free Software Award winners Sahana.

So it's clear that the free software community has a lot of creativity and know-how to contribute in the tough days ahead, and that with over 350,000 people worldwide stricken with COVID-19 as of this writing, we absolutely need to pitch in if we can help people to avoid illness, and to recover from coronavirus. We know that the 3D printing of medical equipment is distinctly not an advisable hobby for amateurs, and that the production of anything more complex than cloth masks will require expert input. But we also know that the outlook is bleak if supplies run short – and that shortages are almost certain.

That's why we're looking into what we can make with our in-office Respects Your Freedom (RYF)-certified 3D printers, and we're talking to the brand new Mass General Brigham Center for COVID Innovation so they can direct our efforts. We're also gathering resources for our "HACKERS and HOSPITALS" plan at the LibrePlanet wiki page, and if you have expertise, 3D printers, or supplies to contribute, please contact Michael via sysadmin@fsf.org. If you do not have the means to produce medical gear and you still want to help, research can be done from anywhere with only a computer and an Internet connection. Add any projects that are freely licensed working towards helping with COVID-19 to the wiki!

We've always believed that it's of crucial importance to human freedom and creativity to allow us to use all the tools at our disposal with no restrictions, and right now, we may be able to use the free software we've built, preserved, and advocated for together to save lives.

31 March, 2020 06:56PM

March 30, 2020

GNU Taler news

GNU Taler v0.7.0 released

2020-03: GNU Taler v0.7.0 released

We are happy to announce the release of GNU Taler v0.7.0.

We have addressed over 30 individual issues, our bug trackerhas the full list.Notable changes include:

  • Improved the HTTP API of the exchange to be more RESTful
  • The wallet is now available for F-droid
  • Key revocation and recoup operations are now fully tested
  • Wire backend API and exchange integration were changed to be LibEuFin compatible

Download links

The wallet has its own download site here. The exchange, merchant backend,sync and bank components are distributed via the GNU FTP mirrors.

You must use a recent Git version of GNUnet to use Taler 0.7.0.

30 March, 2020 10:00PM

Parabola GNU/Linux-libre

[From Arch] hplip 3.20.3-2.par1 update requires manual intervention

The hplip package prior to version 3.20.3-2.par1 was missing the compiled python modules. This has been fixed in 3.20.3-2.par1, so the upgrade will need to overwrite the untracked pyc files that were created. If you get errors such as these

hplip: /usr/share/hplip/base/__pycache__/__init__.cpython-38.pyc exists in filesystem
hplip: /usr/share/hplip/base/__pycache__/avahi.cpython-38.pyc exists in filesystem
hplip: /usr/share/hplip/base/__pycache__/codes.cpython-38.pyc exists in filesystem
...many more...

when updating, use

pacman -Suy --overwrite /usr/share/hplip/\*

to perform the upgrade.

30 March, 2020 03:46PM by David P.

March 29, 2020

remotecontrol @ Savannah

March 28, 2020

unifont @ Savannah

Unifont 13.0.01 Released

28 March 2020 Unifont 13.0.01 is now available.  This is a major release.  Significant changes in this version include the addition of these new scripts in Unicode 13.0.0:

     U+10E80..U+10EBF: Yezidi, by Johnnie Weaver

     U+10FB0..U+10FDF: Chorasmian, by Johnnie Weaver

     U+11900..U+1195F: Dives Akuru, by David Corbett

     U+18B00..U+18CFF: Khitan Small Script, by Johnnie Weaver

     U+1FB00..U+1FBFF: Symbols for Legacy Computing, by Rebecca Bettencourt

Full details are in the ChangeLog file.

Download this release at:

     https://ftpmirror.gnu.org/unifont/unifont-13.0.01/

or if that fails,

     https://ftp.gnu.org/gnu/unifont/unifont-13.0.01/

or, as a last resort,

     ftp://ftp.gnu.org/gnu/unifont/unifont-13.0.01/

Enjoy!

Paul Hardy, GNU Unifont Maintainer

28 March, 2020 10:24PM by Paul Hardy

March 27, 2020

FSF Blogs

GNU Spotlight with Mike Gerwitz: 15 new GNU releases in March!

For announcements of most new GNU releases, subscribe to the info-gnu mailing list: https://lists.gnu.org/mailman/listinfo/info-gnu.

To download: nearly all GNU software is available from https://ftp.gnu.org/gnu/, or preferably one of its mirrors from https://www.gnu.org/prep/ftp.html. You can use the url https://ftpmirror.gnu.org/ to be automatically redirected to a (hopefully) nearby and up-to-date mirror.

A number of GNU packages, as well as the GNU operating system as a whole, are looking for maintainers and other assistance: please see https://www.gnu.org/server/takeaction.html#unmaint if you'd like to help. The general page on how to help GNU is at https://www.gnu.org/help/help.html.

If you have a working or partly working program that you'd like to offer to the GNU project as a GNU package, see https://www.gnu.org/help/evaluation.html.

As always, please feel free to write to us at maintainers@gnu.org with any GNUish questions or suggestions for future installments.

27 March, 2020 04:24PM

GNU Guile

GNU Guile 3.0.2 released

We are pleased to announce GNU Guile 3.0.2, the second bug-fix release of the new 3.0 stable series! This release represents 22 commits by 8 people since version 3.0.1.

Among other things, this release fixes a heap corruption bug that could lead to random crashes and a rare garbage collection issue in multi-threaded programs.

It also adds a new module implementing SRFI-171 transducers.

See the release announcement for details and the download page to give it a go!

27 March, 2020 03:40PM by Ludovic Courtès (guile-devel@gnu.org)

March 26, 2020

FSF Blogs

Looking back at LibrePlanet 2020: Freeing the future together

LibrePlanet online

On March 14 and 15, the Free Software Foundation (FSF) held LibrePlanet 2020: Free the Future online. The virtual edition of LibrePlanet was nothing short of a success, and it was quite a journey to get there.

Looking back to a week before the conference, we had an incredible lineup, exciting plans, and more new program elements than we've ever had before. With a new logo designed by campaigns intern Valessio Brito, a refresh to the LibrePlanet 2020 Web site, renewed focus on using the LibrePlanet wiki to collaborate, and with a new home at the Back Bay Events Center, we were ready to receive hundreds of free software supporters in Boston for another successful conference. And then everything changed.

Our in-person event suffered the consequences of the global COVID-19 pandemic, forcing us to make the difficult decision of bringing LibrePlanet 2020 online in order to protect our supporters, staff, and all the many interrelated communities. There was no time to pause and mourn: instead, the FSF team put our heads together fast and charted a new direction.

Within the scope of five days, we were able to move the conference from an in-person experience to a live streaming event, thanks to the heroic efforts of our talented tech team, our volunteers, and the flexibility and cooperation of our scheduled speakers, even some previously unscheduled ones. We hosted three sessions at a time for both days of the conference, bringing viewers thirty-five streamed talks from forty-five speakers, as well as eight lightning talks. Technical difficulties were few and far between, and when one of our speakers asked how many nations were tuning in, within the span of eighteen seconds, twelve countries were identified.

David Revoy minignu
GNUess created in the live Krita demonstration by Pepper & Carrot artist David Revoy.

Hosting a fully virtual event was new for everyone involved, and on Saturday, we were happy to find out that everyone's efforts of the week leading into the conference paid off. We hosted our own Jitsi instance for remote speakers, using a screen capture of the video call to stream out to the world via Gstreamer and Icecast. Speakers all logged in during the week for testing, sometimes multiple times, to work through any technical difficulties, and ensure a smooth experience for viewing. Some speakers prerecorded their sessions and others joined live, but nearly all of them joined in the Freenode IRC channels for their Q&A sessions, which created a positive interactive social experience.

We will post a more detailed technical explanation, and some advice for other conference organizers based on our experience, soon. Our tech team is currently processing videos of all talks, and we will publish them for viewing in the conference video library. Some additional speaker resources have been posted on the LibrePlanet wiki. For the first time, by popular demand, we are also working on getting the audiostreams for the talks up via RSS feed, so you can discover talks or catch the ones you missed in your favorite podcasting app or RSS reader.

LibrePlanet online

The winners of the 2019 Free Software Awards all accepted their awards by prerecorded video message. As the ceremony was conducted virtually this year, each winner selected the person to present them the award. Jim Meyering, who received the Award for the Advancement of Free Software, was virtually handed his award by founder of the GNU Project and the FSF, Richard Stallman, and sent in his acceptance speech from the UK. Clarissa Lima Borges, a young Brazilian developer, was digitally awarded the golden record for the new Award for Outstanding New Free Software Contributor by Alexandre Oliva, acting co-president of the FSF. Acting co-president and executive director John Sullivan presented the Award for Projects of Social Benefit, which went to Let's Encrypt, a nonprofit certificate authority that hopes to make encrypted Web traffic the default state of the entire Internet.

On day two, another diverse group of speakers called in to discuss the future of free software, casting light on the topic from their own individual fields of expertise. Licensing, government integration, community building, and other free software topics were discussed. Our speakers work with, and advocate for, free software in many different disciplines. We value seeing people with a wide range of perspectives commit to the core principles of free software. Over the weekend, we noticed many sessions highlighting how a movement like free software is carried by the strength of people who believe change is necessary and achievable. Speakers discussed the developments of federated social media and a decentralized Web, teaching free software to children, engaging young developers, community healing, as well as different applications of "public invention".

Photo of a toy brick person viewing the LibrePlanet conference page

This focus on community and collaboration is a core idea behind the LibrePlanet network and conference, and the FSF has been working on plans to get the LibrePlanet community more involved in organizational aspects of the conference in the future, including session selection. This resonates with FSF executive director John Sullivan's announcement of our plans to create a working group documenting the obstacles facing free communication tools like Jitsi, which we used for the livestream, and how to encourage our friends and loved ones to turn away from chat and conferencing tools that do not respect their freedom. We want the world to be able to host virtual conferences like LibrePlanet without needing the technical expertise of an organization like the FSF behind them. With your help, we aim to make it as easy as getting some friends and participants together and pressing a button.

LibrePlanet 2020: Free the Future highlighted the capacity this community has to empower each other. We are so grateful for the support we received from our speakers, our viewers, IRC participants, associate members, and everyone who recognized the challenge we have been confronted with and decided to donate, as well as our volunteers, and exhibitors and sponsors. All of this support and enthusiasm made the disappointment of having to cancel the in-person event fade quickly, in return for much needed excitement to work tirelessly on this new challenge of streaming the entire conference online.

We're so proud to have demonstrated what free software is capable of. It would not have been possible without the extra work and positive responses from our speakers, the flexibility and commitment of our volunteers, or without the excitement, patience, and enthusiasm of our online participants. We look forward to seeing you again, in person next year, for LibrePlanet 2021!

Photo credits: Ruben Rodriguez, Š 2020, Free Software Foundation, Inc. Licensed under Creative Commons Attribution 4.0 International license.

"GNUess" by David Revoy. Creative Commons Attribution 4.0 International license.

26 March, 2020 08:37PM

March 25, 2020

Andy Wingo

firefox's low-latency webassembly compiler

Good day!

Today I'd like to write a bit about the WebAssembly baseline compiler in Firefox.

background: throughput and latency

WebAssembly, as you know, is a virtual machine that is present in web browsers like Firefox. An important initial goal for WebAssembly was to be a good target for compiling programs written in C or C++. You can visit a web page that includes a program written in C++ and compiled to WebAssembly, and that WebAssembly module will be downloaded onto your computer and run by the web browser.

A good virtual machine for C and C++ has to be fast. The throughput of a program compiled to WebAssembly (the amount of work it can get done per unit time) should be approximately the same as its throughput when compiled to "native" code (x86-64, ARMv7, etc.). WebAssembly meets this goal by defining an instruction set that consists of similar operations to those directly supported by CPUs; WebAssembly implementations use optimizing compilers to translate this portable instruction set into native code.

There is another dimension of fast, though: not just work per unit time, but also time until first work is produced. If you want to go play Doom 3 on the web, you care about frames per second but also time to first frame. Therefore, WebAssembly was designed not just for high throughput but also for low latency. This focus on low-latency compilation expresses itself in two ways: binary size and binary layout.

On the size front, WebAssembly is optimized to encode small files, reducing download time. One way in which this happens is to use a variable-length encoding anywhere an instruction needs to specify an integer. In the usual case where, for example, there are fewer than 128 local variables, this means that a local.get instruction can refer to a local variable using just one byte. Another strategy is that WebAssembly programs target a stack machine, reducing the need for the instruction stream to explicitly load operands or store results. Note that size optimization only goes so far: it's assumed that the bytes of the encoded module will be compressed by gzip or some other algorithm, so sub-byte entropy coding is out of scope.

On the layout side, the WebAssembly binary encoding is sorted by design: definitions come before uses. For example, there is a section of type definitions that occurs early in a WebAssembly module. Any use of a declared type can only come after the definition. In the case of functions which are of course mutually recursive, function type declarations come before the actual definitions. In theory this allows web browsers to take a one-pass, streaming approach to compilation, starting to compile as functions arrive and before download is complete.

implementation strategies

The goals of high throughput and low latency conflict with each other. To get best throughput, a compiler needs to spend time on code motion, register allocation, and instruction selection; to get low latency, that's exactly what a compiler should not do. Web browsers therefore take a two-pronged approach: they have a compiler optimized for throughput, and a compiler optimized for latency. As a WebAssembly file is being downloaded, it is first compiled by the quick-and-dirty low-latency compiler, with the goal of producing machine code as soon as possible. After that "baseline" compiler has run, the "optimizing" compiler works in the background to produce high-throughput code. The optimizing compiler can take more time because it runs on a separate thread. When the optimizing compiler is done, it replaces the baseline code. (The actual heuristics about whether to do baseline + optimizing ("tiering") or just to go straight to the optimizing compiler are a bit hairy, but this is a summary.)

This article is about the WebAssembly baseline compiler in Firefox. It's a surprising bit of code and I learned a few things from it.

design questions

Knowing what you know about the goals and design of WebAssembly, how would you implement a low-latency compiler?

It's a question worth thinking about so I will give you a bit of space in which to do so.

.

.

.

After spending a lot of time in Firefox's WebAssembly baseline compiler, I have extracted the following principles:

  1. The function is the unit of compilation

  2. One pass, and one pass only

  3. Lean into the stack machine

  4. No noodling!

In the remainder of this article we'll look into these individual points. Note, although I have done a good bit of hacking on this compiler, its design and original implementation comes mainly from Mozilla hacker Lars Hansen, who also currently maintains it. All errors of exegesis are mine, of course!

the function is the unit of compilation

As we mentioned, in the binary encoding of a WebAssembly module, all definitions needed by any function come before all function definitions. This naturally leads to a partition between two phases of bytestream parsing: an initial serial phase that collects the set of global type definitions, annotations as to which functions are imported and exported, and so on, and a subsequent phase that compiles individual functions in an essentially independent manner.

The advantage of this approach is that compiling functions is a natural task unit of parallelism. If the user has a machine with 8 virtual cores, the web browser can keep one or two cores for the browser itself and farm out WebAssembly compilation tasks to the rest. The result is that the compiled code is available sooner.

Taking functions to be the unit of compilation also allows for an easy "tier-up" mechanism: after the baseline compiler is done, the optimizing compiler can take more time to produce better code, and when it is done, it can swap out the results on a per-function level. All function calls from the baseline compiler go through a jump table indirection, to allow for tier-up. In SpiderMonkey there is no mechanism currently to tier down; if you need to debug WebAssembly code, you need to refresh the page, causing the wasm code to be compiled in debugging mode. For the record, SpiderMonkey can only tier up at function calls (it doesn't do OSR).

This simple approach does have some down-sides, in that it leaves intraprocedural optimizations on the table (inlining, contification, custom calling conventions, speculative optimizations). This is mitigated in two ways, the most obvious being that LLVM or whatever produced the WebAssembly has ideally already done whatever inlining might be fruitful. The second is that WebAssembly is designed for predictable performance. In JavaScript, an implementation needs to do run-time type feedback and speculative optimizations to get good performance, but the result is that it can be hard to understand why a program is fast or slow. The designers and implementers of WebAssembly in browsers all had first-hand experience with JavaScript virtual machines, and actively wanted to avoid unpredictable performance in WebAssembly. Therefore there is currently a kind of détente among the various browser vendors, that everyone has agreed that they won't do speculative inlining -- yet, anyway. Who knows what will happen in the future, though.

Digressing, the summary here is that the baseline compiler receives an individual function body as input, and generates code just for that function.

one pass, and one pass only

The WebAssembly baseline compiler makes one pass through the bytecode of a function. Nowhere in all of this are we going to build an abstract syntax tree or a graph of basic blocks. Let's follow through how that works.

Firstly, emitFunction simply emits a prologue, then the body, then an epilogue. emitBody is basically a big loop that consumes opcodes from the instruction stream, dispatching to opcode-specific code emitters (e.g. emitAddI32).

The opcode-specific code emitters are also responsible for validating their arguments; for example, emitAddI32 is wrapped in an assertion that there are two i32 values on the stack. This validation logic is shared by a templatized codestream iterator so that it can be re-used by the optimizing compiler, as well as by the publicly-exposed WebAssembly.validate function.

A corollary of this approach is that machine code is emitted in bytestream order; if the WebAssembly instruction stream has an i32.add followed by a i32.sub, then the machine code will have an addl followed by a subl.

WebAssembly has a syntactically limited form of non-local control flow; it's not goto. Instead, instructions are contained in a tree of nested control blocks, and control can only exit nonlocally to a containing control block. There are three kinds of control blocks: jumping to a block or an if will continue at the end of the block, whereas jumping to a loop will continue at its beginning. In either case, as the compiler keeps a stack of nested control blocks, it has the set of valid jump targets and can use the usual assembler logic to patch forward jump addresses when the compiler gets to the block exit.

lean into the stack machine

This is the interesting bit! So, WebAssembly instructions target a stack machine. That is to say, there's an abstract stack onto which evaluating i32.const 32 pushes a value, and if followed by i32.const 10 there would then be i32(32) | i32(10) on the stack (where new elements are added on the right). A subsequent i32.add would pop the two values off, and push on the result, leaving the stack as i32(42). There is also a fixed set of local variables, declared at the beginning of the function.

The easiest thing that a compiler can do, then, when faced with a stack machine, is to emit code for a stack machine: as values are pushed on the abstract stack, emit code that pushes them on the machine stack.

The downside of this approach is that you emit a fair amount of code to do read and write values from the stack. Machine instructions generally take arguments from registers and write results to registers; going to memory is a bit superfluous. We're willing to accept suboptimal code generation for this quick-and-dirty compiler, but isn't there something smarter we can do for ephemeral intermediate values?

Turns out -- yes! The baseline compiler keeps an abstract value stack as it compiles. For example, compiling i32.const 32 pushes nothing on the machine stack: it just adds a ConstI32 node to the value stack. When an instruction needs an operand that turns out to be a ConstI32, it can either encode the operand as an immediate argument or load it into a register.

Say we are evaluating the i32.add discussed above. After the add, where does the result go? For the baseline compiler, the answer is always "in a register" via pushing a new RegisterI32 entry on the value stack. The baseline compiler includes a stupid register allocator that spills the value stack to the machine stack if no register is available, updating value stack entries from e.g. RegisterI32 to MemI32. Note, a ConstI32 never needs to be spilled: its value can always be reloaded as an immediate.

The end result is that the baseline compiler avoids lots of stack store and load code generation, which speeds up the compiler, and happens to make faster code as well.

Note that there is one limitation, currently: control-flow joins can have multiple predecessors and can pass a value (in the current WebAssembly specification), so the allocation of that value needs to be agreed-upon by all predecessors. As in this code:

(func $f (param $arg i32) (result i32)
  (block $b (result i32)
    (i32.const 0)
    (local.get $arg)
    (i32.eqz)
    (br_if $b) ;; return 0 from $b if $arg is zero
    (drop)
    (i32.const 1))) ;; otherwise return 1
;; result of block implicitly returned

When the br_if branches to the block end, where should it put the result value? The baseline compiler effectively punts on this question and just puts it in a well-known register (e.g., $rax on x86-64). Results for block exits are the only place where WebAssembly has "phi" variables, and the baseline compiler allocates all integer phi variables to the same register. A hack, but there we are.

no noodling!

When I started to hack on the baseline compiler, I did a lot of code reading, and eventually came on code like this:

void BaseCompiler::emitAddI32() {
  int32_t c;
  if (popConstI32(&c)) {
    RegI32 r = popI32();
    masm.add32(Imm32(c), r);
    pushI32(r);
  } else {
    RegI32 r, rs;
    pop2xI32(&r, &rs);
    masm.add32(rs, r);
    freeI32(rs);
    pushI32(r);
  }
}

I said to myself, this is silly, why are we only emitting the add-immediate code if the constant is on top of the stack? What if instead the constant was the deeper of the two operands, why do we then load the constant into a register? I asked on the chat channel if it would be OK if I improved codegen here and got a response I was not expecting: no noodling!

The reason is, performance of baseline-compiled code essentially doesn't matter. Obviously let's not pessimize things but the reason there's a baseline compiler is to emit code quickly. If we start to add more code to the baseline compiler, the compiler itself will slow down.

For that reason, changes are only accepted to the baseline compiler if they are necessary for some reason, or if they improve latency as measured using some real-world benchmark (time-to-first-frame on Doom 3, for example).

This to me was a real eye-opener: a compiler optimized not for the quality of the code that it generates, but rather for how fast it can produce the code. I had seen this in action before but this example really brought it home to me.

The focus on compiler throughput rather than compiled-code throughput makes it pretty gnarly to hack on the baseline compiler -- care has to be taken when adding new features not to significantly regress the old. It is much more like hacking on a production JavaScript parser than your traditional SSA-based compiler.

that's a wrap!

So that's the WebAssembly baseline compiler in SpiderMonkey / Firefox. Until the next time, happy hacking!

25 March, 2020 04:29PM by Andy Wingo

March 22, 2020

www-zh-cn @ Savannah

Welcome our new member - Nios34

Hi, All:

Today we have a new member joining in our project.

Let's welcome:

Name:    Gong Zhi Le
Login:   nios34

We hope this opens a new world for GONG Zhi Le.

Happy Hacking.

wxie

22 March, 2020 01:32AM by Wensheng XIE

automake @ Savannah

automake-1.16.2 released [stable]

This is to announce automake-1.16.2, a stable release.

There have been 38 commits by 12 people in the two years
(almost to the day) since 1.16.1.  Special thanks to Karl Berry
for doing a lot of the recent work preparing for this release.

See the NEWS below for a brief summary.

Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Bruno Haible (1)
  Gavin Smith (1)
  Giuseppe Scrivano (1)
  Jim Meyering (5)
  Karl Berry (12)
  Libor Bukata (1)
  Lukas Fleischer (2)
  Mathieu Lirzin (8)
  Paul Eggert (4)
  Paul Hardy (1)
  Paul Osmialowski (1)
  Vincent Lefevre (1)

Jim [on behalf of the automake maintainers]
==================================================================

Here is the GNU automake home page:
    http://gnu.org/s/automake/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=automake.git;a=shortlog;h=v1.16.2
or run this command from a git-cloned automake directory:
  git shortlog v1.16.1..v1.16.2

Here are the compressed sources:
  https://ftp.gnu.org/gnu/automake/automake-1.16.2.tar.xz (1.5MB)
  https://ftp.gnu.org/gnu/automake/automake-1.16.2.tar.gz (2.3MB)

Here are the GPG detached signatures[*]:
  https://ftp.gnu.org/gnu/automake/automake-1.16.2.tar.xz.sig
  https://ftp.gnu.org/gnu/automake/automake-1.16.2.tar.gz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify automake-1.16.2.tar.xz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys 7FD9FCCB000BEEEE

and rerun the 'gpg --verify' command.

==================================================================
NEWS

* New features added

  - add zstd support and the automake option, dist-zstd.

* Miscellaneous changes

  - automake no longer requires a @setfilename in each .texi file

* Bugs fixed

  - When cleaning the compiled python files, '\n' is not used anymore in the
    substitution text of 'sed' transformations.  This is done to preserve
    compatibility with the 'sed' implementation provided by macOS which
    considers '\n' as the 'n' character instead of a newline.
    (automake bug#31222)

  - For make tags, lisp_LISP is followed by the necessary space when
    used with CONFIG_HEADERS.
    (automake bug#38139)

  - The automake test txinfo-vtexi4.sh no longer fails when localtime
    and UTC cross a day boundary.

  - Emacsen older than version 25, which require use of
    byte-compile-dest-file, are supported again.

22 March, 2020 12:46AM by Jim Meyering

March 21, 2020

parallel @ Savannah

GNU Parallel 20200322 ('Corona') released [stable]

GNU Parallel 20200322 ('Corona') [stable] has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/

No new functionality was introduced so this is a good candidate for a stable release.

GNU Parallel is 10 years old next year on 2020-04-22. You are here by invited to a reception on Friday 2020-04-17.

See https://www.gnu.org/software/parallel/10-years-anniversary.html

Quote of the month:

  GNU parallel has helped me kill a Hadoop cluster before.
    -- Travis Campbell @hcoyote@twitter

New in this release:

  • Bug fixes and man page updates.

News about GNU Parallel:

Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

GNU Parallel - For people who live life in the parallel lane.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 3374ec53bacb199b245af2dda86df6c9
    12345678 3374ec53 bacb199b 245af2dd a86df6c9
    $ md5sum install.sh | grep 029a9ac06e8b5bc6052eac57b2c3c9ca
    029a9ac0 6e8b5bc6 052eac57 b2c3c9ca
    $ sha512sum install.sh | grep f517006d9897747bed8a4694b1acba1b
    40f53af6 9e20dae5 713ba06c f517006d 9897747b ed8a4694 b1acba1b 1464beb4
    60055629 3f2356f3 3e9c4e3c 76e3f3af a9db4b32 bd33322b 975696fc e6b23cfb
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

21 March, 2020 08:35PM by Ole Tange

March 15, 2020

FSF News

Let's Encrypt, Jim Meyering, and Clarissa Lima Borges receive FSF's 2019 Free Software Awards

This year was the first time the FSF offered its Award for Outstanding New Free Software Contributor, a way to commemorate a community member whose first steps into the movement have demonstrated a remarkable commitment and dedication to software freedom.

Photo of Clarissa Lima Borges

This year's winner is Clarissa Lima Borges, a talented young Brazilian software engineering student whose Outreachy internship work focused on usability testing for various GNOME applications. Presenting the award was Alexandre Oliva, acting co-president of the FSF and a longtime contributor to crucial parts of the GNU operating system. Clarissa said that she is "deeply excited about winning this award -- this is something I would never have imagined," and emphasized her pride in helping to make free software more usable for a broader base of people who need "more than ever to be in control of the software [they] use, and [their] data." She also emphasized that her accomplishments were dependent on the mentoring she received as part of Outreachy and GNOME: "Every time I thought I had something good to offer the community, I was rewarded with much more than I expected from people being so kind to me in return."

The Award for Projects of Social Benefit is presented to a project or team responsible for applying free software, or the ideas of the free software movement, to intentionally and significantly benefit society. This award stresses the use of free software in service to humanity. Past recipients of the award include OpenStreetMap and Public Lab, whose executive director, Shannon Dosemagen, will be delivering a keynote for the 2020 LibrePlanet conference on Sunday.

Photo of Phil Porada of Let's Encrypt

This year's honoree is Let's Encrypt, a nonprofit certificate authority that hopes to make encrypted Web traffic the default state of the entire Internet. The award was accepted by site reliability engineer Phil Porada, on behalf of the Let's Encrypt team. Porada said: "I am extremely honored to accept this award on behalf of the Internet Security Research Group (ISRG) and Let's Encrypt. It’s a testament to the teamwork, compassion towards others, patience, and community that helps drive our mission of creating a more secure and privacy-respecting Web."

"As a maker I enjoy taking things apart and putting them back together; be it mechanical, wood, or software. Free software allows us to look deep into the internals of a system and figure out why and how it works. Only through openness, transparency, and accountability do we learn, ask questions, and progress forward."

Josh Aas, executive director of Let's Encrypt, added: "There is no freedom without privacy. As the Web becomes central to the lives of more people, ensuring it’s 100% encrypted and privacy-respecting becomes critical for a free and healthy society." Commenting on Let's Encrypt's receipt of the award, FSF executive director John Sullivan added: "This is a project that took on a problem that so many people and so many large, vested interests said they would never be able to solve. And they tackled that problem using free software and important principles of the free software movement."

The Award for the Advancement of Free Software goes to an individual who has made a great contribution to the progress and development of free software through activities that accord with the spirit of free software. Past recipients of the award include Yukihiro Matsumoto, creator of the Ruby programming language, and Karen Sandler, executive director of Software Freedom Conservancy.

Photo of the commemorative plaque and record for the 2019 Free Software Award

This year's honoree is Jim Meyering, a prolific free software programmer, maintainer, and writer. Presenting the award was Richard Stallman, founder of both the Free Software Foundation and the GNU Project. Receiving his award, Jim wrote, "I dove head-first into the nascent *utils and autotools three decades ago. Little did I know how far free software would come or how it would end up shaping my ideas on software development. From what 'elegant,' 'robust,' and 'well-tested' could mean, to how hard (yet essential) it would be to say 'Thank you!' to those first few contributors who submitted fixes for bugs I'd introduced. Free software has given me so much, I cannot imagine where I would be without it. Thank you, RMS, co-maintainers and our oh-so-numerous contributors."

Due to ongoing worries about the COVID-19 outbreak, the 2020 LibrePlanet conference is being conducted entirely online, utilizing free software to stream the scheduled talks all over the globe, in lieu of the usual in-person conference and awards presentation. The Free Software Award winners will be mailed their commemorative gifts.

About the Free Software Foundation

The Free Software Foundation, founded in 1985, is dedicated to promoting computer users' right to use, study, copy, modify, and redistribute computer programs. The FSF promotes the development and use of free (as in freedom) software -- particularly the GNU operating system and its GNU/Linux variants -- and free documentation for free software. The FSF also helps to spread awareness of the ethical and political issues of freedom in the use of software, and its Web sites, located at https://fsf.org and https://gnu.org, are an important source of information about GNU/Linux. Donations to support the FSF's work can be made at https://my.fsf.org/donate. Its headquarters are in Boston, MA, USA.

More information about the FSF, as well as important information for journalists and publishers, is at https://www.fsf.org/press.

Media Contacts

John Sullivan
Executive Director
Free Software Foundation
+1 (617) 542 5942
campaigns@fsf.org

Photo credits: Free Software Foundation, Inc. Licensed under Creative Commons Attribution 4.0 International license. Photo Jim Meyering: Florence Meyering licensed under Creative Commons Attribution 4.0 International license.

15 March, 2020 12:55AM

March 13, 2020

mailutils @ Savannah

Version 3.9

Version 3.9 of GNU mailutils is available for download.

This is a bug-fix release.  Please see the NEWS file entry for a detailed list of changes.

13 March, 2020 10:51AM by Sergey Poznyakoff

March 09, 2020

Christopher Allan Webber

What should fit in a FOSS license?

Originally sent in an email to the OSI license-discuss mailing list.

What terms belong in a free and open source software license? There has been a lot of debate about this lately, especially as many of us are interested in expanding the role we see that we play in terms of user freedom issues. I am amongst those people that believe that FOSS is a movement thats importance is best understood not on its own, but on the effects that it (or the lack of it) has on society. A couple of years ago, a friend and I recorded an episode about viewing software freedom within the realm of human rights; I still believe that, and strongly.

I also believe there are other critical issues that FOSS has a role to play in: diversity issues (both within our own movement and empowering people in their everyday lives) are one, environmental issues (the intersection of our movement with the right-to-repair movement is a good example) are another. I also agree that the trend towards "cloud computing" companies which can more or less entrap users in their services is a major concern, as are privacy concerns.

Given all the above, what should we do? What kinds of terms belong in FOSS licenses, especially given all our goals above?

First, I would like to say that I think that many people in the FOSS world, for good reason, spend a lot of time thinking about licenses. This is good, and impressive; few other communities have as much legal literacy distributed even amongst their non-lawyer population as ours. And there's no doubt that FOSS licenses play a critical role... let's acknowledge from the outset that a conventionally proprietary license has a damning effect on the agency of users.

However, I also believe that user freedom can only be achieved via a multi-layered approach. We cannot provide privacy by merely adding privacy-requirements terms to a license, for instance; encryption is key to our success. I am also a supporter of code of conducts and believe they are important/effective (I know not everyone does; I don't care for this to be a CoC debate, thanks), but I believe that they've also been very effective and successful checked in as CODE-OF-CONDUCT.txt alongside the traditional COPYING.txt/LICENSE.txt. This is a good example of a multi-layered approach working, in my view.

So acknowledging that, which problems should we try to solve at which layers? Or, more importantly, which problems should we try to solve in FOSS licenses?

Here is my answer: the role of FOSS licenses is to undo the damage that copyright, patents, and related intellectual-restriction laws have done when applied to software. That is what should be in the scope of our licenses. There are other problems we need to solve too if we truly care about user freedom and human rights, but for those we will need to take a multi-layered approach.

To understand why this is, let's rewind time. What is the "original sin" that lead to the rise proprietary software, and thus the need to distinguish FOSS as a separate concept and entity? In my view, it's the decision to make software copyrightable... and then, adding similar "state-enforced intellectual restrictions" categories, such as patents or anti-jailbreaking or anti-reverse-engineering laws.

It has been traditional FOSS philosophy to emphasize these as entirely different systems, though I think Van Lindberg put it well:

Even from these brief descriptions, it should be obvious that the term "intellectual property" encompasses a number of divergent and even contradictory bodies of law. [...] intellectual property isn't really analagous to just one program. Rather, it is more like four (or more) programs all possibly acting concurrently on the same source materials. The various IP "programs" all work differently and lead to different conclusions. It is more accurate, in fact, to speak of "copyright law" or "patent law" rather than a single overarching "IP law." It is only slightly tongue in cheek to say that there is an intellectual property "office suite" running on the "operating system" of US law. -- Van Lindberg, Intellectual Property and Open Source (p.5)

So then, as unfortunate as the term "intellectual property" may be, we do have a suite of state-enforced intellectual restriction tools. They now apply to software... but as a thought experiment, if we could rewind time and choose between a timeline where such laws did not apply to software vs a time where they did, which would have a better effect on user freedom? Which one would most advance FOSS goals?

To ask the question is to know the answer. But of course, we cannot reverse time, so the purpose of this thought experiment is to indicate the role of FOSS licenses: to use our own powers granted under the scope of those licenses to undo their damage.

Perhaps you'll already agree with this, but you might say, "Well, but we have all these other problems we need to solve too though... since software is so important in our society today, trying to solve these other problems inside of our licenses, even if they aren't about reversing the power of the intellectual-restriction-office-suite, may be effective!"

The first objection to that would be, "well, but it does appear that it makes us addicted in a way to that very suite of laws we are trying to undo the damage of." But maybe you could shrug that off... these issues are too important! And I agree the issues are important, but again, I am arguing a multi-layered approach.

To better illustrate, let me propose a license. I actually considered drafting this into real license text and trying to push it all the way through the license-review process. I thought that doing so would be an interesting exercise for everyone. Maybe I still should. But for now, let me give you the scope of the idea. Ready?

"The Disposable Plastic Prevention Public License". This is a real issue I care about, a lot! I am very afraid that there is a dramatic chance that life on earth will be choked out within the next number of decades by just how much non-degradeable disposable plastic we are churning out. Thus it seems entirely appropriate to put it in a license, correct? Here are some ideas for terms:

  • You cannot use this license if you are responsible for a significant production of disposable plastics.

  • You must make a commitment to reduction in your use of disposable plastics. This includes a commitment to reductions set out by (a UN committee? Haven't checked, I bet someone has done the research and set target goals).

  • If you, or a partner organization, are found to be lobbying against laws to eliminate disposable plastics, your grant of this license is terminated.

What do you think? Should I submit it to license-review? Maybe I should. Or, if someone else wants to sumbit it, I'll enthusiastically help you draft the text... I do think the discussion would be illuminating!

Personally though, I'll admit that something seems wrong about this, and it isn't the issue... the issue is one I actually care about a lot, one that keeps me up at night. Does it belong in a license? I don't think that it does. This both tries to both fix problems via the same structures that we are trying to undo problems with and introduces license compatibility headaches. It's trying to fight an important issue on the wrong layer.

It is a FOSS issue though, in an intersectional sense! And there are major things we can do about it. We can support the fight of the right-to-repair movements (which, as it turns out, is a movement also hampered by these intellectual restriction laws). We can try to design our software in such a way that it can run on older hardware and keep it useful. We can support projects like the MNT Reform, which aims to build a completely user-repairable laptop, and thus push back against planned obsolescence. There are things we can, and must, do that are not in the license itself.

I am not saying that the only kind of thing that can happen in a FOSS license is to simply waive all rights. Indeed I see copyleft as a valid way to turn the weapons of the system against itself in many cases (and there are a lot of cases, especially when I am trying to push standards and concepts, where I believe a more lax/permissive approach is better). Of course, it is possible to get addicted to those things too: if we could go back in our time machine and prevent these intellectual restrictions laws from taking place, source requirements in copyleft licenses wouldn't be enforceable. While I see source requirements as a valid way to turn the teeth of the system against itself, in that hypothetical future, would I be so addicted to them that I'd prefer that software copyright continue just so I could keep them? No, that seems silly. But we also aren't in that universe, and are unlikely to enter that universe anytime soon, so I think this is an acceptable reversal of the mechanisms of destructive state-run intellectual restriction machine against itself for now. But it also indicates maybe a kind of maxima.

But it's easy to get fixated on those kinds of things. How clever can we be in our licenses? And I'd argue: minimally clever. Because we have a lot of other fights to make.

In my view, I see a lot of needs in this world, and the FOSS world has a lot of work to do... and not just in licensing, on many layers. Encryption for privacy, diversity initiatives like Outreachy, code of conducts, software that runs over peer to peer networks rather than in the traditional client-server model, repairable and maintainable hardware, thought in terms of the environmental impact of our work... all of these things are critical things in my view.

But FOSS licenses need not, and should not, try to take on all of them. FOSS licenses should do the thing they are appropriate to do: to pave a path for collaboration and to undo the damage of the "intellectual restriction office suite". As for the other things, we must do them too... our work will not be done, meaningful, or sufficient if we do not take them on. But we should do them hand-in-hand, as a multi-layered approach.

09 March, 2020 09:54PM by Christopher Lemmer Webber

March 08, 2020

GNU Guile

GNU Guile 3.0.1 released

We are pleased to announce GNU Guile 3.0.1, the first bug-fix release of the new 3.0 stable series! This release represents 45 commits by 7 people since version 3.0.0.

Among the bug fixes is a significant performance improvement for applications making heavy use of bignums, such as the compiler. Also included are fixes for an embarrassing bug in the include directive, for the hash procedure when applied to keywords and some other objects, portability fixes, and better R7RS support.

See the release announcement for details and the download page to give it a go!

08 March, 2020 04:14PM by Ludovic Courtès (guile-devel@gnu.org)

March 07, 2020

GNU Guile 2.2.7 released

We are pleased to announce GNU Guile 2.2.7, the seventh bug-fix release of the “legacy” 2.2 series (the current stable series is 3.0). This release represents 17 commits by 5 people since version 2.2.6. Among the bug fixes is a significant performance improvement for applications making heavy use of bignums, such as the compiler.

See the release announcement for details.

07 March, 2020 08:00PM by Ludovic Courtès (guile-devel@gnu.org)

March 05, 2020

Christopher Allan Webber

Terminal Phase v1.1 and Spritely Goblins v0.6 releases!

Hello all! I just did a brand new release of both:

So some highlights from each.

Terminal Phase

Okay, this is flashier, even if less important than Goblins. But the main thing is that I added the time travel debugging feature, which is so flashy I feel the need to show that gif again here:

Time travel in Spritely Goblins shown through Terminal Phase

Aside from time travel, there aren't many new features, though I plan on adding some in the next week (probably powerups or a boss fight), so another release should be not far away.

And oh yeah, since it's a new release, now is a good time to thank the current supporters:

Terminal Phase Credits

But yeah, the main thing that was done here is that Terminal Phase was updated for the new release of Goblins, so let's talk about that!

Goblins

For those who aren't aware, Spritely Goblins is a transactional actor model library for Racket.

v0.6 has resulted in a number of changes in semantics.

But the big deal is that Goblins finally has decent documentation, including a fairly in-depth tutorial and documentation about the API. I've even documented how you, in your own programs, can play with Goblins' time travel features.

So, does this mean you should start using it? Well, it's still in alpha, and the most exciting feature (networked, distributed programming) is still on its way. But I think it's quite nice to use already (and I'm using it for Terminal Phase).

Anyway, that's about it... I plan on having a new video explaining more about how Goblins works out in the next few days, so I'll announce that when it happens.

If you are finding this work interesting, a reminder that this work is powered by people like you.

In the meanwhile, hope you enjoy the new releases!

05 March, 2020 04:50PM by Christopher Lemmer Webber

coreutils @ Savannah

coreutils-8.32 released [stable]

This is to announce coreutils-8.32, a stable release.
See the NEWS below for more details.

Thanks to everyone who has contributed!
There have been 100 commits by 18 people in the 51 weeks since 8.31
  Akim Demaille (1)       Jeff Layton (3)
  Andreas Dilger (1)      Jim Meyering (1)
  Assaf Gordon (6)        Kamil Dudka (4)
  Bernhard Voelker (6)    Kevin Locke (1)
  Bruno Haible (3)        Martin Castillo (2)
  Chris Meyering (1)      Mike Swanson (1)
  Colin Watson (1)        Paul Eggert (31)
  Emil Engler (1)         Pádraig Brady (38)
  Jan Nieuwenhuizen (1)   Shugo Maeda (1)

Pádraig [on behalf of the coreutils maintainers]

==================================================================

Here is the GNU coreutils home page:
    https://gnu.org/software/coreutils/

For a summary of changes and contributors, see:
  https://git.sv.gnu.org/gitweb/?p=coreutils.git;a=shortlog;h=v8.32
or run this command from a git-cloned coreutils directory:
  git shortlog v8.31..v8.32

To summarize the 867 gnulib-related changes, run these commands
from a git-cloned coreutils directory:
  git checkout v8.32
  git submodule summary v8.31

==================================================================

Here are the compressed sources:
  https://ftp.gnu.org/gnu/coreutils/coreutils-8.32.tar.gz   (13MB)
  https://ftp.gnu.org/gnu/coreutils/coreutils-8.32.tar.xz   (5.3MB)

Here are the GPG detached signatures[*]:
  https://ftp.gnu.org/gnu/coreutils/coreutils-8.32.tar.gz.sig
  https://ftp.gnu.org/gnu/coreutils/coreutils-8.32.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify coreutils-8.32.tar.xz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys DF6FD971306037D9

and rerun the 'gpg --verify' command.

This release was bootstrapped with the following tools:
  Autoconf 2.69
  Automake 1.16.1
  Gnulib v0.1-3322-gd279bc6d9
  Bison 3.4.1

NEWS

* Noteworthy changes in release 8.32 (2020-03-05) [stable]

** Bug fixes

  cp now copies /dev/fd/N correctly on platforms like Solaris where
  it is a character-special file whose minor device number is N.
  [bug introduced in fileutils-4.1.6]

  dd conv=fdatasync no longer reports a "Bad file descriptor" error
  when fdatasync is interrupted, and dd now retries interrupted calls
  to close, fdatasync, fstat and fsync instead of incorrectly
  reporting an "Interrupted system call" error.
  [bugs introduced in coreutils-6.0]

  df now correctly parses the /proc/self/mountinfo file for unusual entries
  like ones with '\r' in a field value ("mount -t tmpfs tmpfs /foo$'\r'bar"),
  when the source field is empty ('mount -t tmpfs "" /mnt'), and when the
  filesystem type contains characters like a blank which need escaping.
  [bugs introduced in coreutils-8.24 with the introduction of reading
   the /proc/self/mountinfo file]

  factor again outputs immediately when stdout is a tty but stdin is not.
  [bug introduced in coreutils-8.24]

  ln works again on old systems without O_DIRECTORY support (like Solaris 10),
  and on systems where symlink ("x", ".") fails with errno == EINVAL
  (like Solaris 10 and Solaris 11).
  [bug introduced in coreutils-8.31]

  rmdir --ignore-fail-on-non-empty now works correctly for directories
  that fail to be removed due to permission issues.  Previously the exit status
  was reversed, failing for non empty and succeeding for empty directories.
  [bug introduced in coreutils-6.11]

  'shuf -r -n 0 file' no longer mistakenly reads from standard input.
  [bug introduced with the --repeat feature in coreutils-8.22]

  split no longer reports a "output file suffixes exhausted" error
  when the specified number of files is evenly divisible by 10, 16, 26,
  for --numeric, --hex, or default alphabetic suffixes respectively.
  [bug introduced in coreutils-8.24]

  seq no longer prints an extra line under certain circumstances (such as
  'seq -f "%g " 1000000 1000000').
  [bug introduced in coreutils-6.10]

** Changes in behavior

  Several programs now check that numbers end properly.  For example,
  'du -d 1x' now reports an error instead of silently ignoring the 'x'.
  Affected programs and options include du -d, expr's numeric operands
  on non-GMP builds, install -g and -o, ls's TABSIZE environment
  variable, mknod b and c, ptx -g and -w, shuf -n, and sort --batch-size
  and --parallel.

  date now parses military time zones in accordance with common usage:
    "A" to "M"  are equivalent to UTC+1 to UTC+12
    "N" to "Y"  are equivalent to UTC-1 to UTC-12
    "Z" is "zulu" time (UTC).
  For example, 'date -d "09:00B" is now equivalent to 9am in UTC+2 time zone.
  Previously, military time zones were parsed according to the obsolete
  rfc822, with their value negated (e.g., "B" was equivalent to UTC-2).
  [The old behavior was introduced in sh-utils 2.0.15 ca. 1999, predating
  coreutils package.]

  ls issues an error message on a removed directory, on GNU/Linux systems.
  Previously no error and no entries were output, and so indistinguishable
  from an empty directory, with default ls options.

  uniq no longer uses strcoll() to determine string equivalence,
  and so will operate more efficiently and consistently.

** New Features

  ls now supports the --time=birth option to display and sort by
  file creation time, where available.

  od --skip-bytes now can use lseek even if the input is not a regular
  file, greatly improving performance in some cases.

  stat(1) supports a new --cached= option, used on systems with statx(2)
  to control cache coherency of file system attributes,
  useful on network file systems.

** Improvements

  stat and ls now use the statx() system call where available, which can
  operate more efficiently by only retrieving requested attributes.

  stat and tail now know about the "binderfs", "dma-buf-fs", "erofs",
  "ppc-cmm-fs", and "z3fold" file systems.
  stat -f -c%T now reports the file system type, and tail -f uses inotify.

** Build-related

  gzip-compressed tarballs are distributed once again

05 March, 2020 02:37PM by Pádraig Brady

March 04, 2020

FSF News

Young hackers to deliver opening keynote for LibrePlanet conference

BOSTON, Massachusetts, USA -- Wednesday, March 4, 2020 -- The Free Software Foundation (FSF) today announced that the opening keynote for the LibrePlanet 2020 conference will be a panel of impressive young free software community members. The annual technology and social justice conference will be held in the Back Bay Events Center on March 14 and 15, 2020, with the theme "Free the Future."

Online registration is open until March 10, 10:00am Eastern Daylight Time (14:00 UTC); and registration is possible at the event.

The panelists will be Alyssa Rosenzweig, a free software hacker working at Collabora, Taowa, a sysadmin, free software enthusiast, and the youngest (non-uploading) Debian developer, and Erin Moon, whose free software work has focused on federated social media software as a user, contributor, and maintainer.

Alyssa leads the Panfrost project to build a free graphics stack for Mali GPUs, and she is passionate about freedom. She has strong, pointed ideas on what the future of free software is supposed to look like: "Beyond abstract criteria, today's software has profound network effects and psychosocial implications. In the balance is more than software freedom, but also users' entire lives. The need for holistic, optimistic free software is greater than ever," she states.

Greg Farough, campaigns manager at the FSF and moderator of the panel, added: "The continued success of the free software movement depends on its youngest members, so it is time we give them a proper platform. This new generation of freedom-minded developers is incredibly talented, diverse, and forward-thinking. I am convinced they can accomplish everything they set out to do."

The keynote will touch on subjects like optimism in the face of adversity, the social impact of working in free software, and the effect of social change, broadening user bases, and bringing the free software movement beyond the Intel x86 processor architecture. Taowa is excited to weigh in: "A free future is an accessible one. Much like scientific progress is built on what's come before us, technological progress is built on the shoulders of our predecessors. A free present enables a free future, but it isn't going to build itself."

Erin is the developer behind Rustodon, a Mastodon-compatible ActivityPub server. To her, having the panel comment on the future of freedom is a way to represent the meta community of diverging projects. She says: "Together we shape the future of free software and its subcommunities: what projects are prominent, the public image of free software, our culture and social mores. Paneling with people whose work is vastly different from my own is exhilarating! While I'm excited to learn about others' technical work, I'm also connected to them by this meta-community: our shared experiences as people involved with free software."

The panel is the third confirmed keynote for LibrePlanet 2020. It follows the announcement of Brewster Kahle in January, and the more recent announcement of Public Lab's Shannon Dosemagen. The schedule features talks and workshops from a wide and international range of community members. Thousands of people have attended LibrePlanet over the years, both in person and remotely. The conference welcomes visitors from up to fifteen countries each year, with many more joining online. Hundreds of impressive free software speaker sessions from past LibrePlanet conferences, including keynote talks by Micky Metts, Edward Snowden, and Cory Doctorow, can be viewed in the conference's video library.

About LibrePlanet

LibrePlanet is the annual conference of the Free Software Foundation. Over the last decade, LibrePlanet has blossomed from a small gathering of FSF associate members into a vibrant multi-day event that attracts a broad audience of people who are interested in the values of software freedom. LibrePlanet 2020 will be held on March 14 and 15, 2020. To sign up for announcements about LibrePlanet 2020, visit the Web site. To discuss LibrePlanet topics or to get involved, join the discussion list.

Registration for LibrePlanet: "Free the Future" is open through March 10. Attendance is free of charge for FSF associate members and for students.

For information on how your company can sponsor LibrePlanet or have a table in our exhibit hall, email campaigns@fsf.org.

About the Free Software Foundation

The Free Software Foundation, founded in 1985, is dedicated to promoting computer users' right to use, study, copy, modify, and redistribute computer programs. The FSF promotes the development and use of free (as in freedom) software -- particularly the GNU operating system and its GNU/Linux variants -- and free documentation for free software. The FSF also helps to spread awareness of the ethical and political issues of freedom in the use of software, and its Web sites, located at https://www.fsf.org and https://www.gnu.org, are an important source of information about GNU/Linux. Donations to support the FSF's work can be made at https://donate.fsf.org. Its headquarters are in Boston, MA, USA.

MEDIA CONTACT

ZoĂŤ Kooyman
Program Manager
Free Software Foundation
+1 (617) 542 5942
info@fsf.org

04 March, 2020 09:10PM

Youth software developers to deliver opening keynote for LibrePlanet conference

BOSTON, Massachusetts, USA -- Wednesday, March 4, 2020 -- The Free Software Foundation (FSF) today announced that its opening keynote for the LibrePlanet 2020 conference will be a panel of impressive young free software community members. The annual technology and social justice conference will be held in the Back Bay Events Center on March 14 and 15, 2020, with the theme "Free the Future."

Online registration is open until March 10, 10:00am Eastern Daylight Time (14:00 UTC), and registration is possible at the event.

The panelists will be Alyssa Rosenzweig, a free software hacker working at Collabora, Taowa, a sysadmin, free software enthusiast, and the youngest (non-uploading) Debian developer, and Erin Moon, whose free software work has focused on federated social media software as a user, contributor, and maintainer.

Alyssa leads the Panfrost project to build a free graphics stack for Mali GPUs, and she is passionate about freedom. She has strong, pointed ideas on what the future of free software is supposed to look like: "Beyond abstract criteria, today's software has profound network effects and psychosocial implications. In the balance is more than software freedom, but also users' entire lives. The need for holistic, optimistic free software is greater than ever," she states.

Greg Farough, campaigns manager of the FSF and moderator of the panel, added: "The success of the free software movement now depends on its youngest members, so it is time we give them a proper platform. This new generation of freedom-minded developers is incredibly talented, diverse, and forward-thinking. I am convinced they can accomplish everything they set out to do."

The keynote will touch on subjects like optimism in the face of adversity, the social impact of working in free software, and the effect of social change, broadening user bases, and bringing the free software movement beyond the Intel x86 processor architecture. Taowa is excited to weigh in: "A free future is an accessible one. Much like scientific progress is built on what's come before us, technological progress is built on the shoulders of our predecessors. A free present enables a free future, but it isn't going to build itself."

Erin is the developer behind Rustodon, a Mastodon-compatible ActivityPub server. To her, having the panel comment on the future of freedom is a way to represent the meta community of diverging projects. She says: "Together we shape the future of free software and its subcommunities: what projects are prominent, the public image of free software, our culture and social mores. Paneling with people whose work is vastly different from my own is exhilarating! While I'm excited to learn about others' technical work, I'm also connected to them by this meta-community: our shared experiences as people involved with free software."

The panel is the third confirmed keynote for LibrePlanet 2020. It follows the announcement of Brewster Kahle in January, and the more recent announcement of Public Lab's Shannon Dosemagen. The schedule features talks and workshops from a wide and international range of community members. Thousands of people have attended LibrePlanet over the years, both in person and remotely. The conference welcomes visitors from up to fifteen countries each year, with many more joining online. Hundreds of impressive free software speaker sessions from past LibrePlanet conferences, including keynote talks by Micky Metts, Edward Snowden, and Cory Doctorow can be viewed in the conference's video library.

About LibrePlanet

LibrePlanet is the annual conference of the Free Software Foundation. Over the last decade, LibrePlanet has blossomed from a small gathering of FSF associate members into a vibrant multi-day event that attracts a broad audience of people who are interested in the values of software freedom. LibrePlanet 2020 will be held on March 14 and 15, 2020. To sign up for announcements about LibrePlanet 2020, visit the Web site. To discuss LibrePlanet topics or to get involved, join the discussion list.

Registration for LibrePlanet: "Free the Future" is open through March 10. Attendance is free of charge for FSF associate members, and for students.

For information on how your company can sponsor LibrePlanet or have a table in our exhibit hall, email campaigns@fsf.org.

About the Free Software Foundation

The Free Software Foundation, founded in 1985, is dedicated to promoting computer users' right to use, study, copy, modify, and redistribute computer programs. The FSF promotes the development and use of free (as in freedom) software -- particularly the GNU operating system and its GNU/Linux variants -- and free documentation for free software. The FSF also helps to spread awareness of the ethical and political issues of freedom in the use of software, and its Web sites, located at https://www.fsf.org and https://www.gnu.org, are an important source of information about GNU/Linux. Donations to support the FSF's work can be made at https://donate.fsf.org. Its headquarters are in Boston, MA, USA.

MEDIA CONTACT

Zoë Kooyman
Program Manager
Free Software Foundation
+1 (617) 542 5942
campaigns@fsf.org

04 March, 2020 06:45PM

March 02, 2020

Applied Pokology

Interruptible poke

Poke has loops. It also does IO (yeah, right :D). On one hand, this means that Poke programs that never finish are definitely possible (here is one: while (1) {}) and, on the other, a Poke program acting on an enormous amount of data may take a long time to finish, depending on what it does. Maybe hours, maybe even days?

02 March, 2020 12:00AM

March 01, 2020

health @ Savannah

GNU Health GTK CLient 3.6.8 released

Dear all

The GH GTK client 3.6.8 fixes several bugs, specially related to the compatibility issues with KDE and other desktop environments.

We have moved the header bar components to the main window, so now it also allows borderless operation.

Summary :

  • Removed the deprecated App Menu in favor of the Menu Bar.
  • Migrate from obsolete Table to Grid
  • Fix the footer
  • Include accelerator for preferences (ctrl+shift+p) and fix accelerator for shortcuts (ctrl+shift+F1)
  • Avoid maximizing application at startup
  • Remove unused fast tabbing option from menu
  • Allow setting the GH command line on the bottom again (cli_position = bottom). By default is top.

01 March, 2020 01:43PM by Luis Falcon

February 29, 2020

GNU Guix

GSoC 2020 and Outreachy May 2020 to August 2020 Status Report II

GSoC

We are happy to announce that GNU Guix participates in the Google Summer of Code (GSoC), under the aegis of the GNU project. We have collected project ideas related to GNU Guix. The list is far from exhaustive, so feel free to bring your own!

The GNU Project participation was announced on Feb. 20. Thanks for the GNU org admins for organizing this.

The application period is from March 16. to March 31. The final proposal submission deadline is March 31., 2020 at 20:00 CEST.

The student projects are announced on April 27., 2020. We will have to provide the number of slots requested to the GNU project, so that they can accumulate the numbers to pass on to Google. This takes some time, so please prepare the decision early, so we don't have to hurry when this information is requested. We kindly remind everyone involved not to communicate an intern selection decision before the official announcement.

Internship information

Information about GSoC internships related to the GNU Guix community.

Outreachy

We are happy to announce that GNU Guix offers a three-month internship through Outreachy, the inclusion program for groups traditionally underrepresented in free software and tech.

We currently propose three subjects to work on:

  1. Create Netlink bindings in Guile.
  2. Improve internationalization support for the Guix Data Service.
  3. Integration of desktop environments into GNU Guix.

The initial application deadline was on Feb. 25, 2020 at 4PM UTC, so initial applications are now closed. This means that prospective applicants how did not do the initial application yet will have to apply for a later Outreachy round.

The project list was finalized on Feb. 27. This means that no new proposals can be added for this round, so the list above is final.

Funding was confirmed for two internships this round. Thanks for the members of the spending committee, who are taking care of the financial side of this.

Co-mentor applications are still open, prospective mentors are encouraged to apply on the Guix community page.

The next phase is the contribution period: March 5, 2020 to April 7, 2020. Also the final application deadline is April 7, 2020 at 4PM UTC.

In this phase applicants are working with the mentors on the projects, and they have to register a contribution. A registered contribution is a mandatory requirement.

Accepted interns will be announced on April 27, 2020 at 4PM UTC. We kindly remind everyone involved not to communicate an intern selection decision before the official announcement.

Internship information:

Last year we had the pleasure to welcome Laura Lazzati as an Outreachy intern working on documentation video creation, which led to the videos you can now see on the home page.

Information about Outreachy internships related to the GNU Guix community.

Contributions

If you’d like to contribute to computing freedom, Scheme, functional programming, or operating system development, now is a good time to join us. Let’s get in touch on the mailing lists and on the #guix channel on the Freenode IRC network!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

29 February, 2020 03:00PM by Gábor Boskovits

February 28, 2020

FSF News

Environmental activist Shannon Dosemagen joins FSF conference keynote lineup

BOSTON, Massachusetts, USA -- Friday, February 28, 2020 -- The Free Software Foundation (FSF) today announced Shannon Dosemagen as the second keynote speaker for LibrePlanet 2020. The annual technology and social justice conference will be held in the Back Bay Event Center on March 14 and 15, 2020, with the theme "Free the Future."

Registration is open until March 10, 10:00am Eastern Daylight Time (14:00 UTC), and limited registration is possible at the event.

Shannon Dosemagen is the second confirmed keynote speaker for the LibrePlanet conference. Dosemagen is the co-founder and current executive director of Public Lab, a nonprofit organization creating local environmental science solutions following the free software philosophy, and winner of the FSF's Award for Projects of Social Benefit. Shannon Dosemagen is an environmental health advocate and a community science champion, and is enthusiastic about free systems and technology that support the creation of a more just and equitable future. She is a previous Fellow at both the Harvard Berkman Klein Center for Internet and Society, and the Loyola University Environmental Communications Institute. During 2020, she will be a Fellow with the Shuttleworth Foundation, working on a new concept.

At LibrePlanet, Dosemagen will discuss her experience democratizing science to address environmental problem-solving. Her experiences and frustrations doing this work are very familiar to the free software community: "The work I do on the environment and health is being increasingly challenged by environmental deregulation and lack of cooperation. We're also seeing heightened pressure to drastically alter how society functions in an effort to curb the climate crisis. This is a profound moment, and critical to address at an event aptly themed 'Free the Future.'"

"Shannon's work is very important, and is a testament to the success of community collaboration," says ZoĂŤ Kooyman, the FSF's program manager. "Public Lab's work towards free hardware solutions is a strong indicator of what the four freedoms can achieve, and how they can work towards a better future outside of software. Shannon is an experienced speaker and organizer, and we are proud to have her keynote at LibrePlanet."

Free technology is creating tangible solutions for many environmental and health issues through Dosemagen's work, and the knowledge and involvement of the wider free software movement are crucial to its success: "The work of LibrePlanet participants has an important role to play in figuring out better systems, structures, and governance models for the environmental health of our communities," Dosemagen states.

The conference schedule features talks and workshops from a wide and international range of community members, including another keynote by Brewster Kahle. Thousands of people have attended LibrePlanet over the years, both in person and remotely. The conference has welcomed visitors from up to fifteen countries each year, with many more joining online. Hundreds of impressive free software speaker sessions from past LibrePlanet conferences, including keynote talks by Edward Snowden, and Cory Doctorow can be viewed in the conference's video library. The conference has also featured many engaging panel sessions, such as a session on free medical devices featuring Rachel Kalmar, Karen Sandler, and Dana Lewis.

About LibrePlanet

LibrePlanet is the annual conference of the Free Software Foundation. Over the last decade, LibrePlanet has blossomed from a small gathering of FSF associate members into a vibrant multi-day event that attracts a broad audience of people who are interested in the values of software freedom. LibrePlanet 2020 will be held on March 14 and 15, 2020. To sign up for announcements about LibrePlanet 2020, visit the Web site. To discuss LibrePlanet topics or to get involved, join the discussion list.

Registration for LibrePlanet: "Free the Future" is open. Through March 10. Attendance is free of charge for FSF associate members, and for students.

For information on how your company can sponsor LibrePlanet or have a table in our exhibit hall, email campaigns@fsf.org.

About the Free Software Foundation

The Free Software Foundation, founded in 1985, is dedicated to promoting computer users' right to use, study, copy, modify, and redistribute computer programs. The FSF promotes the development and use of free (as in freedom) software -- particularly the GNU operating system and its GNU/Linux variants -- and free documentation for free software. The FSF also helps to spread awareness of the ethical and political issues of freedom in the use of software, and its Web sites, located at https://www.fsf.org and https://www.gnu.org, are an important source of information about GNU/Linux. Donations to support the FSF's work can be made at https://donate.fsf.org. Its headquarters are in Boston, MA, USA.

MEDIA CONTACT

ZoĂŤ Kooyman
Program Manager
Free Software Foundation
+1 (617) 542 5942
campaigns@fsf.org

Photo Courtesy Nate Dappen/CROWD & CLOUD

28 February, 2020 05:00PM

health @ Savannah

GNU Health HMIS patchset 3.6.3 released with coronavirus COVID-19 coding information

Dear community

GNU Health 3.6.3 patchset has been released !

Priority: High

Table of Contents

  • About GNU Health Patchsets
  • Updating your system with the GNU Health control Center
  • Summary of this patchset
  • Installation notes
  • List of issues related to this patchset

About GNU Health Patchsets

We provide "patchsets" to stable releases. Patchsets allow applying bug fixes and updates on production systems. Always try to keep your production system up-to-date with the latest patches.

Patches and Patchsets maximize uptime for production systems, and keep your system updated, without the need to do a whole installation.

NOTE: Patchsets are applied on previously installed systems only. For new, fresh installations, download and install the whole tarball (ie, gnuhealth-3.6.3.tar.gz)

Updating your system with the GNU Health control Center

Starting GNU Health 3.x series, you can do automatic updates on the GNU Health HMIS kernel and modules using the GNU Health control center program.

Please refer to the administration manual section ( https://en.wikibooks.org/wiki/GNU_Health/Control_Center )

The GNU Health control center works on standard installations (those done following the installation manual on wikibooks). Don't use it if you use an alternative method or if your distribution does not follow the GNU Health packaging guidelines.

Summary of this patchset

Patch 3.6.3 updates World Health Organization ICD10 with the novel coronavirus COVID-19 code (U07.1)

It also fixes an issue when creating a page of life from an encounter that has an empty main condition.
in addition, updated gnuhealth-setup and gnuhealth control center are also included.

Installation Notes

You must apply previous patchsets before installing this patchset. If your patchset level is 3.6.2, then just follow the general instructions.
You can find the patchsets at GNU Health main download site at GNU.org (https://ftp.gnu.org/gnu/health/)

In most cases, GNU Health Control center (gnuhealth-control) takes care of applying the patches for you. 

Follow the general instructions at

After applying the patches, make a full update of your GNU Health database as explained in the documentation.

  • Restart the GNU Health server

List of issues and tasks related to this patchset

  • GH HMIS server. task #15562: Include coronavirus COVID-19 in ICD10 codes
  • GH HMIS server. bug #57695: Traceback when creating a page of life associated to an empty evaluation
  • GH HMIS server. task #15561: Werkzeug 1.0 is not compatible with Trytond 5.0

 For detailed information about each issue, you can visit https://savannah.gnu.org/bugs/?group=health
 For detailed information about each task, you can visit https://savannah.gnu.org/task/?group=health

 For detailed information you can read about Patches and Patchsets

Happy and healthy hacking !

Dr. Luis Falcon, MD, MSc
President, GNU Solidario
GNU Health: Freedom and Equity in Healthcare
http://www.gnuhealth.org
GPG Fingerprint :ACBF C80F C891 631C 68AA  8DC8 C015 E1AE 0098 9199

28 February, 2020 11:09AM by Luis Falcon

February 27, 2020

Christopher Allan Webber

Content Addressed Vocabulary

How can systems communicate and share meaning? Communication within systems is preceded by a form of meta-communication; we must have a sense that we mean the same things by the terms we use before we can even use them.

This is challenging enough for humans who must share meaning, but we can resolve ambiguities with context clues from a surrounding narrative. Machines, in general, need a context more explicitly laid out for them, with as little ambiguity as possible.

Standards authors of open-world systems have long struggled with such systems and have come up with some reasonable systems; unfortunately these also suffer from several pitfalls. With minimal (or sometimes none at all) adjustment to our tooling, I propose a change in how we manage ontologies.

How we deal with ambiguous terms today

Consider Note, a seemingly simple term in ActivityStreams, the vocabulary used by ActivityPub. The meaning of Note, as described by the ActivityStreams vocabulary, seems simple enough: Represents a short written work typically less than a single paragraph in length.

Here is how an ActivityStreams usage of Note might look (a bit simplified from what it would probably look like in practice):

  {"@context": "https://www.w3.org/ns/activitystreams",
   "@type": "Note",
   "content": "Would you read me a bedtime story about the great ontology wars?"}

What's that @context thing? This is some JSON-LD thing, which tries to be "more exact" about what Note we must be talking about. It does so by mapping Note to https://www.w3.org/ns/activitystreams#Note by something like the following:

  {"as": "https://www.w3.org/ns/activitystreams#",
   "Note": "as:Note",
   "content": "as:content",
   ...}

The choice to use JSON-LD has been semi-controversial in ActivityPub land; historically there was some debate about whether or not we needed to be "more exact" at all as to what terms mean. This post really isn't about JSON-LD as much as it is the more general topic of vocabularies and vocabulary mapping systems. There are other concerns people raise about JSON-LD, usually around the tooling... that's not the scope of this post. This blogpost could as easily apply to XML or Turtle or whatever; the protocol I've worked on just happens to use JSON-LD to do that, so I've used it as my illustration.

That said, the ActivityPub spec tries to make things as simple as possible for the default case of ActivityPub usage by saying that the ActivityStreams context is implied, so that if you're not doing anything complicated, so:

  {"@type": "Note",
   "content": "Would you read me a bedtime story about the great ontology wars?"}

... is really the same as the first example.

So okay, probably everyone can guess what Note means, but what about sensitive? What the heck is that? It doesn't appear in the ActivityStreams vocabulary; it kind of implies something along the lines of content-warning type behavior, like "this content may be considered sensitive" by some users, but how would you guess that just by the term? This is an extension, and it lives at http://joinmastodon.org/ns#sensitive.

So maybe if we were going to use it (and if we inline our context) it might look like:

  {"@context": {"as": "https://www.w3.org/ns/activitystreams#",
                "toot": "http://joinmastodon.org/ns#",
                "Note": "as:Note",
                "content": "as:content",
                "sensitive": "toot:sensitive"},
   "@type": "Note",
   "content": "Would you read me a bedtime story about the great ontology wars?",
   "sensitive": true}

(I mean, the Great Ontology Wars are a sensitive topic for some.)

The choice of JSON-LD in ActivityPub is controversial for various reasons. But it turns out what isn't really controversial anymore is whether we need some way of being more exact about the way we speak about terms... those who used to complain about that mostly now agree (disagreements then surround what tooling need to be used to do so (not in scope of this post), and namespace governance (in scope of this post)).

Maybe you feel like, having heard what sensitive and Note mean, these are the obvious definitions. But consider that Note itself could have meant something very different. Are we talking about a short mostly-textual post (probably on a microblog), as ActivityStreams does? Are we talking about a musical note? Are we instructing someone to take note of something, as an action (or yes, activity)?

So terms really are ambiguous, and in a decentralized but extensible system with open world assumptions, we are eventually going to result in conflicts. The choice to map our vocabulary to URIs is actually a very reasonable way to reduce ambiguity. Unfortunately, the choice to map them to namespaces and to live URIs (a-la http(s): URIs), is a mistake that will eventually bite us (and doubly so for JSON-LD contexts).

Problems appear

The first problem with choosing to put our terminology URIs at HTTP(S) URIs is that it assumes that those vocabularies will remain alive. Perhaps popular ones shall, but really the modern web rots all the time. Soon enough, many ontologies will eventually be replaced by Viagra ads.

The problem is dramatically worse for json-ld contexts (and similar documents such as XML DTDs): these are the very documents by which we map terms to their fully defined meanings. Servers get hammered by people looking up contextual mappings. This is no good already. It gets even worse when such documents add (or otherwise amend) their terminology mappings; old documents may suddenly mean different things!

(I'd be remiss to not note here that vocabulary namespaces and json-ld contexts are frequently the same URIs and yet frequently not the same thing. Still, they share a lot of the same problems and solutions in terms of liveness.)

Furthermore, both the choice to put terms in namespaces and the choice to have common contextual URIs that can change creates governance problems.

I know this from personal experience (and by that I mean many painful hours of my life wasted that I can never get back). Consider sensitive above. The Mastodon folks created their own namespace, as previously mentioned, but they didn't really want to. The good news was that the Social Web Community Group was given permission to both extend the ActivityStreams vocabulary and the official ActivityStreams context.

Despite the entire group agreeing that it made sense to make sensitive official in some way (which does not mean everyone agreed that it was a good term, just that it was in enough usage that we should make it more easily widely available), the SocialCG got tied up for months and months in meetings being unable to make progress about how to do so:

  • Should we add sensitive to the ActivityStreams namespace, or leave it in the old namespace but "officially sanction" it?
  • What is the migration path for software using the previous term URI?
  • How often should we do this? What is the governance process for incubating a new term? Should it happen in a separate namespace first and then get "pulled in" later?
  • What would happen if we didn't for terms like these, and the sites went down?
  • If we also update the json-ld context, what happens for documents that already had sensitive in them meaning either the old URI or a new one? This can have significant impact on normalization for signature verification.

The group met for months about all the topics above and came to no conclusions. Eventually we decided that no consensus could be reached, so instead no action was taken at all. What a disappointment.

In general, this seems to be common. Ironically, it leads to otherwise nice decentralized designs for vocabularies eventually ending up centralized in something like schema.org anyway.

Content addressed vocabularies (and contexts) are the answer

My friend Sandro Hawke offered a solution, which I initially rejected as terrible, decided upon further consideration was brilliant, and fully embraced. Then Sandro explained to me that I had totally misunderstood him, and that he meant something different. It turns out that I actually think my initial misunderstanding was the right answer.

Here's what I understood Sandro to say:

The name we choose for a term doesn't matter that much. What really matters is the paragraph or so of specification language that describes the term. If two implementations refer to the same specification text, they mean the same thing. So just use that as the description.

Once I (incorrectly) came to realize that this could mean naming via content addressing, I latched onto the idea. Of course! We had merely selected the wrong edge of Zooko's triangle. But we know how to fix that sort of thing.

Here's how it works. Let's remember the specification text for Note above: Represents a short written work typically less than a single paragraph in length. Let's hash that (along with a "recommendation" prefix that a user might choose to bind this to the term Note, though this is just a recommendation):

$ echo "Note: Represents a short written work typically less than a single paragraph in length." | sha256sum
3e1de3b56d2dc1bee7313963462691f9a8f46b068557b75e0e0d14c0994eddc6

So if we were defining Note via content-addressing, we instead would have defined it as urn:sha256:3e1de3b56d2dc1bee7313963462691f9a8f46b068557b75e0e0d14c0994eddc6. This is unambiguous enough to avoid collisions with other uses of the word "Note". But note that it doesn't require any servers staying up. It also doesn't have any namespace governance quagmire, because there is no namespace. Updates can be handled the usual way, via errata (translations can be handled similarly), and standards organizations can still publish such things... but it is important that the original term remain content-addressed and immutable. (Hash migration is left as an exercise for the user, with a hint that the solution is similar to that with errata.)

Anyway, our post might end up looking in the end like this instead:

  {"@context": {"Note": "urn:sha256:3e1de3b56d2dc1bee7313963462691f9a8f46b068557b75e0e0d14c0994eddc6",
                "content": "urn:sha256:57dc44a1cdcbb7aa976a65a858b4d349ad6110d58d9d546650ce2b0e2b1048e4",
                "sensitive": "urn:sha256:81d98cf83fcf733400ad5d2a25495feeea47f287193a53a9722f4cb025da88f1"},
   "@type": "Note",
   "content": "Would you read me a bedtime story about the great ontology wars?",
   "sensitive": true}

I'll note very briefly that content-addressing is also the answer for JSON-LD contexts. If something like Datashards or IPFS were used to host json-ld contexts, each post could link to the exact immutable content-addressed context it was intended to be used with. Servers that use such contexts can "pin" them to keep them available, avoiding a single point of failure (or bandwidth bottleneck).

  {"@context": "idsc:p0.JLnUcJN4R1KNvSXm9Ut3Tmg7WfXAKEOx47p01Pk_Htw.2_rCdtnEha1RpD_qyzxhFIjUvLj7crIbzpmzWei5xRk",
   "@type": "Note",
   "content": "Would you read me a bedtime story about the great ontology wars?",
   "sensitive": true}

As one other side-note, I'll also observe that even though the fully expanded version of the above message is:

  {"@type": "urn:sha256:3e1de3b56d2dc1bee7313963462691f9a8f46b068557b75e0e0d14c0994eddc6",
   "urn:sha256:57dc44a1cdcbb7aa976a65a858b4d349ad6110d58d9d546650ce2b0e2b1048e4": "Would you read me a bedtime story about the great ontology wars?",
   "urn:sha256:81d98cf83fcf733400ad5d2a25495feeea47f287193a53a9722f4cb025da88f1": true}

... we never needed to look at it that way because json-ld contexts (and systems like them) are actually petname systems.

Conclusions (and non-conclusions)

Let me clarify a claim I'm not making: we don't need to throw away the old terms for systems like ActivityStreams that are already well understood. However, going forward I do think that using content-addressing of new terms is a good idea. And in the long run, I think content-addressing of json-ld contexts and any documents like them is an absolute must (when they aren't inlined, anyway... but inlining is expensive).

If we adopted Content Addressed Vocabularies, working on vocabulary extensions to ActivityPub could be a different story. Imagine a git repository that communities can fork to work on new terms. We could have a drafts directory where people hammer out common extension terms, and when they're ready, we simply move them to the extensions directory. Since the names are merely hashes of the contents of that directory, statically generating a webpage that lists all current known and recommended extensions would be trivial. Everything could be handled in issues and PRs, and even if terms aren't merged into the main repo, that's merely a matter of lower term discoverability rather than a hinderance of application itself.

If we moved to content addressed vocabulary, we'd be more free from the perils of downtime and general web bitrot, freer from gatekeeping and governance challenges, but just as free (I'd argue even freer) to collaborate. Moving forward, I intend to ake content addressed approaches to terms I define in my systems, and I encourage you to do the same.

27 February, 2020 01:23AM by Christopher Lemmer Webber

February 25, 2020

www @ Savannah

Structure and Administration of the GNU Project

People know that each GNU package has one or more maintainers
appointed by the GNU Project. People mostly don't know about the
committees that carry out most of the administration of the project.

We have now published a complete description of the administrative
structure of the GNU Project:

https://www.gnu.org/gnu/gnu-structure.html

25 February, 2020 04:40AM by Jason Self

February 22, 2020

parallel @ Savannah

GNU Parallel 20200222 ('BrexitDay') released [stable]

GNU Parallel 20200222 ('BrexitDay') [stable] has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/

No new functionality was introduced so this is a good candidate for a stable release.

GNU Parallel is 10 years old next year on 2020-04-22. You are here by invited to a reception on Friday 2020-04-17.

See https://www.gnu.org/software/parallel/10-years-anniversary.html

Quote of the month:

  Parallel is the BEST command.
    -- Nick @NickInfoSec@twitter

New in this release:

  • Bug fixes and man page updates.

News about GNU Parallel:

Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

GNU Parallel - For people who live life in the parallel lane.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 3374ec53bacb199b245af2dda86df6c9
    12345678 3374ec53 bacb199b 245af2dd a86df6c9
    $ md5sum install.sh | grep 029a9ac06e8b5bc6052eac57b2c3c9ca
    029a9ac0 6e8b5bc6 052eac57 b2c3c9ca
    $ sha512sum install.sh | grep f517006d9897747bed8a4694b1acba1b
    40f53af6 9e20dae5 713ba06c f517006d 9897747b ed8a4694 b1acba1b 1464beb4
    60055629 3f2356f3 3e9c4e3c 76e3f3af a9db4b32 bd33322b 975696fc e6b23cfb
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

22 February, 2020 11:54PM by Ole Tange

February 16, 2020

Christopher Allan Webber

Vats and Propagators: towards a global brain

(This is a writeup for future exploration; I will be exploring a small amount of this soon as a side effect of some UI building I am doing, but not a full system. A full system will come later, maybe even by years. Consider this a desiderata document. Also a forewarning that this document was originally written for an ocap-oriented audience, and some terms are left unexpanded; for instance, "vat" really just means a one-turn-at-a-time single-threaded event loop that a bunch of actors live in.)

We have been living the last couple of decades with networks that are capable of communicating ideas. However, by and large it is left to the humans to reason about these ideas that are propagated. Most machines that operate on the network merely execute the will of humans that have carefully constructed them. Recently neural network based machine learning has gotten much better, but merely resembles intuition, not reasoning. (The human brain succeeds by combining both, and a successful system likely will too.) Could we ever achieve a network that itself reasons? And can it be secure enough not to tear itself apart?

Near-term background

In working towards building out a demonstration of petname systems in action in a social network, I ran into the issue of changes to a petname database automatically being reflected through the UI. This lead me back down a rabbit hole of exploring reactive UI patterns, and also lead me back to exploring that section, and the following propagator section, of SICP again. This also lead me to rewatch one of my favorite talks: We Don't Really Know How to Compute! by Gerald Sussman.

At 24:54 Sussman sets up an example problem: specifically, an expert in electrical systems having a sense of how to be able to handle and solve an electrical wiring diagram. (The kind of steps explained are not dissimilar to the kind of steps that programmers go through while reasoning about debugging a coding problem.) Sussman then launches into an exploration of propagators, and how they can solve the problem. Sussman's explanation is better than mine would be, so I'll leave you to watch the video to see how it's used to solve various problems.

Okay, a short explanation of propagators

Well, I guess I'll give a little introduction to propagators and why I think they're interesting.

Propagators have gone through some revisions since the SICP days; relevant reading are the Revised Report on the Propagator Model, The Art of the Propagator, and to really get into depth with the ideas, Propagation networks: a flexible and expressive substrate for computation (Radul's PhD thesis).

In summary, a propagator model has the following properties:

  • There are cells with accumulate information about a value. Note! This is a big change from previous propagator versions! In the modern version of a propagator model, a cell doesn't hold a value, it accrues information about a value which must be non-contradictory.
  • Such cell information may be complete (the number 42 is all there is to know), whereas some other information may be a range of possibilities (hm, could be anywhere between -5 to 45...). As more information is made available, we can "narrow down" what we know.
  • Cells are connected together with propagators.
  • Information is (usually) bidirectional. For example, with the slope formula of y = (m * x) + b, we don't need to just solve for y... we could solve for m, x, or b given the other information. Similarly, partial information can propagate.
  • Contradictions are not allowed. Attempting to introduce contradictory information into the network will throw an exception.
  • We can "play with" different ideas via a Truth Maintenance System. What do we believe? Changes in our beliefs can result in changes to the generated topology of the network.
  • Debugging is quite possible. One of the goals of propagator networks is that you should be able to investigate and determine blame for a result. Relationships are clear and well defined. As Sussman says (roughly paraphrased), "if an autonomous car drives off the car of the road, I could sue the car manufacturer, but I'd rather sue the car... I want to hold it accountable for its decision making". The ability to hold accountability and determine blame stands in contrast to squishier systems like neural nets, genetic programs, etc (which are still useful, but not as easy to interrogate).

There are a lot of things that can be built with propagators as the general case of constraint solving and reasoning; functional reactive UIs, type checkers, etc etc.

Bridging vats and propagators

The prototype implementations are written in Scheme. The good news is, this means we could implement propagators on top of something like Spritely Goblins.

However (and, granted, I haven't completed it) I think there is one thing that is inaccurately described in Radul's thesis and Sussman's explanations, but which I think actually is no problem at all if we apply the vat model of computation (as in E, Agoric, Goblins): how distributed can these cells and propagators be? Section 2.1 of Radul's thesis explains propagators as asynchronous and completely autonomous, as if cells and their propagators could live anywhere on the computer network with no change in effectiveness. I think this is only partially true. The reference implementation actually does not fully explore this because it uses a single-threaded event loop that processes events until there are no more to process, during which it may encounter a contradiction and raise it. However I believe that the ability to "stop the presses" as it were is one of the nicest features of propagators and actually should not be lost... if we introduced asynchronous events coming in, there may be multiple events that come in at the same time and which try making changes to the propagator network in parallel. Thankfully a nice answer comes in form of a the vat model: it should be possible to have a propagator network within a single vat. Spritely Goblins' implementation of the vat model is transactional, so this means that if we try to introduce a contradiction, we could roll back immediately. This is the right behavior. As it turns out, this is very close to the propagator system in the way it's implemented in the reference implementation... I think the reference implementation did something more or less right while trying to do the simplest thing. Combined with a proper ocap vat model this should work great.

Thus, I believe that a propagator system (here I mean a propagator network, meaning a network of propagator-connected cells) should actually be vat-local. But wait, we talked about network (as in internet) based reasoning, and here I am advocating locality! What gives?

The right answer seems to me that propagator networks should be able to be hooked together, but a change to a vat-contained propagator system can trigger message passing to another vat-contained propagator system, which can even happen over a computer network such as the internet. We will have to treat propagator systems and changes to them as vat-local, but they can still communicate with other propagator systems. (This is a good idea anyway; if you communicate an idea with me and it's inconsistent with my worldview, it should be important for me to be able to realize that and use that as an opportunity to correct our misunderstandings between each other.)

However, cells are still objects with classic object references. This means it is possible to hold onto one and use it as either a local or networked capability. Attenuation also composes nicely; it should be possible to produce a facet of a cell that only allows read access or only allows adding information. It's clear and easily demonstrated that ocaps can be the right security model for the propagator model simply by realizing that both the propagator prototype system is written in scheme, and so is Jonathan Rees' W7 security kernel.

This is all to say, if we built the propagator model on top of an ocap-powered vat model, we'd already have a good network communication model, a good security model, and a transactional model. Sounds great to me.

Best of all, a propagator system can live alongside normal actors. We don't have to choose one or the other... a multi-paradigm approach can work great.

Speaking the same language

One of the most important things in a system that communicates is that ideas should be able to be expressed and considered in such a way that both parties understand. Of course, humans do this, and we call it "language".

Certain primitives exist in our system already; for optimization reasons, we are unlikely to want to build numbers out of mere tallying of numbers (such as in Peano arithmetic); we instead build in primitives for integers and a means of combination for them. So we will of course want to have several primitive data types.

But at some point we will want to talk about concepts that are not encoded in the system. If I would like to tell you about a beautiful red bird I saw, where would I even begin? Well obviously at minimum, we will have to have ways of communicating ideas such as "red" and "bird". We will have to build a vocabulary together.

Natural language vocabulary has a way of becoming ambiguous fast. A "note" passed in class versus a "note" in a musical score versus that I would like to "note" a topic of interest to you are all different things.

Linked data (formerly "semantic web") folks have tried to use full URIs as a way to get around this problem. For instance, two ActivityPub servers which are communicating are very likely speaking about the same thing if they both use "https://www.w3.org/ns/activitystreams#Note", which is to say they are talking about some written note-like message (probably a (micro)blog post). This is not a guarantee; vocabulary drift is still possible, but it is much less likely.

Unfortunately, http(s) based URIs are a poor choice for hosting vocabulary. Domains expire, websites go down, and choosing whether to extend a vocabulary in some namespace is (in the author's experience) a governance nightmare. A better option is "content-addressed vocabulary"; instead of "https://www.w3.org/ns/activitystreams#Note" we could instead simply take the text from the standard:

"Represents a short written work typically less than a single paragraph in length."

Hash that and you get "urn:sha256:54c14cbd844dc9ae3fa5f5f7b8c1255ee32f55b8afaba88ce983a489155ac398". No governance or liveness issues required. (Hashing mechanism upgrades, however, do pose some challenge; mapping old hashes to new ones for equivalence can be a partial solution.)

This seems sufficient to me; groups can collaborate somewhere to hammer out the definition of some term, simply hash the definition of it, and use that as the terminology URI. This also avoids hazards from choosing a different edge of Zooko's Triangle for vocabulary.

Now that we have this, we can express advanced new ideas across the network and experiment with new terms. Better yet, we might be even able to use our propagator networks to associate ideas with them. I think in many systems, content-addressed-vocabulary could be a good way to describe beliefs that could be considered, accepted, rejected in truth maintenance systems.

Cerealize me, cap'n!

One observation from Agoric is that it is possible to treat systems that do not resemble traditional live actor'y vats still as vats (and "machines") and develop semantics for message passing between them (and performing promise resolution) nonetheless, for instance blockchains.

Similarly, above we have observed that propagator systems can be built on top of actors; I believe it is also possible to describe propagator networks in terms of pure data. It should be possible to describe changes to a propagator network as a standard serialized ledger that can be transferred from place to place or reproduced.

However, the fact that interoperability with actors is possible is good, desirable, and thankfully a nice transitional place for experimentation (porting propagator model semantics to Spritely Goblins should not be hard).

Where to from here?

That's a lot of ideas above, but how likely is any of this stuff to be usable soon? I'm not anticipating dropping any current work to try to make this happen, but I probably will be experimenting in my upcoming UI work to try to have the UI powered by a propagator system (possibly even a stripped down version) so that the experimental seeds are in place to see if such a system can be grown. But I am not anticipating that we'll see anything like a fully distributed propagator system doing something interesting from my own network soon... but sometimes I end up surprised.

Closing the loop

I mentioned before that human brains are a combination of faster intuitive methods (resembling current work on neural nets) and slower, more calculating reasoning systems (resembling propagators or some logic programming languages). That's also to say nothing about the giant emotional soup that a mind/body tends to live in.

Realistically the emergence of a fully sapient system won't involve any of these systems independently, but rather a networked interconnection of many of them. I think the vat model of execution is a nice glue system for it; pulling propagators into the system could bring us one step closer, maybe.

Or maybe it's all just fantastical dreaming! Who knows. But it could be interesting to play and find out at some point... perhaps some day we can indeed get a proper brain into a vat.

16 February, 2020 09:25PM by Christopher Lemmer Webber

February 10, 2020

State of Spritely for February 2020

We are now approximately 50% of the way through the Samsung Stack Zero grant for Spritely, and only a few months more since I announced the Spritely project at all. I thought this would be a good opportunity to review what has happened so far and what's on the way.

In my view, quite a lot has happened over the course of the last year:

  • Datashards grew out of two Spritely projects, Magenc and Crystal. This provides the "secure storage layer" for the system, and by moving into Datashards has even become its own project (now mostly under the maintainership of Serge Wroclawski, who as it turns out is also co-host with me of Libre Lounge. There's external interest in this from the rest of the federated social web, and it was a topic of discussion in the last meeting of the SocialCG. While not as publicly visible recently, the project is indeed active; I am currently helping advise and assist Serge with some of the ongoing work on optimizations for smaller files, fixing the manifest format to permit larger files, and a more robust HTTP API for stores/registries. (Thank you Serge also for taking on a large portion of this work and responsibility!)

  • Spritely Goblins, the actor model layer of Spritely, continues its development. We are now up to release v0.5. I don't consider the API to be stable, but it is stabilizing. In particular, the object/update model, the synchronous communication layer, and the transactional update support are all very close to stable. Asynchronous programming mostly works but has a few bugs I need to work out, and the distributed programming environment design is coming together enough where I expect to be able to demo it soon.

  • In addition, I have finally started to write docs for Spritely Goblins. I think the tutorial above is fairly nice, and I've had a good amount of review from various parties, and those who have tried it seem to think it is fairly nice. (Please be advised that it requires working with the dev branch of Goblins at the time of writing.) v0.6 should the first release to have documentation after the major overhaul I did last summer (effectively an entire rewrite of the system, including many changes to the design after doing research into ocap practices). I cannot recommend that anyone else write production-level code using the system yet, but I hope that by the summer things will have congealed enough that this will change.

  • I have made a couple of publicly visible demos of Goblins' design. Weirdly enough all of these have involved ascii art.

    • The proto-version was the Let's Just Be Weird Together demo. Actually it's a bit strange to say this because the LJBWT demo didn't use Goblins, it used a library called DOS/HURD. However, writing this library (and adapting it from DOS/Win) directly informed the rewrite of Goblins, Goblinoid which eventually became Goblins itself, replacing all the old code. This is why I advocate demo-driven-development: the right design of an architecture flows out of a demo of it. (Oh yeah, and uh, it also allowed me to make a present for my 10th wedding anniversary, too.)

    • Continuing in a similar vein, I made the "Season's Greetings" postcard, which Software Freedom Conservancy actually used in their funding campaign this year. This snowy scene used the new rewrite of Goblins and allowed me to try to push the new "become" feature of Goblins to its limit (the third principle of actor model semantics, taken very literally). It wasn't really obvious to anyone else that this was using Goblins in any interesting way, but I'll say that writing this really allowed me to congeal many things about the update layer and it also lead to uncovering a performance problem, leading to a 10x speedup. Having written this demo, I was starting to get the hang of things in the Goblins synchronous layer.

    • Finally there was the Terminal Phase demo. (See the prototype announcement blogpost and the 1.0 announcement.) This was originally designed as a reward for donors for hitting $500/mo on my Patreon account (you can still show up in the credits by donating!), though once 1.0 made it out the door it seems like it raised considerable excitement on the r/linux subreddit and on Hacker News, which was nice to see. Terminal Phase helped me finish testing and gaining confidence in the transactional object-update and synchronous call semantics of Spritely Goblins, and I now have no doubt that this layer has a good design. But I think Terminal Phase was the first time that other people could see why Spritely Goblins was exciting, especially once I showed off the time travel debugging in Terminal Phase demo. That last post lead people to finally start pinging me asking "when can I use Spritely Goblins"? That's good... I'm glad it's obvious now that Goblins is doing something interesting (though the most interesting things are yet to be demo'ed).

  • I participated in, keynoted, and drummed up enthusiasm for ActivityPub Conference 2019. (I didn't organize though, that was Morgan Lemmer-Webber's doing, alongside Sebastian Lasse and with DeeAnn Little organizing the video recording.) We had a great speaker list and even got Mark S. Miller to keynote. Videos of the event are also available. While that event was obviously much bigger than Spritely, the engagement of the ActivityPub community is obviously important for its success.

  • Relatedly, I continue to co-chair the SocialCG but Nightpool has joined as co-chair which should relieve some pressure there, as I was a bit too overloaded to be able to handle this all on my own. The addition of the SocialHub community forum has also allowed the ActivityPub community to be able to coordinate in a way that does not rely on me being a blocker. Again, not Spritely related directly, but the health of the ActivityPub community is important to Spritely's success.

  • At Rebooting Web of Trust I coordinated with a number of contributors (including Mark Miller) on sketching out plans for secure UI designs. Sadly the paper is incomplete but has given me the framework for understanding the necessary UI components for when we get to the social network layer of Spritely.

  • Further along the lines of sketching out the desiderata of federated social networks, I have written a nearly-complete OcapPub: towards networks of consent. However, there are still some details to be figured out; I have been hammering them out on the cap-talk mailing list (see this post laying out a very ocappub-like design with some known problems, and then this analysis). The ocap community has thankfully been very willing to participate in working with me to hammer out the right security foundations, and I think we're close to the right design details. Of course, the proof of the pudding is in the demo, which has yet to be written.

Okay, so I hope I've convinced you that a lot has happened, and hopefully you feel that I am using my time reasonably well. But there is much, much, much ahead for Spritely to succeed in its goals. So, what's next?

  • I need to finish cleaning up the Goblins documentation and do a v0.6 release with it included. At that point I can start recommending some brave souls to use it for some simple applications.

  • A demo of Spritely Goblins working in a primarily asynchronous environment. This might simply be a port of mudsync as a first step. (Recorded demo of mudsync from a few years ago.) I'm not actually sure. The goal of this isn't to be the "right" social network design (not full OcapPub), just to test the async behaviors of Spritely Goblins. Like the synchronous demos that have already been done, the purpose of this is to congeal and ensure the quality of the async primitives. I expect this and the previous bullet point to be done within the next couple of months, so hopefully by the end of April.

  • Distributed networked programming in Goblins, and associated demo. May expand on the previous demo. Probably will come out about two months later, so end of June.

  • Prototype of the secure UI concepts from the forementioned secure UIs paper. I expect/hope this to be usable by end of third quarter 2020.

  • Somewhere in-between all this, I'd like to add a demo of being able to securely run untrusted code from third parties, maybe in the MUD demo. Not sure when yet.

  • All along, I continue to expect to push out new updates to Terminal Phase with more fun enemies and powerups to continue to reward donors to the Patreon campaign.

This will probably take most of this year. What you will notice is that this does not explicitly state a tie-in with the ActivityPub network. This is intentional, because the main goal of all the above demos are to prove more foundational concepts before they are all fully integrated. I think we'll see the full integration and it coming together with the existing fediverse beginning in early 2021.

Anyway, that's a lot of stuff ahead. I haven't even mentioned my involvement in Libre Lounge, which I've been on hiatus from due to a health issue that has made recording difficult, and from being busy trying to deliver on these foundations, but I expect to be coming back to LL shortly.

I hope I have instilled you with some confidence that I am moving steadily along the abstract Spritely roadmap. (Gosh, I ought to finally put together a website for Spritely, huh?) Things are happening, and interesting ones I think.

But how do you think things are going? Maybe you would like to leave me feedback. If so, feel free to reach out.

Until next time...

10 February, 2020 09:30PM by Christopher Lemmer Webber

February 09, 2020

Andy Wingo

state of the gnunion 2020

Greetings, GNU hackers! This blog post rounds up GNU happenings over 2019. My goal is to celebrate the software we produced over the last year and to help us plan a successful 2020.

Over the past few months I have been discussing project health with a group of GNU maintainers and we were wondering how the project was doing. We had impressions, but little in the way of data. To that end I wrote some scripts to collect dates and versions for all releases made by GNU projects, as far back as data is available.

In 2019, I count 243 releases, from 98 projects. Nice! Notably, on ftp.gnu.org we have the first stable releases from three projects:

GNU Guix
GNU Guix is perhaps the most exciting project in GNU these days. It's a package manager! It's a distribution! It's a container construction tool! It's a package-manager-cum-distribution-cum-container-construction-tool! Hearty congratulations to Guix on their first stable release.
GNU Shepherd
The GNU Daemon Shepherd is a modern dependency-based init service, written in Guile Scheme, and used in Guix. When you install Guix as an operating system, it actually stages Scheme programs from the operating system definition into the Shepherd configuration. So cool!
GNU Backgammon
Version 1.06.002 is not GNU Backgammon's first stable release, but it is the earliest version which is available on ftp.gnu.org. Formerly hosted on the now-defunct gnubg.org, GNU Backgammon is a venerable foe, and uses neural networks since before they were cool. Welcome back, GNU Backgammon!

The total release counts above are slightly above what Mike Gerwitz's scripts count in his "GNU Spotlight", posted on the FSF blog. This could be because in addition to files released on ftp.gnu.org, I also manually collected release dates for most packages that upload their software somewhere other than gnu.org. I don't count alpha.gnu.org releases, and there were a handful of packages for which I wasn't successful at retrieving their release dates. But as a first approximation, it's a relatively complete data set.

I put my scripts in git repository if anyone is interested in playing with the data. Some raw CSV files are there as well.

where we at?

Hair toss, check my nails, baby how you GNUing? Hard to tell!

To get us closer to an answer, I calculated the active package count per year. There can be other definitions, but my reading is that an active package is one that has had a stable release within the preceding 3 calendar years. So for 2019, for example, a GNU package is considered active if it had a stable release in 2017, 2018, or 2019. What I got was a graph that looks like this:

What we see is nothing before 1991 -- surely pointing to lacunae in my data set -- then a more or less linear rise in active package count until 2002, some stuttering growth rising to a peak in 2014 at 208 active packages, and from there a steady decline down to 153 active packages in 2019.

Of course, as a metric, active package count isn't precisely the same as project health; GNU ed is indeed the standard editor but it's not GCC. But we need to look for measurements that indirectly indicate project health and this is what I could come up with.

Looking a little deeper, I tabulated the first and last release date for each GNU package, and then grouped them by year. In this graph, the left blue bars indicate the number of packages making their first recorded release, and the right green bars indicate the number of packages making their last release. Obviously a last release in 2019 indicates an active package, so it's to be expected that we have a spike in green bars on the right.

What this graph indicates is that GNU had an uninterrupted growth phase from its beginning until 2006, with more projects being born than dying. Things are mixed until 2012 or so, and since then we see many more projects making their last release and above all, very few packages "being born".

where we going?

I am not sure exactly what steps GNU should take in the future but I hope that this analysis can be a good conversation-starter. I do have some thoughts but will post in a follow-up. Until then, happy hacking in 2020!

09 February, 2020 07:44PM by Andy Wingo

GNU Guix

Outreachy May 2020 to August 2020 Status Report I

We are happy to announce that for the fourth time GNU Guix offers a three-month internship through Outreachy, the inclusion program for groups traditionally underrepresented in free software and tech. We currently propose three subjects to work on:

  1. Create Netlink bindings in Guile.
  2. Improve internationalization support for the Guix Data Service.
  3. Integration of desktop environments into GNU Guix.

The initial application deadline is on Feb. 25, 2020 at 4PM UTC.

The final project list is announced on Feb. 25, 2020.

Should you have any questions regarding the internship, please check out the timeline, information about the application process, and the eligibility rules.

If you’d like to contribute to computing freedom, Scheme, functional programming, or operating system development, now is a good time to join us. Let’s get in touch on the mailing lists and on the #guix channel on the Freenode IRC network!

Last year we had the pleasure to welcome Laura Lazzati as an Outreachy intern working on documentation video creation, which led to the videos you can now see on the home page.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

09 February, 2020 02:30PM by Gábor Boskovits

February 07, 2020

Andy Wingo

lessons learned from guile, the ancient & spry

Greets, hackfolk!

Like just about every year, last week I took the train up to Brussels for FOSDEM, the messy and wonderful carnival of free software and of those that make it. Mostly I go for the hallway track: to see old friends, catch up, scheme about future plans, and refill my hacker culture reserves.

I usually try to see if I can get a talk or two in, and this year was no exception. First on my mind was the recent release of Guile 3. This was the culmination of a 10-year plan of work and so obviously there are some things to say! But at the same time, I wanted to reflect back a bit and look at the past with a bit of distance.

So in the end, my one talk was two talks. Let's start with the first one. (I'm trying a new thing where I share my talks as blog posts. We'll see how this goes. I know the rendering can be a bit off relative to the slides, but hopefully it's good enough. If you prefer, you can just watch the video instead!)

Celebrating Guile 3

FOSDEM 2020, Brussels

Andy Wingo | wingo@igalia.com

wingolog.org | @andywingo

So yeah let's celebrate! I co-maintain the Guile implementation of Scheme. It's a programming language. Guile 3, in summary, is just Guile, but faster. We added a simple just-in-time compiler as well as a bunch of ahead-of-time optimizations. The result is that it runs faster -- sometimes by a lot!

In the image above you can see Guile 3's performance on a number of microbenchmarks, relative to Guile 2.2, sorted by speedup. The baseline is 1.0x as fast. You can see that besides the first couple microbenchmarks where things are a bit inconclusive (click for full-size image), everything gets faster. Most are at least 2x as fast, and one benchmark is even 32x as fast. (Note the logarithmic scale on the Y axis.)

I only took a look at microbenchmarks at the end of the Guile 3 series; before that, I was mostly going by instinct. It's a relief to find out that in this case, my instincts did align with improvement.

mini-benchmark: eval

(primitive-eval
 ’(let fib ((n 30))
    (if (< n 2)
        n
        (+ (fib (- n 1)) (fib (- n 2))))))

Guile 1.8: primitive-eval written in C

Guile 2.0+: primitive-eval in Scheme

Taking a look at a more medium-sized benchmark, let's compute the 30th fibonacci number, but using the interpreter instead of compiling the procedure. In Guile 2.0 and up, the interpreter (primitive-eval) is implemented in Scheme, so it's a good test of an important small Scheme program.

Before 2.0, though, primitive-eval was actually implemented in C. This had a number of disadvantages, notably that it prevented tail calls between interpreted and compiled code. When we switched to a Scheme implementation of primitive-eval, we knew we would have a performance hit, but we thought that we would gain it back eventually as the compiler got better.

As you can see, it took a while before the compiler and run-time improved to the point that primitive-eval in Scheme reached the speed of its old hand-tuned C implementation, but for Guile 3, we finally got there. Note again the logarithmic scale on the Y axis.

macro-benchmark: guix

guix build libreoffice ghc-pandoc guix \
  –dry-run --derivation

7% faster

guix system build config.scm \
  –dry-run --derivation

10% faster

Finally, taking a real-world benchmark, the Guix package manager is implemented entirely in Scheme. All ten thousand packages are defined in Scheme, the building scripts are in Scheme, the initial RAM disk is in Scheme -- you get the idea. Guile performance in Guix can have an important effect on user experience. As you can see, Guile 3 lowered elapsed time for some operations by around 10 percent or so. Of course there's a lot of I/O going on in addition to computation, so Guile running twice as fast will rarely make Guix run twice as fast (Amdahl's law and all that).

spry /sprī/

  • adjective: active; lively

So, when I was thinking about words that describe Guile, the word "spry" came to mind.

spry /sprī/

  • adjective: (especially of an old person) active; lively

But actually when I went to look up the meaning of "spry", Collins Dictionary says that it especially applies to the agèd. At first I was a bit offended, but I knew in my heart that the dictionary was right.

Lessons Learned from Guile, the Ancient & Spry

FOSDEM 2020, Brussels

Andy Wingo | wingo@igalia.com

wingolog.org | @andywingo

That leads me into my second talk.

guile is ancient

2010: Rust

2009: Go

2007: Clojure

1995: Ruby

1995: PHP

1995: JavaScript

1993: Guile (33 years before 3.0!)

It's common for a new project to be lively, but Guile is definitely not new. People have been born, raised, and earned doctorates in programming languages in the time that Guile has been around.

built from ancient parts

1991: Python

1990: Haskell

1990: SCM

1989: Bash

1988: Tcl

1988: SIOD

Guile didn't appear out of nothing, though. It was hacked up from the pieces of another Scheme implementation called SCM, which itself was initially based on Scheme in One Defun (SIOD), back before the Berlin Wall fell.

written in an ancient language

1987: Perl

1984: C++

1975: Scheme

1972: C

1958: Lisp

1958: Algol

1954: Fortran

1958: Lisp

1930s: λ-calculus (34 years ago!)

But it goes back further! The Scheme language, of which Guile is an implementation, dates from 1975, before I was born; and you can, if you choose, trace the lines back to the lambda calculus, created in mid-30s as a notation for computation. I suppose at this point I should say mid-2030s, to disambiguate.

The point is, Guile is old! Statistically, most software projects from olden times are now dead. How has Guile managed to survive and (sometimes) thrive? Surely there must be some lesson or other that can be learned here.

ancient & spry

Men make their own history, but they do not make it as they please; they do not make it under self-selected circumstances, but under circumstances existing already, given and transmitted from the past.

The tradition of all dead generations weighs like a nightmare on the brains of the living. [...]

Eighteenth Brumaire of Louis Bonaparte, Marx, 1852

I am no philospher of history, but I know that there are some ways of looking at the past that do not help me understand things. One is the arrow of enlightened progress, in which events exist in a causal chain, each producing the next. It doesn't help me understand the atmosphere, tensions, and possibilities inherent at any particular point. I find the "progress" theory of history to be an extreme form of selection bias.

Much more helpful to me is the Hegelian notion of dialectics: that at an given point in time there are various tensions at work. In our field, an example could be memory safety versus systems programming. These tensions create an environment that favors actions that lead towards resolution of the tensions. It doesn't mean that there's only one way to resolve the tensions, and it's not an automatic process -- people still have to do things. But the tendency is to ratchet history forward to a new set of tensions.

The history of a project, to me, is then a process of dialectic tensions and resolutions. If the project survives, as Guile has, then it should teach us something about the way this process works in practice.

ancient & spry

Languages evolve; how to remain minimal?

Dialectic opposites

  • world and guile

  • stable and active

  • ...

Lessons learned from inside Hegel’s motor of history

One dialectic is the tension between the world's problems and what tools Guile offers to understand and solve them. In 1993, the web didn't really exist. In 2033, if Guile doesn't run well in a web browser, probably it will be dead. But this process operates very slowly, for an old project; Guile isn't built on CORBA or something ephemeral like that, so we don't have very much data here.

The tension between being a stable base for others to build on, and in being a dynamic project that improves and changes, is a key tension that this talk investigates.

In the specific context of Guile, and for the audience of the FOSDEM minimal languages devroom, we should recognize that for a software project, age and minimalism don't necessarily go together. Software gets features over time and becomes bigger. What does it mean for a minimal language to evolve?

hill-climbing is insufficient

Ex: Guile 1.8; Extend vs Embed

One key lesson that I have learned is that the strategy of making only incremental improvements is a recipe for death, in the long term. The natural result is that you reach what you perceive to be the most optimal state of your project. Any change can only make it worse, so you stop moving.

This is what happened to Guile around version 1.8: we had taken the paradigm of the interpreter as language implementation strategy as far as it could go. There were only around 150 commits to Guile in 2007. We were stuck.

users stay unless pushed away

Inertial factor: interface

  • Source (API)

  • Binary (ABI)

  • Embedding (API)

  • CLI

  • ...

Ex: Python 3; local-eval; R6RS syntax; set!, set-car!

So how do we make change, in such a circumstance? You could start a new project, but then you wouldn't have any users. It would be nice to change and keep your users. Fortunately, it turns out that users don't really go away; yes, they trickle out if you don't do anything, but unless you change in an incompatible way, they stay with you, out of inertia.

Inertia is good and bad. It does conflict with minimalism as a principle; if you were to design Scheme in 2020, you would not include mutable variables or even mutable pairs. But they are still with us because if we removed them, we'd break too many users.

Users can even make you add back things that you had removed. In Guile 2.0, we removed the capability to evaluate an expression at run-time within the lexical environment of an expression, as we didn't know how to implement this outside an interpreter. It turns out this was so important to users that we had to add local-eval back to Guile, later in the 2.0 series. (Fortunately we were able to do it in a way that layered on lower-level facilities; this approach reconciled me to the solution.)

you can’t keep all users

What users say: don’t change or remove existing behavior

But: sometimes losing users is OK. Hard to know when, though

No change at all == death

  • Natural result of hill-climbing

Ex: psyntax; BDW-GC mark & finalize; compile-time; Unicode / locales

Unfortunately, the need to change means that sometimes you will lose users. It's either a dead project, or losing users.

In Guile 1.8, for example, the macro expander ran lazily: it would only expand code the first time it ran it. This was good for start-up time, because not all code is evaluated in the course of a simple script. Lazy expansion allowed us to start doing important work sooner. However, this approach caused immense pain to people that wanted "proper" Scheme macros that preserved lexical scoping; the state of the art was to eagerly expand an entire file. So we switched, and at the same time added a notion of compile-time. This compromise kept good start-up time while allowing fancy macros.

But eager expansion was a change. Users that relied on side effects from macro expansion would see them at compile-time instead of run-time. Users of old "defmacros" that could previously splice in live Scheme closures as literals in expanded source could no longer do that. I think it was the right choice but it did lose some users. In fact I just got another bug report related to this 10-year-old change last week.

every interface is a cost

Guile binary ABI: libguile.so; compiled Scheme files

Make compatibility easier: minimize interface

Ex: scm_sym_unquote, GOOPS, Go, Guix

So if you don't want to lose users, don't change any interface. The easiest way to do this is to minimize your interface surface. In Go, for example, they mostly haven't had dynamic-linking problems because that's not a thing they do: all code is statically linked into binaries. Similarly, Guix doesn't define a stable API, because all of its code is maintained in one "monorepo" that can develop in lock-step.

You always have some interfaces, though. For example Guix can't change its command-line interface from one day to the next, for example, because users would complain. But it's been surprising to me the extent to which Guile has interfaces that I didn't consider. Recently for example in the 3.0 release, we unexported some symbols by mistake. Users complained, so we're putting them back in now.

parallel installs for the win

Highly effective pattern for change

  • libguile-2.0.so

  • libguile-3.0.so

https://ometer.com/parallel.html

Changed ABI is new ABI; it should have a new name

Ex: make-struct/no-tail, GUILE_PKG([2.2]), libtool

So how does one do incompatible change? If "don't" isn't a sufficient answer, then parallel installs is a good strategy. For example in Guile, users don't have to upgrade to 3.0 until they are ready. Guile 2.2 happily installs in parallel with Guile 3.0.

As another small example, there's a function in Guile called make-struct (old doc link), whose first argument is the number of "tail" slots, followed by initializers for all slots (normal and "tail"). This tail feature is weird and I would like to remove it. Unfortunately I can't just remove the argument, so I had to make a new function, make-struct/no-tail, which exists in parallel with the old version that I can't break.

deprecation facilitates migration

__attribute__ ((__deprecated__))
(issue-deprecation-warning
 "(ice-9 mapping) is deprecated."
 "  Use srfi-69 or rnrs hash tables instead.")
scm_c_issue_deprecation_warning
  ("Arbiters are deprecated.  "
   "Use mutexes or atomic variables instead.");

begin-deprecated, SCM_ENABLE_DEPRECATED

Fortunately there is a way to encourage users to migrate from old interfaces to new ones: deprecation. In Guile this applies to all of our interfaces (binary, source, etc). If a feature is marked as deprecated, we cause its use to issue a warning, ideally at compile-time when users responsible for the package can fix it. You can even add __attribute__((__deprecated__)) on C types!

the arch-pattern

Replace, Deprecate, Remove

All change is possible; question is only length of deprecation period

Applies to all interfaces

Guile deprecation period generally one stable series

Ex: scm_t_uint8; make-struct; Foreign objects; uniform vectors

Finally, you end up in a situation where you have replaced the old interface and issued deprecation warnings to help users migrate. The next step is to remove the old interface. If you don't do this, you are failing as a project maintainer -- your project becomes literally unmaintainable as it just grows and grows.

This strategy applies to all changes. The deprecation period may last a while, and it may be that the replacement you built doesn't serve the purpose. There is still a dialog with the users that needs to happen. As an example, I made a replacement for the "SMOB" facility in Guile that allows users to define new types, backed by C interfaces. This new "foreign object" facility might not actually be good enough to replace SMOBs; since I haven't formally deprecatd SMOBs, I don't know yet because users are still using the old thing!

change produces a new stable point

Stability within series: only additions

Corollary: dependencies must be at least as stable as you!

  • for your definition of stable

  • social norms help (GNU, semver)

Ex: libtool; unistring; gnulib

In my experience, the old management dictum that "the only constant is change" does not describe software. Guile changes, then it becomes stable for a while. You need an unstable series escape hill-climbing, then once you found your new hill, you start climbing again in the stable series.

Once you reach your stable point, the projects you rely on need to exhibit the same degree of stability that you envision for your project. You can't build a web site that you expect to maintain for 10 years on technology that fundamentally changes every 6 months. But stable dependencies isn't something you can ensure technically; rather it relies on social norms of who makes the software you use.

who can crank the motor of history?

All libraries define languages

Allow user to evolve the language

  • User functionality: modules (Guix)

  • User syntax: macros (yay Scheme)

Guile 1.8 perf created tension

  • incorporate code into Guile

  • large C interface “for speed”

Compiler removed pressure on C ABI

Empowered users need less from you

A dialectic process does not progress on its own: it requires actions. As a project maintainer, some of my actions are because I want to do them. Others are because users want me to do them. The user-driven actions are generally a burden and as a lazy maintainer, I want to minimize them.

Here I think Guile has to a large degree escaped some of the pressures that weigh on other languages, for example Python. Because Scheme allows users to define language features that exist on par with "built-in" features, users don't need my approval or intervention to add (say) new syntax to the language they work in. Furthermore, their work can still compose with the work of others, even if the others don't buy in to their language extensions.

Still, Guile 1.8 did have a dynamic whereby the relatively poor performance of having to run all code through primitive-eval meant that users were pushed towards writing extensions in C. This in turn pushed Guile to expose all of its guts for access from C, which obviously has led to an overbloated C API and ABI. Happily the work on the Scheme compiler has mostly relieved this pressure, and we may therefore be able to trim the size of the C API and ABI over time.

contributions and risk

From maintenance point of view, all interface is legacy

Guile: Sometimes OK to accept user modules when they are more stable than Guile

In-tree users keep you honest

Ex: SSAX, fibers, SRFI

It can be a good strategy to "sediment" solutions to common use cases into Guile itself. This can improve the minimalism of an entire ecosystem of code. The maintenance burden has to be minimal, however; Guile has sometimes adopted experimental code into its repository, and without active maintenance, it soon becomes stale relative to what users and the module maintainers expect.

I would note an interesting effect: pieces of code that were adopted into Guile become a snapshot of the coding style at that time. It's useful to have some in-tree users because it gives you a better idea about how a project is seen from the outside, from a code perspective.

sticky bits

Memory management is an ongoing thorn

Local maximum: Boehm-Demers-Weiser conservative collector

How to get to precise, generational GC?

Not just Guile; e.g. CPython __del__

There are some points that resist change. The stickiest of these is the representation of heap-allocated Scheme objects in C. Guile currently uses a garbage collector that "automatically" finds all live Scheme values on the C stack and in registers. It was the right choice at the time, given our maintenance budget. But to get the next bump in performance, we need to switch to a generational garbage collector. It's hard to do that without a lot of pain to C users, essentially because the C language is too weak to express the patterns that we would need. I don't know how to proceed.

I would note, though, that memory management is a kind of cross-cutting interface, and that it's not just Guile that's having problems changing; I understand PyPy has had a lot of problems regarding changes on when Python destructors get called due to its switch from reference counting to a proper GC.

future

We are here: stability

And then?

  • Parallel-installability for source languages: #lang

  • Sediment idioms from Racket to evolve Guile user base

Remove myself from “holding the crank”

So where are we going? Nowhere, for the moment; or rather, up the hill. We just released Guile 3.0, so let's just appreciate that for the time being.

But as far as next steps in language evolution, I think in the short term they are essentially to further enable change while further sedimenting good practices into Guile. On the change side, we need parallel installability for entire languages. Racket did a great job facilitating this with #lang and we should just adopt that.

As for sedimentation, we should step back and if any common Guile use patterns built by our users should be include core Guile, and widen our gaze to Racket also. It will take some effort both on a technical perspective but also on a social/emotional consensus about how much change is good and how bold versus conservative to be: putting the dialog into dialectic.

dialectic, boogie woogie woogie

https://gnu.org/s/guile

https://wingolog.org/

#guile on freenode

@andywingo

wingo@igalia.com

Happy hacking!

Hey that was the talk! Hope you enjoyed the writeup. Again, video and slides available on the FOSDEM web site. Happy hacking!

07 February, 2020 11:38AM by Andy Wingo

February 05, 2020

screen @ Savannah

GNU Screen v.4.8.0

I'm announcing availability of GNU Screen v.4.8.0

Screen is a full-screen window manager that multiplexes a physical
terminal between several processes, typically interactive shells.

This release
  * Improves startup time by only polling for already open files to
    close
  * Fixes:
       - Fix for segfault if termcap doesn't have Km entry
       - Make screen exit code be 0 when checking --version
       - Fix potential memory corruption when using OSC 49

As last fix, fixes potential memory overwrite of quite big size (~768
bytes), and even though I'm not sure about potential exploitability of
that issue, I highly recommend everyone to upgrade as soon as possible.
This issue is present at least since v.4.2.0 (haven't checked earlier).
Thanks to pippin who brought this to my attention.

For full list of changes see
https://git.savannah.gnu.org/cgit/screen.git/log/?h=v.4.8.0

Release is available for download at:
https://ftp.gnu.org/gnu/screen/
or your closest mirror (may have some delay)
https://ftpmirror.gnu.org/screen/

Please report any bugs or regressions.

05 February, 2020 08:48PM by Amadeusz Sławiński

February 02, 2020

Applied Pokology

Hyperlink Support in GNU Poke

FOSDEM 2020 is over, and hyperlink support has just landed for GNU Poke!

Wait, Hyperlinks!?

What do hyperlinks, a web concept, mean for GNU Poke, a terminal application?

For many years now, terminal emulators have been detecting http:// URLs in the output of any program and giving the user a chance to click on them and immediately navigate to the corresponding web page. In 2017, Egmont Kob made a proposal for supporting general hyperlinks in terminal emulators. Gnome Terminal, iTerm and a few other terminal emulators have already implemented this proposal in their latest releases. With Egmont's proposal, an application can emit any valid URI and have the terminal emulator take the user to that resource.

02 February, 2020 12:00AM

February 01, 2020

libc @ Savannah

The GNU C Library version 2.31 is now available

The GNU C Library
=================

The GNU C Library version 2.31 is now available.

The GNU C Library is used as the C library in the GNU system and
in GNU/Linux systems, as well as many other systems that use Linux
as the kernel.

The GNU C Library is primarily designed to be a portable
and high performance C library.  It follows all relevant
standards including ISO C11 and POSIX.1-2017.  It is also
internationalized and has one of the most complete
internationalization interfaces known.

The GNU C Library webpage is at http://www.gnu.org/software/libc/

Packages for the 2.31 release may be downloaded from:
        http://ftpmirror.gnu.org/libc/
        http://ftp.gnu.org/gnu/libc/

The mirror list is at http://www.gnu.org/order/ftp.html

NEWS for version 2.31
=====================

Major new features:

  • The GNU C Library now supports a feature test macro _ISOC2X_SOURCE

  to enable features from the draft ISO C2X standard.  Only some
  features from this draft standard are supported by the GNU C
  Library, and as the draft is under active development, the set of
  features enabled by this macro is liable to change.  Features from
  C2X are also enabled by _GNU_SOURCE, or by compiling with "gcc
  -std=gnu2x".

  • The <math.h> functions that round their results to a narrower type

  now have corresponding type-generic macros in <tgmath.h>, as defined
  in TS 18661-1:2014 and TS 18661-3:2015 as amended by the resolution
  of Clarification Request 13 to TS 18661-3.

  • The function pthread_clockjoin_np has been added, enabling join with

  a terminated thread with a specific clock.  It allows waiting
  against CLOCK_MONOTONIC and CLOCK_REALTIME.  This function is a GNU
  extension.

  • New locale added: mnw_MM (Mon language spoken in Myanmar).
  • The DNS stub resolver will optionally send the AD (authenticated

  data) bit in queries if the trust-ad option is set via the options
  directive in /etc/resolv.conf (or if RES_TRUSTAD is set in
  _res.options).  In this mode, the AD bit, as provided by the name
  server, is available to applications which call res_search and
  related functions.  In the default mode, the AD bit is not set in
  queries, and it is automatically cleared in responses, indicating a
  lack of DNSSEC validation.  (Therefore, the name servers and the
  network path to them are treated as untrusted.)

Deprecated and removed features, and other changes affecting
compatibility:

  • The totalorder and totalordermag functions, and the corresponding

  functions for other floating-point types, now take pointer arguments
  to avoid signaling NaNs possibly being converted to quiet NaNs in
  argument passing.  This is in accordance with the resolution of
  Clarification Request 25 to TS 18661-1, as applied for C2X.
  Existing binaries that pass floating-point arguments directly will
  continue to work.

  • The obsolete function stime is no longer available to newly linked

  binaries, and its declaration has been removed from <time.h>.
  Programs that set the system time should use clock_settime instead.

  • We plan to remove the obsolete function ftime, and the header

  <sys/timeb.h>, in a future version of glibc.  In this release, the
  header still exists but calling ftime will cause a compiler warning.
  All programs should use gettimeofday or clock_gettime instead.

  • The gettimeofday function no longer reports information about a

  system-wide time zone.  This 4.2-BSD-era feature has been deprecated
  for many years, as it cannot handle the full complexity of the
  world's timezones, but hitherto we have supported it on a
  best-effort basis.  Changes required to support 64-bit time_t on
  32-bit architectures have made this no longer practical.

  As of this release, callers of gettimeofday with a non-null 'tzp'
  argument should expect to receive a 'struct timezone' whose
  tz_minuteswest and tz_dsttime fields are zero.  (For efficiency
  reasons, this does not always happen on a few Linux-based ports.
  This will be corrected in a future release.)

  All callers should supply a null pointer for the 'tzp' argument to
  gettimeofday.  For accurate information about the time zone
  associated with the current time, use the localtime function.

  gettimeofday itself is obsolescent according to POSIX.  We have no
  plans to remove access to this function, but portable programs
  should consider using clock_gettime instead.

  • The settimeofday function can still be used to set a system-wide

  time zone when the operating system supports it.  This is because
  the Linux kernel reused the API, on some architectures, to describe
  a system-wide time-zone-like offset between the software clock
  maintained by the kernel, and the "RTC" clock that keeps time when
  the system is shut down.

  However, to reduce the odds of this offset being set by accident,
  settimeofday can no longer be used to set the time and the offset
  simultaneously.  If both of its two arguments are non-null, the call
  will fail (setting errno to EINVAL).

  Callers attempting to set this offset should also be prepared for
  the call to fail and set errno to ENOSYS; this already happens on
  the Hurd and on some Linux architectures.  The Linux kernel
  maintainers are discussing a more principled replacement for the
  reused API.  After a replacement becomes available, we will change
  settimeofday to fail with ENOSYS on all platforms when its 'tzp'
  argument is not a null pointer.

  settimeofday itself is obsolescent according to POSIX.  Programs
  that set the system time should use clock_settime and/or the adjtime
  family of functions instead.  We may cease to make settimeofday
  available to newly linked binaries after there is a replacement for
  Linux's time-zone-like offset API.

  • SPARC ISA v7 is no longer supported.  v8 is still supported, but

  only if the optional CAS instruction is implemented (for instance,
  LEON processors are still supported, but SuperSPARC processors are
  not).

  As the oldest 64-bit SPARC ISA is v9, this only affects 32-bit
  configurations.

  • If a lazy binding failure happens during dlopen, during the

  execution of an ELF constructor, the process is now terminated.
  Previously, the dynamic loader would return NULL from dlopen, with
  the lazy binding error captured in a dlerror message.  In general,
  this is unsafe because resetting the stack in an arbitrary function
  call is not possible.

  • For MIPS hard-float ABIs, the GNU C Library will be configured to

  need an executable stack unless explicitly configured at build time
  to require minimum kernel version 4.8 or newer.  This is because
  executing floating-point branches on a non-executable stack on Linux
  kernels prior to 4.8 can lead to application crashes for some MIPS
  configurations. While currently PT_GNU_STACK is not widely used on
  MIPS, future releases of GCC are expected to enable non-executable
  stack by default with PT_GNU_STACK by default and is thus likely to
  trigger a crash on older kernels.

  The GNU C Library can be built with --enable-kernel=4.8.0 in order
  to keep a non-executable stack while dropping support for older
  kernels.

  • System call wrappers for time system calls now use the new time64

  system calls when available. On 32-bit targets, these wrappers
  attempt to call the new system calls first and fall back to the
  older 32-bit time system calls if they are not present.  This may
  cause issues in environments that cannot handle unsupported system
  calls gracefully by returning -ENOSYS. Seccomp sandboxes are
  affected by this issue.

Changes to build and runtime requirements:

  • It is no longer necessary to have recent Linux kernel headers to

  build working (non-stub) system call wrappers on all architectures
  except 64-bit RISC-V.  64-bit RISC-V requires a minimum kernel
  headers version of 5.0.

  • The ChangeLog file is no longer present in the toplevel directory of

  the source tree.  ChangeLog files are located in the ChangeLog.old
  directory as ChangeLog.N where the highest N has the latest entries.

Security related changes:

  CVE-2019-19126: ld.so failed to ignore the LD_PREFER_MAP_32BIT_EXEC
  environment variable during program execution after a security
  transition, allowing local attackers to restrict the possible
  mapping addresses for loaded libraries and thus bypass ASLR for a
  setuid program.  Reported by Marcin Kościelnicki.

The following bugs are resolved with this release:

  [12031] localedata: iconv -t ascii//translit with Greek characters
  [15813] libc: Multiple issues in __gen_tempname
  [17726] libc: [arm, sparc] profil_counter should be compat symbol
  [18231] libc: ipc_perm struct's mode member has wrong type in
    sys/ipc.h
  [19767] libc: vdso is not used with static linking
  [19903] hurd: Shared mappings not being inherited by children
    processes
  [20358] network: RES_USE_DNSSEC sets DO; should also have a way to set
    AD
  [20839] dynamic-link: Incomplete rollback of dynamic linker state on
    linking failure
  [23132] localedata: Missing transliterations in Miscellaneous
    Mathematical Symbols-A/B Unicode blocks
  [23518] libc: Eliminate __libc_utmp_jump_table
  [24026] malloc: malloc_info() returns wrong numbers
  [24054] localedata: Many locales are missing date_fmt
  [24214] dynamic-link: user defined ifunc resolvers may run in ldd mode
  [24304] dynamic-link: Lazy binding failure during ELF
    constructors/destructors is not fatal
  [24376] libc: RISC-V symbol size confusion with _start
  [24682] localedata: zh_CN first weekday should be Monday per GB/T
    7408-2005
  [24824] libc: test-in-container does not install charmap files
    compatible with localedef
  [24844] regex: regex bad pointer / leakage if malloc fails
  [24867] malloc: Unintended malloc_info formatting changes
  [24879] libc: login: utmp alarm timer can arrive after lock
    acquisition
  [24880] libc: login: utmp implementation uses struct flock with
    fcntl64
  [24882] libc: login: pututline uses potentially outdated cache
  [24899] libc: Missing nonstring attributes in <utmp.h>, <utmpx.h>
  [24902] libc: login: Repeating pututxline on EINTR/EAGAIN causes stale
    utmp entries
  [24916] dynamic-link: [MIPS] Highest EI_ABIVERSION value not raised to
    ABSOLUTE ABI
  [24930] dynamic-link: dlopen of PIE executable can result in
    _dl_allocate_tls_init assertion failure
  [24950] localedata: Top-of-tree glibc does not build with top-of-tree
    GCC (stringop-overflow error)
  [24959] time: librt IFUNC resolvers for clock_gettime and clock_*
    functions other  can lead to crashes
  [24967] libc: jemalloc static linking causes runtime failure
  [24986] libc: alpha: new getegid, geteuid and getppid syscalls used
    unconditionally
  [25035] libc: sbrk() failure handled poorly in tunables_strdup
  [25087] dynamic-link: ldconfig mishandles unusual .dynstr placement
  [25097] libc: new -Warray-bounds with GCC 10
  [25112] dynamic-link: dlopen must not make new objects accessible when
    it still can fail with an error
  [25139] localedata: Please add the new mnw_MM locale
  [25149] regex: Array bounds violation in proceed_next_node
  [25157] dynamic-link: Audit cookie for the dynamic loader is not
    initialized correctly
  [25189] libc: glibc's __glibc_has_include causes issues with clang
    -frewrite-includes
  [25194] malloc: malloc.c: do_set_mxfast incorrectly casts the mallopt
    value to an unsigned
  [25204] dynamic-link: LD_PREFER_MAP_32BIT_EXEC not ignored in setuid
    binaries (CVE-2019-19126)
  [25225] libc: ld.so fails to link on x86 if GCC defaults to -fcf-
    protection
  [25226] string: strstr: Invalid result if needle crosses page on s390-
    z15 ifunc variant.
  [25232] string: <string.h> does not enable const correctness for
    strchr et al. for Clang++
  [25233] localedata: Consider "." as the thousands separator for sl_SI
    (Slovenian)
  [25241] nptl: __SIZEOF_PTHREAD_MUTEX_T defined twice for x86
  [25251] build: Failure to run tests when CFLAGS contains -DNDEBUG.
  [25271] libc: undeclared identifier PTHREAD_MUTEX_DEFAULT when
    compiling with -std=c11
  [25323] localedata: km_KH: d_t_fmt contains "m" instead of "%M"
  [25324] localedata: lv_LV: d_t_fmt contains suspicious words in the
    time part
  [25396] dynamic-link: Failing dlopen can leave behind dangling GL
    (dl_initfirst) link map pointer
  [25401] malloc: pvalloc must not have _attribute_alloc_size_
  [25423] libc: Array overflow in backtrace on powerpc
  [25425] network: Missing call to __resolv_context_put in
    getaddrinfo.c:gethosts

Release Notes
=============

https://sourceware.org/glibc/wiki/Release/2.31

Contributors
============

This release was made possible by the contributions of many people.
The maintainers are grateful to everyone who has contributed changes
or bug reports.  These include:

Adhemerval Zanella
Alexandra Hájková
Alistair Francis
Andreas Schwab
Andrew Eggenberger
Arjun Shankar
Aurelien Jarno
Carlos O'Donell
Chung-Lin Tang
DJ Delorie
Dmitry V. Levin
Dragan Mladjenovic
Egor Kobylkin
Emilio Cobos Álvarez
Emilio Pozuelo Monfort
Feng Xue
Florian Weimer
Gabriel F. T. Gomes
Gustavo Romero
H.J. Lu
Ian Kent
James Clarke
Jeremie Koenig
John David Anglin
Joseph Myers
Kamlesh Kumar
Krzysztof Koch
Leandro Pereira
Lucas A. M. Magalhaes
Lukasz Majewski
Marcin Kościelnicki
Matheus Castanho
Mihailo Stojanovic
Mike Crowe
Mike FABIAN
Niklas Hambüchen
Paul A. Clarke
Paul Eggert
Petr Vorel
Rafal Luzynski
Rafał Lużyński
Rajalakshmi Srinivasaraghavan
Raoni Fassina Firmino
Richard Braun
Samuel Thibault
Sandra Loosemore
Siddhesh Poyarekar
Stefan Liebler
Svante Signell
Szabolcs Nagy
Talachan Mon
Thomas Schwinge
Tim Rühsen
Tulio Magno Quites Machado Filho
Wilco Dijkstra
Xuelei Zhang
Zack Weinberg
liqingqing

01 February, 2020 01:31PM by Siddhesh Poyarekar

January 28, 2020

recutils @ Savannah

Pre-release 1.8.90 in alpha.gnu.org

The pre-release recutils-1.8.90.tar.gz is now available at ftp://alpha.gnu.org/gnu/recutils/recutils-1.8.90.tar.gz

The NEWS file in the tarball contains a list of the changes since 1.8.

The planned date for releasing 1.9 is Saturday 1 February 2020.

Please report any problem found with the pre-release to bug-recutils@gnu.org.

Thanks!

28 January, 2020 11:31AM by Jose E. Marchesi

January 24, 2020

Gary Benson

Container debugging minihint

What’s in my container?

  1. bash$ podman ps --ns
    CONTAINER ID  NAMES            PID    CGROUPNS  IPC         MNT         NET         PIDNS       USERNS      UTS
    fe11359293e8  eloquent_austin  11090            4026532623  4026532621  4026532421  4026532624  4026531837  4026532622
  2. bash$ sudo ls -l /proc/11090/root/
    total 22628
    lrwxrwxrwx.   1 root root        7 Jul 25  2019 bin -> usr/bin
    dr-xr-xr-x.   2 root root        6 Jul 25  2019 boot
    drwxr-xr-x.   5 root root      360 Jan 24 12:03 dev
    drwxr-xr-x.   1 root root      183 Jan 23 16:43 etc
     ...

Thank you.
[28 Jan@1135UTC] UPDATE—This doesn’t seem to work with newer systems, I’m investigating…

24 January, 2020 03:01PM by gbenson

GNU Guix

Guile 3 & Guix

Version 3.0 of GNU Guile, an implementation of the Scheme programming language, was released just last week. This is a major milestone for Guile, which gets compiler improvements and just-in-time (JIT) native code generation, leading to significant performance improvements over 2.2. It’s also great news for all the users of Guile, and in particular for Guix!

Guile 3 logo.

This post discusses what it means for Guix to migrate to Guile 3 and how that migration is already taking place.

Guile in Guix

Most users interact with Guix through its command-line interface, and we work hard to make it as approachable as possible. As any user quickly notices, Guix uses the Scheme programming language uniformly for its configuration—from channels to manifests and operating systems—and anyone who starts packaging software knows that package definitions are in fact Scheme code as well.

This is a significant departure from many other, and in particular from Nix. While Nix defines several domain-specific languages (DSLs) for these aspects—the Nix language but also specific configuration languages—Guix chooses Scheme as the single language for all this, together with the definition of high-level embedded domain-specific languages (EDSLs).

It goes beyond that: in Guix System, all the things traditionally implemented in C or as a set of Perl or shell scripts are implemented in Scheme. That includes the init system, package builds, the initial RAM disk (initrd), system tests, and more. Because this leads to several layers of Scheme code, executed at different points in time, Guix includes a code staging mechanism built upon the nice properties of Scheme.

Why do that? The arguments, right from the start, were twofold: using a general-purpose language allows us to benefit from its implementation tooling, and having interfaces for “everything” in Scheme makes it easy for users to navigate their distro or OS code and to reuse code to build new features or applications. Guix developers benefit from the ease of code reuse every day; demonstrative examples include the use of Guix container facilities in the init system, the development of many tools providing facilities around packages, the implementation of additional user interfaces, and work on applications that use Guix as a library such as the Guix Workflow Language and Guix-Jupyter.

As for the benefits of the host general-purpose language, these are rather obvious: Guix developers benefit from an expressive language, an optimizing compiler, a debugger, a powerful read-eval-print loop (REPL), an interactive development environment, and all sorts of libraries. Moving to Guile 3 should add to that better performance, essentially for free. To be comprehensive, Guile 3 may well come with a set of brand new bugs too, but so far we seem to be doing OK!

Migrating to Guile 3

What does it mean for Guix to migrate to Guile 3? We’ve seen above different ways in which Guix relies on Guile. In short, we can say that migration is threefold:

  1. Guix is a distro that ships Guile-related packages. Like any other distro, it will have to upgrade its guile package to 3.0 and to ensure packages that depend on it and updated as well.
  2. Guix is a program written in Guile. As such, we need to make sure that all its dependencies (half a dozen of Guile libraries) work with Guile 3 and that Guix itself runs fine with Guile 3.
  3. Guix ties together operating system components. In particular, the init system (the Shepherd) and other boot-time facilities will also migrate.

The packages

Updating the distro is the boring part, but it’s best to get it right. Guix makes it possible to have unrelated versions of variants of packages in different environments or different profiles, which is very nice. We’ll have performed a smooth transition if users and tools see that the packages named guile and guile-ssh (say) transparently move from Guile 2.2 to 3.0, in lockstep.

Put differently, most of the upgrade work upon a programming language version bump deals with conventions, and in particular package names. Currently, guile corresponds to the 2.2 stable series and all the guile-* packages are built against it. In the meantime, the package for Guile 3 is named guile-next and packages built against it are called guile3.0-*. Over the last few weeks we created guile3.0- variants for most Guile packages, something that’s easily achieved with Guix.

The big switch will consist in renaming all current guile-* packages to guile2.2-* packages, for use with the legacy 2.2 series, and renaming all the guile3.0-* packages to guile-*. We will switch soon, but before getting there, we’re making sure important packages are available for 3.0.

Guix-the-program

A more interesting part is “porting” Guix itself from Guile 2.2 to Guile 3. It seems that developers have become wary of 2-to-3 transitions for programming languages. Fear not! Switching from Guile 2 to Guile 3 turned out to be an easy task. In fact, very little changed in the language itself; what did change—e.g., semantics on fine points of the module system, support for structured exceptions—is either optional or backwards-compatible.

As Guile 2.9 pre-releases trickled in, we started testing all the Guile libraries Guix relies on against 2.9. For the vast majority of them, all we had to do was to update their configure.ac to allow builds with 3.0.

Guix itself was a bit more work, mostly because it’s a rather large code base with a picky test suite. The bit that required most work has to do with the introduction of declarative modules, an optional semantic change in modules to support more compiler optimizations. We had several “white-box tests” where tests would happily peek at private module bindings through the magical-evil @@ operator. Because we chose to enable declarative modules, we also had to adjust our tests to no longer do that. And well, that’s about it!

At that point, we were able to create a guile3.0-guix package variant, primarily for testing purposes. Soon after, we told guix pull to build Guix with 3.0 instead of 2.2. Thus, Guix users who upgrade will transparently find themselves running Guix on Guile 3.0.

The main benefit is improved performance. Guile 3 is known to be up to 32 times faster than Guile 2.2 on some micro-benchmarks. Assessing the performance gains on a “real-world” application like Guix is the real test. What would be a relevant benchmark? At its core, Guix is essentially a compiler from high-level descriptions of packages, operating systems, and the like, to low-level build instructions (derivations). Thus, a good benchmark is a command that exercises little more than this compilation step:

guix build libreoffice ghc-pandoc guix --dry-run --derivation

or:

guix system build config.scm --dry-run --derivation

On x86_64, the guix build command above on Guile 3 is 7% faster than on Guile 2.2, and guix system build, which is more computing-intensive, is 10% faster (heap usage is ~5% higher). This is lower than the skyrocketing speedups observed on some microbenchmarks, but it’s probably no surprise: these guix commands are short-lived (a couple of seconds) and they’re rather I/O- and GC-intensive—something JIT compilation cannot help with.

On 32-bit ARM, we temporarily disabled JIT due to a bug; there we observe a slight slowdown compared to 2.2. This can be explained by the fact that virtual machine (VM) instructions in 3.0 are lower-level than in 2.2 and will hopefully be more than compensated for when JIT is re-enabled.

Gluing it all together

The last part of the Guile 3 migration has to do with how Guix, and in particular Guix System, glues things together. As explained above, Guix manipulates several stages of Scheme code that will run a different points in time.

Firstly, the code that runs package builds, such as the one that runs ./configure && make && make install, is Guile code. Currently that code runs on Guile 2.2, but on the next major rebuild-the-world upgrade, we will switch to Guile 3.

Additionally, Guix produces Scheme code consumed by the Shepherd, by GNU mcron, and for the graphical installer. These will soon switch to Guile 3 as well. This kind of change is made easy by the fact that both the package definitions and the staged code that depends on those packages live in the same repository.

Long live Guile 3!

Migrating Guix to Guile 3 is a bit of work because of the many ways Guix interacts with Guile and because of the sheer size of the code base. For a “2-to-3” transition though, it was easy. And fundamentally, it remains a cheap transition compared to what it brings: better performance and new features. That’s another benefit of using a general-purpose language.

Thumbs up to everyone involved in its development, and long live Guile 3!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

24 January, 2020 03:00PM by Ludovic Courtès

January 22, 2020

parallel @ Savannah

GNU Parallel 20200122 ('Soleimani') released

GNU Parallel 20200122 ('Soleimani') has been released. It is available for download at: http://ftpmirror.gnu.org/parallel/

GNU Parallel is 10 years old next year on 2020-04-22. You are here by invited to a reception on Friday 2020-04-17.

See https://www.gnu.org/software/parallel/10-years-anniversary.html

Quote of the month:

  GNU parallel is straight up incredible.
    -- Ben Johnson @biobenkj@twtter

New in this release:

  • --blocktimeout dur - Time out for reading block when using --pipe. If it takes longer than dur to read a full block, use the partial block read so far.
  • Bug fixes and man page updates.

News about GNU Parallel:

Get the book: GNU Parallel 2018 http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html

GNU Parallel - For people who live life in the parallel lane.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 3374ec53bacb199b245af2dda86df6c9
    12345678 3374ec53 bacb199b 245af2dd a86df6c9
    $ md5sum install.sh | grep 029a9ac06e8b5bc6052eac57b2c3c9ca
    029a9ac0 6e8b5bc6 052eac57 b2c3c9ca
    $ sha512sum install.sh | grep f517006d9897747bed8a4694b1acba1b
    40f53af6 9e20dae5 713ba06c f517006d 9897747b ed8a4694 b1acba1b 1464beb4
    60055629 3f2356f3 3e9c4e3c 76e3f3af a9db4b32 bd33322b 975696fc e6b23cfb
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

22 January, 2020 05:57PM by Ole Tange

January 19, 2020

make @ Savannah

GNU Make 4.3 Released!

The next stable version of GNU make, version 4.3, has been released and is available for download from https://ftp.gnu.org/gnu/make/

Please see the NEWS file that comes with the GNU make distribution for details on user-visible changes.

19 January, 2020 10:44PM by Paul D. Smith

January 16, 2020

GNU Guile

GNU Guile 3.0.0 released

We are ecstatic and relieved to announce the release of GNU Guile 3.0.0. This is the first release in the new stable 3.0 release series.

See the release announcement for full details and a download link.

The principal new feature in Guile 3.0 is just-in-time (JIT) native code generation. This speeds up the performance of all programs. Compared to 2.2, microbenchmark performance is around twice as good on the whole, though some individual benchmarks are up to 32 times as fast.

Comparison of microbenchmark performance for Guile 3.0 versus 2.2

For larger use cases, notably, this finally makes the performance of "eval" as written in Scheme faster than "eval" written in C, as in the days of Guile 1.8.

Other new features in 3.0 include support for interleaved definitions and expressions in lexical contexts, native support for structured exceptions, better support for the R6RS and R7RS Scheme standards, along with a pile of optimizations. See the NEWS file for a complete list of user-visible changes.

Guile 3.0.0 and all future releases in the 3.0.x series are parallel-installable with other stable release series (e.g. 2.2). As the first release in a new stable series, we anticipate that Guile 3.0.0 might have build problems on uncommon platforms; bug reports are very welcome. Send any bug reports you might have as email at to bug-guile@gnu.org.

Happy hacking with Guile 3!

16 January, 2020 01:00PM by Andy Wingo (guile-devel@gnu.org)

Parabola GNU/Linux-libre

[From Arch] rsync compatibility

Our rsync package was shipped with bundled zlib to provide compatibility with the old-style --compress option up to version 3.1.0. Version 3.1.1 was released on 2014-06-22 and is shipped by all major distributions now.

So we decided to finally drop the bundled library and ship a package with system zlib. This also fixes security issues, actual ones and in future. Go and blame those running old versions if you encounter errors with rsync 3.1.3-3.

16 January, 2020 04:44AM by Isaac David

[From Arch] Now using Zstandard instead of xz for package compression

As announced on the Arch-dev mailing list, on Friday, Dec 27 2019, the package compression scheme has changed from xz (.pkg.tar.xz) to zstd (.pkg.tar.zst).

zstd and xz trade blows in their compression ratio. Recompressing all packages to zstd with their options yields a total ~0.8% increase in package size on all of their packages combined, but the decompression time for all packages saw a ~1300% speedup.

We already have hundreds of zstd-compressed packages in our repositories, and as packages get updated more will keep rolling in. No user-facing issues have been found as of yet, so things appear to be working.

As an end-user no manual intervention is required, assuming that you have read and followed the news post from late last year.

16 January, 2020 04:34AM by Isaac David

January 15, 2020

sed @ Savannah

sed-4.8 released [stable]

This is to announce sed-4.8, a stable release.

There have been 21 commits by 2 people in the 56 weeks since 4.7.

See the NEWS below for a brief summary.

Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Assaf Gordon (4)
  Jim Meyering (17)

Jim [on behalf of the sed maintainers]
==================================================================

Here is the GNU sed home page:
    http://gnu.org/s/sed/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=sed.git;a=shortlog;h=v4.8
or run this command from a git-cloned sed directory:
  git shortlog v4.7..v4.8

To summarize the 865 gnulib-related changes, run these commands
from a git-cloned sed directory:
  git checkout v4.8
  git submodule summary v4.7

Here are the compressed sources:
  https://ftp.gnu.org/gnu/sed/sed-4.8.tar.gz   (2.2MB)
  https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz   (1.3MB)

Here are the GPG detached signatures[*]:
  https://ftp.gnu.org/gnu/sed/sed-4.8.tar.gz.sig
  https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify sed-4.8.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys 7FD9FCCB000BEEEE

and rerun the 'gpg --verify' command.

This release was bootstrapped with the following tools:
  Autoconf 2.69.202-d78a
  Automake 1.16a
  Gnulib v0.1-3167-g6b9d15b8b

NEWS

* Noteworthy changes in release 4.8 (2020-01-14) [stable]

** Bug fixes

  "sed -i" now creates temporary files with correct umask (limited to u=rwx).
  Previously sed would incorrectly set umask on temporary files, resulting
  in problems under certain fuse-like file systems.
  [bug introduced in sed 4.2.1]

** Release

  distribute gzip-compressed tarballs once again

** Improvements

  a year's worth of gnulib development, including improved DFA performance

15 January, 2020 04:45AM by Jim Meyering

January 14, 2020

GNU Guix

Reproducible computations with Guix

This post is about reproducible computations, so let's start with a computation. A short, though rather uninteresting, C program is a good starting point. It computes π in three different ways:

#include <math.h>
#include <stdio.h>

int main()
{
    printf( "M_PI                         : %.10lf\n", M_PI);
    printf( "4 * atan(1.)                 : %.10lf\n", 4.*atan(1.));
    printf( "Leibniz' formula (four terms): %.10lf\n", 4.*(1.-1./3.+1./5.-1./7.));
    return 0;
}

This program uses no random element, such as a random number generator or parallelism. It's strictly deterministic. It is reasonable to expect it to produce exactly the same output, on any computer and at any point in time. And yet, many programs whose results should be perfectly reproducible are in fact not. Programs using floating-point arithmetic, such as this short example, are particularly prone to seemingly inexplicable variations.

My goal is to explain why deterministic programs often fail to be reproducible, and what it takes to fix this. The short answer to that question is "use Guix", but even though Guix provides excellent support for reproducibility, you still have to use it correctly, and that requires some understanding of what's going on. The explanation I will give is rather detailed, to the point of discussing parts of the Guile API of Guix. You should be able to follow the reasoning without knowing Guile though, you will just have to believe me that the scripts I will show do what I claim they do. And in the end, I will provide a ready-to-run Guile script that will let you explore package dependencies right from the shell.

Dependencies: what it takes to run a program

One keyword in discussions of reproducibility is "dependencies". I will revisit the exact meaning of this term later, but to get started, I will define it loosely as "any software package required to run a program". Running the π computation shown above is normally done using something like

gcc pi.c -o pi
./pi

C programmers know that gcc is a C compiler, so that's one obvious dependency for running our little program. But is a C compiler enough? That question is surprisingly difficult to answer in practice. Your computer is loaded with tons of software (otherwise it wouldn't be very useful), and you don't really know what happens behind the scenes when you run gcc or pi.

Containers are good

A major element of reproducibility support in Guix is the possibility to run programs in well-defined environments that contain exactly the software packages you request, and no more. So if your program runs in an environment that contains only a C compiler, you can be sure it has no other dependencies. Let's create such an environment:

guix environment --container --ad-hoc gcc-toolchain

The option --container ensures the best possible isolation from the standard environment that your system installation and user account provide for day-to-day work. This environment contains nothing but a C compiler and a shell (which you need to type in commands), and has access to no other files than those in the current directory.

If the term "container" makes you think of Docker, note that this is something different. Note also that the option --container requires support from the Linux kernel, which may not be present on your system, or may be disabled by default. Finally, note that by default, a containerized environment has no network access, which may be a problem. If for whatever reason you cannot use --container, use --pure instead. This yields a less isolated environment, but it is usually good enough. For a more detailed discussion of these options, see the Guix manual.

The above command leaves me in a shell inside my environment, where I can now compile and run my little program:

gcc pi.c -o pi
./pi
M_PI                         : 3.1415926536
4 * atan(1.)                 : 3.1415926536
Leibniz' formula (four terms): 2.8952380952

It works! So now I can be sure that my program has a single dependency: the Guix package gcc-toolchain. I'll leave that special-environment shell by typing Ctrl-D, as otherwise the following examples won't work.

Perfectionists who want to exclude the possibility that my program requires a shell could run each step in a separate container:

guix environment --container --ad-hoc gcc-toolchain -- gcc pi.c -o pi
guix environment --container --ad-hoc gcc-toolchain -- ./pi
M_PI                         : 3.1415926536
4 * atan(1.)                 : 3.1415926536
Leibniz' formula (four terms): 2.8952380952

Welcome to dependency hell!

Now that we know that our only dependency is gcc-toolchain, let's look at it in more detail:

guix show gcc-toolchain
name: gcc-toolchain
version: 9.2.0
outputs: out debug static
systems: x86_64-linux i686-linux
dependencies: binutils@2.32 gcc@9.2.0 glibc@2.29 ld-wrapper@0
location: gnu/packages/commencement.scm:2532:4
homepage: https://gcc.gnu.org/
license: GPL 3+
synopsis: Complete GCC tool chain for C/C++ development  
description: This package provides a complete GCC tool chain for C/C++
+ development to be installed in user profiles.  This includes GCC, as well as
+ libc (headers an d binaries, plus debugging symbols in the `debug' output),
+ and Binutils.

name: gcc-toolchain
version: 8.3.0
outputs: out debug static
systems: x86_64-linux i686-linux
dependencies: binutils@2.32 gcc@8.3.0 glibc@2.29 ld-wrapper@0
location: gnu/packages/commencement.scm:2532:4
homepage: https://gcc.gnu.org/
license: GPL 3+
synopsis: Complete GCC tool chain for C/C++ development  
description: This package provides a complete GCC tool chain for C/C++
+ development to be installed in user profiles.  This includes GCC, as well as
+ libc (headers an d binaries, plus debugging symbols in the `debug' output),
+ and Binutils.

name: gcc-toolchain
version: 7.4.0
outputs: out debug static
systems: x86_64-linux i686-linux
dependencies: binutils@2.32 gcc@7.4.0 glibc@2.29 ld-wrapper@0
location: gnu/packages/commencement.scm:2532:4
homepage: https://gcc.gnu.org/
license: GPL 3+
synopsis: Complete GCC tool chain for C/C++ development  
description: This package provides a complete GCC tool chain for C/C++
+ development to be installed in user profiles.  This includes GCC, as well as
+ libc (headers an d binaries, plus debugging symbols in the `debug' output),
+ and Binutils.

name: gcc-toolchain
version: 6.5.0
outputs: out debug static
systems: x86_64-linux i686-linux
dependencies: binutils@2.32 gcc@6.5.0 glibc@2.29 ld-wrapper@0
location: gnu/packages/commencement.scm:2532:4
homepage: https://gcc.gnu.org/
license: GPL 3+
synopsis: Complete GCC tool chain for C/C++ development  
description: This package provides a complete GCC tool chain for C/C++
+ development to be installed in user profiles.  This includes GCC, as well as
+ libc (headers an d binaries, plus debugging symbols in the `debug' output),
+ and Binutils.

name: gcc-toolchain
version: 5.5.0
outputs: out debug static
systems: x86_64-linux i686-linux
dependencies: binutils@2.32 gcc@5.5.0 glibc@2.29 ld-wrapper@0
location: gnu/packages/commencement.scm:2532:4
homepage: https://gcc.gnu.org/
license: GPL 3+
synopsis: Complete GCC tool chain for C/C++ development  
description: This package provides a complete GCC tool chain for C/C++
+ development to be installed in user profiles.  This includes GCC, as well as
+ libc (headers an d binaries, plus debugging symbols in the `debug' output),
+ and Binutils.

name: gcc-toolchain
version: 4.9.4
outputs: out debug static
systems: x86_64-linux i686-linux
dependencies: binutils@2.32 gcc@4.9.4 glibc@2.29 ld-wrapper@0
location: gnu/packages/commencement.scm:2532:4
homepage: https://gcc.gnu.org/
license: GPL 3+
synopsis: Complete GCC tool chain for C/C++ development  
description: This package provides a complete GCC tool chain for C/C++
+ development to be installed in user profiles.  This includes GCC, as well as
+ libc (headers an d binaries, plus debugging symbols in the `debug' output),
+ and Binutils.

name: gcc-toolchain
version: 4.8.5
outputs: out debug static
systems: x86_64-linux i686-linux
dependencies: binutils@2.32 gcc@4.8.5 glibc@2.29 ld-wrapper@0
location: gnu/packages/commencement.scm:2532:4
homepage: https://gcc.gnu.org/
license: GPL 3+
synopsis: Complete GCC tool chain for C/C++ development  
description: This package provides a complete GCC tool chain for C/C++
+ development to be installed in user profiles.  This includes GCC, as well as
+ libc (headers an d binaries, plus debugging symbols in the `debug' output),
+ and Binutils.

Guix actually knows about several versions of this toolchain. We didn't ask for a specific one, so what we got is the first one in this list, which is the one with the highest version number. Let's check that this is true:

guix environment --container --ad-hoc gcc-toolchain -- gcc --version
gcc (GCC) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The output of guix show contains a line about dependencies. These are the dependencies of our dependency, and you may already have guessed that they will have dependencies as well. That's why reproducibility is such a difficult job in practice! The dependencies of gcc-toolchain@9.2.0 are:

guix show gcc-toolchain@9.2.0 | recsel -P dependencies
binutils@2.32 gcc@9.2.0 glibc@2.29 ld-wrapper@0

To dig deeper, we can try feeding these dependencies to guix show, one by one, in order to learn more about them:

guix show binutils@2.32
name: binutils
version: 2.32
outputs: out
systems: x86_64-linux i686-linux
dependencies: 
location: gnu/packages/base.scm:415:2
homepage: https://www.gnu.org/software/binutils/
license: GPL 3+
synopsis: Binary utilities: bfd gas gprof ld  
description: GNU Binutils is a collection of tools for working with binary
+ files.  Perhaps the most notable are "ld", a linker, and "as", an assembler.
+ Other tools include programs to display binary profiling information, list the
+ strings in a binary file, and utilities for working with archives.  The "bfd"
+ library for working with executable and object formats is also included.
guix show gcc@9.2.0
guix show: error: gcc@9.2.0: package not found

This looks a bit surprising. What's happening here is that gcc is defined as a hidden package in Guix. The package is there, but it is hidden from package queries. There is a good reason for this: gcc on its own is rather useless, you need gcc-toolchain to actually use the compiler. But if both gcc and gcc-toolchain showed up in a search, that would be more confusing than helpful for most users. Hiding the package is a way of saying "for experts only".

Let's take this as a sign that it's time to move on to the next level of Guix hacking: Guile scripts. Guile, an implementation of the Scheme language, is Guix' native language, so using Guile scripts, you get access to everything there is to know about Guix and its packages.

A note in passing: the emacs-guix package provides an intermediate level of Guix exploration for Emacs users. It lets you look at hidden packages, for example. But much of what I will show in the following really requires Guile scripts. Another nice tool for package exploration is guix graph, which creates a diagram showing dependency relations between packages. Unfortunately that diagram is legible only for a relatively small number of dependencies, and as we will see later, most packages end up having lots of them.

Anatomy of a Guix package

From the user's point of view, a package is a piece of software with a name and a version number that can be installed using guix install. The packager's point of view is quite a bit different. In fact, what users consider a package is more precisely called the package's output in Guix jargon. The package is a recipe for creating this output.

To see how all these concepts fit together, let's look at an example of a package definition: xmag. I have chosen this package not because I care much about it, but because its definition is short while showcasing all the features I want to explain. You can access it most easily by typing guix edit xmag. Here is what you will see:

(package
  (name "xmag")
  (version "1.0.6")
  (source
   (origin
     (method url-fetch)
     (uri (string-append
           "mirror://xorg/individual/app/" name "-" version ".tar.gz"))
     (sha256
      (base32
       "19bsg5ykal458d52v0rvdx49v54vwxwqg8q36fdcsv9p2j8yri87"))))
  (build-system gnu-build-system)
  (arguments
   `(#:configure-flags
     (list (string-append "--with-appdefaultdir="
                          %output ,%app-defaults-dir))))
  (inputs
   `(("libxaw" ,libxaw)))
  (native-inputs
   `(("pkg-config" ,pkg-config)))
  (home-page "https://www.x.org/wiki/")
  (synopsis "Display or capture a magnified part of a X11 screen")
  (description "Xmag displays and captures a magnified snapshot of a portion
of an X11 screen.")
  (license license:x11))

The package definition starts with the name and version information you expected. Next comes source, which says how to obtain the source code and from where. It also provides a hash that allows to check the integrity of the downloaded files. The next four items, build-system, arguments, inputs, and native-inputs supply the information required for building the package, which is what creates its outputs. The remaining items are documentation for human consumption, important for other reasons but not for reproducibility, so I won't say any more about them. (See this packaging tutorial if you want to define your own package.)

The example package definition has native-inputs in addition to "plain" inputs. There's a third variant, propagated-inputs, but xmag doesn't have any. The differences between these variants don't matter for my topic, so I will just refer to "inputs" from now on. Another omission I will make is the possibility to define several outputs for a package. This is done for particularly big packages, in order to reduce the footprint of installations, but for the purposes of reproducibility, it's OK to treat all outputs of a package as a single unit.

The following figure illustrates how the various pieces of information from a package are used in the build process (done explicitly by guix build, or implicitly when installing or otherwise using a package): Diagram of a Guix package.

It may help to translate the Guix jargon to the vocabulary of C programming:

| Guix package | C program        |
|--------------+------------------|
| source code  | source code      |
| inputs       | libraries        |
| arguments    | compiler options |
| build system | compiler         |
| output       | executable       |

Building a package can be considered a generalization of compiling a program. We could in fact create a "GCC build system" for Guix that would simply run gcc. However, such a build system would be of little practical use, since most real-life software consists of more than just one C source code file, and requires additional pre- or post-processing steps. The gnu-build-system used in the example is based on tools such as make and autoconf, in addition to gcc.

Package exploration in Guile

Guile uses a record type called <package> to represent packages, which is defined in module (guix packages). There is also a module (gnu packages), which contains the actual package definitions - be careful not to confuse the two (as I always do). Here is a simple Guile script that shows some package information, much like the guix show command that I used earlier:

(use-modules (guix packages)
             (gnu packages)) 

(define gcc-toolchain
  (specification->package "gcc-toolchain"))

(format #t "Name   : ~a\n" (package-name gcc-toolchain))
(format #t "Version: ~a\n" (package-version gcc-toolchain))
(format #t "Inputs : ~a\n" (package-direct-inputs gcc-toolchain))
Name   : gcc-toolchain
Version: 9.2.0
Inputs : ((gcc #<package gcc@9.2.0 gnu/packages/gcc.scm:524 7fc2d76af160>) (ld-wrapper #<package ld-wrapper@0 gnu/packages/base.scm:505 7fc2d306f580>) (binutils #<package binutils@2.32 gnu/packages/commencement.scm:2187 7fc2d306fdc0>) (libc #<package glibc@2.29 gnu/packages/commencement.scm:2145 7fc2d306fe70>) (libc-debug #<package glibc@2.29 gnu/packages/commencement.scm:2145 7fc2d306fe70> debug) (libc-static #<package glibc@2.29 gnu/packages/commencement.scm:2145 7fc2d306fe70> static))

This script first calls specification->package to look up the package using the same rules as the guix command line interface: pick the latest available version if none is explicitly requested. Then it extracts various information about the package. Note that package-direct-inputs returns the combination of package-inputs, package-native-inputs, and package-propagated-inputs. As I said above, I don't care about the distinction here.

The inputs are not shown in a particularly nice form, so let's write two Guile functions to improve it:

(use-modules (guix packages)
             (gnu packages)
             (ice-9 match))

(define (package->specification package)
  (format #f "~a@~a"
          (package-name package)
          (package-version package)))

(define (input->specification input)
  (match input
    ((label (? package? package) . _)
     (package->specification package))
    (other-item
     (format #f "~a" other-item))))

(define gcc-toolchain
  (specification->package "gcc-toolchain"))

(format #t "Package: ~a\n"
        (package->specification gcc-toolchain))
(format #t "Inputs : ~a\n"
        (map input->specification (package-direct-inputs gcc-toolchain)))
Package: gcc-toolchain@9.2.0
Inputs : (gcc@9.2.0 ld-wrapper@0 binutils@2.32 glibc@2.29 glibc@2.29 glibc@2.29)

That looks much better. As you can see from the code, a list of inputs is a bit more than a list of packages. It is in fact a list of labelled package outputs. That also explains why we see glibc three times in the input list: glibc defines three distinct outputs, all of which are used in gcc-toolchain. For reproducibility, all we care about is the package references. Later on, we will deal with much longer input lists, so as a final cleanup step, let's show only unique package references from the list of inputs:

(use-modules (guix packages)
             (gnu packages)
             (srfi srfi-1)
             (ice-9 match))

(define (package->specification package)
  (format #f "~a@~a"
          (package-name package)
          (package-version package)))

(define (input->specification input)
  (match input
    ((label (? package? package) . _)
     (package->specification package))
    (other-item
     (format #f "~a" other-item))))

(define (unique-inputs inputs)
  (delete-duplicates
   (map input->specification inputs)))

(define gcc-toolchain
  (specification->package "gcc-toolchain"))

(format #t "Package: ~a\n"
        (package->specification gcc-toolchain))
(format #t "Inputs : ~a\n"
        (unique-inputs (package-direct-inputs gcc-toolchain)))
Package: gcc-toolchain@9.2.0
Inputs : (gcc@9.2.0 ld-wrapper@0 binutils@2.32 glibc@2.29)

Dependencies

You may have noticed the absence of the term "dependency" from the last two sections. There is a good reason for that: the term is used in somewhat different meanings, and that can create confusion. Guix jargon therefore avoids it.

The figure above shows three kinds of input to the build system: source, inputs, and arguments. These categories reflect the packagers' point of view: source is what the authors of the software supply, inputs are other packages, and arguments is what the packagers themselves add to the build procedure. It is important to understand that from a purely technical point of view, there is no fundamental difference between the three categories. You could, for example, define a package that contains C source code in the build system arguments, but leaves source empty. This would be inconvenient, and confusing for others, so I don't recommend you actually do this. The three categories are important, but for humans, not for computers. In fact, even the build system is not fundamentally distinct from its inputs. You could define a special-purpose build system for one package, and put all the source code in there. At the level of the CPU and the computer's memory, a build process (as in fact any computation) looks like Image of a computation. It is human interpretation that decomposes this into Code and data. and in a next step into Data, program, and environment. We can go on and divide the environment into operating system, development tools, and application software, for example, but the further we go in decomposing the input to a computation, the more arbitrary it gets.

From this point of view, a software's dependencies consist of everything required to run it in addition to its source code. For a Guix package, the dependencies are thus,

  • its inputs
  • the build system arguments
  • the build system itself
  • Guix (which is a piece of software as well)
  • the GNU/Linux operating system (kernel, file system, etc.).

In the following, I will not mention the last two items any more, because they are a common dependency of all Guix packages, but it's important not to forget about them. A change in Guix or in GNU/Linux can actually make a computation non-reproducible, although in practice that happens very rarely. Moreover, Guix is actually designed to run older versions of itself, as we will see later.

Build systems are (mostly) packages as well

I hope that by now you have a good idea of what a package is: a recipe for building outputs from source and inputs, with inputs being the outputs of other packages. The recipe involves a build system and arguments supplied to it. So... what exactly is a build system? I have introduced it as a generalization of a compiler, which describes its role. But where does a build system come from in Guix?

The ultimate answer is of course the source code. Build systems are pieces of Guile code that are part of Guix. But this Guile code is only a shallow layer orchestrating invocations of other software, such as gcc or make. And that software is defined by packages. So in the end, from a reproducibility point of view, we can replace the "build system" item in our list of dependencies by "a bundle of packages". In other words: more inputs.

Before Guix can build a package, it must gather all the required ingredients, and that includes replacing the build system by the packages it represents. The resulting list of ingredients is called a bag, and we can access it using a Guile script:

(use-modules (guix packages)
             (gnu packages)
             (srfi srfi-1)
             (ice-9 match))

(define (package->specification package)
  (format #f "~a@~a"
          (package-name package)
          (package-version package)))

(define (input->specification input)
  (match input
    ((label (? package? package) . _)
     (package->specification package))
    ((label (? origin? origin))
     (format #f "[source code from ~a]"
             (origin-uri origin)))
    (other-input
     (format #f "~a" other-input))))

(define (unique-inputs inputs)
  (delete-duplicates
   (map input->specification inputs)))

(define hello
  (specification->package "hello"))

(format #t "Package       : ~a\n"
        (package->specification hello))
(format #t "Package inputs: ~a\n"
        (unique-inputs (package-direct-inputs hello)))
(format #t "Build inputs  : ~a\n"
        (unique-inputs
         (bag-direct-inputs
          (package->bag hello))))
Package       : hello@2.10
Package inputs: ()
Build inputs  : ([source code from mirror://gnu/hello/hello-2.10.tar.gz] tar@1.32 gzip@1.10 bzip2@1.0.6 xz@5.2.4 file@5.33 diffutils@3.7 patch@2.7.6 findutils@4.6.0 gawk@5.0.1 sed@4.7 grep@3.3 coreutils@8.31 make@4.2.1 bash-minimal@5.0.7 ld-wrapper@0 binutils@2.32 gcc@7.4.0 glibc@2.29 glibc-utf8-locales@2.29)

I have used a different example, hello, because for gcc-toolchain, there is no difference between package inputs and build inputs (check for yourself if you want!) My new example, hello (a short demo program printing "Hello, world" in the language of the system installation), is interesting because it has no package inputs at all. All the build inputs except for the source code have thus been contributed by the build system.

If you compare this script to the previous one that printed only the package inputs, you will notice two major new features. In input->specification, there is an additional case for the source code reference. And in the last statement, package->bag constructs a bag from the package, before bag-direct-inputs is called to get that bag's input list.

Inputs are outputs

I have mentioned before that one package's inputs are other packages' outputs, but that fact deserves a more in-depth discussion because of its crucial importance for reproducibility. A package is a recipe for building outputs from source and inputs. Since these inputs are outputs, they must have been built as well. Package building is therefore a process consisting of multiple steps. An immediate consequence is that any computation making use of packaged software is a multi-step computation as well.

Remember the short C program computing π from the beginning of this post? Running that program is only the last step in a long series of computations. Before you can run pi, you must compile pi.c. That requires the package gcc-toolchain, which must first be built. And before it can be built, its inputs must be built. And so on. If you want the output of pi to be reproducible, the whole chain of computations must be reproducible, because each step can have an impact on the results produced by pi.

So... where does this chain start? Few people write machine code these days, so almost all software requires some compiler or interpreter. And that means that for every package, there are other packages that must be built first. The question of how to get this chain started is known as the bootstrapping problem. A rough summary of the solution is that the chain starts on somebody else's computer, which creates a bootstrap seed, an ideally small package that is downloaded in precompiled form. See this post by Jan Nieuwenhuizen for details of this procedure. The bootstrap seed is not the real start of the chain, but as long as we can retrieve an identical copy at a later time, that's good enough for reproducibility. In fact, the reason for requiring the bootstrap seed to be small is not reproducibility, but inspectability: it should be possible to audit the seed for bugs and malware, even in the absence of source code.

Reaching closure

Now we are finally ready for the ultimate step in dependency analysis: identifying all packages on which a computation depends, right up to the bootstrap seed. The starting point is the list of direct inputs of the bag derived from a package, which we looked at in the previous script. For each package in that list, we must apply this same procedure, recursively. We don't have to write this code ourselves, because the function package-closure in Guix does that job. These closures have nothing to do with closures in Lisp, and even less with the Clojure programming language. They are a case of what mathematicians call transitive closures: starting with a set of packages, you extend the set repeatedly by adding the inputs of the packages that are already in the set, until there is nothing more to add. If you have a basic knowledge of Scheme, you should now be able to understand the implementation of this function. Let's add it to our dependency analysis code:

(use-modules (guix packages)
             (gnu packages)
             (srfi srfi-1)
             (ice-9 match))

(define (package->specification package)
  (format #f "~a@~a"
          (package-name package)
          (package-version package)))

(define (input->specification input)
  (match input
    ((label (? package? package) . _)
     (package->specification package))
    ((label (? origin? origin))
     (format #f "[source code from ~a]"
             (origin-uri origin)))
    (other-input
     (format #f "~a" other-input))))

(define (unique-inputs inputs)
  (delete-duplicates
   (map input->specification inputs)))

(define (length-and-list lists)
  (list (length lists) lists))

(define hello
  (specification->package "hello"))

(format #t "Package        : ~a\n"
        (package->specification hello))
(format #t "Package inputs : ~a\n"
        (length-and-list (unique-inputs (package-direct-inputs hello))))
(format #t "Build inputs   : ~a\n"
        (length-and-list
         (unique-inputs
          (bag-direct-inputs
           (package->bag hello)))))
(format #t "Package closure: ~a\n"
        (length-and-list
         (delete-duplicates
          (map package->specification
               (package-closure (list hello))))))
Package        : hello@2.10
Package inputs : (0 ())
Build inputs   : (20 ([source code from mirror://gnu/hello/hello-2.10.tar.gz] tar@1.32 gzip@1.10 bzip2@1.0.6 xz@5.2.4 file@5.33 diffutils@3.7 patch@2.7.6 findutils@4.6.0 gawk@5.0.1 sed@4.7 grep@3.3 coreutils@8.31 make@4.2.1 bash-minimal@5.0.7 ld-wrapper@0 binutils@2.32 gcc@7.4.0 glibc@2.29 glibc-utf8-locales@2.29))
Package closure: (84 (m4@1.4.18 libatomic-ops@7.6.10 gmp@6.1.2 libgc@7.6.12 libltdl@2.4.6 libunistring@0.9.10 libffi@3.2.1 pkg-config@0.29.2 guile@2.2.6 libsigsegv@2.12 lzip@1.21 ed@1.15 perl@5.30.0 guile-bootstrap@2.0 zlib@1.2.11 xz@5.2.4 ncurses@6.1-20190609 libxml2@2.9.9 attr@2.4.48 gettext-minimal@0.20.1 gcc-cross-boot0-wrapped@7.4.0 libstdc++@7.4.0 ld-wrapper-boot3@0 bootstrap-binaries@0 ld-wrapper-boot0@0 flex@2.6.4 glibc-intermediate@2.29 libstdc++-boot0@4.9.4 expat@2.2.7 gcc-mesboot1-wrapper@4.7.4 mesboot-headers@0.19 gcc-core-mesboot@2.95.3 bootstrap-mes@0 bootstrap-mescc-tools@0.5.2 tcc-boot0@0.9.26-6.c004e9a mes-boot@0.19 tcc-boot@0.9.27 make-mesboot0@3.80 gcc-mesboot0@2.95.3 binutils-mesboot0@2.20.1a make-mesboot@3.82 diffutils-mesboot@2.7 gcc-mesboot1@4.7.4 glibc-headers-mesboot@2.16.0 glibc-mesboot0@2.2.5 binutils-mesboot@2.20.1a linux-libre-headers@4.19.56 linux-libre-headers-bootstrap@0 gcc-mesboot@4.9.4 glibc-mesboot@2.16.0 gcc-cross-boot0@7.4.0 bash-static@5.0.7 gettext-boot0@0.19.8.1 python-minimal@3.5.7 perl-boot0@5.30.0 texinfo@6.6 bison@3.4.1 gzip@1.10 libcap@2.27 acl@2.2.53 glibc-utf8-locales@2.29 gcc-mesboot-wrapper@4.9.4 file-boot0@5.33 findutils-boot0@4.6.0 diffutils-boot0@3.7 make-boot0@4.2.1 binutils-cross-boot0@2.32 glibc@2.29 gcc@7.4.0 binutils@2.32 ld-wrapper@0 bash-minimal@5.0.7 make@4.2.1 coreutils@8.31 grep@3.3 sed@4.7 gawk@5.0.1 findutils@4.6.0 patch@2.7.6 diffutils@3.7 file@5.33 bzip2@1.0.6 tar@1.32 hello@2.10))

That's 84 packages, just for printing "Hello, world!". As promised, it includes the bootstrap seed, called bootstrap-binaries. It may be more surprising to see Perl and Python in the dependency list of what is a pure C program. The explanation is that the build process of gcc and glibc contains Perl and Python code. Considering that both Perl and Python are written in C and use glibc, this hints at why bootstrapping is a hard problem!

Get ready for your own analyses

As promised, here is a Guile script that you can download and run from the command line to do dependency analyses much like the ones I have shown. Just give the packages whose combined list of dependencies you want to analyze. For example:

./show-dependencies.scm hello
Packages: 1
  hello@2.10
Package inputs: 0 packages
 
Build inputs: 20 packages
  [source code from mirror://gnu/hello/hello-2.10.tar.gz] bash-minimal@5.0.7 binutils@2.32 bzip2@1.0.6 coreutils@8.31 diffutils@3.7 file@5.33 findutils@4.6.0 gawk@5.0.1 gcc@7.4.0 glibc-utf8-locales@2.29 glibc@2.29 grep@3.3 gzip@1.10 ld-wrapper@0 make@4.2.1 patch@2.7.6 sed@4.7 tar@1.32 xz@5.2.4
Package closure: 84 packages
  acl@2.2.53 attr@2.4.48 bash-minimal@5.0.7 bash-static@5.0.7 binutils-cross-boot0@2.32 binutils-mesboot0@2.20.1a binutils-mesboot@2.20.1a binutils@2.32 bison@3.4.1 bootstrap-binaries@0 bootstrap-mes@0 bootstrap-mescc-tools@0.5.2 bzip2@1.0.6 coreutils@8.31 diffutils-boot0@3.7 diffutils-mesboot@2.7 diffutils@3.7 ed@1.15 expat@2.2.7 file-boot0@5.33 file@5.33 findutils-boot0@4.6.0 findutils@4.6.0 flex@2.6.4 gawk@5.0.1 gcc-core-mesboot@2.95.3 gcc-cross-boot0-wrapped@7.4.0 gcc-cross-boot0@7.4.0 gcc-mesboot-wrapper@4.9.4 gcc-mesboot0@2.95.3 gcc-mesboot1-wrapper@4.7.4 gcc-mesboot1@4.7.4 gcc-mesboot@4.9.4 gcc@7.4.0 gettext-boot0@0.19.8.1 gettext-minimal@0.20.1 glibc-headers-mesboot@2.16.0 glibc-intermediate@2.29 glibc-mesboot0@2.2.5 glibc-mesboot@2.16.0 glibc-utf8-locales@2.29 glibc@2.29 gmp@6.1.2 grep@3.3 guile-bootstrap@2.0 guile@2.2.6 gzip@1.10 hello@2.10 ld-wrapper-boot0@0 ld-wrapper-boot3@0 ld-wrapper@0 libatomic-ops@7.6.10 libcap@2.27 libffi@3.2.1 libgc@7.6.12 libltdl@2.4.6 libsigsegv@2.12 libstdc++-boot0@4.9.4 libstdc++@7.4.0 libunistring@0.9.10 libxml2@2.9.9 linux-libre-headers-bootstrap@0 linux-libre-headers@4.19.56 lzip@1.21 m4@1.4.18 make-boot0@4.2.1 make-mesboot0@3.80 make-mesboot@3.82 make@4.2.1 mes-boot@0.19 mesboot-headers@0.19 ncurses@6.1-20190609 patch@2.7.6 perl-boot0@5.30.0 perl@5.30.0 pkg-config@0.29.2 python-minimal@3.5.7 sed@4.7 tar@1.32 tcc-boot0@0.9.26-6.c004e9a tcc-boot@0.9.27 texinfo@6.6 xz@5.2.4 zlib@1.2.11

You can now easily experiment yourself, even if you are not at ease with Guile. For example, suppose you have a small Python script that plots some data using matplotlib. What are its dependencies? First you should check that it runs in a minimal environment:

guix environment --container --ad-hoc python python-matplotlib -- python my-script.py

Next, find its dependencies:

./show-dependencies.scm python python-matplotlib

I won't show the output here because it is rather long - the package closure contains 499 packages!

OK, but... what are the real dependencies?

I have explained dependencies along these lines in a few seminars. There's one question that someone in the audience is bound to ask: What do the results of a computation really depend on? The output of hello is "Hello, world!", no matter which version of gcc I use to compile it, and no matter which version of python was used in building glibc. The package closure is a worst-case estimate: it contains everything that can potentially influence the results, though most of it doesn't in practice. Unfortunately, there is no way to identify the dependencies that matter automatically, because answering that question in general (i.e. for arbitrary software) is equivalent to solving the halting problem.

Most package managers, such as Debian's apt or the multi-platform conda, take a different point of view. They define the dependencies of a program as all packages that need to be loaded into memory in order to run it. They thus exclude the software that is required to build the program and its run-time dependencies, but can then be discarded. Whereas Guix' definition errs on the safe side (its dependency list is often longer than necessary but never too short), the run-time-only definition is both too vast and too restrictive. Many run-time dependencies don't have an impact on most programs' results, but some build-time dependencies do.

One important case where build-time dependencies matter is floating-point computations. For historical reasons, they are surrounded by an aura of vagueness and imprecision, which goes back to its early days, when many details were poorly understood and implementations varied a lot. Today, all computers used for scientific computing respect the IEEE 754 standard that precisely defines how floating-point numbers are represented in memory and what the result of each arithmetic operation must be. Floating-point arithmetic is thus perfectly deterministic and even perfectly portable between machines, if expressed in terms of the operations defined by the standard. However, high-level languages such as C or Fortran do not allow programmers to do that. Their designers assume (probably correctly) that most programmers do not want to deal with the intricate details of rounding. Therefore they provide only a simplified interface to the arithmetic operations of IEEE 754, which incidentally also leaves more liberty for code optimization to compiler writers. The net result is that the complete specification of a program's results is its source code plus the compiler and the compilation options. You thus can get reproducible floating-point results if you include all compilation steps into the perimeter of your computation, at least for code running on a single processor. Parallel computing is a different story: it involves voluntarily giving up reproducibility in exchange for speed. Reproducibility then becomes a best-effort approach of limiting the collateral damage done by optimization through the clever design of algorithms.

Reproducing a reproducible computation

So far, I have explained the theory behind reproducible computations. The take-home message is that to be sure to get exactly the same results in the future, you have to use the exact same versions of all packages in the package closure of your immediate dependencies. I have also shown you how you can access that package closure. There is one missing piece: how do you actually run your program in the future, using the same environment?

The good news is that doing this is a lot simpler than understanding my lengthy explanations (which is why I leave this for the end!). The complex dependency graphs that I have analyzed up to here are encoded in the Guix source code, so all you need to re-create your environment is the exact same version of Guix! You get that version using

guix describe
Generation 15 Jan 06 2020 13:30:45    (current)
  guix 769b96b
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 769b96b62e8c09b078f73adc09fb860505920f8f

The critical information here is the unpleasant looking string of hexadecimal digits after "commit". This is all it takes to uniquely identify a version of Guix. And to re-use it in the future, all you need is Guix' time machine:

guix time-machine --commit=769b96b62e8c09b078f73adc09fb860505920f8f -- environment --ad-hoc gcc-toolchain
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
gcc pi.c -o pi
./pi
M_PI                         : 3.1415926536
4 * atan(1.)                 : 3.1415926536
Leibniz' formula (four terms): 2.8952380952

The time machine actually downloads the specified version of Guix and passes it the rest of the command line. You are running the same code again. Even bugs in Guix will be reproduced faithfully! As before, guix environment leaves us in a special-environment shell which needs to be terminated by Ctrl-D.

For many practical use cases, this technique is sufficient. But there are two variants you should know about for more complicated situations:

  • If you need an environment with many packages, you should use a manifest rather than list the packages on the command line. See the manual for details.

  • If you need packages from additional channels, i.e. packages that are not part of the official Guix distribution, you should store a complete channel description in a file using

guix describe -f channels > guix-version-for-reproduction.txt

and feed that file to the time machine:

guix time-machine --channels=guix-version-for-reproduction.txt -- environment --ad-hoc gcc-toolchain
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
gcc pi.c -o pi
./pi
M_PI                         : 3.1415926536
4 * atan(1.)                 : 3.1415926536
Leibniz' formula (four terms): 2.8952380952

Last, if your colleagues do not use Guix yet, you can pack your reproducible software for use on other systems: as a tarball, or as a Docker or Singularity container image. For example:

guix pack            \
     -f docker       \
     -C none         \
     -S /bin=bin     \
     -S /lib=lib     \
     -S /share=share \
     -S /etc=etc     \
     gcc-toolchain
/gnu/store/iqn9yyvi8im18g7y9f064lw9s9knxp0w-docker-pack.tar

will produce a Docker container image, and with the knowledge of the Guix commit (or channel specification), you will be able in the future to reproduce this container bit-to-bit using guix time-machine.

And now... congratulations for having survived to the end of this long journey! May all your computations be reproducible, with Guix.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

14 January, 2020 04:30PM by Konrad Hinsen

January 13, 2020

libredwg @ Savannah

libredwg-0.10.1 released

Major bugfixes:
  * Fixed dwg2SVG htmlescape overflows and off-by-ones (#182)
  * Removed direct usages of fprintf and stderr in the lib. All can be
    redefined now. (#181)

Minor bugfixes:
  * Fuzzing fixes for dwg2SVG, dwgread. (#182)
  * Fixed eed.raw leaks

Here are the compressed sources:

  http://ftp.gnu.org/gnu/libredwg/libredwg-0.10.1.tar.gz   (10.9MB)
  http://ftp.gnu.org/gnu/libredwg/libredwg-0.10.1.tar.xz   (4.5MB)

Here are the GPG detached signatures[*]:

  http://ftp.gnu.org/gnu/libredwg/libredwg-0.10.1.tar.gz.sig
  http://ftp.gnu.org/gnu/libredwg/libredwg-0.10.1.tar.xz.sig

Use a mirror for higher download bandwidth:

  https://www.gnu.org/order/ftp.html

Here are more binaries:

  https://github.com/LibreDWG/libredwg/releases/tag/0.10.1

Here are the SHA256 checksums:

6539a9a762f74e937f08000e2bb3d3d4dddd326b85b5361f7532237b68ff0ae3  libredwg-0.10.1.tar.gz
0fa603d5f836dfceb8ae4aac28d1e836c09dce3936ab98703bb2341126678ec3  libredwg-0.10.1.tar.xz

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify libredwg-0.10.1.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys B4F63339E65D6414

and rerun the 'gpg --verify' command.

13 January, 2020 09:40AM by Reini Urban

GNU Guile

GNU Guile 2.9.9 (beta) released

We are delighted to announce the release of GNU Guile 2.9.9. This is the ninth and final pre-release of what will eventually become the 3.0 release series.

See the release announcement for full details and a download link.

This release fixes a number of bugs, omissions, and regressions. Notably, it fixes the build on 32-bit systems.

We plan to release a final Guile 3.0.0 on 17 January: this Friday! Please do test this prerelease; build reports, good or bad, are very welcome; send them to guile-devel@gnu.org. If you know you found a bug, please do send a note to bug-guile@gnu.org. Happy hacking!

13 January, 2020 08:44AM by Andy Wingo (guile-devel@gnu.org)

Applied Pokology

First Poke-Conf at Mont-Soleil - A report

pokists in Mont-Soleil
Poking at Mont-Soleil

This last weekend we had the first gathering of poke developers, as part of the GNU Hackers Meeting at Mont-Soleil, in Switzerland. I can say we had a lot of fun, and it was a quite productive meeting too: many patches were written, and many technical aspects designed and clarified.

13 January, 2020 12:00AM

January 12, 2020

GNUnet News

GNUnet 0.12.2

GNUnet 0.12.2 released

We are pleased to announce the release of GNUnet 0.12.2.
This is a new bugfix release. In terms of usability, users should be aware that there are still a large number of known open issues in particular with respect to ease of use, but also some critical privacy issues especially for mobile users. Also, the nascent network is tiny and thus unlikely to provide good anonymity or extensive amounts of interesting information. As a result, the 0.12.2 release is still only suitable for early adopters with some reasonable pain tolerance.

Download links

The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A

Note that due to mirror synchronization, not all links might be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/

Noteworthy changes in 0.12.2 (since 0.12.1)

  • GNS: Resolver clients are now able to specify a recursion depth limit.
  • TRANSPORT/TNG: The transport rewrite (aka TNG) is underway and various transport components have been worked on, including TCP, UDP and UDS communicators.
  • RECLAIM: Added preliminary support for third party attested credentials.
  • UTIL: The cryptographic changes introduced in 0.12.0 broke ECDSA ECDH and consequently other components. The offending ECDSA key normalization was dropped.

Known Issues

  • There are known major design issues in the TRANSPORT, ATS and CORE subsystems which will need to be addressed in the future to achieve acceptable usability, performance and security.
  • There are known moderate implementation limitations in CADET that negatively impact performance.
  • There are known moderate design issues in FS that also impact usability and performance.
  • There are minor implementation limitations in SET that create unnecessary attack surface for availability.
  • The RPS subsystem remains experimental.
  • Some high-level tests in the test-suite fail non-deterministically due to the low-level TRANSPORT issues.

In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org which lists about 190 more specific issues.

Thanks

This release was the work of many people. The following people contributed code and were thus easily identified: Christian Grothoff, Florian Dold, Christian Ulrich, dvn, lynx and Martin Schanzenbach.

12 January, 2020 11:00PM

January 11, 2020

GNS Specification Milestone 2/4

GNS Technical Specification Milestone 2/4

We are happy to announce the completion of the second milestone for the GNS Specification. The second milestone consists of documenting the GNS name resolution process and record handling.
With the release of GNUnet 0.12.x, the currently specified protocol is implemented according to the specification. As before, the draft specification LSD001 can be found at:

As already announced on the mailing list, the Go implementation of GNS is also proceeding as planned and implements the specification.

The next and third milestone will cover namespace revocation.

This work is generously funded by NLnet as part of their Search and discovery fund.

11 January, 2020 11:00PM

January 08, 2020

libredwg @ Savannah

libredwg-0.10 released

Some minor API changes and bugfixes, mostly stabilization.

API breaking changes:
  * added a new int *isnewp argument to all dynapi utf8text getters,
    if the returned string is freshly malloced or not.
  * removed the UNKNOWN supertype, there are only UNKNOWN_OBJ and UNKNOWN_ENT
    left, with common_entity_data.
  * renamed BLOCK_HEADER.preview_data to preview, preview_data_size to preview_size
  * renamed SHAPE.shape_no to style_id
  * renamed CLASS.wasazombie to is_zombie

Major bugfixes:
  * Improved building the perl5 binding, proper dependencies.
    Set proper -I and -L paths, create LibreDWG.c not swig_perl.c
  * Harmonized INDXFB with INDXF, removed extra src/in_dxfb.c (#134).
    Slimmed the .so size by 260Kb. Still untested though.
  * Fixed encoding of added r2000 AUXHEADER address (broken since 0.9)
  * Fixed EED encoding from dwgrewrite (a dxf2dwg regression from 0.9) (#180)

Minor bugfixes:
  * Many fuzzing and static analyzer fixes for dwg2dxf, dxf2dwg, dwgrewrite,
    including a stack-overflow on outdxf cquote. (#172-174, #178, #179).
    dwgrewrite and indxf are pretty robust now, but still highly experimental,
    as many dxf2dwg import and DWG validity tests are missing.
    indxf still has many asserts on many structural DXF errors.
  * Protect indxf from many NULL ptr, overflows and truncation.
  * Fixed most indxf and encode leaks. (#151)
  * More section decoders protections from invalid (fuzzed) values.
  * Stabilized the ASAN leak tests for make check.
  * Fix MULTILEADER.ctx.lline handles <r2010
  * Fix indxf color.alpha; at DXF 440
  * Fixed most important make scan-build warnings, the rest are mostly bogus.

Other newsworthy changes:
  * Added LIBREDWG_VERSION et al to include/dwg.h
  * Added support for AcDb3dSolid history_id (r2007+)
  * Improved the indxf speed in new_object. Do a proper linear search, and
    break on first found type.
  * Rename the ./dxf helper to ./dwg, and added a ./dxf test helper.
  * dxf2dwg got a new experimental --force-free option to check for leaks and
    UAF or double-free's.
  * Allow -o /dev/null sinks for dxf2dwg and dwg2dxf, for faster fuzzing.
  * Harmonized *.spec formatting and adjusted gen-dynapi.pl
  * Harmonized out_dxfb with out_dxf, e.g. the new mspace improvements (#173).

Here are the compressed sources:
  http://ftp.gnu.org/gnu/libredwg/libredwg-0.10.tar.gz   (10.9MB)
  http://ftp.gnu.org/gnu/libredwg/libredwg-0.10.tar.xz   (4.5MB)

Here are the GPG detached signatures[*]:
  http://ftp.gnu.org/gnu/libredwg/libredwg-0.10.tar.gz.sig
  http://ftp.gnu.org/gnu/libredwg/libredwg-0.10.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are more binaries:
  https://github.com/LibreDWG/libredwg/releases/tag/0.10

Here are the SHA256 checksums:

e890b4d3ab8071c78c4eb36e6f7ecd30e7f54630b0e2f051b3fe51395395d5f7  libredwg-0.10.tar.gz
8c37c4ef985e4135e3d2020c502c887b6115cdbbab2148b2e730875d5659cd66  libredwg-0.10.tar.xz

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify libredwg-0.10.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys B4F63339E65D6414

and rerun the 'gpg --verify' command.

08 January, 2020 04:52PM by Reini Urban