This is quite a long post. The executive summary is that freedesktop.org now
hosts an instance of GitLab, which is
generally available and now our preferred platform for hosting going forward.
We think it offers a vastly better service, and we needed to do it in order to
offer the projects we host the modern workflows they have been asking for.
In parallel, we’re working on making our governance, including policies,
processes and decision making, much more transparent.
Founded by Havoc Pennington in 2000, freedesktop.org is now old enough to vote.
From the initial development of the cross-desktop XDG specs, to supporting
critical infrastructure such as NetworkManager, and now as the home to
open-source graphics development (the kernel DRM tree, Mesa, Wayland, X.Org, and
more), it’s long been a good home to a lot of good work.
We don’t provide day-to-day technical direction or enforce set rules: it’s a
very loose collection of projects which we each trust to do their own thing,
some with nothing in common but where they’re hosted.
Unfortunately, that hosting hasn’t really grown up a lot since the turn of the
millennium. Our account system was forked (and subsequently heavily hacked) from
Debian’s old LDAP-based system in 2004. Everyone needing direct Git commit
access to projects, or the ability to upload to web space, has to file a bug in
Bugzilla, where after a trip
through the project maintainer, eventually an admin will get around to pulling
their SSH and GPG (!) keys and adding an account by hand.
Similarly, creating or reconfiguring a Git repository also requires manual admin
intervention, where on request one of us will SSH into the Git server and do
whatever is required. Beyond Git and cgit for viewing, we provide Bugzilla for
issue tracking, Mailman and Patchwork for code review and discussion, and
ikiwiki for tracking. For our sins, we also have an FTP server running
somewhere. None of these services are really integrated with each other;
separate accounts and separate sets of permissions are required.
Maintaining these disparate services is a burden on both admins and projects.
Projects are frequently blocked on admins adding users and changing their SSH
keys, changing Git hooks, adding people to Patchwork, manually applying more
duct tape to the integration between these services, and fixing the duct tape
when it breaks (which is surprisingly often). As a volunteer admin for the
service, doing these kinds of things is not exactly the reason we get out of
bed in the morning; it also consumes so much time treading water that we
haven’t been able to enable new features and workflows for the projects we
Seeking better workflows
As of writing, around one third of the non-dormant projects on fd.o have at some
point migrated their development elsewhere; mostly to GitHub. Sometimes this was
because the other sites were a more natural home (e.g. to sibling projects), and
sometimes just because they offered a better workflow (integration between issue
tracking and commits, web-based code review, etc). Other projects which would
have found fd.o a natural home have gone straight to hosting externally, though
they may use some of our services - particularly mailing lists.
Not everyone wants to make use of these features, and not everyone will. For
example, the kernel might well never move away from
email for issue tracking and code review. But
the evidence shows us that many others do want to, and our platform will be a
non-starter for them unless we provide the services they want.
A bit over three years ago, I set up an instance of Phabricator at Collabora to
replace our mix of Bugzilla, Redmine, Trac, and JIRA. It was a great fit for how
we worked internally, and upstream seemed like a good fit too; though they were
laser-focused on their usecases, their extremely solid data storage and
processing model made it quite easy to extend, and projects like MediaWiki,
Haskell, LLVM and more were beginning to switch over to use it as their tracker.
I set up an instance on fd.o, and we started to use it for a couple of trial
projects: some issue tracking and code review for Wayland and Weston,
development of PiTiVi, and so on.
The first point we seriously discussed it more widely was at XDC 2016 in
Helsinki, where Eric Anholt gave a talk about our broken
infrastructure, cleverly disguised as
something about test suites. It became clear that we had wide interest in and
support for better infrastructure, though with some reservation about particular
workflows. There was quite a bit of hallway discussion afterwards, as Eric and
Adam Jackson in particular tried out Phabricator and gave some really good
feedback on its usability. At that point, it was clear that some fairly major UI
changes were required to make it usable for our needs, especially for drive-by
contributors and new users.
Last year, GNOME went through a similar
Carlos and some of the other members being more familiar with GitLab, myself and
Emmanuele Bassi made the case for using Phabricator, based on our experiences
with it at Collabora and Endless respectively. At the time, our view was that
whilst GitLab’s code review was better, the issue tracking (being much like
GitHub’s) would not really scale to our needs. This was mostly based on having
last evaluated GitLab during the 8.x series; whilst the discussions were going
on, GitLab were making giant strides in issue tracking throughout 9.x.
With GitLab coming up to par on issue tracking, both Emmanuele and I ended up
fully supporting GNOME’s decision to base their infrastructure on GitLab. The UI
changes required to Phabricator were not really tractable for the resources we
had, the code review was and will always be fundamentally
unsuitable being based around the
Subversion-like model of reviewing large branches in one go, and upstream were
also beginning to move to a much more closed community model.
By contrast, one of the things which really impressed us about GitLab was how
openly they worked, and how open they were to collaboration. Early on in GNOME’s
journey to GitLab, they dropped their old
replace it with a DCO, and Eliran Mesika
from GitLab’s partnership team came to GUADEC to listen and understand how GNOME
worked and what they needed from GitLab. Unfortunately this was too early in the
process for us, but Robert McQueen later introduced us, and Eliran and I started
talking about how they could help freedesktop.org.
One of our bigger issues was infrastructure. Not only were our services getting
long in the tooth, but so were the machines they ran on. In order to stand up a
large new service, we’d need new physical machines, but a fleet of new machines
was beyond the admin time we had. It also didn’t solve issues such as everyone’s
favourite: half of Europe can’t route to fd.o for half an hour most mornings due
to obscure network issues with our host we’ve had no success diagnosing or
GitLab Inc. listened to our predicament and suggested a solution to help us:
that they would sponsor our hosting on Google Cloud Platform for an initial
period to get us on our feet. This involves us running the completely
open-source GitLab Community Edition on infrastructure we control ourselves,
whilst freeing us from having to worry about failing and full disks or creaking
networks. (As with GNOME, we politely declined the offer of a license to the
pay-for GitLab Enterprise Edition; we wanted to be fully in control of our
infrastructure, and on a level playing field with the rest of the open-source
They have also offered us support, from helping a cloud idiot understand how to
deploy and maintain services on Kubernetes, to taking the time to listen and
understand our workflows and improve GitLab for our uses. Much of the fruit of
this is already visible in GitLab through feedback from us and GNOME, though
there is always more to come. In particular, one area we’re looking at is
integration with mailing lists and placing tags in commit messages, so
developers used to mail-based workflows can continue to consume the firehose
through email, rather than being required to use the web UI for everything.
Last Christmas, we gave ourselves the present of standing up
gitlab.freedesktop.org on GCP, and set about
gradually making it usable and maintainable for our projects. Our first hosted
project was Panfrost, who were running on
either non-free services or non-collaborative hosted services. We wanted to help
them out by getting them on to fd.o, but they didn’t want to use the services we
had at the time, and we didn’t want to add new projects to those services
Over time, as we stabilised the deployment and fleshed out the feature set, we
added a few smaller projects, who understood the experimental nature and gave us
space to make some mistakes, have some down time, and helped us smooth out the
rough edges. Some of the blocker here was migrating bugs: though we reused
GNOME’s bztogl script, we needed some
adjustments for our different setups, as well as various bugfixes.
Not long ago, we migrated Mesa’s repository
hosting as well as
Weston for both repository
tracking and issue tracking which are our biggest projects to date.
What we offer to projects
With GitLab, we offer everything you would expect from
gitlab.com (their hosted offering), or everything you
would expect from GitHub with the usual external services such as Travis CI.
This includes issue tracking integrated with repository management (close issues
by pushing), merge requests with online review and merge, a comprehensive CI
suite with shared runners
available to all, custom sites built with whatever toolchain you like, external
web hooks to integrate with other services, and a well-documented stable API
which allows you to use external clients like git
In theory, we’ve always provided most of the above services. Most of these - if
you ignore the lack of integration between them - were more or less fine for
projects running their own standalone infrastructure. But they didn’t scale to
something like fd.o, where we have a very disparate family of projects sharing
little in common, least of all common infrastructure and practices. For example,
we did have a Jenkins deployment for a while, but it became very clear very
early that this did not scale out to
was impossible for us to empower projects to run their own CI without fatally
Anyone familiar with the long wait for an admin to add an account or change an
SSH key will be relieved to hear that this is no longer. Anyone can make an
account on our GitLab instance using an email address and password, or with
trusted external identity providers (currently Google, gitlab.com, GitHub, or
Twitter) rather than having another username and password. We delegate
permission management to project owners: if you want to give someone commit
rights to your project, go right ahead. No need to wait for us.
We also support such incredible leading-edge security features as two-factor
TOTP authentication for your account, Recaptcha to protect against spammers, and
ways of deleting spam which don’t involve an admin sighing into a SQL console
for half an hour, trying to not accidentally delete all the content.
Having an integrated CI system allows our projects to run test pipelines on
merge requests, giving people fast feedback about any required changes without
human intervention, and making sure distcheck works all the time, rather than
just the week before release. We can capture and store logs, binaries and more
The same powerful system is also the engine for GitLab Pages: you can use static
site generators like Jekyll and Hugo, or have a very spartan, hand-written
site but also host auto-generated documentation. The
choice is yours: running everything in (largely) isolated containers means that
you can again do whatever you like with your own sites, without having to ask
admins to set up some duct-taped triggers from Git repositories, then ask them
to fix it when they’ve upgraded Python and everything has mysteriously stopped
Migration to GitLab, and legacy services
Now that we have a decent and battle-tested service to offer, we can look to
what this means for our other services.
Phabricator will be decommissioned immediately; a read-only archive will be
taken of public issues and code reviews and maintained as static pages forever,
and a database dump will also be kept. But we do not plan to bring this service
back, as all the projects using it have already migrated away from it.
Similarly, Jenkins has already been decommissioned and deactivated some time
Whilst we are encouraging projects to migrate their issue tracking away from
Bugzilla and helping those who do, we realise a lot of projects have built their
workflows around Bugzilla. We will continue to maintain our Bugzilla
installation and support existing projects with its use, though we are not
offering Bugzilla to new projects anymore, and over the long term would like to
see Bugzilla eventually retired.
Patchwork (already currently maintained by Intel for their KMS and Mesa work) is
in the same boat, complicated by the fact that the kernel might never move away
from patches carved into stone tablets.
Hopefully it goes without saying that our mailing lists are going to be
long-lived, even if better issue tracking and code review does mean they’re a
little less-trafficked than before.
Perhaps most importantly, we have anongit and cgit. anongit is not provided
by GitLab, as they rightly prefer to serve repositories over https. Given
that, for all existing projects we are maintaining anongit.fd.o as a
read-only mirror of GitLab; there are far too many distributions, build
scripts, and users out there with anongit URIs to discontinue the service.
Over time we will encourage these downstreams to move to HTTPS to lessen the
pressure, but this will continue to live for quite some time. Having cgit
live alongside anongit is fairly painless, so we will keep it running whilst
it isn’t a burden.
Lastly, annarchy.fd.o (aka people.fd.o) is currently offered as a
general-purpose shell host. People use this to manage their Git repositories
on people.fd.o and their files publicly served there. Since it is also the
primary web host for most projects, both people and scripts use it to deploy
files to sites. Some people use it for random personal file storage, to run
various scripts and even as a personal IRC host. We are trying to transition
these people away from using annarchy for this, as it is difficult for us
to provide totally arbitrary resources to everyone who has at one point had
an account with one of our member projects. Running a source of lots of IRC
traffic is also a good way to make yourself deeply unpopular with many hosts.
Migrating your projects
After being iterated and fleshed out, we are happy to offer to migrate all the
projects. For each project, we will ask you to file an
using the migration template. This gives you a checklist with all the
information we need to migrate your GitLab repositories, as well as your
existing Bugzilla bugs.
Every user with a freedesktop.org SSH account already has an account created
for them on GitLab, with access to the same groups. In order to recover
access to the migrated accounts, you can request a password-reset link by
entering the email address you signed up with into the ‘forgotten password’
box on the GitLab front page.
More information is available on the freedesktop GitLab
and of course the admins are happy to help if you have any problems with this.
The usual failure mode is that your email address has changed since you signed
up: we’ve had one user who needed it changed as they were still using a Yahoo!
Governance and process
Away from technical issues, we’re also looking to inject a lot more transparency
into our processes. For instance, why do we host kernel graphics
development, but not
new filesystems? What do
we look for (both good and bad), and why is that? What is freedesktop.org even
for, and who is it serving?
This has just been folk knowledge for some time; passed on by oral legend over
IRC as verbal errata to out-of-date wiki pages. Just as with technical
issues, this is not healthy for anyone: it’s more difficult for people to
get involved and give us the help we so clearly need, it’s more difficult for
our community to find out what they can expect from us and how we can help them,
and it’s impossible for anyone to measure how good a job we’re actually doing.
One of the reasons we haven’t done a great job at this is because just keeping
the Rube Goldberg machine of our infrastructure running exhausts basically all
the time we have to deal with fd.o. The time we spend changing someone’s SSH
keys by hand, or debugging a Git
post-receive hook, is time we’re not spending
on the care and feeding of our community.
We’ve spent the past couple of years paying down our technical debt, and the
community equivalent thereof. Our infrastructure is much less error-prone than
it was: we’ve gone from fighting fires to being able to prepare the new GitLab
infrastructure and spend time shepherding projects through it. Now that we have
a fair few projects on GitLab and they’ve been able to serve themselves, we’ve
been able to take some time for community issues.
Writing down our processes is still a work in
something we’ve made a little more headway on is governance. Currently fd.o’s
governance is myself, Keith and Tollef
discussing things and coming to some kind of conclusion. Sometimes that’s in
recorded public fora, sometimes over email with searchable archives, sometimes
just over IRC message or verbally with no public record of what happened.
Given that there’s a huge overlap between our mission and that of the X.Org
Foundation (which is a lot more than just X11!), one
idea we’re exploring is to bring fd.o under the Foundation’s oversight, with
clear responsibility, accountability, and delegated roles. The combination of
the two should give our community much more insight into what we’re doing and
why - as well as, crucially, the chance to influence it.
Of course, this is all conditional on fd.o speaking to our member projects, and
the Foundation speaking to its individual members, and getting wide agreement.
There will be a lot of tuning required - not least, the Foundation’s bylaws
would need a change which needs a formal vote from the membership - but this at
least seems like a promising avenue.