Planet Collabora

March 23, 2017

Simon McVittie

GTK hackfest 2017: D-Bus communication with containers

At the GTK hackfest in London (which accidentally became mostly a Flatpak hackfest) I've mainly been looking into how to make D-Bus work better for app container technologies like Flatpak and Snap.

The initial motivating use cases are:

  • Portals: Portal authors need to be able to identify whether the container is being contacted by an uncontained process (running with the user's full privileges), or whether it is being contacted by a contained process (in a container created by Flatpak or Snap).

  • dconf: Currently, a contained app either has full read/write access to dconf, or no access. It should have read/write access to its own subtree of dconf configuration space, and no access to the rest.

At the moment, Flatpak runs a D-Bus proxy for each app instance that has access to D-Bus, connects to the appropriate bus on the app's behalf, and passes messages through. That proxy is in a container similar to the actual app instance, but not actually the same container; it is trusted to not pass messages through that it shouldn't pass through. The app-identification mechanism works in practice, but is Flatpak-specific, and has a known race condition due to process ID reuse and limitations in the metadata that the Linux kernel maintains for AF_UNIX sockets. In practice the use of X11 rather than Wayland in current systems is a much larger loophole in the container than this race condition, but we want to do better in future.

Meanwhile, Snap does its sandboxing with AppArmor, on kernels where it is enabled both at compile-time (Ubuntu, openSUSE, Debian, Debian derivatives like Tails) and at runtime (Ubuntu, openSUSE and Tails, but not Debian by default). Ubuntu's kernel has extra AppArmor features that haven't yet gone upstream, some of which provide reliable app identification via LSM labels, which dbus-daemon can learn by querying its AF_UNIX socket. However, other kernels like the ones in openSUSE and Debian don't have those. The access-control (AppArmor mediation) is implemented in upstream dbus-daemon, but again doesn't work portably, and is not sufficiently fine-grained or flexible to do some of the things we'll likely want to do, particularly in dconf.

After a lot of discussion with dconf maintainer Allison Lortie and Flatpak maintainer Alexander Larsson, I think I have a plan for fixing this.

This is all subject to change: see fd.o #100344 for the latest ideas.

Identity model

Each user (uid) has some uncontained processes, plus 0 or more containers.

The uncontained processes include dbus-daemon itself, desktop environment components such as gnome-session and gnome-shell, the container managers like Flatpak and Snap, and so on. They have the user's full privileges, and in particular they are allowed to do privileged things on the user's session bus (like running dbus-monitor), and act with the user's full privileges on the system bus. In generic information security jargon, they are the trusted computing base; in AppArmor jargon, they are unconfined.

The containers are Flatpak apps, or Snap apps, or other app-container technologies like Firejail and AppImage (if they adopt this mechanism, which I hope they will), or even a mixture (different app-container technologies can coexist on a single system). They are containers (or container instances) and not "apps", because in principle, you could install com.example.MyApp 1.0, run it, and while it's still running, upgrade to com.example.MyApp 2.0 and run that; you'd have two containers for the same app, perhaps with different permissions.

Each container has an container type, which is a reversed DNS name like org.flatpak or io.snapcraft representing the container technology, and an app identifier, an arbitrary non-empty string whose meaning is defined by the container technology. For Flatpak, that string would be another reversed DNS name like com.example.MyGreatApp; for Snap, as far as I can tell it would look like example-my-great-app.

The container technology can also put arbitrary metadata on the D-Bus representation of a container, again defined and namespaced by the container technology. For instance, Flatpak would use some serialization of the same fields that go in the Flatpak metadata file at the moment.

Finally, the container has an opaque container identifier identifying a particular container instance. For example, launching com.example.MyApp twice (maybe different versions or with different command-line options to flatpak run) might result in two containers with different privileges, so they need to have different container identifiers.

Contained server sockets

App-container managers like Flatpak and Snap would create an AF_UNIX socket inside the container, bind() it to an address that will be made available to the contained processes, and listen(), but not accept() any new connections. Instead, they would fd-pass the new socket to the dbus-daemon by calling a new method, and the dbus-daemon would proceed to accept() connections after the app-container manager has signalled that it has called both bind() and listen(). (See fd.o #100344 for full details.)

Processes inside the container must not be allowed to contact the AF_UNIX socket used by the wider, uncontained system - if they could, the dbus-daemon wouldn't be able to distinguish between them and uncontained processes and we'd be back where we started. Instead, they should have the new socket bind-mounted into their container's XDG_RUNTIME_DIR and connect to that, or have the new socket set as their DBUS_SESSION_BUS_ADDRESS and be prevented from connecting to the uncontained socket in some other way. Those familiar with the kdbus proposals a while ago might recognise this as being quite similar to kdbus' concept of endpoints, and I'm considering reusing that name.

Along with the socket, the container manager would pass in the container's identity and metadata, and the method would return a unique, opaque identifier for this particular container instance. The basic fields (container technology, technology-specific app ID, container ID) should probably be added to the result of GetConnectionCredentials(), and there should be a new API call to get all of those plus the arbitrary technology-specific metadata.

When a process from a container connects to the contained server socket, every message that it sends should also have the container instance ID in a new header field. This is OK even though dbus-daemon does not (in general) forbid sender-specified future header fields, because any dbus-daemon that supported this new feature would guarantee to set that header field correctly, the existing Flatpak D-Bus proxy already filters out unknown header fields, and adding this header field is only ever a reduction in privilege.

The reasoning for using the sender's container instance ID (as opposed to the sender's unique name) is for services like dconf to be able to treat multiple unique bus names as belonging to the same equivalence class of contained processes: instead of having to look up the container metadata once per unique name, dconf can look it up once per container instance the first time it sees a new identifier in a header field. For the second and subsequent unique names in the container, dconf can know that the container metadata and permissions are identical to the one it already saw.

Access control

In principle, we could have the new identification feature without adding any new access control, by keeping Flatpak's proxies. However, in the short term that would mean we'd be adding new API to set up a socket for a container without any access control, and having to keep the proxies anyway, which doesn't seem great; in the longer term, I think we'd find ourselves adding a second new API to set up a socket for a container with new access control. So we might as well bite the bullet and go for the version with access control immediately.

In principle, we could also avoid the need for new access control by ensuring that each service that will serve contained clients does its own. However, that makes it really hard to send broadcasts and not have them unintentionally leak information to contained clients - we would need to do something more like kdbus' approach to multicast, where services know who has subscribed to their multicast signals, and that is just not how dbus-daemon works at the moment. If we're going to have access control for broadcasts, it might as well also cover unicast.

The plan is that messages from containers to the outside world will be mediated by a new access control mechanism, in parallel with dbus-daemon's current support for firewall-style rules in the XML bus configuration, AppArmor mediation, and SELinux mediation. A message would only be allowed through if the XML configuration, the new container access control mechanism, and the LSM (if any) all agree it should be allowed.

By default, processes in a container can send broadcast signals, and send method calls and unicast signals to other processes in the same container. They can also receive method calls from outside the container (so that interfaces like org.freedesktop.Application can work), and send exactly one reply to each of those method calls. They cannot own bus names, communicate with other containers, or send file descriptors (which reduces the scope for denial of service).

Obviously, that's not going to be enough for a lot of contained apps, so we need a way to add more access. I'm intending this to be purely additive (start by denying everything except what is always allowed, then add new rules), not a mixture of adding and removing access like the current XML policy language.

There are two ways we've identified for rules to be added:

  • The container manager can pass a list of rules into the dbus-daemon at the time it attaches the contained server socket, and they'll be allowed. The obvious example is that an org.freedesktop.Application needs to be allowed to own its own bus name. Flatpak apps' implicit permission to talk to portals, and Flatpak metadata like org.gnome.SessionManager=talk, could also be added this way.

  • System or session services that are specifically designed to be used by untrusted clients, like the version of dconf that Allison is working on, could opt-in to having contained apps allowed to talk to them (effectively making them a generalization of Flatpak portals). The simplest such request, for something like a portal, is "allow connections from any container to contact this service"; but for dconf, we want to go a bit finer-grained, with all containers allowed to contact a single well-known rendezvous object path, and each container allowed to contact an additional object path subtree that is allocated by dconf on-demand for that app.

Initially, many contained apps would work in the first way (and in particular sockets=session-bus would add a rule that allows almost everything), while over time we'll probably want to head towards recommending more use of the second.

Related topics

Access control on the system bus

We talked about the possibility of using a very similar ruleset to control access to the system bus, as an alternative to the XML rules found in /etc/dbus-1/system.d and /usr/share/dbus-1/system.d. We didn't really come to a conclusion here.

Allison had the useful insight that the XML rules are acting like a firewall: they're something that is placed in front of potentially-broken services, and not part of the services themselves (which, as with firewalls like ufw, makes it seem rather odd when the services themselves install rules). D-Bus system services already have total control over what requests they will accept from D-Bus peers, and if they rely on the XML rules to mediate that access, they're essentially rejecting that responsibility and hoping the dbus-daemon will protect them. The D-Bus maintainers would much prefer it if system services took responsibility for their own access control (with or without using polkit), because fundamentally the system service is always going to understand its domain and its intended security model better than the dbus-daemon can.

Analogously, when a network service listens on all addresses and accepts requests from elsewhere on the LAN, we sometimes work around that by protecting it with a firewall, but the optimal resolution is to get that network service fixed to do proper authentication and access control instead.

For system services, we continue to recommend essentially this "firewall" configuration, filling in the ${} variables as appropriate:

    <policy user="${the daemon uid under which the service runs}">
        <allow own="${the service's bus name}"/>
    <policy context="default">
        <allow send_destination="${the service's bus name}"/>

We discussed the possibility of moving towards a model where the daemon uid to be allowed is written in the .service file, together with an opt-in to "modern D-Bus access control" that makes the "firewall" unnecessary; after some flag day when all significant system services follow that pattern, dbus-daemon would even have the option of no longer applying the "firewall" (moving to an allow-by-default model) and just refusing to activate system services that have not opted in to being safe to use without it. However, the "firewall" also protects system bus clients, and services like Avahi that are not bus-activatable, against unintended access, which is harder to solve via that approach; so this is going to take more thought.

For system services' clients that follow the "agent" pattern (BlueZ, polkit, NetworkManager, Geoclue), the correct "firewall" configuration is more complicated. At some point I'll try to write up a best-practice for these.

New header fields for the system bus

At the moment, it's harder than it needs to be to provide non-trivial access control on the system bus, because on receiving a method call, a service has to remember what was in the method call, then call GetConnectionCredentials() to find out who sent it, then only process the actual request when it has the information necessary to do access control.

Allison and I had hoped to resolve this by adding new D-Bus message header fields with the user ID, the LSM label, and other interesting facts for access control. These could be "opt-in" to avoid increasing message sizes for no reason: in particular, it is not typically useful for session services to receive the user ID, because only one user ID is allowed to connect to the session bus anyway.

Unfortunately, the dbus-daemon currently lets unknown fields through without modification. With hindsight this seems an unwise design choice, because header fields are a finite resource (there are 255 possible header fields) and are defined by the D-Bus Specification. The only field that can currently be trusted is the sender's unique name, because the dbus-daemon sets that field, overwriting the value in the original message (if any).

To make it safe to rely on the new fields, we would have to make the dbus-daemon filter out all unknown header fields, and introduce a mechanism for the service to check (during connection to the bus) whether the dbus-daemon is sufficiently new that it does so. If connected to an older dbus-daemon, the service would not be able to rely on the new fields being true, so it would have to ignore the new fields and treat them as unset. The specification is sufficiently vague that making new dbus-daemons filter out unknown header fields is a valid change (it just says that "Header fields with an unknown or unexpected field code must be ignored", without specifying who must ignore them, so having the dbus-daemon delete those fields seems spec-compliant).

This all seemed fine when we discussed it in person; but GDBus already has accessors for arbitrary header fields by numeric ID, and I'm concerned that this might mean it's too easy for a system service to be accidentally insecure: It would be natural (but wrong!) for an implementor to assume that if g_message_get_header (message, G_DBUS_MESSAGE_HEADER_FIELD_SENDER_UID) returned non-NULL, then that was guaranteed to be the correct, valid sender uid. As a result, fd.o #100317 might have to be abandoned. I think more thought is needed on that one.

Unrelated topics

As happens at any good meeting, we took the opportunity of high-bandwidth discussion to cover many useful things and several useless ones. Other discussions that I got into during the hackfest included, in no particular order:

  • .desktop file categories and how to adapt them for AppStream, perhaps involving using the .desktop vocabulary but relaxing some of the hierarchy restrictions so they behave more like "tags"
  • how to build a recommended/reference "app store" around Flatpak, aiming to host upstream-supported builds of major projects like LibreOffice
  • how Endless do their content-presenting and content-consuming apps in GTK, with a lot of "tile"-based UIs with automatic resizing and reflowing (similar to responsive design), and the applicability of similar widgets to GNOME and upstream GTK
  • whether and how to switch GNOME developer documentation to Hotdoc
  • whether pies, fish and chips or scotch eggs were the most British lunch available from Borough Market
  • the distinction between stout, mild and porter

More notes are available from the GNOME wiki.


The GTK hackfest was organised by GNOME and hosted by Red Hat and Endless. My attendance was sponsored by Collabora. Thanks to all the sponsors and organisers, and the developers and organisations who attended.

March 23, 2017 06:07 PM

February 27, 2017 - Robert Foss

Removing the Chromebook Write-Protect screw

This post will look specifically at removing the WP screw from a Chell (HP Chromebook 13 G1) device, and verifying that it has been successfully removed.

To actually flash firmware to Chromebook machines, a device called a Servo is needed. While these devices aren't available publicly, they can be produced freely or possibly requested from Google if you are contributing code to the ChromiumOS project.

Removing the Write-Protect screw

Alt text

So this is what the WP screw looks like on a Chell Chromebook. This may or may not be what you will find in other devices. But if you take a close look, you will notice that the copper pad that the the screw attaches against is split into parts that are bridged by a screw being inserted.

Disable Write-Protect

So this is the part that requires a Servo device. And a ChromiumOS checkout, for some help setting one up, have a look at my previous post.

# Go to your ChromiumOS checkout
cd /opt/chromiumos

# Enter dev environment

# Set device variable
export BOARD=chell

# Connect to Chromebook using a Servo device
sudo servod -b $BOARD &

# Disable WP
# This step may vary depending on the hardware of your actual Chromebook
dut-control fw_wp:off
sudo /usr/sbin/flashrom -p ft2232_spi:type=servo-v2 --wp-disable
sudo /usr/sbin/flashrom -p ec --wp-disable


ChromiumOS Servo
Setting up a ChromiumOS dev environment


This post has been a part of work undertaken by my employer Collabora.

by Robert Foss at February 27, 2017 03:55 PM

February 24, 2017

Gustavo Padovan

Slides from my talk at ELC!

For those who are interested here are the slides of my talk at Embedded Linux Conference about the Unification of the Android and Mainline Kernel Graphics Stack. ELC in Portland was a great conference this year, with lots of great talks and folks to chat with. The video of the talks will be published soon, so stay tuned fo updates on that. See you around!

by Gustavo Padovan at February 24, 2017 04:31 PM

February 22, 2017 - Robert Foss

Precompiling APK files during Android AOSP build

Enable precompilation

cd /opt/aosp_checkout/

find . -name

# Edit the that you are using for your build
cd device/linaro/generic/

# Add the config option

# Propagate the new settings
make defconfig
make all

Increase system partition size

Depending on the previous system partition size and how many APKs that are built, you may need to increase the system partition size.

nano configs/defconfig

# Change the below variable to something big enough to house all of you binaries

# Propagate the new settings
make defconfig
make all


This post has been a part of work undertaken by my employer Collabora.

by Robert Foss at February 22, 2017 03:55 PM

February 21, 2017

Gustavo Padovan

Collabora Contributions to Linux Kernel 4.10

Linux Kernel v4.10 is out and this time Collabora contributed a total of 39 patches by 10 different developers. You can read more about the v4.10 merge window on part 1, part 2 and part 3.

Now here is a look at the changes made by Collaborans. To begin with Daniel Stone fixed an issue when waiting for fences on the i915 driver, while Emil Velikov added support to read the PCI revision for sysfs to improve the starting time in some applications.

Emilio López added a set of selftests for the Sync File Framework and Enric Balletbo i Serra added support for the ChromeOS Embedded Controller Sensor Hub. Fabien Lahoudere added support for the NVD9128 simple panel and enabled ULPI phy for USB on i.MX.

Gabriel Krisman fixed a spurious CARD_INT interrupts for SD cards that was preventing one of our kernelCI machines to boot. On the graphics side Gustavo Padovan added Explicit Synchronization support to DRM/KMS.

Martyn Welch added GPIO support for CP2105 USB serial device while Nicolas Dufresne fixed Exynos4 FIMC to roundup imagesize to row size for tiled formats, otherwise there would be enough space to fit the last row of the image. Last but not least, Tomeu Vizoso added debugfs interface to capture frames CRCs, which is quite helpful for debugging and automated graphics testing.

And now the complete list of Collabora contributions:

Daniel Stone (1):

Emil Velikov (1):

Emilio López (7):

Enric Balletbo i Serra (3):

Fabien Lahoudere (4):

Gabriel Krisman Bertazi (1):

Gustavo Padovan (18):

Martyn Welch (1):

Nicolas Dufresne (1):

Tomeu Vizoso (2):

by Gustavo Padovan at February 21, 2017 04:02 PM

February 16, 2017 - Robert Foss

Setting up a ChromiumOS dev environment

Set up environment

export DEV_DIR="/opt"
mkdir -p $DEV_DIR
export CHROMIUM_DIR="$DEV_DIR/chromiumos"
mkdir -p $CHROMIUM_DIR
export PATH="$DEV_DIR/depot_tools:$PATH"

# The BOARD variable used here is specific for the Chromebook that is
# being targeted, a more generic target like "amd64-generic" could
# be more useful for you needs.
export BOARD=chell

# The USB_DEVICE variable refers to the USB device that will be used
# for flashing ChromiumOS onto a Chromebook.
# Make sure that this device does not contain anything important!
export USB_DEVICE="/dev/sda"

Install dependencies

sudo apt install git-core gitk git-gui subversion curl
git clone

Get ChromiumOS source

repo init -u
repo sync -j25

Build ChromiumOS

cros_sdk -- ./build_packages --board=${BOARD}
cros_sdk -- ./build_image --board=${BOARD}

Flash ChromiumOS to storage medium

cros_sdk -- cros flash --board=${BOARD} usb:/$USB_DEVICE

Install ChromiumOS on Chromebook

Enter Chromebook into dev-mode

This part is highly device specific, and depends on how the manufacturer of your device has chosen to implement the dev-mode switch.

A partial list of devices and how to enter them into dev-mode can be found here.

Flash ChromiumOS to Chromebook

Fire up your Chrombook device and hit Ctrl+Alt+Back, followed by 'chronos' and hit enter. Followed by the below command to install the ChromiumOS build that was just flashed.


Debug an application

On Chromebook

Again fire up your Chrombook device and hit Ctrl+Alt+Back, followed by 'chronos' and hit enter.

# Remount the root drive read / write
sudo mount -o remount,rw /

# Open port so that gdbserver can be reached
sudo /sbin/iptables -A INPUT  -p tcp --dport 1234 -j ACCEPT

# Run gdb server, listening on port 1234 (opened in iptables command above)
sudo gdbserver :1234 chrome

On dev machine

# On x86
cros_sdk -- sudo USE=expat emerge cross-i686-pc-linux-gnu/gdb
# On ARMv7
cros_sdk -- sudo USE=expat emerge cross-armv7a-cros-linux-gnueabi/gdb

cros_sdk -- i686-pc-linux-gnu-gdb "/build/$BOARD/opt/google/chrome/chrome"
(gdb) set sysroot /build/$BOARD/
(gdb) target remote IP_ADDR_CHROMEBOOK:1234
(gdb) continue


This is a bit of a rough outline, and is only suitable for Chromebook devices that already are in dev-mode.


This post has been a part of work undertaken by my employer Collabora.


ChromiumOS Depo Tools
ChromiumOS Quick Start
ChromiumOS Dev Mode
ChromiumOS Debug

by Robert Foss at February 16, 2017 09:31 AM

January 26, 2017

Gustavo Padovan

Mainline Explicit Fencing – part 3

In the last two articles we talked about how Explicit Fencing can help the graphics pipeline in general and what happened on the effort to upstream the Android Sync Framework. Now on the third post of this series we will go through the Explicit Fencing implementation on DRM and other elements of the graphics stack.

The DRM implementation lays down on top of two kernel infrastructures, struct dma_fence, which represents the fence and struct sync file that provides the file descriptors to be shared with userspace (as it was discussed in the previous articles). With fencing the display infrastructure needs to wait for a signal on that fence before displaying the buffer on the screen. On a Explicit Fencing implementation that fence is sent from userspace to the kernel. The display infrastructure also sends back to userspace a fence, encapsulated in a struct sync_file, that will be signalled when the buffer is scanned out on the screen. The same process happens on the rendering side.

It is mandatory to use of Atomic Modesetting and here is not plan to support legacy APIs. The fence that DRM will wait on needs to be passed via the IN_FENCE_FD property for each DRM plane, that means it will receive one sync_file fd containing one or more dma_fence per plane. Remember that in DRM a plane directly relates to a framebuffer so one can also say that there is one sync_file per framebuffer.

On the other hand for the fences created by the kernel that are sent back to userspace the OUT_FENCE_PTR property is used. It is a DRM CRTC property because we only create one dma_fence per CRTC as all the buffers on it will be scanned out at the same time. The kernel sends this fence back to userspace by writing the fd number to the pointer provided in the OUT_FENCE_PTR property. Note that, unlike from what Android did, when the fence signals it means the previous buffer – the buffer removed from the screen – is free for reuse. On Android when the signal was raised it meant the current buffer was freed. However, the Android folks have patched SurfaceFlinger already to support the Mainline semantics when using Explicit Fencing!

Nonetheless, that is only one side of the equation and to have the full graphics pipeline running with Explicit Fencing we need to support it on the rendering side as well. As every rendering driver has its own userspace API we need to add Explicit Fencing support to every single driver there. The freedreno driver already has its Explicit Fencing support  mainline and there is work in progress to add support to i915 and virtio_gpu.

On the userspace side Mesa already has support for the EGL_ANDROID_native_fence_sync needed to use Explicit Fencing on Android. Libdrm incorporated the headers to access the sync file IOCTL wrappers. On Android, libsync now has support for both the old Android Sync and Mainline Sinc File APIs. And finally, on drm_hwcomposer, patches to use Atomic Modesetting and Explicit Fencing are available but they are not upstream yet.

Validation tests for both Sync Files and fences on the Atomic API were written and added to IGT.

by Gustavo Padovan at January 26, 2017 03:23 PM

December 22, 2016

Tomeu Vizoso

Slides on the Chamelium board

Yesterday I gave a short talk about the Chamelium board from the ChromeOS team, and thought that the slides could be useful for others as this board gets used more and more outside of Google.

If you are interested in how this board can help you automate the testing of your display (and not only!) code and hardware, a new mailing list has been created to discuss its uses. We at Collabora will be happy to help you integrate this board in your CI lab as well.

Thanks go to Intel for sponsoring the preparation of these slides and for allowing me to share them under an open license.

And of course, thanks to Google's ChromeOS team for releasing the hardware design with an open hardware license along with the code they are running on it and with it.

by Tomeu Vizoso ( at December 22, 2016 08:58 AM

December 14, 2016

Gustavo Padovan

Collabora Contributions to Linux Kernel 4.9

Linux Kernel 4.9 was released this week and once more Collabora developers took part on the kernel development cycle. This time we contributed 37 patches by 11 different developers, our highest number of single contributors in a kernel release ever. Remember that in the previous release we had our highest number total contributions. The numbers shows how Collabora have been increasing its commitment in contributing to the upstream kernel community.

For those who want to see an overall report of what was happened in the 4.9 kernel take a look  on the always good LWN articles: part 1, 2  and 3.

As for Collabora contributions most of our work was in the DRM and DMABUF subsystems. Andrew Shadura and Daniel Stone added to fixes to the AMD and i915 drivers respectively. Emilio López added the missing install of sync_file.h uapi.

Gustavo Padovan advanced a few more steps on the goal to add explicit fencing to the DRM subsystem, besides a few improvements to Sync File and the virtio_gpu driver he also de-staged the SW_SYNC validation framework that helps with Sync File testing.

Peter Senna added drm_bridge support to imx-ldb device while Tomeu Vizoso improved drm_bridge support on RockChip’s analogic-dp and added documentation about validation of the DRM subsystem.

Outside of the Graphics world we had Enric Balletbo i Serra adding support to upload firmware on the ziirave watchdog device. Fabien Lahoudere and Martyn Welch enabled and improved DMA support for i.MX53 UARTs, allowing the device tree to decide whether DMA is used or not. Martyn also added a fake VMEbus (Versa Module Europa bus) to help with VME driver development.

On the Bluetooth, subsystem Frédéric Dalleau fixed an error code for SCO connections, that was causing big timeout and failures on SCO connections requests. Finally Robert Foss worked to clear the pipeline on errors for cdc-wdm USB devices.

Andrew Shadura (1):

Daniel Stone (1):

Emilio López (2):

Enric Balletbo i Serra (1):

Fabien Lahoudere (3):

Frédéric Dalleau (1):

Gustavo Padovan (14):

Martyn Welch (4):

Peter Senna Tschudin (1):

Robert Foss (2):

Tomeu Vizoso (7):

by Gustavo Padovan at December 14, 2016 05:12 PM

December 07, 2016 - Robert Foss

Running an Android Unit Test

A similar approach can be used for any Android module.

cd aosp
source build/ && \
lunch linaro_arm64-userdebug && \
mmm system/core/libsync/tests && \
adb root && \
adb remount && \
adb sync && \
adb shell /data/nativetest64/sync-unit-tests/sync-unit-tests

by Robert Foss at December 07, 2016 07:23 PM

November 21, 2016

Gustavo Noronha Silva

A tale of cylinders and shadows

Like I wrote before, we at Collabora have been working on improving WebKitGTK+ performance for customer projects, such as Apertis. We took the opportunity brought by recent improvements to WebKitGTK+ and GTK+ itself to make the final leg of drawing contents to screen as efficient as possible. And then we went on investigating why so much CPU was still being used in some of our test cases.

The first weird thing we noticed is performance was actually degraded on Wayland compared to running under X11. After some investigation we found a lot of time was being spent inside GTK+, painting the window’s background.

Here’s the thing: the problem only showed under Wayland because in that case GTK+ is responsible for painting the window decorations, whereas in the X11 case the window manager does it. That means all of that expensive blurring and rendering of shadows fell on GTK+’s lap.

During the web engines hackfest, a couple of months ago, I delved deeper into the problem and noticed, with Carlos Garcia’s help, that it was even worse when HiDPI displays were thrown into the mix. The scaling made things unbearably slower.

You might also be wondering why would painting of window decorations be such a problem, anyway? They should only be repainted when a window changes size or state anyway, which should be pretty rare, right? Right, that is one of the reasons why we had to make it fast, though: the resizing experience was pretty terrible. But we’ll get back to that later.

So I dug into that, made a few tries at understanding the issue and came up with a patch showing how applying the blur was being way too expensive. After a bit of discussion with our own Pekka Paalanen and Benjamin Otte we found the root cause: a fast path was not being hit by pixman due to the difference in scale factors on the shadow mask and the target surface. We made the shadow mask scale the same as the surface’s and voilà, sane performance.

I keep talking about this being a performance problem, but how bad was it? In the following video you can see how huge the impact in performance of this problem was on my very recent laptop with a HiDPI display. The video starts with an Epiphany window running with a patched GTK+ showing a nice demo the WebKit folks cooked for CSS animations and 3D transforms.

After a few seconds I quickly alt-tab to the version running with unpatched GTK+ – I made the window the exact size and position of the other one, so that it is under the same conditions and the difference can be seen more easily. It is massive.

Yes, all of that slow down was caused by repainting window shadows! OK, so that solved the problem for HiDPI displays, made resizing saner, great! But why is GTK+ repainting the window even if only the contents are changing, anyway? Well, that turned out to be an off-by-one bug in the code that checks whether the invalidated area includes part of the window decorations.

If the area being changed spanned the whole window width, say, it would always cause the shadows to be repainted. By fixing that, we now avoid all of the shadow drawing code when we are running full-window animations such as the CSS poster circle or gtk3-demo’s pixbufs demo.

As you can see in the video below, the gtk3-demo running with the patched GTK+ (the one on the right) is using a lot less CPU and has smoother animation than the one running with the unpatched GTK+ (left).

Pretty much all of the overhead caused by window decorations is gone in the patched version. It is still using quite a bit of CPU to animate those pixbufs, though, so some work still remains. Also, the overhead added to integrate cairo and GL rendering in GTK+ is pretty significant in the WebKitGTK+ CSS animation case. Hopefully that’ll get much better from GTK+ 4 onwards.

by kov at November 21, 2016 05:04 PM

November 15, 2016

Héctor Orón Martínez

Open Build Service in Debian needs YOU! ☞

“Open Build Service is a generic system to build and distribute packages from sources in an automatic, consistent and reproducible way.”


openSUSE distributions’ build system is based on a generic framework named Open Build Service (OBS), I have been using these tools in my work environment, and I have to say, as Debian developer, that it is a great tool. In the current blog post I plan for you to learn the very basics of such tool and provide you with a tutorial to get, at least, a Debian package building.


Fig 1 – Open Build Service Architecture

The figure above shows Open Build Service, from now on OBS, software architecture. There are several parts which we should differenciate:

  • Web UI / API (obs-api)
  • Backend (obs-server)
  • Build daemon / worker (obs-worker)
  • CLI tool to manage API (osc)

Each one of the above packages can be installed in separated machines as a distributed architecture, it is very easy to split the system into several machines running the services, however in the tutorial below everything installs in one machine.


The backend is composed of several scripts written either in shell or Perl. There are several services running in the backend:

  • Source service
  • Repository service
  • Scheduler service
  • Dispatcher service
  • Warden service
  • Publisher service
  • Signer service
  • DoD service

The backend manages source packages (any format such RPM, DEB, …) and schedules them for a build in the worker. Once the package is built it can be published in a repository for the wider audience or kept unpublished and used by other builds.


System can have several worker machines which are encharged to perform the package builds. There are different options that can be configured (see /etc/default/obsworker) such enabling switch, number of worker instances, jobs per instance. This part of the system is written in shell and/or Perl language.


The frontend allows in a clickable way to get around most options OBS provides: setup projects, upload/branch/delete packages, submit review requests, etc. As an example, you can see a live instance running at

The frontend parts are really a Ruby-on-rails web application, we (mainly thanks to Andrew Lee with ruby team help) have tried to get it nicely running, however we have had lots of issues due to javascripts or rubygems malfunctioning. Current webui is visible and provides some package status, however most actions do not work properly, or configurations cannot be applied as editor does not save changes, projects or packages in a project are not listed either. If you are a Ruby-on-rails expert or if you are able to help us out with some of the webui issues we get at Debian that would be really appreciated from our side.


OSC is a managing command line tool, written in Python, that interfaces with OBS API to be able to perform actions, edit configurations, do package reviews, etc.


Now that we have done a general overview of the system, let me introduce you to OBS with a practical tutorial.

TUTORIAL: Build a Debian package against Debian 8.0 using Download On Demand (DoD) service.

by zumbi at November 15, 2016 11:05 AM

November 08, 2016

Tomeu Vizoso

How continuous integration can help you keep pace with the Linux kernel

Almost all of Collabora's customers use the Linux kernel on their products. Often they will use the exact code as delivered by the SBC vendors and we'll work with them in other pars of their software stack. But it's becoming increasingly common for our customers to adapt the kernel sources to the specific needs of their particular products.

A very big problem most of them have is that the kernel version they based on isn't getting security updates any more because it's already several years old. And the reason why companies are shipping kernels so old is that they have been so heavily modified compared to the upstream versions, that rebasing their trees on top of newer mainline releases is so expensive that is very hard to budget and plan for it.

To avoid that, we always recommend our customers to stay close to their upstreams, which implies rebasing often on top of new releases (typically LTS releases, with long term support). For the budgeting of that work to become possible, the size of the delta between mainline and downstream sources needs to be manageable, which is why we recommend contributing back any changes that aren't strictly specific to their products.

But even for those few companies that already have processes in place for upstreaming their changes and are rebasing regularly on top of new LTS releases, keeping up with mainline can be a substantial disruption of their production schedules. This is in part because new bugs will be in the new mainline release, and new bugs will be in the downstream changes as they get applied to the new version.

Those companies that are already keeping close to their upstreams typically have advanced QA infrastructure that will detect those bugs long before production, but a long stabilization phase after every rebase can significantly slow product development.

To improve this situation and encourage more companies to keep their efforts close to upstream we at Collabora have been working for a few years already in continuous integration of FOSS components across a diverse array of hardware. The initial work was sponsored by Bosch for one of their automotive projects, and since the start of 2016 Google has been sponsoring work on continuous integration of the mainline kernel.

One of the major efforts to continuously integrate the mainline Linux kernel codebase is, which builds several configurations of different trees and submits boot jobs to several labs around the world, collating the results. This is being of great help already in detecting at a very early stage any changes that either break the builds, or prevent a specific piece of hardware from completing the boot stage.

Though can easily detect when an update to a source code repository has introduced a bug, such updates can have several dozens of new commits, and without knowing which specific commit introduced the bug, we cannot identify culprits to notify of the problem. This means that either someone needs to monitor the dashboard for problems, or email notifications are sent to the owners of the repositories who then have to manually look for suspicious commits before getting in contact with their author.

To address this limitation, Google has asked us to look into improving the existing code for automatic bisection so it can be used right away when a regression is detected, so the possible culprits are notified right away without any manual intervention.

Another area in which is currently lacking is in the coverage of the testing. Build and boot regressions are very annoying for developers because they impact negatively everybody who work in the affected configurations and hardware, but the consequences of regressions in peripheral support or other subsystems that aren't involved critically during boot can still make rebases much costlier.

At Collabora we have had a strong interest in having the DRM subsystem under continuous integration and some time ago started a R&D project for making the test suite in IGT generically useful for all the DRM drivers. IGT started out being i915-specific, but as most of the tests exercise the generic DRM ABI, they could as well test other drivers with a moderate amount of effort. Early in 2016 Google started sponsoring this work and as of today submitters of new drivers are using it to validate their code.

Another related effort has been the addition to DRM of a generic ABI for retrieving CRCs of frames from different components in the graphics pipeline, so two frames can be compared when we know that they should match. And another one is adding support to IGT for the Chamelium board, which can simulate several display connections and hotplug events.

A side-effect of having continuous integration of changes in mainline is that when downstreams are sending back changes to reduce their delta, the risk of introducing regressions is much smaller and their contributions can be accepted faster and with less effort.

We believe that improved QA of FOSS components will expand the base of companies that can benefit from involvement in development upstream and are very excited by the changes that this will bring to the industry. If you are an engineer who cares about QA and FOSS, and would like to work with us on projects such as, LAVA, IGT and Chamelium, get in touch!

by Tomeu Vizoso ( at November 08, 2016 02:26 PM

October 28, 2016

Pekka Paalanen

Waltham: a generic Wayland-style IPC over network

I have recently been occupied with a new project (and being with a cold all this week), so I have not been much present in the Wayland community. Now I can finally say what I and Emilio have been up to: Waltham! For more information, please see our annoucement.

by pq ( at October 28, 2016 11:49 AM

October 20, 2016

Héctor Orón Martínez

Build a Debian package against Debian 8.0 using Download On Demand (DoD) service

In the previous post Open Build Service software architecture has been overviewed. In the current blog post, a tutorial on setting up a package build with OBS from Debian packages is presented.


  • Generate a test environment by creating Stretch/SID VM
  • Enable experimental repository
  • Install OBS server, api, worker and osc CLI packages
  • Ensure all OBS services are running
  • Create an OBS project for Download on Demand (DoD)
  • Create an OBS project linked to DoD
  • Adding a package to the project
  • Troubleshooting OBS

Generate a test environment by creating Stretch/SID VM

Really, use whatever suits you best, but please create an untrusted test environment for this one.

In the current tutorial it assumes “$hostname” is “stretch”, which should be stretch or sid suite.

Be aware that copy & paste configuration files from current post might lead you into broken characters (i.e. “).

Debian Stretch weekly netinst CD

Enable experimental repository

# echo "deb experimental main" >> /etc/apt/sources.list.d/experimental.list
# apt-get update

Install and setup OBS server, api, worker and osc CLI packages

# apt-get install obs-server obs-api obs-worker osc

In the install process mysql database is needed, therefore if mysql server is not setup, a password needs to be provided.
When OBS API database ‘obs-api‘ is created, we need to pick a password for it, provide “opensuse”. The ‘obs-api’ package will configure apache2 https webserver (creating a dummy certificate for “stretch”) to serve OBS webui.
Add “stretch” and “obs” aliases to “localhost” entry in your /etc/hosts file.
Enable worker by setting ENABLED=1 in /etc/default/obsworker
Try to connect to the web UI https://stretch/
Login into OBS webui, default login credentials: Admin/opensuse).
From command line tool, try to list projects in OBS

 $ osc -A https://stretch ls

Accept dummy certificate and provide credentials (defaults: Admin/opensuse)
If the install proceeds as expected follow to the next step.

Ensure all OBS services are running

# backend services
obsrun     813  0.0  0.9 104960 20448 ?        Ss   08:33   0:03 /usr/bin/perl -w /usr/lib/obs/server/bs_dodup
obsrun     815  0.0  1.5 157512 31940 ?        Ss   08:33   0:07 /usr/bin/perl -w /usr/lib/obs/server/bs_repserver
obsrun    1295  0.0  1.6 157644 32960 ?        S    08:34   0:07  \_ /usr/bin/perl -w /usr/lib/obs/server/bs_repserver
obsrun     816  0.0  1.8 167972 38600 ?        Ss   08:33   0:08 /usr/bin/perl -w /usr/lib/obs/server/bs_srcserver
obsrun    1296  0.0  1.8 168100 38864 ?        S    08:34   0:09  \_ /usr/bin/perl -w /usr/lib/obs/server/bs_srcserver
memcache   817  0.0  0.6 346964 12872 ?        Ssl  08:33   0:11 /usr/bin/memcached -m 64 -p 11211 -u memcache -l
obsrun     818  0.1  0.5  78548 11884 ?        Ss   08:33   0:41 /usr/bin/perl -w /usr/lib/obs/server/bs_dispatch
obsserv+   819  0.0  0.3  77516  7196 ?        Ss   08:33   0:05 /usr/bin/perl -w /usr/lib/obs/server/bs_service
mysql      851  0.0  0.0   4284  1324 ?        Ss   08:33   0:00 /bin/sh /usr/bin/mysqld_safe
mysql     1239  0.2  6.3 1010744 130104 ?      Sl   08:33   1:31  \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/ --socket=/var/run/mysqld/mysqld.sock --port=3306

# web services
root      1452  0.0  0.1 110020  3968 ?        Ss   08:34   0:01 /usr/sbin/apache2 -k start
root      1454  0.0  0.1 435992  3496 ?        Ssl  08:34   0:00  \_ Passenger watchdog
root      1460  0.3  0.2 651044  5188 ?        Sl   08:34   1:46  |   \_ Passenger core
nobody    1465  0.0  0.1 444572  3312 ?        Sl   08:34   0:00  |   \_ Passenger ust-router
www-data  1476  0.0  0.1 855892  2608 ?        Sl   08:34   0:09  \_ /usr/sbin/apache2 -k start
www-data  1477  0.0  0.1 856068  2880 ?        Sl   08:34   0:09  \_ /usr/sbin/apache2 -k start
www-data  1761  0.0  4.9 426868 102040 ?       Sl   08:34   0:29 delayed_job.0
www-data  1767  0.0  4.8 425624 99888 ?        Sl   08:34   0:30 delayed_job.1
www-data  1775  0.0  4.9 426516 101708 ?       Sl   08:34   0:28 delayed_job.2
nobody    1788  0.0  5.7 496092 117480 ?       Sl   08:34   0:03 Passenger RubyApp: /usr/share/obs/api
nobody    1796  0.0  4.9 488888 102176 ?       Sl   08:34   0:00 Passenger RubyApp: /usr/share/obs/api
www-data  1814  0.0  4.5 282576 92376 ?        Sl   08:34   0:22 delayed_job.1000
www-data  1829  0.0  4.4 282684 92228 ?        Sl   08:34   0:22 delayed_job.1010
www-data  1841  0.0  4.5 282932 92536 ?        Sl   08:34   0:22 delayed_job.1020
www-data  1855  0.0  4.9 427988 101492 ?       Sl   08:34   0:29 delayed_job.1030
www-data  1865  0.2  5.0 492500 102964 ?       Sl   08:34   1:09 clockworkd.clock
www-data  1899  0.0  0.0  87100  1400 ?        S    08:34   0:00 /usr/bin/searchd --pidfile --config /usr/share/obs/api/config/production.sphinx.conf
www-data  1900  0.1  0.4 161620  8276 ?        Sl   08:34   0:51  \_ /usr/bin/searchd --pidfile --config /usr/share/obs/api/config/production.sphinx.conf

# OBS worker
root      1604  0.0  0.0  28116  1492 ?        Ss   08:34   0:00 SCREEN -m -d -c /srv/obs/run/worker/boot/screenrc
root      1605  0.0  0.9  75424 18764 pts/0    Ss+  08:34   0:06  \_ /usr/bin/perl -w ./bs_worker --hardstatus --root /srv/obs/worker/root_1 --statedir /srv/obs/run/worker/1 --id stretch:1 --reposerver http://obs:5252 --jobs 1

Create an OBS project for Download on Demand (DoD)

Create a meta project file:

$ osc -A https://stretch:443 meta prj Debian:8 -e

<project name=”Debian:8″>
<title>Debian 8 DoD</title>
<description>Debian 8 DoD</description>
<person userid=”Admin” role=”maintainer”/>
<repository name=”main”>
<download arch=”x86_64″ url=”” repotype=”deb”/>

Visit webUI to check project configuration

Create a meta project configuration file:

$ osc -A https://stretch:443 meta prjconf Debian:8 -e

Add the following file, as found at

Repotype: debian

# create initial user
Preinstall: base-passwd
Preinstall: user-setup

# required for preinstall images
Preinstall: perl

# preinstall essentials + dependencies
Preinstall: base-files base-passwd bash bsdutils coreutils dash debconf
Preinstall: debianutils diffutils dpkg e2fslibs e2fsprogs findutils gawk
Preinstall: gcc-4.9-base grep gzip hostname initscripts insserv libacl1
Preinstall: libattr1 libblkid1 libbz2-1.0 libc-bin libc6 libcomerr2 libdb5.3
Preinstall: libgcc1 liblzma5 libmount1 libncurses5 libpam-modules
Preinstall: libpcre3 libsmartcols1
Preinstall: libpam-modules-bin libpam-runtime libpam0g libreadline6
Preinstall: libselinux1 libsemanage-common libsemanage1 libsepol1 libsigsegv2
Preinstall: libslang2 libss2 libtinfo5 libustr-1.0-1 libuuid1 login lsb-base
Preinstall: mount multiarch-support ncurses-base ncurses-bin passwd perl-base
Preinstall: readline-common sed sensible-utils sysv-rc sysvinit sysvinit-utils
Preinstall: tar tzdata util-linux zlib1g

Runscripts: base-passwd user-setup base-files gawk

VMinstall: libdevmapper1.02.1

Order: user-setup:base-files

# Essential packages (this should also pull the dependencies)
Support: base-files base-passwd bash bsdutils coreutils dash debianutils
Support: diffutils dpkg e2fsprogs findutils grep gzip hostname libc-bin 
Support: login mount ncurses-base ncurses-bin perl-base sed sysvinit 
Support: sysvinit-utils tar util-linux

# Build-essentials
Required: build-essential
Prefer: build-essential:make

# build script needs fakeroot
Support: fakeroot
# lintian support would be nice, but breaks too much atm
#Support: lintian

# helper tools in the chroot
Support: less kmod net-tools procps psmisc strace vim

# everything below same as for Debian:6.0 (apart from the version macros ofc)

# circular dependendencies in openjdk stack
Order: openjdk-6-jre-lib:openjdk-6-jre-headless
Order: openjdk-6-jre-headless:ca-certificates-java

Keep: binutils cpp cracklib file findutils gawk gcc gcc-ada gcc-c++
Keep: gzip libada libstdc++ libunwind
Keep: libunwind-devel libzio make mktemp pam-devel pam-modules
Keep: patch perl rcs timezone

Prefer: cvs libesd0 libfam0 libfam-dev expect

Prefer: gawk locales default-jdk
Prefer: xorg-x11-libs libpng fam mozilla mozilla-nss xorg-x11-Mesa
Prefer: unixODBC libsoup glitz java-1_4_2-sun gnome-panel
Prefer: desktop-data-SuSE gnome2-SuSE mono-nunit gecko-sharp2
Prefer: apache2-prefork openmotif-libs ghostscript-mini gtk-sharp
Prefer: glib-sharp libzypp-zmd-backend mDNSResponder

Prefer: -libgcc-mainline -libstdc++-mainline -gcc-mainline-c++
Prefer: -libgcj-mainline -viewperf -compat -compat-openssl097g
Prefer: -zmd -OpenOffice_org -pam-laus -libgcc-tree-ssa -busybox-links
Prefer: -crossover-office -libgnutls11-dev

# alternative pkg-config implementation
Prefer: -pkgconf
Prefer: -openrc
Prefer: -file-rc

Conflict: ghostscript-library:ghostscript-mini

Ignore: sysvinit:initscripts

Ignore: aaa_base:aaa_skel,suse-release,logrotate,ash,mingetty,distribution-release
Ignore: gettext-devel:libgcj,libstdc++-devel
Ignore: pwdutils:openslp
Ignore: pam-modules:resmgr
Ignore: rpm:suse-build-key,build-key
Ignore: bind-utils:bind-libs
Ignore: alsa:dialog,pciutils
Ignore: portmap:syslogd
Ignore: fontconfig:freetype2
Ignore: fontconfig-devel:freetype2-devel
Ignore: xorg-x11-libs:freetype2
Ignore: xorg-x11:x11-tools,resmgr,xkeyboard-config,xorg-x11-Mesa,libusb,freetype2,libjpeg,libpng
Ignore: apache2:logrotate
Ignore: arts:alsa,audiofile,resmgr,libogg,libvorbis
Ignore: kdelibs3:alsa,arts,pcre,OpenEXR,aspell,cups-libs,mDNSResponder,krb5,libjasper
Ignore: kdelibs3-devel:libvorbis-devel
Ignore: kdebase3:kdebase3-ksysguardd,OpenEXR,dbus-1,dbus-1-qt,hal,powersave,openslp,libusb
Ignore: kdebase3-SuSE:release-notes
Ignore: jack:alsa,libsndfile
Ignore: libxml2-devel:readline-devel
Ignore: gnome-vfs2:gnome-mime-data,desktop-file-utils,cdparanoia,dbus-1,dbus-1-glib,krb5,hal,libsmbclient,fam,file_alteration
Ignore: libgda:file_alteration
Ignore: gnutls:lzo,libopencdk
Ignore: gnutls-devel:lzo-devel,libopencdk-devel
Ignore: pango:cairo,glitz,libpixman,libpng
Ignore: pango-devel:cairo-devel
Ignore: cairo-devel:libpixman-devel
Ignore: libgnomeprint:libgnomecups
Ignore: libgnomeprintui:libgnomecups
Ignore: orbit2:libidl
Ignore: orbit2-devel:libidl,libidl-devel,indent
Ignore: qt3:libmng
Ignore: qt-sql:qt_database_plugin
Ignore: gtk2:libpng,libtiff
Ignore: libgnomecanvas-devel:glib-devel
Ignore: libgnomeui:gnome-icon-theme,shared-mime-info
Ignore: scrollkeeper:docbook_4,sgml-skel
Ignore: gnome-desktop:libgnomesu,startup-notification
Ignore: python-devel:python-tk
Ignore: gnome-pilot:gnome-panel
Ignore: gnome-panel:control-center2
Ignore: gnome-menus:kdebase3
Ignore: gnome-main-menu:rug
Ignore: libbonoboui:gnome-desktop
Ignore: postfix:pcre
Ignore: docbook_4:iso_ent,sgml-skel,xmlcharent
Ignore: control-center2:nautilus,evolution-data-server,gnome-menus,gstreamer-plugins,gstreamer,metacity,mozilla-nspr,mozilla,libxklavier,gnome-desktop,startup-notification
Ignore: docbook-xsl-stylesheets:xmlcharent
Ignore: liby2util-devel:libstdc++-devel,openssl-devel
Ignore: yast2:yast2-ncurses,yast2-theme-SuSELinux,perl-Config-Crontab,yast2-xml,SuSEfirewall2
Ignore: yast2-core:netcat,hwinfo,wireless-tools,sysfsutils
Ignore: yast2-core-devel:libxcrypt-devel,hwinfo-devel,blocxx-devel,sysfsutils,libstdc++-devel
Ignore: yast2-packagemanager-devel:rpm-devel,curl-devel,openssl-devel
Ignore: yast2-devtools:perl-XML-Writer,libxslt,pkgconfig
Ignore: yast2-installation:yast2-update,yast2-mouse,yast2-country,yast2-bootloader,yast2-packager,yast2-network,yast2-online-update,yast2-users,release-notes,autoyast2-installation
Ignore: yast2-bootloader:bootloader-theme
Ignore: yast2-packager:yast2-x11
Ignore: yast2-x11:sax2-libsax-perl
Ignore: openslp-devel:openssl-devel
Ignore: java-1_4_2-sun:xorg-x11-libs
Ignore: java-1_4_2-sun-devel:xorg-x11-libs
Ignore: kernel-um:xorg-x11-libs
Ignore: tetex:xorg-x11-libs,expat,fontconfig,freetype2,libjpeg,libpng,ghostscript-x11,xaw3d,gd,dialog,ed
Ignore: yast2-country:yast2-trans-stats
Ignore: susehelp:susehelp_lang,suse_help_viewer
Ignore: mailx:smtp_daemon
Ignore: cron:smtp_daemon
Ignore: hotplug:syslog
Ignore: pcmcia:syslog
Ignore: avalon-logkit:servlet
Ignore: jython:servlet
Ignore: ispell:ispell_dictionary,ispell_english_dictionary
Ignore: aspell:aspel_dictionary,aspell_dictionary
Ignore: smartlink-softmodem:kernel,kernel-nongpl
Ignore: OpenOffice_org-de:myspell-german-dictionary
Ignore: mediawiki:php-session,php-gettext,php-zlib,php-mysql,mod_php_any
Ignore: squirrelmail:mod_php_any,php-session,php-gettext,php-iconv,php-mbstring,php-openssl

Ignore: simias:mono(log4net)
Ignore: zmd:mono(log4net)
Ignore: horde:mod_php_any,php-gettext,php-mcrypt,php-imap,php-pear-log,php-pear,php-session,php
Ignore: xerces-j2:xml-commons-apis,xml-commons-resolver
Ignore: xdg-menu:desktop-data
Ignore: nessus-libraries:nessus-core
Ignore: evolution:yelp
Ignore: mono-tools:mono(gconf-sharp),mono(glade-sharp),mono(gnome-sharp),mono(gtkhtml-sharp),mono(atk-sharp),mono(gdk-sharp),mono(glib-sharp),mono(gtk-sharp),mono(pango-sharp)
Ignore: gecko-sharp2:mono(glib-sharp),mono(gtk-sharp)
Ignore: gnome-libs:libgnomeui
Ignore: nautilus:gnome-themes
Ignore: gnome-panel:gnome-themes
Ignore: gnome-panel:tomboy

Substitute: utempter

%ifnarch s390 s390x ppc ia64
Substitute: java2-devel-packages java-1_4_2-sun-devel
 %ifnarch s390x
Substitute: java2-devel-packages java-1_4_2-ibm-devel
Substitute: java2-devel-packages java-1_4_2-ibm-devel xorg-x11-libs-32bit

Substitute: yast2-devel-packages docbook-xsl-stylesheets doxygen libxslt perl-XML-Writer popt-devel sgml-skel update-desktop-files yast2 yast2-devtools yast2-packagemanager-devel yast2-perl-bindings yast2-testsuite

# SUSE compat mappings
Substitute: gcc-c++ gcc
Substitute: libsigc++2-devel libsigc++-2.0-dev
Substitute: glibc-devel-32bit
Substitute: pkgconfig pkg-config

%ifarch %ix86
Substitute: kernel-binary-packages kernel-default kernel-smp kernel-bigsmp kernel-debug kernel-um kernel-xen kernel-kdump
%ifarch ia64
Substitute: kernel-binary-packages kernel-default kernel-debug
%ifarch x86_64
Substitute: kernel-binary-packages kernel-default kernel-smp kernel-xen kernel-kdump
%ifarch ppc
Substitute: kernel-binary-packages kernel-default kernel-kdump kernel-ppc64 kernel-iseries64
%ifarch ppc64
Substitute: kernel-binary-packages kernel-ppc64 kernel-iseries64
%ifarch s390
Substitute: kernel-binary-packages kernel-s390
%ifarch s390x
Substitute: kernel-binary-packages kernel-default

%define debian_version 800

%debian_version 800

Visit webUI to check project configuration

Create an OBS project linked to DoD

$ osc -A https://stretch:443 meta prj test -e

<project name=”test”>
<person userid=”Admin” role=”maintainer”/>
<repository name=”Debian_8.0″>
<path project=”Debian:8″ repository=”main”/>

Visit webUI to check project configuration

Adding a package to the project

$ osc -A https://stretch:443 co test ; cd test
$ mkdir hello ; cd hello ; apt-get source -d hello ; cd - ; 
$ osc add hello 
$ osc ci -m "New import" hello

The package should go to dispatched state then get in blocked state while it downloads build dependencies from DoD link, eventually it should start building. Please check the journal logs to check if something went wrong or gets stuck.

Visit webUI to check hello package build state

OBS logging to the journal

Check in the journal logs everything went fine:

$ sudo journalctl -u obsdispatcher.service -u obsdodup.service -u obsscheduler@x86_64.service -u obsworker.service -u obspublisher.service


Currently we are facing few issues with web UI:

And there are more issues that have not been reported, please do ‘reportbug obs-api‘.

by zumbi at October 20, 2016 07:58 AM

October 17, 2016

Gustavo Padovan

Mainline Explicit Fencing – part 2

In the first post we covered the main concepts behind Explicit Synchronization for the Linux Kernel. Now in the second post of the series we are going to look to the Android Sync Framework, the first (out-of-tree) Explicit Fencing implementation for the Linux Kernel.

The Sync Framework was the Android solution to implement Explicit Fencing in AOSP. It uses file descriptors to communicate fencing information between userspace and kernel and between userspace process.

In the Sync Framework it all starts with the creation of a Sync Timeline, a struct created for each driver context to represent a monotonically increasing counter. It is the Sync Timeline who will guarantee the ordering between fences in the same Timeline. The driver contexts could be different GPU rings, or different Displays on your hardware.

Sync Timeline

Sync Timeline

Then we have Sync Points(sync_pt), the name Android gave to fences, they represent a specific value in the Sync Timeline. When created the Sync Point is initialized in the Active state, and when it signals, i.e., the job it was associated to finishes, it transits to the Signaled state and informs the Sync Timeline to update the value of the last signaled Sync Point.

Sync Point

Sync Point

To export and import Sync Points to/from userspace the Sync Fence struct is used. Under the hood the the Sync Fence is a Linux file and we use thte Sync Fence to store Sync Point information. To exported to userspace a unused file descriptor(fd) is associated to the Sync Fence file. Drivers can then use the file descriptor to pass the Sync Point information around.

Sync Fence

Sync Fence

The Sync Fence is usually created just after the Sync Point creation, it then travel through the pipeline, via userspace, until the driver that is going to wait for the Sync Fence to signal. The Sync Fence signal when the Sync Point inside it signals.

One of the most important features of the Android Sync Framework is the ability to merge Sync Fences into a new Sync Fence containing all Sync Points from both Sync Fences. It can contain as many Sync Points as your resource allows. A merged Sync Fence will only signal when all its Sync Points signals.

Sync Fence with Merged fences

Sync Fence with Merged Fences. Here we merge two Sync Points into one Sync File.

When it comes to userspace API the Sync Framework has implements three ioctl calls. The first one is to wait on sync_fence to signal. There is also a call to merge two sync_fences into a third and new sync_fence. And finally there is a also a call to grab information about the sync_fence and all its sync_points.

The Sync Fences fds are passed to/from the kernel in the calls to ask the kernel to render or display a buffer.

This was intended to be a overview of the Sync Framework as we will see some of these concepts on the next article where we will talk about the effort to add explict fencing on mainline kernel. If you want to learn more about the Sync Framework you can find more info here and here.

by Gustavo Padovan at October 17, 2016 04:45 PM

October 05, 2016

Gustavo Noronha Silva

Web Engines Hackfest 2016!

I had a great time last week and the web engines hackfest! It was the 7th web hackfest hosted by Igalia and the 7th hackfest I attended. I’m almost a local Galician already. Brazilian Portuguese being so close to Galician certainly helps! Collabora co-sponsored the event and it was great that two colleagues of mine managed to join me in attendance.

It had great talks that will eventually end up in videos uploaded to the web site. We were amazed at the progress being made to Servo, including some performance results that blew our minds. We also discussed the next steps for WebKitGTK+, WebKit for Wayland (or WPE), our own Clutter wrapper to WebKitGTK+ which is used for the Apertis project, and much more.

Zan giving his talk on WPE (former WebKitForWayland)Zan giving his talk on WPE (former WebKitForWayland)

One thing that drew my attention was how many Dell laptops there were. Many collaborans (myself included) and igalians are now using Dells, it seems. Sure, there were thinkpads and macbooks, but there was plenty of inspirons and xpses as well. It’s interesting how the brand make up shifted over the years since 2009, when the hackfest could easily be mistaken with a thinkpad shop.

Back to the actual hackfest: with the recent release of Gnome 3.22 (and Fedora 25 nearing release), my main focus was on dealing with some regressions suffered by users experienced after a change that made putting the final rendering composited by the nested Wayland compositor we have inside WebKitGTK+ to the GTK+ widget so it is shown on the screen.

One of the main problems people reported was applications that use WebKitGTK+ not showing anything where the content was supposed to appear. It turns out the problem was caused by GTK+ not being able to create a GL context. If the system was simply not able to use GL there would be no problem: WebKit would then just disable accelerated compositing and things would work, albeit slower.

The problem was WebKit being able to use an older GL version than the minimum required by GTK+. We fixed it by testing that GTK+ is able to create GL contexts before using the fast path, falling back to the slow glReadPixels codepath if not. This way we keep accelerated compositing working inside WebKit, which gives us nice 3D transforms and less repainting, but take the performance hit in the final “blit”.

Introducing "WebKitClutterGTK+"Introducing “WebKitClutterGTK+”

Another issue we hit was GTK+ not properly updating its knowledge of the window’s opaque region when painting a frame with GL, which led to some really interesting issues like a shadow appearing when you tried to shrink the window. There was also an issue where the window would not use all of the screen when fullscreen which was likely related. Both were fixed.

André Magalhães also worked on a couple of patches we wrote for customer projects and are now pushing upstream. One enables the use of more than one frontend to connect to a remote web inspector server at once. This can be used to, for instance, show the regular web inspector on a browser window and also use IDE integration for setting breakpoints and so on.

The other patch was cooked by Philip Withnall and helped us deal with some performance bottlenecks we were hitting. It improves the performance of painting scroll bars. WebKitGTK+ does its own painting of scrollbars (we do not use the GTK+ widgets for various reasons). It turns out painting scrollbars can be quite a hit when the page is being scrolled fast, if not done efficiently.

Emanuele Aina had a great time learning more about meson to figure out a build issue we had when a more recent GStreamer was added to our jhbuild environment. He came out of the experience rather sane, which makes me think meson might indeed be much better than autotools.

Igalia 15 years cakeIgalia 15 years cake

It was a great hackfest, great seeing everyone face to face. We were happy to celebrate Igalia’s 15 years with them. Hope to see everyone again next year =)

by kov at October 05, 2016 12:23 PM

October 03, 2016

Gustavo Padovan

Collabora Contributions to Linux Kernel 4.8

Linux Kernel 4.8 is out and once more Collabora engineers did a significant contribution to the Kernel. For the 4.8 Collabora contributed 101 patches by 8 engineers, our record to date in single kernel release! We’ve also seen the first contribution from Frederic Dalleau since he joined Collabora. covered the new features of the new kernel in three different posts, here, here and here.

On the Collabora side of the contributions we touched a few different areas in the kernel. Bob Ham, who recently left Collabora, added support for the Alea I Random Number Generator, while Enric Balletbo improved the audio support on the Rockchip rk3288 SoC. Frederic Dalleau fixed an important memory leak on the Bluetooth stack.

Gustavo Padovan continued his work add Explicit Synchronization for Buffer Sharing on the kernel. In this release he added fence_array support and prepared the SW_SYNC interfaces for de-staging, SW_SYNC meant to be used for Explict Syncronization testing. He also worked in removing some of the legacy functions from drm_irq.c from the kernel.

Helen Koike added some improvements and clean ups to the ASoC subsystem mainly on the max9877 and tpa6130a2 drivers. Nicolas Dufresne fixed the bytes per line calculation on YUV planes on the uvcvideo driver.

Thierry Escande added many improvements the NFC digital layer and Tomeu Vizoso added a new helper for the ChromeOS Embedded Controller and improved usage of DRM Core APIs on the Rockchip driver. He also fixed an issue with the Analogix DP on Rockchip that was not enabling clocks in the correct order.

Bob Ham (2):

Enric Balletbo i Serra (8):

Frederic Dalleau (1):

Gustavo Padovan (50):

Helen Koike (8):

Nicolas Dufresne (1):

Thierry Escande (26):

Tomeu Vizoso (5):

by Gustavo Padovan at October 03, 2016 01:59 PM

September 25, 2016

Gustavo Padovan

My talk about Mainline Explicit Fencing at XDC 2016!

Last week I was at XDC in Helsinki where I presented about the Explicit Fencing work we’ve been doing on the Mainline Linux Kernel in the lastest few months. There was a livestream of all presentations during the conference and recorded sections are available. You can check the video of my presentation. Check out the slides too.

If you want to check the code we’ve been writing they are available here:

Linux Kernel:




Soon we will get Explicit Fencing on Android’s drm_hwcomposer as well so expect updates on this blog with more information about that. :)

Also I would like to take the opportunity to thank Collabora for sponsoring my travel to XDC and Martin Peres for organizing such a great conference. It was my first time attending XDC and my time there was absolutely great, I  have learnt a lot about what the Graphics community have been doing lately and I met the people doing this work. I was happy to see a lot of interest from many people around the Explicit Fencing work we’ve doing.


by Gustavo Padovan at September 25, 2016 06:23 PM

September 23, 2016 - Robert Foss

XDC 2016

Alt text

XDC 2016 was hosted in Helsinki at Haaga-Helia. The full program was filmed and is archived here.

[Slides] FastUIDraw - High Performance 2D renderer for GPUs

Kevin Rogovin gave an excellent talk about FastUIDraw, which is a highly optimiser 2d renderer for GPUs.

By agressively targetting GPUs only and limiting the feature set to what is required by a browser FastUIDraw performs >9.3x faster than Cairo-CPU and >4.8x times the previous GPU state of the art SKIA-GL.

Hopefully FastUIDraw can be incorporated into upstream of ChromiumOS and Android.

[Slides] 2D Performance

Martin Perez gave a talk about 2D performance and power consumption of the Xserver.

The xf86-video-intel driver showed quite mixed performance numbers of 1.51-32.6x times the CPU performance. The FPS/Watt measurements showed 0.73-15.1x times the CPU efficiency.

When looking at Cairo traces the power efficiency is actually lower using GPU accelaration than with CPU acceleration. This is somewhat expected in that the Cairo workload is not very high throughput, so the overhead of doing 2D operations is relatively high compared to the actual work.

Overall toolkits are moving away from letting the Xserver do 2D rendering, for reasons of portability and performance.

[Slides] libglvnd Status Update

Andy Ritger gave a talk about the current libglvnd status. The goal of libglvnd is to allow different graphics libraries from potentially different vendors to coexist on a filesystem and in a process. GLX/EGl/OpenGL/OpenGLES/GL are all supported by libglvnd.

Currently mesa supports libglvnd for OpenGL and GLX, with EGL support being in the pipeline.

[Slides] drm_hwcomposer

Sean Paul and Zach Reizner gave an exceptionally well timed talk about the Android && Chromium drm_hwcomposer project. drm_hwcomposer is an implementation of the hwcomposer (HWC) API ontop of the Linux DRM/KMS.

The talk detailed the implications of HWC2 and explicit fencing. A large part of the HWC1 implementation can be removed from drm_hwcomposer since it's made redundant by the fencing support in HWC2. So, for example the worker threads (DrmCompositorWorker and FrameWorker) are no longer necessary and can be removed.

[Slides] Status update of Nouveau

Samuel Pitoiset, Karol Herbst, Pierre Moreau and Martin Perez gave a talk about what has happened in Nouveau land the last year.

Hardware support is as always taking steps forward, with Fermi support is scheduled for Linux v4.9+.

A call to arms for Nouveau compiler optimizations was issued by Martin Perez. There is a lot of low hanging fruit for optimizations is available, and the compiler is overall in a good shape. So feel free to contact Martin or the Nouveau project if you are interested or curious.

As of the Maxwell generation of GPUs the firmware needed now has to be signed, GM20x/GP100 firmware has been released, but support for loading firmware on Tegra has not been provided by NVidia.

Martin made a rather clear point of needing to see some cooperation from NVidia in order for the Nouveau project to be able to make progress with new and upcoming NVidia hardware.


Thanks to the X.Org Foundation and the board of directors for arranging XDC 2016. This post has been a part of work undertaken by my employer Collabora.

by Robert Foss at September 23, 2016 06:20 PM

September 22, 2016

Gustavo Noronha Silva

WebKitGTK+ 2.14 and the Web Engines Hackfest

Next week our friends at Igalia will be hosting this year’s Web Engines Hackfest. Collabora will be there! We are gold sponsors, and have three developers attending. It will also be an opportunity to celebrate Igalia’s 15th birthday \o/. Looking forward to meet you there! =)

Carlos Garcia has recently released WebKitGTK+ 2.14, the latest stable release. This is a great release that brings a lot of improvements and works much better on Wayland, which is becoming mature enough to be used by default. In particular, it fixes the clipboard, which was one of the main missing features, thanks to Carlos Garnacho! We have also been able to contribute a bit to this release =)

One of the biggest changes this cycle is the threaded compositor, which was implemented by Igalia’s Gwang Yoon Hwang. This work improves performance by not stalling other web engine features while compositing. Earlier this year we contributed fixes to make the threaded compositor work with the web inspector and fixed elements, helping with the goal of enabling it by default for this release.

Wayland was also lacking an accelerated compositing implementation. There was a patch to add a nested Wayland compositor to the UIProcess, with the WebProcesses connecting to it as Wayland clients to share the final rendering so that it can be shown to screen. It was not ready though and there were questions as to whether that was the way to go and alternative proposals were floating around on how to best implement it.

At last year’s hackfest we had discussions about what the best path for that would be where collaborans Emanuele Aina and Daniel Stone (proxied by Emanuele) contributed quite a bit on figuring out how to implement it in a way that was both efficient and platform agnostic.

We later picked up the old patchset, rebased on the then-current master and made it run efficiently as proof of concept for the Apertis project on an i.MX6 board. This was done using the fancy GL support that landed in GTK+ in the meantime, with some API additions and shortcuts to sidestep performance issues. The work was sponsored by Robert Bosch Car Multimedia.

Igalia managed to improve and land a very well designed patch that implements the nested compositor, though it was still not as efficient as it could be, as it was using glReadPixels to get the final rendering of the page to the GTK+ widget through cairo. I have improved that code by ensuring we do not waste memory when using HiDPI.

As part of our proof of concept investigation, we got this WebGL car visualizer running quite well on our sabrelite imx6 boards. Some of it went into the upstream patches or proposals mentioned below, but we have a bunch of potential improvements still in store that we hope to turn into upstreamable patches and advance during next week’s hackfest.

One of the improvements that already landed was an alternate code path that leverages GTK+’s recent GL super powers to render using gdk_cairo_draw_from_gl(), avoiding the expensive copying of pixels from the GPU to the CPU and making it go faster. That improvement exposed a weird bug in GTK+ that causes a black patch to appear when shrinking the window, which I have a tentative fix for.

We originally proposed to add a new gdk_cairo_draw_from_egl() to use an EGLImage instead of a GL texture or renderbuffer. On our proof of concept we noticed it is even more efficient than the texturing currently used by GTK+, and could give us even better performance for WebKitGTK+. Emanuele Bassi thinks it might be better to add EGLImage as another code branch inside from_gl() though, so we will look into that.

Another very interesting igalian addition to this release is support for the MemoryPressureHandler even on systems with no cgroups set up. The memory pressure handler is a WebKit feature which flushes caches and frees resources that are not being used when the operating system notifies it memory is scarce.

We worked with the Raspberry Pi Foundation to add support for that feature to the Raspberry Pi browser and contributed it upstream back in 2014, when Collabora was trying to squeeze as much as possible from the hardware. We had to add a cgroups setup to wrap Epiphany in, back then, so that it would actually benefit from the feature.

With this improvement, it will benefit even without the custom cgroups setups as well, by having the UIProcess monitor memory usage and notify each WebProcess when memory is tight.

Some of these improvements were achieved by developers getting together at the Web Engines Hackfest last year and laying out the ground work or ideas that ended up in the code base. I look forward to another great few days of hackfest next week! See you there o/

by kov at September 22, 2016 05:03 PM

Jeremy Whiting

GSettings vs QSettings

A few weeks ago after discussing with Luke Yelavich about what to work on in speech-dispatcher next I decided to take a stab at making it use GSettings for its settings. (You can see the work in progress here if you like.) I've used GSettings before for work projects so thought it would be a good/easy thing to take on.

There are many advantages of using GSettings over plain ini-style files.
  • Type checking (You can't enter a string for a numeric setting for example).
  • Notification of setting changes.
  • Command-line changing of settings.
  • Default values for settings defined in the schema(s).

On that wip branch speech-dispatcher itself has been changed to use GSettings and also reacts to many setting changes dynamically. It doesn't react to changing the port type or port number or unix socket path dynamically, since we have no mechanism to tell client applications that it is changing. There are also GSettings schemas for the output modules, just need to make them read their settings from GSettings instead of the old ini-style .conf files. spd-conf also has been modified to write to GSettings rather than .conf files. That change alone reduced the spd-conf python script by quite a few lines of code and made it a lot easier to read.

As I was doing this work I got thinking about the differences between GSettings and QSettings. Besides one being glib/c based and the other being Qt/C++ they are really pretty similar. There are a few differences though:
  • QSettings doesn't emit signals when a setting changes. (I found a few forum posts asking why this is with possible workarounds. Nothing built into QSettings though).
  • QSettings doesn't have a schema for the settings themselves. There's no way to introspect a setting file to see what settings are possible. It just depends what keys the application reads.
  • QSettings doesn't have a command-line tool to set the settings. Since QSettings is cross platform it uses the Registry by default on Windows, PList files by default on macOS, and ini-style files on linux
  • QSettings does have type checking, but no range checking or anything like that.

I was a bit disappointed that QSettings that I've used for many many years is lacking these seemingly obvious and probably useful features. I wonder if we as a community implemented these features in QSettings if the Qt company would accept them.

by Jeremy Whiting ( at September 22, 2016 04:09 PM

September 06, 2016

Andrew Shadura

Manual control of OpenEmbedded -dbg packages

In December last year, OpenEmbebbed introduced automatic debug packages. Prior to that, you’d need to manually construct FILES_${PN}-dbg variable in your recipe. If you need to retain manual control over precisely what does into debug packages, set an undocumented NOAUTOPACKAGEDEBUG variable to 1, the same way Qt recipe does:

FILES_${PN}-dev = "${includedir}/${QT_DIR_NAME}/Qt/*"
FILES_${PN}-dbg = "/usr/src/debug/"
FILES_${QT_BASE_NAME}-demos-doc = "${docdir}/${QT_DIR_NAME}/qch/qt.qch"

P.S. Knowing this would have saved me and my colleagues days of work.

September 06, 2016 12:28 PM

Gustavo Padovan

Mainline Explicit Fencing – part 1

When it comes to buffer sharing synchronization in the kernel there are two ways of doing it: Implicit Fencing and Explicit Fencing. The difference between them relies on the fact that the kernel may or may not share synchronization information with userspace, it will either be implicit, with no fencing information provided, or explicit with all information available to userspace.

The fencing synchronization mechanism allows the sharing of buffers without the risk of a driver or userspace to read an incomplete buffer or write to a buffer that is still under use somewhere else in the system. The fencing provides ordering to these operations to make reads or writes happen only when the buffer is not used by other drivers anymore. For example,when a GPU job is queued a fence is associated to the buffer in the job, that fence can be used by other drivers for synchronization purposes, they won’t use the buffer a signal from the fence is received. The signal means the buffers is now free to be used. Similarly we can have the same setting for the GPU driver to wait the buffer to come out of the screen to render on it again.

The central piece here is the fence, an element that is attached to each buffer whenever a request involving the buffer is sent to the kernel. The fence can be used by userspace or other drivers to wait for the work to finish. So once the work is finished the fence signals and the waiter can proceed and do whatever they want with the buffer.

While Implicit Fencing  helps a lot with buffer synchronization there are a few cases where the whole desktop compositing could stall. Imagine the following compositor flow: there are 3 buffers to process, A, B and C. A and B are sent for rendering in parallel while C is going to be composed of both A and B. But the compositor will only be notified when both buffers are rendered thus if B takes too long the compositing of the whole desktop will be blocked waiting for B and C won’t be displayed in time.

A compositor processing two buffers in parallel

A compositor processing two buffers in parallel, with Implicit Fencing if B takes too long the desktop compositor freezes.

However with Explicit Fencing the compositor should have one fence for each buffer and will be notified when each buffer is rendered. So if A renders fast and B takes too long the compositor can decide not wait for B and proceed with the scanout of C with buffer A but an old version of B. The fencing information allows the compositor to be smart and take decisions to avoid the screen to freeze for example.

As of today the Linux Kernel only has generic APIs for Implicit Fencing, although some drivers have Explicit Fencing already their APIs are device specific. Android currently has its own implementation through the Android Sync Framework – which will be explained in the next article.

Explicit Fencing works on a Consumer-Producer fashion. In an GPU rendering + scanout to the screen pipeline it would synchronize between the kernel drivers, so when submitting a new rendering job to the GPU(Producer side) userspace would get back a fence related to that buffer submitted. That means userspace doesn’t need to block waiting for the job to complete, a signal is sent when the job is finished. As userspace doesn’t need to block it and has a fence of the buffer it then can proceed right away with the syscall to ask the display hardware(Consumer) to scanout the buffer that is yet to be processed. With explicit fencing the kernel is taught to wait for the fence to signal, before starting the scanout process.

A new fence is returned to userspace when the buffer is submitted to the kernel for scanout on the display hardware, that fence will signal when the buffer is not being displayed anymore, thus is ready for reuse by another rendering job. When the userspace gets this fence back it can submit a new rendering job to the GPU without waiting. The wait is done on the kernel side by the GPU driver, once the fence signals the rendering on that buffer can be initiated.

Explicit Fencing

The fence travels all the way to userspace and the next element on the pipeline. The yellow arrows represents the fences on userspace.

Last but not least, debugability of the graphics pipeline is improved. Having access to the fence in userspace helps a lot understanding what is happening in the pipeline. Previously, with Implicit Fencing there was no infomation available, so it was hard to figure out what was happening on the pipeline, also each vendor was trying to implement their own Implicit Fencing mechanism. Now with an standard Explicit Fencing mechanism it easier to build debug/tracing infrastructure that can be used to investigate issues in any system.

The next article will explain the Android Sync Framework and later the work on mainline to support explicit fencing will be described.

by Gustavo Padovan at September 06, 2016 10:00 AM

August 31, 2016

Helen Koike

LinuxCon NA 2016 - Highlights

After visiting FISL this summer, my travels have now taken me to LinuxCon NA 2016 in Toronto.
As everyone knows, the hot topic of the moment is containers, and they were everywhere at LinuxCon. Several companies are working in this market, there are even hardware optimized for getting the best performance on containers!
However, besides containers, there were several other different subjects of which I had more contact with:
memory-driven computers, workqueues, bluetooth, graphics, file systems, power saving (check the talk highlights below).
I also met several amazing people working in different fields and contributing with the free software community.

The place:

The infrastructure of the event was great, wifi worked everywhere. There was breakfast for attendees and snacks during the small breaks during the day.
In the main Hall, there were several couches and tables, and the conference rooms were great.

Each morning there were keynotes that were hosted in a big fancy room.  These were also streamed to the main hall so other people could watch.
In the afternoons there were several talks happening in parallel in smaller rooms.

The women lunch:

On the first day there was a women's only lunch event promoted by Intel, which was populated by 100+ women from the tech field. I've never seen so many of us reunited like that!
It was a great event to socialize and learn where everybody works. Several of them work directly with coding, but not the majority.
It was a pleasure to meet everyone and I am looking forward to see even more women in tech.

Booths (Hightlights):



In this booth I met The Machine, which is based in a Memory-driven computer architecture that promises to revolutionize how we know computers today.
The main memory is based in memristors, which can be viewed as a non-volatile RAM, so instead of having our basic model of Caches/Main Memory/Disk we would only have one memory based on memristors, all connected through a photonic fabric instead of a copper bus.
This changes our current programing model. HP have a github available with a framework where you can emulate the hardware, test and start programing for it.


Diamanti is a company that offers a hardware based solution to optimize containers and virtual environments, as mentioned in my NVMe post, I am working in a patch to optimize performance of shared NVMe device for a guest system in software while Diamanti, instead of sharing a NVMe device by software, make their hardware pretend there are multiple NVMe devices and they attach each of this devices directly to a container or virtual machine, thus from a software point of view, the container controls the device without having the VMM interfering.
They also do the same this for other peripherals beside NVMe as network cards.


Besides the Linux distribution (Ubuntu), this booth was presenting Juju, which is a tool to manage your services in the cloud, and also LXD, an hypervisor for containers


As most of you know, the Docker project is a great tool to create containers, which are something in the middle of a virtual machine and a chroot, it uses the kernel from the host.
Docker is also the name of the company (I thought is was only the name of the project) and they use LXC as a base to create containers.
The company provides services for other companies using the Docker project as setting up the infrastructure, setting a private Docker Hub, providing support, etc.


Why was Microsoft was in LinuxCon? To declare its love for Linux! :)
In this booth I obtained many stickers written "Microsoft loves Linux".I guess they decided to stop fighting old battles and be friends with Linux in the server market.


CoreOs is a Linux distribution mostly meant to be a lightweight host system for docker containers.
Kubernetes is a tool for managing containers, automating deployment and scaling. So used in conjunction with CoreOs is a good match.

Talks (Hightlights):


Btrfs with High Speed Devices - Chris Mason, Facebook:

Currently the maintainer of Btrfs, Chris Mason talked about this file system, tools to debug and how to identify bottlenecks.
One of the bottlenecks was btree locking, where he presented a patch that has a new locking scheme that optimizes the file system.

Open Source Bluetooth Device Firmware for IoT and Makers - Marcel Holtmann, Intel:

In this talk, Marcel Hltmann gave a great overview of the Bluetooth stack and mentioned that Bluetooth 5.0 is coming with support for mesh network.
As he is the maintainer of the Bluetooth stack on Linux, he talked about BlueZ and other Bluetooth tools in Linux.
For IoT and Makers who usually use an nRF51/nRF52 Bluetooth chip with the proprietary SoftDevice firmware, Marcel talked about how we could use Zyphis or MyNewt (which are open source) instead of SoftDevice and how he managed to get it working on Arduino 101.

Async Execution with Workqueue - Bhaktipriya Shridhar, Linux Kernel

Bhaktipriya Shridhar gave a talk about her Outreachy project on workqueues and how she managed to migrated several drivers from the old API to the new one.
Workqueues is a mechanism in Linux Kernel to execute pieces of code in asynchronous fashion, in short: if you have a function to execute and you don't want to wait for it to return, you can add it in the workqueue.
Internally, the kernel has two API's, the old one, with several issues as proliferation of kernel threads (it could run out of process IDs before even executing user space), deadlocks (if wasn't handled correctly) and unnecessary context switches. And the new API, the Concurrency Managed Workqueue (cmwq), which solves most of these issues.


Kernel Internship Report and Outreachy Panel - Moderated by Karen Sandler, Software Freedom Conservancy; Helen Fornazier, Rik Van Riel & Bhaktipriya Shridhar

Outreachy is 3 month internship meant to promote the presence of minorities in free software community.
If you know what GSoC is, Outreachy is similar with small differences in the projects (not necessary about coding), the selection phase, who can participate, etc.

In the panel we had 2 former mentors Rik Van Riel and Tiffany Antopolski (who is also a former intern), Bhaktipriya Shridhar (current intern in the linux kernel), myself as former intern and Karen Sandler as host and part of the organization of the Outreachy program at Software Freedom Conservancy.
Each one shared their experience as a mentor or as an intern.

CPUfreq and The Scheduler: Revolution in CPU Power Management - Rafael J. Wysocki, Intel OTC

To save power when the system can't go idle, CPUFreq can decrease or increase the clock frequency of the CPU based in the current work load.
Rafael Wysocki (ACPI core maintainer) explained the architecture of the old system that was based on timers, that would sample the load from time to time and update the clock frequency accordingly. The new system provides a much better result by using a Scheduler-driven mechanism instead of timers, using data from the scheduler to make decisions on the next frequency.


Bringing Android Explicit Fencing to Mainline - Gustavo Padovan, Collabora Ltd.

In this talk, Gustavo Padovan explained how graphic fences are exposed to userspace to synchronize buffer sharing and increase performance compared to the implicit fencing where userspace is not aware.


The gala party:

In the last day we had a great gala for the 25th anniversary of Linux.

I had the pleasure to have a great conversation with Eduardo Habkost from Red Hat who has worked with virtualization for 10+ years and gave me a great explanation on how Qemu connects with KVM.

This conference was not only about getting updates around the Linux community, but I also had the pleasure to meet great people and finally meet in person several people who I only knew through IRC, and was able to confirm that they were not bots! xD

Special thanks to Allison Lortie and William Hua from Canonical who showed us the city and made it such a pleasant trip.

by Helen Fornazier ( at August 31, 2016 06:34 PM

August 30, 2016 - Robert Foss

Building Android for Qemu

Alt text

Developing Linux for Android on Qemu allows you to do some things that are not necessarily possible using the stock emulator. For my purposes I need access to a GPU and be able to modify the driver, which is where Virgilrenderer and Qemu comes in handy.

The guide below helps you compile Android and run it on top of Qemu with Mesa/Virgilrenderer supplying a virtual GPU. Because of this, the following guide is aimed at Linux hosts.

This guide is based on Rob Herrings fantastic guide, but has been slightly streamlined and had physical hardware support stripped out.

Install dependencies

These dependencies were available on Ubuntu 16.04, some alternative packages might be needed for other distributions.

sudo apt install autoconf gcc-aarch64-linux-gnu libaio-dev libbluetooth-dev libbrlapi-dev libbz2-dev libcap-dev libcap-ng-dev libcurl4-gnutls-dev libepoxy-dev libfdt-dev libgbm-dev libgles2-mesa-dev libglib2.0-dev libibverbs-dev libjpeg8-dev liblzo2-dev libncurses5-dev libnuma-dev librbd-dev librdmacm-dev libsasl2-dev libsdl1.2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh2-1-dev libtool libusb-1.0-0 libusb-1.0-0-dev libvde-dev libvdeplug-dev libvte-2.90-dev libxen-dev valgrind xfslibs-dev xutils-dev zlib1g-dev

Set up paths

Naturally all of the paths below are configurable, this is just what I used.

export PROJECT_PATH="/opt/qemu_android"
export VIRGLRENDERER_PATH="${PROJECT_PATH}/virglrenderer"
export QEMU_PATH="${PROJECT_PATH}/qemu"
export LINUX_PATH="${PROJECT_PATH}/linux"
export ANDROID_PATH="${PROJECT_PATH}/android"
export ANDROID_TOOLS_PATH="${PROJECT_PATH}/android-tools"


Virglrenderer creates a virtual 3D GPU, that allows the Qemu guest to use the graphics capabilities of the host machine.

git clone git:// ${VIRGLRENDERER_PATH}
sudo make install


Qemu is a full system emulator, and supports a multitude of machine architectures. We're going to to use x86_64 but also build support for arm64/aarch64.

git clone git:// ${QEMU_PATH}
mkdir ${QEMU_PATH}/build
cd ${QEMU_PATH}/build
../configure --target-list=aarch64-softmmu,x86_64-softmmu --enable-gtk --with-gtkabi=3.0 --enable-kvm
make -j

Linux kernel

Build trunk of mainline linux kernel.

Important: The below instructions use upstream/master but during testing of this guide, and the fences branch was used due to SW_SYNC not yet being included in upstream. Inclusion is targeted for v4.9.

git clone git:// ${LINUX_PATH}
wget -O ${LINUX_PATH}/.config
make oldconfig
make -j

Important: If you decide not to use the .config linked in this step, a few Kconfig options need to be set:



Build the Android Open Source Project.

Important: When running source build/ make sure that you are using bash. I had issues running lunch using zsh.

repo init -u -b master
cd .repo
git clone -b android-6.0 local_manifests
cd ..
repo sync -j
cd device/linaro/generic
make defconfig
make all
cd ../../..
# The following snippet must be run in bash
source build/
# Select linaro_x86_64-userdebug
make -j
# We don't need to use bash any longer

As of this writing DRM fences related patches by Gustavo Padovan have yet to be included into AOSP, and therefore have to be included manually until it is upstreamed. After switching to this branch, the AOSP project has to be rebuilt again.

cd $ANDROID_PATH/system/core/
git remote add padovan git://
git fetch padovan
git checkout padovan/master


Fetch the make boot image script. This script later assembles the boot image, boot.img.


Run Qemu machine

When running the below script, make sure that the all of the paths from step two have been exported.

wget -O ${PROJECT_PATH}/
chmod +x ${PROJECT_PATH}/
${PROJECT_PATH}/ x86_64


Hopefully this guide will have enabled you build the required software and run Android on Qemu with a virtual GPU. The post was has been a part of work undertaken by my employer Collabora.

by Robert Foss at August 30, 2016 01:22 PM

Building Android for Qemu with Mesa and Virgil3D

Alt text

Developing Linux for Android on Qemu allows you to do some things that are not necessarily possible using the stock emulator. For my purposes I need access to a GPU and be able to modify the driver, which is where Virgilrenderer and Qemu comes in handy.

The guide below helps you compile Android and run it on top of Qemu with Mesa/Virgilrenderer supplying a virtual GPU. Because of this, the following guide is aimed at Linux hosts.

This guide is based on Rob Herrings fantastic guide, but has been slightly streamlined and had physical hardware support stripped out.

Install dependencies

These dependencies were available on Ubuntu 16.04, some alternative packages might be needed for other distributions.

sudo apt install autoconf gcc-aarch64-linux-gnu libaio-dev libbluetooth-dev libbrlapi-dev libbz2-dev libcap-dev libcap-ng-dev libcurl4-gnutls-dev libepoxy-dev libfdt-dev libgbm-dev libgles2-mesa-dev libglib2.0-dev libibverbs-dev libjpeg8-dev liblzo2-dev libncurses5-dev libnuma-dev librbd-dev librdmacm-dev libsasl2-dev libsdl1.2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh2-1-dev libtool libusb-1.0-0 libusb-1.0-0-dev libvde-dev libvdeplug-dev libvte-2.90-dev libxen-dev valgrind xfslibs-dev xutils-dev zlib1g-dev libusbredirhost-dev usbredirserver

Set up paths

Naturally all of the paths below are configurable, this is just what I used.

export PROJECT_PATH="/opt/qemu_android"
export VIRGLRENDERER_PATH="${PROJECT_PATH}/virglrenderer"
export QEMU_PATH="${PROJECT_PATH}/qemu"
export LINUX_PATH="${PROJECT_PATH}/linux"
export ANDROID_PATH="${PROJECT_PATH}/android"
export ANDROID_TOOLS_PATH="${PROJECT_PATH}/android-tools"


Virglrenderer creates a virtual 3D GPU, that allows the Qemu guest to use the graphics capabilities of the host machine.

git clone git:// ${VIRGLRENDERER_PATH}
make -j7
sudo make install


Qemu is a full system emulator, and supports a multitude of machine architectures. We're going to to use x86_64 but also build support for arm64/aarch64.

git clone git:// ${QEMU_PATH}
mkdir ${QEMU_PATH}/build
cd ${QEMU_PATH}/build
../configure --target-list=aarch64-softmmu,x86_64-softmmu --enable-gtk --with-gtkabi=3.0 --enable-kvm --enable-spice --enable-usb-redir --enable-libusb
make -j7

Linux kernel

Build trunk of mainline linux kernel.

Important: The below instructions use upstream/master but during testing of this guide, and the fences branch was used due to SW_SYNC not yet being included in upstream. Inclusion is targeted for v4.9.

git clone git:// ${LINUX_PATH}
wget -O ${LINUX_PATH}/.config
make oldconfig
make -j7

Important: If you decide not to use the .config linked in this step, a few Kconfig options need to be set:



Build the Android Open Source Project.

Important: When running source build/ make sure that you are using bash. I had issues running lunch using zsh.

repo init -u -b master
cd .repo
git clone -b android-6.0 local_manifests
cd ..
repo sync -j20
cd device/linaro/generic
make defconfig
make all
cd ../../..
# The following snippet must be run in bash
source build/
# Select linaro_x86_64-userdebug
make -j7
# We don't need to use bash any longer

As of this writing DRM fences related patches by Gustavo Padovan have yet to be included into AOSP, and therefore have to be included manually until it is upstreamed. After switching to this branch, the AOSP project has to be rebuilt again.

cd $ANDROID_PATH/system/core/
git remote add padovan git://
git fetch padovan
git checkout padovan/master


Fetch the make boot image script. This script later assembles the boot image, boot.img.


Run Qemu machine

When running the below script, make sure that the all of the paths from step two have been exported.

wget -O ${PROJECT_PATH}/
chmod +x ${PROJECT_PATH}/
${PROJECT_PATH}/ x86_64


Hopefully this guide will have enabled you build the required software and run Android on Qemu with a virtual GPU. This post has been a part of work undertaken by my employer Collabora.

by Robert Foss at August 30, 2016 01:22 PM

August 25, 2016

Gustavo Padovan

Slides for my LinuxCon talk on Mainline Explicit Fencing

For those of you that are interested here are the slides of the my presentation at LinuxCon North America this week. The conference was great with very good talks and very interesting meetings on the hallway track.

My presentation covered the effort to create the Explicit Fencing mechanism on the Linux Kernel which is to be used mainly by the Graphics pipeline. In short, Explicit Fencing is a way to give userspace information about the current state of shared buffers inside the kernel. This is done through fences, that can then be passed around to userspace and/or other kernel drivers for synchronization purposes. This allows both userspace and kernel to wait for kernel jobs to finish without blocking. It also significantly helps the compositor take more efficient and smart decisions on scheduling frames to display on the screen. I’ll be posting an article with more details on it soon. :)

Finally I would like to thank Collabora for sponsoring my travel to LinuxCon.

by Gustavo Padovan at August 25, 2016 03:45 PM - Robert Foss

Ethernet device stress testing

Alt text

During testing of power management patches for usb ethernet dongles, a script was needed to stress test connecting/disconnecting/reconnecting these devices.

Luckily a script like that already exists as a part of the chromiumos project, and can be found here.

That script does however not run standalone and requires a remote device (chromebook) to execute on. So I took the liberty of changing it to support local testing. The modified version can be found here.

This might come in handy for someone, if not, the script will at least be archived on this site.


$ sudo pip2.7 install autotest
$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DORMANT group default qlen 1000
    link/ether 48:e2:44:f6:e8:5b brd ff:ff:ff:ff:ff:ff
27: enx000ec689ab9e: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:0e:c6:89:ab:9e brd ff:ff:ff:ff:ff:ff
$ export INTERFACE=enx000ec689ab9e
$ export NUM_ITERATIONS=10

by Robert Foss at August 25, 2016 11:05 AM

August 23, 2016

Helen Koike

Increased performance of emulated NVMe devices

Nowadays, in Google Cloud Engine (GCE), it is possible to attach a local SSD with the NVMe interface to your virtual machine. Unfortunately, you only get a good number of iops (input/output operations per second) if you instantiate a machine with nvme-backports-debian-7-wheezy image; other available distributions on GCE will have a lower number of iops.

It turns out that Google's Virtual Machine Monitor (aka Hypervisor) implements a custom NVMe command that allows it to increase up to 4 times the number of  iops (note: this is from what I've tested so far, but it seems to be possible to get up to 5 times faster according to the original commit message; check the  Technical Details sessions to see how this is possible), however the kernel you use needs to support it and this is not yet the case with the mainline kernel.

This is not exclusive to GCE as Google released a patch not only to the kernel  but also to the qemu and is available here.

Collabora has been helping update, refactor and review the patches to the Linux Kernel to send it upstream, however since this is not yet an official nvme standard, it shouldn't be merged into Kernel mainline, as its specification may still receive changes.

Seeing as it considerably increases performance, the feature is in the process of being discussed and proposed to the NVMe workgroup with Collabora's help.
While the seems interested in adding an official extension to stardarize it, as published in the mailing list, nothing has been defined yet, as this is a very recent discussion and it can take up to a year to be ratified by the NVMe workgroup.

So, for the time being, you can get a more recent version of the patch and install the driver yourself here:

How it works?

Technical details


The NVMe interface basicaly works with command queues. The drive writes a command in a region known to both (driver and device controller) and then updates the tail of the queue, writting to an MMIO register called doorbell.

In an environment with several guest OSes on top of a VMM sharing a resource, communication between the guest OS and the real device is usually trapped by the VMM. As an MMIO is usually a syncronous acces to the device, it means that every MMIO access will cause a trap.

Example of emulated device in the VMM
The main idea here is to decrease the number of traps to the VMM by reducing the number of writtes to the doorbells.

This is achieved in two ways:
    1) Batching; or
    2) Letting the VMM pull the current doorbell value when it is already in execution.

The first one is easy, we can wait X commands to be written in the queue to ring the doorbell.
The second one is a bit more complicated. The guest OS needs to inform the emulated device in the VMM where it can pull the doorbell values, and the emulated NVMe device needs to inform the guest OS that it can restart the counter of X.

This is what this new feature does:
It adds a new command in the NVMe interface where the driver can send to the NVMe device controller two memory buffers:
1) A buffer where the real doorbell values are: Instead of writting to the MMIO  doorbell, the driver writtes the value in this buffer; and
2) Another buffer with a hint from the controller about how many commands the driver can write in the queue without ringing the doorbell.

The exact technical details may still change in the future, especially on how to properly implement the second item above. It is also very likely that Google's patches won't be compliant with the future ratified standard.

For the time being though, you can use the Collabora tree. Please let me know if you have any comments/feedback!

by Helen Fornazier ( at August 23, 2016 09:25 PM