Planet Collabora

Things have been going really well here at the GStreamer Hackfest in Malaga. Thanks to the help of Ara and Yaiza from Nido Malaga, we have a great venue in downtown Malaga and they have also helped us greatly with sorting out food.
We have a great group of people here and are making great progress, and by tomorrow I hope we will have screenshots of quite a few applications running with GStreamer 0.11, for instance both Rhythmbox and Jokosher for instance is already screen shootable, if not fully functional

GStreamer Hackfest Malaga 2012

Also making good progress on Transmageddon, even if the move to GObject Introspection bindings are making things a bit more complicated. Screenshot below of the progress so far.

Transmageddon at Hackfest in Malaga 2012

Also a big thanks to Fluendo who is sponsoring the lunches at the hackfest and Collabora who is sponsoring tonight’s dinner. Ensuring that no hacker is left hungry during this hackfest.

Update: Yaiza took these photos from the hackfest

With all the talk generated by Arun Raghavans blog post comparing PulseAudio and Audioflinger I thought it would be good to follow up with an interview with Arun about the latest developments in PulseAudio and the way forward for the project. You can find the PulseAudio interview here. I also made a new page listing all the Collabora developer interviews done so far. Enjoy

Tomorrow I will be heading off to attend the GStreamer Application Porting Hackfest in Malaga, Spain. I think we have managed to pull together an absolutely incredible group of people for this event and I have great hopes that by next weekend we will have squashed a ton of bugs in GStreamer 0.11/1.0 and also have initial ports of a long range of important applications and bindings. This is the first time in GStreamer history that we are trying to hold a hackfest focused on application developers, but hopefully it will be the first of many and that they can become a good way for the core GStreamer community and the application development community to interact and collaborate more closely.

Also want to say a special thanks to the community members attending the event on their own and also to the companies sending their employees to the hackfest; Collabora, Fluendo, Flumotion and Igalia and finally a special thanks to the GNOME Foundation for sponsoring some of the attendees.

Hopefully I will be able to post some screenshots of a fully functional GStreamer 1.0 Transmageddon next weekend

Here’s a “quick” status update on the 4.00 HEN (Homebrew ENabler) for PS3.

Following my clarifications from almost 2 months ago here, there has been a lot of progress. We have not been slacking off, we’re a group of about 10 developers working together for the last 2 months, for sometimes 15 hours everyday in order to bring back homebrew support to the latest version of the PS3.

There are three major parts to the HEN, first, getting the packages to install on the PS3, that part is done, completed, tested, debugged, etc.. the second part is to get the apps to run, that one still has major issues… the last part is something I will not discuss for now (it’s a surprise) but it’s about 60% to 70% done (and it has nothing to do with peek&poke and has nothing to do with backup managers or anything like that. This is and will stay a piracy-free solution for the PS3).

Now, running apps is the biggest challenge that we’ve been working on for the past 2 months. As some of you know, if you’ve been following me on Twitter, we originally had hoped for Mathieulh to give us the “npdrm hash algorithm” that was necessary to run the apps, but he was reluctant, he kept doing his usual whore so people would kiss his feet (or something else) so he’d feel good about himself. But in the end, he said that he refuses to give us the needed “npdrm hash algorithm” to make it work… So what I initially thought would be “this will be released next week” ended up taking a lot more time than expected, and we’re still nowhere near ready to make it work.

Mathieulh kept tossing his usual “riddles” which he thinks are “very helpful for those who have a brain”, and which pisses off anyone who actually does… so he told us that the solution to all our problems was to look in appldr of the 3.56 firmware.. and that it was something lv1 was sending appldr which made the “hash check” verified or not… so we spent one month and a lot of sweat and after killing a few of our brain cells out of exhaustion, we finally concluded that it was all bullshit. After one month of reading assembly code and checking and double-checking our results, we finally were able to confirm that that hash algorithm was NOT in the 3.56 firmware like he told us (at all).

He said  that it was an AES OMAC hash, but after tracking all the uses of the OMAC functions in appldr, we found that it was not used for the “hash”…  he then said “oh, I meant HMAC“, so we do that again and again come up with the same conclusion, then we’re sure it’s not in appldr, and then he says “ah no, it’s in lv1“.. have a look for yourself to what he decided to write : http://www.ps3devwiki.com/index.php?title=Talk:KaKaRoTo_Kind_of_%C2%B4Jailbreak%C2%B4

That happened after the huge twitter fight I had with him for being his usual arrogant ass and claiming that he “shared” something (For your information, the code that he shared was not his own, I have proof of that too (can’t show you the proof because even if I don’t respect him, I gave him my word to not share what he gave me, and I respect my word) since he forgot to remove the name of the original developer from one of the files… also it was completely useless and was not used at all, just made me waste a day reading the crappy undocumented code. So why is he still trying to force his “advice” through these riddles even after we had that fight? Well to sabotage us and make us lose all those months of hard work!

So anyways, we had all accepted that Mathieulh was full of shit (we knew before, but we gave him the benefit of the doubt) and decided to continue working without considering any of his useless riddles. So we then tried to exploit/decrypt the 3.60+ firmware in order to get the algorithm from there.

Now, a few more weeks later, we finally have succeeded in fully understanding that missing piece from the “npdrm hash algorithm”,  and here it is for everyone’s pleasure with some prerequisite explanation :

A game on the PS3 is an executable file in a format called a “SELF“file (kind of like .exe on windows), those “self” files are cryptographically signed and encrypted.. For PSN games (games that do not run from a bluray disc), they need to have an additional security layer called “NPDRM”. So a “npdrm self” is basically an executable that is encrypted and signed, then re-encrypetd again with some additional information. On 3.55 and lower, we were able to encrypt and sign our own self files so they would look like original (made by sony) “npdrm self” files, and the PS3 would run them without problem. However, it wasn’t really like an original file.. a real NPDRM self file had some additional information that the PS3 simply ignored, it did not check for that information, so we could put anything in it, and it worked. Since the 3.60 version, the PS3 now also validates this additional information, so it can now differentiate between NPDRM self files created by sony and the ones that we create ourselves for homebrew. That’s the “npdrm hash algorithm” that we have been trying to figure out, because once we can duplicate that information in the proper manner, then the PS3 will again think that those files are authentic and will let us play them.

Another important point to explain, I said a few times that the files are “signed”.. this means that there is an “ECDSA signature” in the file which the PS3 can verify. The ECDSA signature is something that allows the PS3 to verify if the file has been modified or not.. it is easy to validate the signature, but impossible to create one without having access to the “private keys” (think of it like a real signature, you can see your dad’s signature and recognize it, but you can’t sign it exactly like him, and you can recognize if your brother tried to forge his signature). So how were we able to sign the self files that were properly authenticated on 3.55? That’s because this “ECDSA signature” is just a very complicated mathematical equation (my head still hurts trying to fully understand it, but I might blog about it in the future and try to explain it in simple terms if people are interested), and one very important part of this mathematical equation is that you need to use a random number to generate the signature, but Sony had failed and used the same number every time.. by doing that, it was easy to just find the private key (which allows us to forge perfectly the signature) by doing some mathematical equation on it. So to summarize, a “signed file” is a file which is digitally signed with an “ECDSA signature” that cannot be forged, unless you have the “private key” for it, which is impossible to obtain usually, but we were able to obtain it because Sony failed in implementing it properly.

Now, back on topic.. so what is this missing “npdrm hash algorithm” that we need? well it turns out that the “npdrm self” has a second signature, so it’s a “encrypted and signed self file” with an additional layer of security (the NPDRM layer) which re-encrypts it and re-signs it again. That second signature was not verified in 3.55 and is now verified since the 3.60 version of the PS3 firmware.

One important thing to note is that Sony did NOT make the same mistake with this signature, they always used a random number, so it it technically impossible to figure out the private key for it. To be more exact, this is the exact same case as the .pkg packages you install on the PS3, you need to patch the firmware (making it cfw) so that those .pkg files can be installed, and that’s because the .pkg files are signed with an ECDSA signature for which no one was able to get the private key. That’s why we call them “pseudo-retail packages” or “unsigned packages”.

The signature on the NPDRM self file uses the exact same ECDSA curve and the same key as the one used in PS3 .pkg files, so no one has (or could have) the private key for it. What this means is that, even though we finally figured out the missing piece and we now know how the NPDRM self is built, we simply cannot duplicate it.

The reason we wasted 2 months on this is because Mathieulh lied by saying that he can do it.. remember when the 4.0 was out and I said “I can confirm that my method still works” then he also confirmed that his “npdrm hash algorithm” still works too? well he didn’t do anything to confirm, he just lied about it because there is no way that he could have verified it because he doesn’t have the private key.

I said I will provide proof of the lies that Mathieulh gave us, so here they are : he said it’s in 3.56, that was a lie, he said it’s an AES OMAC, that was a lie,  he said it’s an HMAC, that was a lie, he said it’s in appldr, that was a lie, he said it’s in lv1, that was a lie, he said that he can do it, that was a lie, he said that “it takes one hour to figure it out if you have a brain”, that was a lie, he said that he verified it to work on 4.0, that was a lie, he said that he had the algorithm/keys, that was a lie, he said that once we know the algorithm used, we can reproduce it, that was a lie, he kept referring to it as “the hash”, that was wrong. The proof ? It’s an ECDSA signature, it’s not a hash (two very different terms for different things), it was verified by vsh.self, it was not in lv2, or lv1, or appldr, and the private key is unaccessible, so there is no way he could build his own npdrm self files. Now you know the real reason why he refused to “share” what he had.. it’s because he didn’t have it…

So why do all this? was it because his arrogance didn’t allow him to admit not knowing something? or was it because he wanted to make us lose all this time? To me, it looks like pure sabotage, it was misleading information to steer us away from the real part of the code that holds the solution…. That is of course, if we are kind enough to assume that he knew what/where it was in the first place.  In the end, he wasn’t smart enough to only lie about things that we could not verify.. now we know (we always knew, but now we have proof to back it) that he’s a liar, and I do not think that anyone will believe his lies anymore.

Enough talking about liars and drama queens, back to the 4.0 HEN solution… so what next? well, we now know that we can’t sign the file, so we can’t run our apps on 3.60+ (it can work on 3.56 though). What we will do is look for a different way, a completely new exploit that would allow the files we install to actual run on the PS3. We will also be looking for possible “signature collisions” and for that we will need the help of the community, hopefully there is a collision (same random number used twice) which will allow us to calculate the private key, and if that happens, then we can move forward with a release.

When will the “jailbreak” be released? If I knew, I’d tell you, but I don’t know.. I would have said in last november, then december, then before christmas, then before new year, etc… but as you can see, it’s impossible to predict what we will find.. we might get lucky and have it ready in a couple of days, or we may not and it will not be ready for another couple of months.. so all you need to do is : BE PATIENT (and please stop asking me about an estimated release date)!

I would like to thank the team who helped on this task for all this time and who never got discouraged, and I’d like to thank an anonymous contributor who recently joined us and who was instrumental in figuring it all out. We all believe that freedom starts with knowledge, and that knowledge should be open and available to all, that is why we are sharing this information with the world. We got the confirmation (by finding the public key used and verifying the signatures) yesterday and since sharing this information will not help Sony in any way to block our efforts in a future release, we have decided to share it with you.  We believe in transparency, we believe in openness, we believe in a free world, and we want you to be part of it.

If you want to know more about this ECDSA signature algorithm, read this interesting paper that explains it in detail, and you can also watch Team Fail0verflow’s CCC presentation that first explained Sony’s mistake in their implementation, which made custom firmwares possible.

Thanks for reading,

KaKaRoTo

Breaking with the “every six months or so maybe” release tradition, here’s the second Bustle release of the month. What’s the new hotness this time? You can record D-Bus traffic by just clicking File → New, and watch the diagram being drawn before your very eyes. After years of on-off development, Bustle can finally liberate you from needing to open a terminal to monitor D-Bus traffic¹. Here’s a super-brief video tour.

Grab it for x86_64 or i486 today! Unlike the previous release, these work on both Debian and Fedora and have a strong change of working on pretty much anything with a modern-ish Glib and Gtk+2². (Thanks to my fellow Collaboran Guillaume for testing these tarballs, and to Scott Tsai for his suggestion.) Source and so on at the usual location.

Today’s surprising Bustle-assisted observation is that switching to and from the Shell overview causes the Shell to retrieve /desktop/gnome/shell/windows/button_layout from GConf 29 times. (Presumably that number is proportional to the number of windows I have open?) This was extremely useful for testing the live-logging feature, but is perhaps not ideal in many other ways.

1. “Occupy dbus-monitor” is arguably an unexciting slogan.
2. …inasmuch as any version of Gtk+2 can be called “modern”, that is.

In my last post I mentioned how it is possible to achieve a form of "reinversion of control" by using (green) threads. Some commenters noted how this is effectively a solved problem, as demonstrated for example by Erlang, as well as the numerous variations on CSP currently gaining a lot of popularity.

I don’t disagree with that, but it’s just not the point of this series of posts. This is about understanding the computational structure of event-driven code, and see how it’s possible to transform it into a less awkward form without introducing concurrency (or at least not in the traditional sense of the term).

Using threads to solve what is essentially a control flow problem is cheating. And you pay in terms of increased complexity, and code which is harder to reason about, since you introduced a whole lot of interleaving opportunities and possible race conditions. Using a non-preemptive concurrency abstraction with manual yield directives (like my Python gist does) will solve that, but then you’d have to think of how to schedule your coroutines, so that is also not a complete solution.

Programmable semicolons

To find an alternative to the multitask-based approach, let’s focus on two particular lines of the last example:

    reply = start_request();
    get_data(reply)

where I added an explicit semicolon at the end of the first line. A semicolon is an important component of an imperative program, even though, syntactically, it is often omitted in languages like Python. It corresponds to the sequencing operator: execute the instruction on the left side, then pass the result to the right side and execute that.

If the instruction on the left side corresponds to an asynchronous operation, we want to alter the meaning of sequencing. Given a sequence of statements of the form

    x = A(); B(x)

we want to interpret that as: call A, then return control back to the main loop; when A is finished, bind its result to x, then run B.

So what we want is to be able to override the sequencing operator: we want programmable semicolons.

The continuation monad

Since it is often really useful to look at the types of functions to understand how exactly they fit together, we’ll leave Python and start focusing on Haskell for our running example.

We can make a very important observation immediately by looking at the type of the callback registration function that our framework offers, and try to interpret it in the context of controlled side effects (i.e. the IO monad). For Qt, it could look something like:

1

    connect :: Object -> String -> (a -> IO ()) -> IO ()

to be used, for example, like this:

1
2

    connect httpReply "finished()" $ \_ -> do
        putStrLn "request finished"

so the first argument is the object, the second is the C++ signature of the signal, and the third is a callback that will be invoked by the framework whenever the specified signal is emitted. Now, we can get rid of all the noise of actually connecting to a signal, and define a type representing just the act of registering a callback.

1

    newtype Event a = Event { on :: (a -> IO ()) -> IO () }

Doesn’t that look familiar? It is exactly the continuation monad transformer applied to the IO monad! The usual monad instance for ContT perfectly captures the semantics we are looking for:

1
2
3
4
5

    instance Monad Event where
      return x = Event $ \k -> k x
      e >>= f = Event $ \k ->
        on e $ \x ->
          on (f x) k

The return function simply calls the callback immediately with the provided value, no actual connection is performed. The bind operator represents our custom semicolon: we connect to the first event, and when that fires, we take the value it yielded, apply it to f, and connect to the resulting event.

Now we can actually translate the Python code of the previous example to Haskell:

1
2
3
4
5
6
7
8
9
10

    ex :: Event ()
    ex = forever $ do
      result <- untilRight . replicate 2 $ do
        reply <- startRequest
        either (return . Left) (liftM Right . getData) reply
      either handleError displayData result

    untilRight :: Monad m => [m (Either a b)] -> m (Either a b)
    untilRight [m] = m
    untilRight (m : ms) = m >>= either (const (untilRight ms)) (return . Right)

Again, this could be cleaned up by adding some error reporting functionality into the monad stack.

Implementing the missing functions in terms of connect is straightforward. For example, startRequest will look something like this:

1
2
3
4
5

    startRequest :: Event (Either String Reply)
    startRequest = Event $ \k -> do
      reply <- AccessManager.get "http://example.net"
      connect reply "finished()" $ \_ -> k (Right reply)
      connect reply "error(QString)" $ \e -> k (Left e)

where I took the liberty of glossing over some irrelevant API details.

How do we run such a monad? Well, the standard runContT does the job:

1
2

    runEvent :: Event () -> IO ()
    runEvent e = on $ \k -> return ()

so

1

    runEvent ex

will run until the first connection, return control to the main loop, resume when an event occurs, and so on.

Conclusion

I love the simplicity and elegance of this approach, but unfortunately, it is far from a complete solution. So far we have only dealt with "one-shot" events, but what happens when an event fires multiple times? Also, as this is still very imperative in nature, can we do better? Is it possible to employ a more functional style, with emphasis on composability?

I’ll leave the (necessarily partial) answers to those questions for a future post.

AdaCamp

Saturday was the very first AdaCamp, an unconference set up by the Ada Initiative to talk about issues facing women in open stuff, here in Melbourne.

The event was well attended by awesome feminists from a breadth of fields and some geographic diversity within Australia (including a surprising number of people from Perth). There were open source people, Wikimedia people, librarians and academics.

There were 21 sessions in total, in 3 streams, so I don’t know everything awesome that was discussed. I think the most important session I attended was the first one of the morning on Imposter syndrome. Also the session on getting women involved with open source, where I talked about the GNOME Outreach Programme for Women.

[Unfortunately I forgot to bring a notebook with me, and tiredness has caused my memory to let me down.]

Brianna and I supplied the morning and afternoon tea, which was various combinations of vegan, gluten-free and fructose-free. Some people asked for the recipes used, which I blogged here.

Haecksen

I drove to Ballarat yesterday to give my panel at the Haecksen miniconf at linux.conf.au¹ on gender-focused outreach in open source.

It was, in my opinion, an extremely good panel. I was originally daunted by appearing on a panel with Pia Waugh, Leslie Hawthorn and Selena Deckmann (all of whom are amazingly talented), but I think I did okay. We covered a breadth of mentoring from school age girls, to open source mentoring with Summer of Code and specific programmes like GOPW with a little bit about mentoring in organisations. Unfortunately I feel like the panel ran out of time just when it was getting interesting, and I didn’t get to talk about optimising your mentoring and the differences I’ve found in mentoring people from high-context culture vs low-context cultures [1, 2].

We also talked a bit about improving your diversity (which was a bit off-topic, more of a carry-on from the talk preceding the panel). Key point for me was: people from minorities tend not to ask (or apply), if you want diversity, you have to seek it out. Furthermore, people tend to know people who are like them, to seek out diversity, ask for recommendations from people who are least like the people you already have.

~

Unfortunately, I’m not at linux.conf.au itself, so if you’re looking for me, I’m not there. If you want to talk to a Collaboran about what we can do for you or how awesome it is to work for us, you can find Daniel Stone or Dario Freddi at the conference.

1. Many thanks to my employer Collabora for letting me be there.

Despite what some stats may say, my biggest contribution to GNOME is not in bugs or code but in the organization of beer related events!

So I'm pleased to announce that, like each year, we'll have a GNOME Beer party on the Saturday night of FOSDEM (4th Feb). People seemed happy of the location of last year, so we decided to stay at "La Bécasse" in the city center. Feel free to add yourself to the wiki if you are planning to attend.

See you at FOSDEM!

I’ve been meaning to try this for a while, and we’ve heard a number of requests from the community as well. Recently, I got some time here at Collabora to give it a go — that is, to get PulseAudio running on an Android device and see how it compares with Android’s AudioFlinger.

The Contenders

Let’s introduce our contenders first. For those who don’t know, PulseAudio is pretty much a de-facto standard part of the Linux audio stack. It sits on top of ALSA which provides a unified way to talk to the audio hardware and provides a number of handy features that are useful on desktops and embedded devices. I won’t rehash all of these, but this includes a nice modular framework, a bunch of power saving features, flexible routing, and lots more. PulseAudio runs as a daemon, and clients usually use the libpulse library to communicate with it.

In the other corner, we have Android’s native audio system — AudioFlinger. AudioFlinger was written from scratch for Android. It provides an API for playback/recording as well as a control mechanism for implementing policy. It does not depend on ALSA, but instead allows for a sort of HAL that vendors can implement any way they choose. Applications generally play audio via layers built on top of AudioFlinger. Even if you write a native application, it would use OpenSL ES implementation which goes through AudioFlinger. The actual service runs as a thread of the mediaserver daemon, but this is merely an implementation detail.

Note: all my comments about AudioFlinger and Android in general are based on documentation and code for Android 4.0 (Ice Cream Sandwich).

The Arena

My test-bed for the tests was the Galaxy Nexus running Android 4.0 which we shall just abbreviate to ICS. I picked ICS since it is the current platform on which Google is building, and hopefully represents the latest and greatest in AudioFlinger development. The Galaxy Nexus runs a Texas Instruments OMAP4 processor, which is also really convenient since this chip has pretty good support for running stock Linux (read on to see how useful this was).

Preparations

The first step in getting PulseAudio on Android was deciding between using the Android NDK like a regular application or integrate into the base Android system. I chose the latter — even though this was a little more work initially, it made more sense in the long run since PulseAudio really belongs to the base-system.

The next task was to get the required dependencies ported to Android. Fortunately, a lot of the ground work for this was already done by some of the awesome folks at Collabora. Derek Foreman’s androgenizer tool is incredibly handy for converting an autotools-based build to Android–friendly makefiles. With Reynaldo Verdejo and Alessandro Decina’s prior work on GStreamer for Android as a reference, things got even easier.

The most painful bit was libltdl, which we use for dynamically loading modules. Once this was done, the other dependencies were quite straightforward to port over. As a bonus, the Android source already ships an optimised version of Speex which we use for resampling, and it was easy to reuse this as well.

As I mentioned earlier, vendors can choose how they implement their audio abstraction layer. On the Galaxy Nexus, this is built on top of standard ALSA drivers, and the HAL talks to the drivers via a minimalist tinyalsa library. My first hope was to use this, but there was a whole bunch of functions missing that PulseAudio needed. The next approach was to use salsa-lib, which is a stripped down version of the ALSA library written for embedded devices. This too had some missing functions, but these were fewer and easy to implement (and are now upstream).

Now if only life were that simple. :) I got PulseAudio running on the Galaxy Nexus with salsa-lib, and even got sound out of the HDMI port. Nothing from the speakers though (they’re driven by a TI twl6040 codec). Just to verify, I decided to port the full alsa-lib and alsa-utils packages to debug what’s happening (by this time, I’m familiar enough with androgenizer for all this to be a breeze). Still no luck. Finally, with some pointers from the kind folks at TI (thanks Liam!), I got current UCM configuration files for OMAP4 boards, and some work-in-progress patches to add UCM support to PulseAudio, and after a couple of minor fixes, wham! We have output. :)

(For those who don’t know about UCM — embedded chips are quite different from desktops and expose a huge amount of functionality via ALSA mixer controls. UCM is an effort to have a standard, meaningful way for applications and users to use these.)

In production, it might be handy to write light-weight UCM support for salsa-lib or just convert the UCM configuration into PulseAudio path/profile configuration (bonus points if it’s an automated tool). For our purposes, though, just using alsa-lib is good enough.

To make the comparison fair, I wrote a simple test program that reads raw PCM S16LE data from a file and plays it via the AudioTrack interface provided by AudioFlinger or the PulseAudio Asynchronous API. Tests were run with the brightness fixed, wifi off, and USB port connected to my laptop (for adb shell access).

All tests were run with the CPU frequency pegged at 350 MHz and with 44.1 and 48 kHz samples. Five readings were recorded, and the median value was finally taken.

Round 1: CPU

First, let’s take a look at how the two compare in terms of CPU usage. The numbers below are the percentage CPU usage taken as the sum of all threads of the audio server process and the audio thread in the client application using top (which is why the granularity is limited to an integer percentage).

At 44.1 kHz, the two are essentially the same. Both cases are causing resampling to occur (the native sample rate for the device is 48 kHz). Resampling is done using the Speex library, and we’re seeing minuscule amounts of CPU usage even at 350 MHz, so it’s clear that the NEON optimisations are really paying off here.

The astute reader would have noticed that since the device’ native sample rate is 48 kHz, the CPU usage for 48 kHz playback should be less than for 44.1 kHz. This is true with PulseAudio, but not with AudioFlinger! The reason for this little quirk is that AudioFlinger provides 44.1 kHz samples to the HAL (which means the stream is resampled there), and then the HAL needs to resample it again to 48 kHz to bring it to the device’ native rate. From what I can tell, this is a matter of convention with regards to what audio HALs should expect from AudioFlinger (do correct me if I’m mistaken about the rationale).

So round 1 leans slightly in favour of PulseAudio.

Round 2: Memory

Comparing the memory consumption of the server process is a bit meaningless, because the AudioFlinger daemon thread shares an address space with the rest of the mediaserver process. For the curious, the resident set size was: AudioFlinger — 6,796 KB, PulseAudio — 3,024 KB. Again, this doesn’t really mean much.

We can, however, compare the client process’ memory consumption. This is RSS in kilobytes, measured using top.

The memory consumption is comparable between the two, but leans in favour of AudioFlinger.

Round 3: Power

I didn’t have access to a power monitor, so I decided to use a couple of indirect metrics to compare power utilisation. The first of these is PowerTOP, which is actually a Linux desktop tool for monitoring various power metrics. Happily, someone had already ported PowerTOP to Android. The tool reports, among other things, the number of wakeups-from-idle per second for the processor as a whole, and on a per-process basis. Since there are multiple threads involved, and PowerTOP’s per-process measurements are somewhat cryptic to add up, I used the global wakeups-from-idle per second. The “Idle” value counts the number of wakeups when nothing is happening. The actual value is very likely so high because the device is connected to my laptop in USB debugging mode (lots of wakeups from USB, and the device is prevented from going into a full sleep).

The second, similar, data point is the number of interrupts per second reported by vmstat. These corroborate the numbers above:

PulseAudio’s power-saving features are clearly highlighted in this comparison. AudioFlinger causes about three times the number of wakeups per second that PulseAudio does. Things might actually be worse on older hardware with less optimised drivers than the Galaxy Nexus (I’d appreciate reports from running similar tests on a Nexus S or any other device with ALSA support to confirm this).

For those of you who aren’t familiar with PulseAudio, the reason we manage to get these savings is our timer-based scheduling mode. In this mode, we fill up the hardware buffer as much as possible and go to sleep (disabling ALSA interrupts while we’re at it, if possibe). We only wake up when the buffer is nearing empty, and fill it up again. More details can be found in this old blog post by Lennart.

Round 4: Latency

I’ve only had the Galaxy Nexus to actually try this out with, but I’m pretty certain I’m not the only person seeing latency issues on Android. On the Galaxy Nexus, for example, the best latency I can get appears to be 176 ms. This is pretty high for certain types of applications, particularly ones that generate tones based on user input.

With PulseAudio, where we dynamically adjust buffering based on what clients request, I was able to drive down the total buffering to approximately 20 ms (too much lower, and we started getting dropouts). There is likely room for improvement here, and it is something on my todo list, but even out-of-the-box, we’re doing quite well.

Round 5: Features

With the hard numbers out of the way, I’d like to talk a little bit about what else PulseAudio brings to the table. In addition to a playback/record API, AudioFlinger provides mechanism for enforcing various bits of policy such as volumes and setting the “active” device amongst others. PulseAudio exposes similar functionality, some as part of the client API and the rest via the core API exposed to modules.

From SoC vendors’ perspective, it is often necessary to support both Android and standard Linux on the same chip. Being able to focus only on good quality ALSA drivers and knowing that this will ensure quality on both these systems would be a definite advantage in this case.

The current Android system leaves power management to the audio HAL. This means that each vendor needs to implement this themselves. Letting PulseAudio manage the hardware based on requested latencies and policy gives us a single point of control, greatly simplifying the task of power-management and avoiding code duplication.

There are a number of features that PulseAudio provides that can be useful in the various scenarios where Android is used. For example, we support transparently streaming audio over the network, which could be a handy way of supporting playing audio from your phone on your TV completely transparently and out-of-the-box. We also support compressed formats (AC3, DTS, etc.) which the ongoing Android-on-your-TV efforts could likely take advantage of.

Edit: As someone pointed out on LWN, I missed one thing — AudioFlinger has an effect API that we do not yet have in PulseAudio. It’s something I’d definitely like to see added to PulseAudio in the future.

Ding! Ding! Ding!

That pretty much concludes the comparison of these two audio daemons. Since the Android-side code is somewhat under-documented, I’d welcome comments from readers who are familiar with the code and history of AudioFlinger.

I’m in the process of pushing all the patches I’ve had to write to the various upstream projects. A number of these are merely build system patches to integrate with the Android build system, and I’m hoping projects are open to these. Instructions on building this code will be available on the PulseAudio Android wiki page.

For future work, it would be interesting to write a wrapper on top of PulseAudio that exposes the AudioFlinger audio and policy APIs — this would basically let us run PulseAudio as a drop-in AudioFlinger replacement. In addition, there are potential performance benefits that can be derived from using Android-specific infrastructure such as Binder (for IPC) and ashmem (for transferring audio blocks as shared memory segments, something we support on desktops using the standard Linux SHM mechanism which is not available on Android).

If you’re an OEM who is interested in this work, you can get in touch with us — details are on the Collabora website.

I hope this is useful to some of you out there!

Thanks to Collabora, I’ve received an N9. So I’ve build ssh-contact package for it. It was easy since the N9 already has all its dependencies. I’ve simply taken debian’s package and rebuild it in scratchbox with the harmattan SDK.

Here are the packages:
ssh-contact_0.6-1_all.deb
ssh-contact-client_0.6-1_armel.deb
ssh-contact-service_0.6-1_armel.deb

To install them, you can follow the steps from here for example.

The problem

Most of modern programming is based on events. Event-driven frameworks are the proven and true abstraction to express any kind of asynchronous and interactive behavior, like in GUIs or client-server architectures.

The core idea is inversion of control: the main loop is run by the framework, users only have to register some form of “callbacks”, and the framework will take care of calling them at the appropriate times.

This solves many issues that a straightforward imperative/procedural approach would present, eliminates the need for any kind of polling, and creates all sorts of opportunities for general-purpose optimizations inside the framework, with no impact on the complexity of user code. All of this without introducing any concurrency.

There are drawbacks, however. Event-driven code is hideous to write in most languages, especially those lacking support for first class closures. More importantly, event-driven code is extremely hard to reason about. The very nature of this callback-based approach makes it impossible to use a functional style, and even the simplest of interactions requires some form of mutable state which has to be maintained across callback calls.

As a very simple example, suppose we want to perform a GET request and retrieve some data, handling any HTTP error that might occur. In a generic event-driven frameworkm, we would need to implement a simple state machine whose graph will look somewhat like this:

Each state (except the initial one) corresponds to a callback. The transitions are determined by the framework. To avoid starting more than one request at a time, we will need to explicitly keep track of the current state.

Now let’s try to make a simple change to our program: suppose we want to retry requests when they fail, but not more than once. Now the state machine becomes more complicated, since we need to add extra nodes for the non-fatal error condition.

In our hypotetical event-driven code, we need to keep track of whether we already encountered an error, check this flag at each callback to perform the right action, and update it appropriately. Moreover, this time the code isn’t even shaped exactly like the state machine, because we reuse the same callback for multiple nodes. To test our code exhaustively, we need to trace every possible path through the graph and reproduce it.

Now assume we want to allow simultaneous requests… you get the idea. The code gets unwieldy pretty fast. Small changes in requirements have devastating consequences in terms of the state graph. In practice, what happens most of the times is that the state graph is kept implicit, which makes the code impossible to test reliably, and consequently impossible to modify.

Towards a solution

A very simple but effective solution can be found by observing that state graphs like those of the previous examples have a very clear operational interpretation in the context of the equivalent synchronous code.

A single forward transition from A to B can be simply modelled as the sequence A;B, i.e. execute A, then execute B. Extra outward transitions from a single node can be mapped to exceptions, while backward arrows can be thought of as looping constructs.

Our second state machine can then be translated to the following pseudopython:

while True:
    for i in xrange(2):
        error = None
        try:
            reply = start_request()
            data = get_data(reply)
            break
        except Exception as e:
            error = get_error(e)
    if error:
        handle_error(error)
    else:
        display_data(data)

This code is straightforward. It could be made cleaner by splitting it up in a couple of extra functions and removing local state, but that’s beside the point. Note how easy it is now to generalize to an arbitrary number of retries.

So the key observation is that we can transform asynchronous code into synchronous-looking code, provided that we attach the correct semantics to sequencing of operations, exceptions and loops.

Now the question becomes: is it possible to do so?

We could turn functions like start_request and get_data into blocking operations that can throw. This will work locally, but it will break asynchronicity, so it’s not an option.

One way to salvage this transformation is to run the code in its own thread. Asynchronous operations will block, but won’t hang the main loop, and the rest of the program will continue execution.

However, we need to be careful with the kind of threads that we use. Since we don’t need (and don’t want!) to run multiple threads simultaneously, but we need to spawn a thread for each asynchronous operation, we have to make sure that the overhead is minimal, context switching is fast, and we’re not paying the cost of scheduling and synchronization.

Here you can find a sketched solution along these lines that I wrote in python. It’s based on the greenlet library, which provides cooperative multithreading.

In the next post I will talk about alternative solutions, as well as how to extend the idea further, and make event-driven code more declarative and less procedural.

It’s a cold evening here in Cambridge, but I’m being kept warm at Collabora Towers, sipping a revitalizing mug of fresh applicative functor soup.

A mere five months after I demoed its features at the Desktop Summit in Berlin, here’s Bustle 0.3.1. Whereas previous versions of Bustle only recorded and showed you the names, senders and destinations of D-Bus messages, this version also records and shows you the contents of the messages.

The statistics page also takes advantage of this new information: you can now get statistics about the sizes of messages in the log. Grab your copy today from the usual location. Beside the source, I’ve also uploaded a 64-bit binary tarball to save you some compilation time. Give me a shout if you have trouble with it. 32-bit version to follow when I get my chroots straightened out.

I have good news and bad news. Good news: here’s a 32-bit binary tarball. Bad news: seems like Debian and Fedora have differently-sonamed libpcaps. Why is distributing software so tedious?

After updating GStreamer and doing a couple of small fixes I managed to make Transmageddon work with the GTK3 and the 0.11 branch of GStreamer. Obligatory screenshot below. As you might guess from looking at the screenshot there are still some issues that needs solving, but
I am happy that I managed to get this far.

Transmageddon running GTK3 and GStreamer 0.11/1.0

Hopefully it is a sign that the upcoming GStreamer hackfest in Malaga will be a great successful everyone who is participating.

I hope the remainder of the porting effort will be relatively simple as I would love to get back to working on real features instead of just updating old functionality to use a new backend to do the same. Having had a need for Transmageddon for a couple of work related tasks recently a couple of items, like batch job programming has moved up my priorities list.

Just posted another interview on the Collabora website, this time with Gustavo Noronha Silva talking about WebKit and some of the work he and Collabora are doing around that project. So be sure to check it out if you want to learn more about things like WebKit and Clutter integration and how WebKit impacts the GNOME platform.

I have been experimenting with using Yesod to throw together a web application or two. My experience so far has been broadly positive—if you like computers to check things for you, I recommend it.¹ That said, watching the full chain of dependencies fly past was moderately entertaining:²

parsec-3.1.2

An excellent parser-combinator library, widely imitated. This wouldn’t be funny, except…

attoparsec-0.10.1.0

Another excellent parser-combinator library, inspired by parsec.

base-unicode-symbols-0.2.2.3

This defines a bunch of Unicode aliases for standard functions with boring ASCII names. Why write:

x `elem` xs

when you could write:

x ∈ xs

Invaluable!

utf8-light-0.4.0.1 and utf8-string-0.3.7

Two UTF-8 encoding libraries!

semigroups-0.8

“In mathematics, a semigroup is an algebraic structure consisting of a set together with an associative binary operation. A semigroup generalizes a monoid in that there might not exist an identity element. It also (originally) generalized a group (a monoid with all inverses) to a type where every element did not have to have an inverse, thus the name semigroup.”

1. assuming you like deciphering compiler error messages when the computer says no, that is
2. for a quiet Wednesday morning…

I guess it is time to announce that since yesterday I am working at Collabora, a UK-based company very well known for its work in several free software projects, like Telepathy, Farstream, GStreamer and others.

Haven’t had much time really to transition (and relax) from Canonical to Collabora, apart from last week, which I spent skiing, but hey, new year, new life, as we say in Spain, so the sooner you start with your new life, the better.

Collabora Ltd, my employer for the past few years, was kind enough to give every employee the option of a Nokia N9 as a sort of Christmas bonus.

Two years ago Collabora gifted Nokia N900s to employees and the microphone in mine had just ceased to function so the new N9 was a very welcome replacement.

A number of us at Collabora have been working on MeeGo and Maemo before it though I myself am a relative newcomer having worked on the camera software since February 2011. I’m certainly proud to have worked on the N9 as it is a solid and well-performing phone despite still being a single core ARM Cortex-A8.

The user interface follows an interaction method Nokia have dubbed ‘swipe’. The name tells all - most interaction methods for moving between screens and applications involve swipes from an edge of the screen. It’s very simple and works well to traverse the environment.

There are three screens on a virtual carousel - an application grid common to most mobile phones, an open tasks grid with thumbnail tiles and a feed of updates from various sources (Facebook, Twitter, etc.). You can read more in reviews or have a look at some videos if you wish.

With the N900, honestly I didn’t feel comfortable using it too much as it was quite slow to do lots of things, especially web browsing. Browsing on the N9 is smooth and fast. As such, I now have some mobile internet allowance on a contract so I’m sure I’ll be playing around with connected programs much more on the N9.

One very nice feature on the phone when moving from the N900 was that I could transfer all my contacts and calendar and some other things via bluetooth. It would have been super nice if I could have also transferred photos, music, old conversations and more, but just the simplicity and capability to transfer contacts is awesome.

Scrolling is silky smooth, the camera is fast and high quality for both still images and video, call quality seems pretty good but I haven’t spoken to anyone with a newer phone yet to be able to tell properly, the interface is shiny and simple, the hardware is spot on in terms of size and the screen is beautiful. Battery life is very good, though will obviously diminish more quickly if you’re playing 3D games or so. Integration with web services seems pretty good though I want to see if I can make it automatically upload photos to Google Picasa, Flickr or so. It can act as a wifi hotspot. The maps app is slow to start but locks onto my location quickly and allows one to download maps for offline/minimal mobile bandwidth use which is a huge plus when travelling. There’s even a digital compass built in, though I would be interested to test its accuracy.

The only weakness I see so far is the limited number of interesting apps available. It has Angry Birds, Need for Speed: Shift, Solitaire, sudoku and some other games I haven’t yet tried.

My personal verdict, with the current PR1.1 release of the software is that it is a beautiful, solid and well-connected phone that I will certainly enjoy using and as a consequence will see far more use. Also, if the N900 is anything to go by and if any more updates make it out of the door, the phone will improve further with time. :)

Since it’s annoyingly hard to work out what muxer to use for m4a files, and I don’t have a profile for encodebin, here’s the gst-launch command to make m4a or m4r (ringtone) files.

gst-launch filesrc location=input.file ! decodebin2 ! audioconvert ! faac ! ffmux_mp4 ! filesink location=output.m4a

This is a neat trick, and so I thought worth reblogging. I was handed a corrupt SD memory card yesterday and asked if I could try and get some Christmas photos off it.

Typically, the first thing I did was image the card with ddrescue (package gddrescue in Debian/Ubuntu), confirming that the partition table was intact, it looks like the problem was a corrupted FAT, so I wanted to run fsck on the partition.

Back in the day this would have involved either dd-ing the partition out of the image, or re-imaging the card, both slow, or screwing around with loopback offsets. Instead I discovered a tool called kpartx which maps the partitions from the image into the device mapper, giving a device like /dev/mapper/loop0p1, which can be used like a regular block device.

~

Unfortunately while different, both FATs were damaged, and has been incorrectly repaired by something, and I ended up recovering the files with photorec instead.

Happy new year, everyone!

Once again my former employer, Fugro Seismic Imaging, is looking for two new R&D staff for their office in Perth, Western Australia (one to replace someone who’s moving on, and another to grow the team).

The team there is approximately half a dozen people, generally open-source experienced, who are primarily developing graphical front ends to geophysics applications (there’s also some other stuff, in general it’s pretty cool). You’ll almost certainly have to know C. Knowing any of C++, Python, Perl, GTK+, Qt or Fortran also a benefit. You don’t have to know any geophysics, but knowing a bit of maths and physics helps. You’ll easily pick up what you don’t know as you go along.

Pay and conditions are very good. FSI have previously provided sponsored work visas for skilled applicants looking to work in Australia.

If you’re interested, send your CV to the regional manager, Kelly Beauglehole.

So, I intended to be completely away from my computer during my vacations, but hey. I have been interested in this new extension Apple added to the C language a little while ago which introduces the equivalent of closures to C. Today I spent a few minutes looking into it and writing a few tests with the help of clang.

Here’s something I came up with, to use a block as the callback for a GIO asynchronous call:

#include <Block.h>
#include <gio/gio.h>

typedef void (^Block)();

static void async_result_cb(GObject* source,
                            GAsyncResult* res,
                            gpointer data)
{
    Block block = (Block)data;
    block(res);
}

int main(int argc, char** argv)
{
    g_type_init();

    if (argc != 2) {
        g_error("Blah.");
        return 1;
    }

    GMainLoop* loop = g_main_loop_new(NULL, TRUE);
    GFile* file = g_file_new_for_path(argv[1]);

    g_file_query_info_async(file,
                            G_FILE_ATTRIBUTE_STANDARD_CONTENT_TYPE,
                            G_FILE_QUERY_INFO_NONE, G_PRIORITY_DEFAULT,
                            NULL, async_result_cb, (gpointer) ^ (GAsyncResult* res) {
        GError* error = NULL;
        GFileInfo* info = g_file_query_info_finish(file, res, &error);

        if (error) {
            g_error("Failed: %s", error->message);
            g_error_free(error);
            return;
        }

        g_message("Content Type: %s",
                  g_file_info_get_attribute_string(info, G_FILE_ATTRIBUTE_STANDARD_CONTENT_TYPE));

        g_object_unref(info);
        g_main_loop_quit(loop);
    });

    g_main_loop_run(loop);
    g_object_unref(file);

    return 0;
}

Pretty neat, don’t you think? To build you need to use clang and have the blocks runtime installed (libblocksruntime-dev in Debian). Here’s the command I use:

$ clang -fblocks -o gio gio.c -lBlocksRuntime `pkg-config --cflags --libs gio-2.0`

Looking like the SELinux but less boring, the AppArmor is a Linux security module (LSM) which provides mandatory access control (MAC). The first distro to adopt the utilization of AppArmor was SUSE in SUSE Linux Enterprise Server 10 and in openSUSE 10.1. It is part of Ubuntu since the version 8.04 and the adoption increase version to version since more profiles are created.

Other software that is part of more and more applications each day is the D-Bus, adopted by GNOME and KDE as an inter-process communication mechanism, the usage of D-Bus allows the communication between different applications. It is used, for example, to provide the communication between a software Core with the UI. Due to the nature of the communication of certain applications (sensible data) is indispensable to have some control about who can acquire some interface or who can listen or send some message.

D-Bus daemon has support to mediate SELinux messages and there is also a D-Bus internal mechanism that has some control over the use of the bus, but none of this is related to AppArmor. There are some experiments that show that it is possible however the necessary patches (Kernel, libapparmor and D-Bus daemon) were not submitted to be part of the respective projects, as explained in the earlier post.

The patches on the experiment enable apparmor parser to understand the tag dbus, as illustrated on the example bellow (line 15). More information about the experiment and the syntax of the file can be seen in: https://lists.ubuntu.com/archives/apparmor/2011-September/001541.html

/home/zimmerle/hello.py flags=(complain) {
  #include <abstractions/base>

  /usr/bin/python2.7 ix,
  /usr/include/python2.7/pyconfig.h r,
  /usr/local/lib/python2.7/dist-packages/ r,
  /usr/share/pyshared/PIL.pth r,
  /usr/share/pyshared/lazr.restfulclient-0.11.2-nspkg.pth r,
  /usr/share/pyshared/lazr.uri-1.0.2-nspkg.pth r,
  /usr/share/pyshared/pygst.pth r,
  /usr/share/pyshared/pygtk.pth r,
  /usr/share/pyshared/ubuntu-sso-client.pth r,
  /usr/share/pyshared/ubuntuone-client.pth r,

  dbus bar.foo.hello acquire,
}

In order to ensure the functionality of the suggestion made in the post: D-Bus Loadable security module support, I decided to modify the AppArmor D-Bus daemon patches to make them compatible with the suggested model. And it is working like a charm.

The code of the current experiment can be fetched from:

http://cgit.collabora.com/git/user/zimmerle/dbus-apparmor-lsm.git/

Note that in this experiment I had to use the D-Bus internal functions/headers. I made little hacks in order to get it working but apparently, this is a good way to go.

While I was thinking about LSM mediations of the D-Bus messages, I found out a nice work that is being developed by the Ubuntu sec team in order to support the AppArmor mediation on D-Bus message exchange and service acquisition.

Having a chat with John Johansen (from Unbuntu sec team), he said that he was missing a loadable module support on the D-Bus. Allowing the support of different Linux Security Modules mediation without messing up the D-Bus daemon code, which does make sense.

I started to implement a little PoC about this loadable support, which consists in the following: the LSM modules can be dynamically loadable at the d-bus daemon startup. By copying a D-Bus LMS module to a given directory (which can be specified at the d-bus configuration) it will be loaded and registered.

The idea is to have independent modules, if possible use only the D-Bus functions provided by libdbus, however, of course, if needed symbols can be copied from libdbus-internal.a.

Despite the fact that the modules can be independent of the D-Bus internals, they must have at least one known function, this function should be named as “pre_init“, and receives the pointer to the D-Bus internal function “register_security“. The “register_security” function should be called by the module if it is loaded successfully. The “pre_init” function must return a “dbus_bool_t“: true if everything goes right or false if not. Note that audit can be also initialized by this function.

The function “register_security” receives as parameter a pointer to the structure “security_validations” that is part of dbus-security.h. The structure is illustrated bellow:

struct security_validations
{
 char *name;
 dbus_bool_t (*bus_security_allows_send) (DBusConnection *,
                                         DBusConnection*,
                                         const char *,
                                         const char *,
                                         const char *,
                                         const char *,
                                         const char *,
                                         const char *,
                                         const char *,
                                         DBusError *);
 dbus_bool_t (*bus_security_allows_acquire_service) (DBusConnection *,
                                                    const char *,
                                                    const char *,
                                                    DBusError *);
 dbus_bool_t (*shutdown) (void);
};

The structure “security_validations” defines the hooks and the name of the security module and also the function to shutdown the mediation. Two main hooks were needed, the first is the one responsible to mediate the message exchanges and the second is the responsible to avoid unauthorized process to acquire some service. The shutdown hook is not less important, but less used. Shutdown is only called when the D-Bus daemon is hanging out.

The current implementation of SELinux mediation needs more hooks to work than what I am offering in this PoC. Since the SELinux implementation has some performance improvements by doing caching, it will be necessary to create new hooks to gather some information before deciding whether some message is ok to go or not, but this may be a later discussion.

The patched D-Bus code is available at:

http://cgit.collabora.com/git/user/zimmerle/dbus-lsm.git/

And there is a dummy module at:

http://cgit.collabora.com/git/user/zimmerle/dbus-dummy-lsm.git/

For those who want to give it a try but doesn’t want to install unstable distro, I’ve added the needed pieces in our telepathy ppa.

One of my side projects for these last months was to enroll on the online Introduction to AI class, with Peter Norvig and Sebastian Thrun, professors at Stanford. Through it I also learned about the Kahn Academy. I must say that getting to know these efforts made me feel similar to when I found Free Software: it’s hard to believe that such great things exist!

I learned some really cool stuff, and was also introduced to the amazing work of Sebastian Thrun with self-driving cars, it was an awesome experience! Last weekend I took the final exam, and today I got the certificate of accomplishment. It was delivered as a signed PDF which can be checked with a certificate they provided, pretty neat. I’m very happy, and motivated to enroll on more such courses in the future =). Now it’s time to cool down, though. My vacations start today, and on the weekend I’ll travel to the sunny Fortaleza, in northeastern Brazil, to enjoy some nice beaches and get some tan. See you next year!

Statement of Accomplishment - AIClass 2011