WizBlog

OpenCL Calculation and Reduction

2010-02-25T21:35:08Z

Otherwise known as "calculating a long list of numbers then adding them all up". For a GPGPU (OpenCL) simulation program at work, I needed to calculate around 160 numbers which would be averaged to produce one result for storage in a 1024x1024 element array. That's 160 numbers for each of 1024x1024 pixels, which would be a lot to store as an intermediate result for a later step of averaging on the GPU, or (heaven forbid) to be copied back to system memory.

The magic word to search for in tackling this is reduction, and there's plenty of hardcore compsci knowledge about how to make it go as fast as possible in a parallel environment. But, basically the trick is to have 160x1024x1024 threads operate in groups of 160 (one group for each of the 1024x1024 overall elements). Threads cooperating like this can share memory, and each thread writes its individual value to an array in that local memory. Then, one of the 160 threads adds up all the values and does a single write of the final average value to the global array. For the kernel to test if it's running the "chosen thread" is as simple as something like this:

if ( get_local_id(0) == 0 )

The only bit of "funny business" is that each of the 160 threads has to have finished calculating before the results can be added. That's done with this statement, which guarantees that all previous local memory writes have completed for all threads:

barrier(CLK_LOCAL_MEM_FENCE);

This is a really simple example: for one thread to do all of the averaging is a waste of resources when the reduction itself could be parallelised. In that case, one thread would (say) add up values 0-79 while another added up 80-159, then one of those threads would (after another barrier) add up the remaining two values. It's easy to see how it can be broken down more and more, and there are variations which make better use of the GPU resources, avoid memory conflicts, and so on.

So, if you'd ever heard of the thread groups and local memory used in OpenCL (also CUDA) and wondered what they were good for, now you know..

NVidia's OpenCL Programming Guide has a lot of discussion of this topic, and there's loads more to be found around the web.

OpenStreetMap's Role in Haiti

2010-02-25T20:12:49Z

This is very cool. Volunteer contributors used satellite imagery, which was made available especially, to create detailed maps of Haiti which the aid workers used to help find their way.

The Mysterious Missing Milliamps

2010-02-10T19:28:06Z

My new A7+ Freerunner arrived a few days ago. Before long I'd flashed it with the latest version of SHR-testing, made sure stuff worked, and moved my main SIM over to it. Then I set about installing all the most cutting-edge unstable userspace and 2.6.32 kernel stuff on the old A6 Freerunner ready to have some fun. But I quickly noticed that something wasn't right - it wouldn't charge properly. During my first Sunday hacking session, it didn't seem to be able to charge from a USB connection to my computer, and seemed to be getting quite warm to the touch as well.

The current_now sysfs node indicated that the device was using many hundreds of milliamps more current than it should have been. So much, in fact, that the current provided to it over USB by my laptop wasn't enough to charge the battery - with barely enough power to keep the thing running the battery slowly discharged. With the higher current provided by the mains charger, the battery would charge, but more slowly that normal. But this only seemed to happen with the 2.6.32 kernel.

Lots of discussion on IRC (mostly with DocScrutinizer) followed, and it emerged that one possibility was a short circuit in the uSD card slot. Sure enough: I'd moved my uSD card from the A6 to the A7+ Freerunner (since the new one didn't come with a card), leaving the A6's SD slot empty. On my device, some of the pins protruded up, perhaps enough to short out against the metal SD slot lid:

I cut a roughly 1x1.5cm rectangular piece of plastic from the packaging of some small halogen bulbs I'd bought the previous day, and put it into the slot as a "dummy". Testing again: no current drain. Dummy removed, the drain came back, and so on. The Doc had hit the nail on the head - the culprit was found!

But there was one more piece to the puzzle. Why didn't the drain seem to happen with the older 2.6.29 kernel? Turns out that a small detail had been left out from the kernel during an earlier merge. Because of this, the Glamo SD engine's power supply was left on all the time, when normally it should get switched off when no SD card was present. The resulting waste of a small amount of power became a huge power drain when the short circuit was added. Hopefully the SD electronics aren't too fried..

Gitolite

2010-01-26T23:13:18Z

I've just spent a few hours rearranging my Git repositories to use Gitolite, something I've been meaning to do for a long time. This gives me much better control over access permissions, potentially letting me give other people access to my repository without handing over shells and/or real user accounts.

I've done some testing, but things might have broken. If they have, address your complaints to the usual address..

New Toys

2010-01-25T18:18:06Z

So, I'm now the proud owner of a Lenovo Thinkpad T500 with WSXGA+ screen (1680x1050), 7200rpm HDD, Core 2 Duo, UK keyboard and 9-cell battery. Exactly to specification, and within budget :D

I've also ordered myself a second Freerunner. This was a bit less of a budgeted expense, but I have no real shortage of money at the moment (thanks to not drinking regularly). I realised that the reason for my recent lack of productivity wasn't time as such, rather the faff involved with switching into a very unstable environment for development then having to go back to a usable setup at the end of a "session". Developing in "sessions" like this seems to be a nice way to avoid getting anything done at all - I had the same problem at the start of my DRM work when we changed to Linux 2.6.29 from 2.6.24. It's much better to be able to work in a semi-continuous stream as time allows.

There's another reason for this purchase though. I'm affected strongly by the infamous Freerunner buzz problem in Germany, whereas I didn't notice it back in the UK. I was going to send my FR in to get both the buzz and #1024 (standby time) fixes done, but I've decided instead just to buy a new one with both fixes already. Then I'll use the new one day-to-day while my current one becomes a development platform, installed with all the latest and most unstable software I can find, so that I can stomp on the nastiest bugs with some degree of comfort.

And there's one more new toy: A 32TB RAID6 array with 4 optical fibre channel connections for storing and analysing our data on at work. All my analyses just went from being I/O limited to being firmly CPU bound..

Code Offsetting

2010-01-23T09:56:09Z

I just came across The Alliance for Code Excellence, where you can offset bad code you've written in the past with donations which support open-source projects which "decrease the propagation of bad code".

New Year

2010-01-08T21:05:16Z

2009 was an awful lot better than 2008 for me, for many reasons. 2008 had a lot of negative things of various types, but I won't go into detail about those. 2009 began with my falling into what could possibly be described as my dream job (particle accelerators and computers!), the completion of my PhD thesis and passing of the subsequent examination before being whisked off my feet by the beginning of an international adventure. I suspect the adventure will be ongoing for the rest of my life. When I moved out of my last long-term accommodation in Cambridge back in May, I knew that things would never be the same again.

I won't go into more detail than that, because his journal has become more about technical things and "productive" aspects of my life rather than personal wibbling. Along which lines, I hope to soon be able (/allowed) to post details of the exciting science I've been doing...

New Laptop Time

2009-12-22T22:53:29Z

{I am getting} {scary messages} in {curly brackets} in my laptop's {dmesg}. The universal warning signal for imminent hard drive death and data loss. I get dropouts of about 30 seconds at a time with no hard drive activity (before the kernel realises and reset the link), during which time the computer is mostly frozen (no HDD I/O possible), and this seems to be happening more and more often. In addition, the power connector is broken - the central pin in the laptop's connector snapped off. Since the pin stays fixed in the hole in the adaptor's plug, it still just about works if it's carefully pushed in and the cable wrapped round to put pressure in the right way. However, I don't know how long either of these will hold out.

Of course I'm backed up to the hilt with distributed version control, so I'm not in immediate danger of losing anything particularly important. However, it's apparent that I'll need to buy a new laptop in the near future. At the moment I'm looking at a Lenovo Thinkpad T500 with WSXGA+ (1680x1050) screen and Radeon graphics, but does anyone have any other suggestions? My non-negotiable requirements are:

Linux-friendly wifi and graphics.
Dual core, or at least HT. This really does make a huge difference.
Widescreen. 1680x1050 with a 15.4" screen gives a resolution I like.
UK keyboard layout (i.e. UK market, ideally with delivery to Germany possible).
DVD drive.

Ideally it would also have:

Decent battery life, or the possibility to buy spare or larger batteries during the next few years once the original one becomes a plastic box of jelly.
VGA output.
Fast-ish hard drive (7200rpm or higher. I'm not sure how much of a difference this makes, but I do a lot of compiling and so on. No need to go overboard with solid-state disks for hundreds of extra pounds.

I'm not too bothered about:

Bluetooth (I don't use it at the moment).
Huge hard drive - I get on fine with only about 80Gb at the moment.

Any suggestions on a comment or email to this address..

Trip to Stanford

2009-12-20T11:11:21Z

On Friday I got back to Hamburg from San Francisco, where I'd been taking part in experiments at the Stanford Linear Accelerator Cente. We used the LCLS to zap various kinds of object and measure the resulting signals. Without going into too many gory details, I'm happy to be able to say that the experiment was a great success and we got several terabytes of useful data, which we'll be analysing for many months to come.

I hardly had any time to escape the campus or to explore, but I did manage to find some sites of geeky interest. Here's a photo of the research stations at the end of the two mile long linear accelerator, which is hidden by the trees at the back of the picture. The large concrete building on the right is End Station A, where the first experimental evidence for the existence of quarks was recorded around 1966. Their experiment was like a much larger version of Rutherford's scattering experiment with alpha particles and gold foil. Today, End Station A contains test experiments to prepare for the International Linear Collider. I wasn't able to go inside - it's probably possible, subject to the particle accelerator's beam being directed elsewhere and talking to the right people, but there wasn't time and I wasn't keen to push the limits of my security pass (I also couldn't find the door..).

The similar concrete monstrosity on the left is End Station B, which probably contains similarly cool things (remember what I wrote about concrete?). The long building which comes out between and runs off the picture to the left is the LCLS beam transport hall. Standing where I took this picture, below me and to the left would be the undulator hall, which contains magnets which cause the electron beam to emit X-ray pulses of unimaginable brightness and brevity. The tunnel emerges on the other side of the hill I'm standing on, where the electrons and X-rays move into the building where we worked. The electron beam is dumped (basically by directing it into the ground) and the X-rays make their way into our experiment.

Comments Enabled

2009-11-20T21:51:29Z

Due to popular demand, I've enabled comments and trackbacks here. Please don't all spam me at once... :)

Internal Memory Bottlenecks and Their Removal

2009-11-15T16:53:12Z

While debugging something different late last night, I noticed some flags in one of Glamo's registers which looked interesting: FIFO settings for the LCD engine. This reminded me of an observation by Lars a few weeks ago that the LCD engine seems to conflict with Glamo's 2D engine on memory accesses, leading to slower performance of accelerated 2D operations when the screen is switched on. So I turned the FIFO up to "8 stages" (from 1) to see what happened. The result was much faster 2D operations - literally twice the speed!

At "8 stages", the price of this was that the display became jittery and unstable. However, the same speed improvement is seen at the "4 stages" setting. I've also seen some occasional artifacts with this setting, so I'm using 2 stages at the moment, where the speed is still right up there. I'll be testing some more and seeing if things can be tuned even more.

Because we don't make the maximum use possible of the 2D engine, this doesn't immediately translate into a huge increase in the UI speed. But the differences are very obvious with x11perf and some of my test programs. The program I showed in the screenshot recently jumped from 45-48fps right up to 95-98fps!

"Look Ma, No Busywaits!"

2009-11-13T20:47:39Z

When the CPU needs to do something which depends on a result which the GPU is currently working on, it has to wait for the GPU to catch up. One of the biggest problems with the current architecture of xf86-video-glamo, both DRM and non-DRM versions, is that they do this waiting by spinning in a tight loop, each time checking the current status of the GPU, until it's caught up. This isn't great for a few reasons. It makes no use of the parallelism between the CPU and the GPU, so precious CPU time is being wasted while something more useful could be being done. If there's nothing else to do, then the CPU could be sleeping - reducing power consumption.

Most GPUs, including Glamo, have a mechanism for being a little smarter. The kernel can ask the chip to trigger an interrupt when a certain point in the command queue has been reached. When a process needs to wait, the kernel can send it to sleep and watch out for the interrupt. When it happens, the process can be quickly woken back up in a low-latency fashion, meaning that the process gets back to work with very little latency.

This week, I've been implementing this kind of thing for the Glamo DRM driver. It goes a bit like this:

Process submits some rendering commands via one of the command submission ioctls.
Kernel driver places rendering commands on Glamo's command queue.
Process needs to wait for the GPU to catch up, so calls the wait ioctl.
Kernel driver puts an extra sequence of commands, called a fence, onto the command queue. A unique number is associated with the fence. The number is recorded by the kernel.
When the GPU processes the fence, it raises the interrupt and places a unique number into a certain register.
The interrupt handler checks this number, and wakes up the corresponding process.

I wrote a test program which tells Glamo to fill the whole screen with colour as fast as it can, waiting for the GPU to catch up each time. The task was to make the program run with close to zero CPU usage while still getting the full framerate that I could get using busywaits. The task was achieved successfully, and here's a screenshot to prove it. The framerate - just below 50fps when doing fills of the entire VGA screen - was exactly the same with busywaits. It even went up a little (to 50-51fps) when I improved the interrupt handling.

Things aren't always so great. When the command sequence to be executed is very short, the overheads of fencing and scheduling become significant, and the overall rate drops. However, it shouldn't be too difficult to design some kind of heuristic to use busywaits as a low-latency strategy in such cases.

There are still a few problems to iron out. The fence mechanism seems to be able to fall out of sync with things, leading to processes waiting for too long (or even forever). But when it works, some things do seem to feel a little faster in general use.

Geeks may be interested in the actual code.

Concrete Blocks

2009-11-07T21:31:37Z

Concrete blocks. Absolutely everywhere. That's my dominant impression of this international particle accelerator research centre after the first month. Concrete blocks shielding the outside world from radiation emitted by the shiny things hiding behind them. And generally, the bigger the pile of concrete blocks, the cooler the thing that's lurking behind.

Here are some photos from today's open day at DESY. Most of the things shown (everything apart from FLASH and XFEL) have nothing to do with what I work on, but they're still exciting to look at. The HERA and PETRA tunnels aren't normally open, least of all to the public, and there probably won't be another opportunity to see them for years. In pictures 38, 40, 42, 45, 46 and 51, you can see the sequence of bits of pipes and coils which guided electrons from PETRA, physically above HERA, into HERA's electron ring. HERA was switched off in September 2007, but almost all of it is still in the tunnels. You can also see wider views of the machine. The cylindrical pipe thing on the top is the superconducting ring of magnets which guided protons, and the pink boxy thing underneath is a normally conducting ring of magnets for the electrons. You can even see what's underneath the pink metal cover, but it's not very exciting. Then there's a spin rotator which alters the polarisation of the electrons. A bit further down, you can see the electron and proton rings being brought closer together (the electron beam pipe is the thin bronze-coloured thing just in front of the yellow thing), and then going through the final focusing magnets before colliding with one another in the next room. Not that you can see anything except concrete blocks, because that bit is just way too cool.

And it needs a whole lot of cryogenic stuff to make it work.

PETRA was previously used for particle physics, before being turned into a pre-accelerator for HERA and more recently (last year or so) into a synchrotron radiation source for (e.g.) protein crystallography. This thing is still used - in fact it's one of the most modern synchrotron X-ray sources in the world - but it wasn't switched on while we were in the tunnel, otherwise we would have been fried. Naturally it's hidden behind a huge wall of concrete blocks.

There are plenty more photos to see beyond the ones linked here..!

Mains Adaptors

2009-10-21T21:54:04Z

I brought some UK-style multiblock extension leads over from the UK, and replaced the plugs with continental European plugs bought for €0.99 shortly after I arrived here.

There are no words which adequately describe how superior the resulting devices are compared to those one-piece plastic adaptor things you buy at the airport.

It Doesn't Feel...

2009-10-14T12:38:50Z

... like I'm in a different country. Not in the same way that it does (or did, previously) when I visited other countries. Maybe it's because almost everything at work is done in English, or maybe it's because I've had such a long time to get used to the idea before coming. Or maybe it's because things aren't actually that much different over here from what I'm used to.