How Do You Do Computational Photography at HD Video Rates?

Increasingly, a GPU is misnamed as a “graphics” processing unit. They are really specialized architecture highly parallel compute engines. You can use these compute engines for graphics, of course, but people are inventive and find ways of using GPUs for other tasks that can take advantage of the highly parallel architecture. For example, in EDA, Daniel Payne talked to G-Char at DAC in June who use GPUs to accelerate SPICE circuit simulation.

Users don’t actually want to program on the “bare metal” of the GPU. Firstly, often the hardware interface to the GPU is not fully revealed in the documentation since it is only intended to be accessed through libraries. And secondly, whatever the GPU interface is to this GPU, it will not be the same to the next generation GPU, never mind to a competitor’s GPU. So standards have sprung up not just for using GPUs for graphics but also using GPUs for general purpose computation. The earliest of these was OpenCLand Imagination were a pioneer in using mobile GPU compute, being the first mobile IP vendor to achieve OpenCL conformance.

Indeed, to encourage wider adoption of GPU compute in mobile, Imagination has delivered a number of OpenCL extensions related to advanced camera interoperability. They can be used with the Samsung Exynos Soc which is the application processor in the Galaxy S4 and some other smartphones, and is also available on a development board. These new OpenCL API extensions enable developers to implement Instagram-like functionality on real-time camera data, including computational photography and video processing, while offloading the main CPU.

The challenge to doing all this, especially in HD (and 4K is coming soon too), is that video generates a lot of data to swallow, and it has tight constraints: if you don’t like this frame there is another one coming real soon.

Historically, computational photography like this has been done on the main CPU, which is fine for still images (you can’t press the shutter 60 times a second even if the camera would let you) but it doesn’t work for HD video because the processor gets too hot. It is no good adding extra cores either, since the CPU will just overheat and shut down. But modern application processors in smartphones contain a GPU (often an Imagination PowerVR). The trick is to use a heterogeneous solution that combines these blocks and lives under the power and thermal budgets.

In an application, for example, performing real-time airbrushing on a teleconference, there is a camera, the CPU, the GPU and a video codec involved, four components all requiring access to the same image data in memory. Historically, all OpenCL implementations in the market created a behind-the-scenes copy of the image data while transferring its ownership between components. This increases memory traffic, burns power and reduces performance. This negates, or perhaps even eliminates, the advantage of using the GPU in the first place.

Imagination has been working with partners to develop a set of extensions that allow images to be shared between multiple components sharing the same system memory. So no increased memory traffic, lower power and better processing performance, based on Kronos (the OpenGL group) EGL images which handles issues related to binding and synchronization.

See a video explaining and showing it all in action (2 mins)

To accelerate adoption, Imagination is releasing the PowerVR GPU compute SDK and programming guidelines for PowerVR Series5XT GPUs. In the future they will release an SDK supporting PVRTrace for PowerVR Series6 GPUs, which will allow compute and graphics workloads to be tracked simultaneously.

More details are in the Imagination blog here.