Bring high end camera image quality to smartphone

We have to go back to 2008 to understand why Super Resolution is desperately needed by smartphone users, expecting to take high quality pictures with their smartphone, at least as good quality as with their camera. It’s in 2008 that smartphone worldwide shipments have surpassed standalone compact camera shipments… and we don’t expect this trend to reverse!

When you buy a smartphone, you probably don’t buy it first for the camera function, but you are happy to learn that you benefit from a 41M pixel sensor in your phone, and you think –like I do- that such a pixel count should provide top quality image. In fact, the CMOS sensor size (the chip size) in smartphone is growing smaller than in a camera so, when the number of pixel is growing, the pixel size becomes very small, and becomes more sensitive to noise and low light. Two other effects, simply based on geometry consideration, will again affect the image quality in smartphone compared with a camera, whatever the number of pixel your sensor is, as we will see with these two pictures:

The first effect is due to the distance between the sensor and the lens: as we can see, there is at least one order of magnitude difference, if no more, when comparing the smartphone with the high end camera.

The second difference is simply linked with the lens sizeitself. It’s only geometry, but optic laws are totally based on geometry. Even if it looks cruel, it’s a matter of fact and, since Euclide has issued the first rules and theorems, we can’t change these laws… But, that we can do is digitalizing the signal, then process it, using DSP algorithms, that’s the solution proposed by CEVA, called “Super Resolution”.

The principle looks obvious, like any great idea: instead of trying to take one high quality image with your smartphone (which is almost impossible due to the geometry), you take four images, with a low resolution sensor (say 5MPixel), and you process it, to finally generate high resolution image. As you can enhance the resolution by 2X by axis, you can generate up to 20 MPixel quality image, starting with your 5 MPixel sensor.
The process starts by enhancing image quality by:

• Extracting image details
• Reducing Noise and Luma & Chroma channels
• Accurate Sharpenning
• Ghost Blur Removal

Then you run the algorithm stages:

• Course Registration
• Fine registration
• Image fusion, including ghost removal

Super Resolution is described in technical papers for a while, but based on iterative process requiring high bandwidth, up to 10, 000 operations by pixel. CEVA has greatly improved this complexity, down to less than 100 operations by pixel on a PC and finally, using CEVA MM3101 DSP core, up to 16 cycles/pixel. For example, in a 28nm process, the CEVA-MM3101 processor is able to take four 5MPixel images and fuse them into a single high-resolution 20MPixel image in a fraction of a second, while consuming less than 30mW.

Jeff Bier, founder of the Embedded Vision Alliance (www.Embedded-Vision.com), commented: “Smartphones are the most commonly used devices for capturing still images and video, but the slim form factor of these devices places severe limitations on the quality of captured images. CEVA’s Super Resolution algorithm, coupled with the CEVA-MM3101 imaging and vision processor, is an excellent example of how clever computer vision algorithms can be combined with optimized processor architectures to overcome physical limitations of imaging systems.”

Comparing image quality on a PC is not an easy task, but we can understand, from the above picture, that CEVA SR looks better than a PC application, and far better than Bi-cubic. And we agree with Eran Briman, vice president of marketing at CEVA, commenting: “Our new Super Resolution algorithm for the CEVA-MM3101 platform marks the first time that this technology is available in software for embedded applications. It is a testament to both the expertise of our highly skilled software engineers and to the low power capabilities of our CEVA-MM3101 platform, which comprises the hardware platform together with optimized algorithms, software components, kernel libraries, software multimedia framework and a complete development environment. We continue to lead the industry in the embedded imaging and vision domain and the addition of this latest high performance software component to our platform furthers illustrates the strength of our IP portfolio for advanced multimedia applications.”

The CEVA-MM3101 offers SoC designers an unrivalled IP platform for integrating advanced imaging and vision capabilities into any device. Coupled with CEVA’s internally developed computational photography and imaging expert algorithms such as dynamic range correction (DRC), color enhancement, digital image stabilizer and now super resolution, CEVA’s customers are equipped with a full development platform for image enhancement and computer vision applications for any end market, including mobile, home and automotive.

For more information, visit www.ceva-dsp.com/CEVA-MM3101.html.

Eric Esteve from IPNEST

lang: en_US