Real-Time AI-driven Image Signal Processing with Reduced Memory Footprint and Processing Latency

In our day to day lives, we all benefit from image signal processing (ISP), whether everyone realizes it or not. ISP is the technique of processing image data captured by an imaging device. It involves a series of algorithms that transform raw image data into a usable image by correcting for distortions, removing noise, adjusting brightness and contrast, and enhancing features. So, ISP by itself is not something new.

What is new though is leveraging artificial intelligence (AI) to enhance ISP to yield better results than traditional ISP can. Firstly, in the field of digital photography, AI can significantly enhance ISP capabilities. Traditional ISPs are effective at processing images, but AI can take this to the next level. For instance, AI can help in noise reduction, improving the image’s clarity, especially under low-light conditions. Moreover, AI algorithms can recognize various scenes or objects in the image, enabling automatic adjustments to different parameters like brightness, contrast, and saturation for optimal results.

In the world of autonomous vehicles, AI-enhanced ISPs can process real-time images to understand the vehicle’s surroundings better, aiding in decision-making. Traditional ISPs can struggle with different lighting conditions or object detection at high speeds, but AI can improve upon these aspects, enhancing the vehicle’s ability to react to potential hazards and improve overall road safety.

Lastly, in the realm of surveillance and security, AI-enhanced ISPs can process images from CCTV footage more effectively. AI can help detect suspicious activities, recognize faces, or identify objects left unattended, providing real-time alerts and enhancing overall security measures.

But how to implement AI-based ISP? It is easier said than done. There are a lot of challenges to overcome.  AI algorithms are fast advancing, requiring an AI-ISP solution to be programmable. AI models require a lot of computational power. Conventional  techniques combining separately developed ISP and NPU often require a lot of memory to store entire frames of images for processing. Accessing DDR-SDRAMs consume a lot of power. And this loosely coupled frame-based solution will introduce latencies measured in frames, which is unacceptable for many applications.

What is needed for today’s applications is real-time processing at low latency and low power consumption. Making the solution DDR-less is even more attractive as it will reduce the system power requirements significantly. Of course, NPUs are key to an AI-based ISP solution. But there is a lot more to arriving at a DDR-less, low latency, low power AI-ISP solution. This was the topic of Mankit Lo’s presentation at the recent Embedded Vision Summit conference. Mankit, who is the Chief Architect, NPU IP Development at VeriSilicon walked the audience through the various aspects that need to be addressed.

Solution Requirements

ISP Requirements

In traditional hardware for ISP, there are many modules in the pipeline stages to correct the potential artifacts of the imaging system. To perform AI-ISP, the chosen ISP should be flexible enough to allow the customer to pick and choose the modules to replace or enhance.

NPU Requirements

As ISP tasks are computationally huge and intense, the task is usually partitioned to be executed on many NPU cores. There is a lot of image overlap on the input side going into the NPU cores. Even for a 3×3 convolution layer inside the neural network, the overlapping requirement for just a few pixels could result in a huge overlap at the whole network level. The overlap needs to be minimized for reducing the memory, power, and computing demand on the system. The way to do it is through layer-level overlap sharing.

What is needed is an NPU that can handle raster lines and become part of the ISP processing pipeline, making the solution DDR-less, low latency, and low power. The NPU needs to be programmable to handle a changing AI network model landscape and deliver very good performance. It also needs to be able to support Per Layer Overlap Sharing which results in not requiring any overlap on the image input side.

VeriSilicon Offers

Specific to the topic of AI-ISP, VeriSilicon provides ISP, NPU, and the FLEXA-PSI interface to seamlessly connect these IPs. Refer to the block diagram of a VeriSilicon AI-ISP solution.

The customers need to just add their custom AI algorithms to the mix to complete their unique solution. VeriSilicon can also provide algorithms relating to the ISP such as noise reduction, demosaicing, and different types of detection such as facial detection and scene detection, etc.

Click here to learn more about VeriSilicon’s NPU IP offering.

Click here to learn more about VeriSilicon’s ISP IP offering.

VeriSilicon offers custom silicon services and a very broad portfolio of IP for many different markets. Learn more at VeriSilicon.com

Also Read:

VeriSilicon’s VeriHealth Chip Design Platform for Smart Healthcare Applications

VeriSilicon’s AI-ISP Breaks the Limits of Traditional Computer Vision Technologies