A Remote Touchscreen-like Control Experience for TVs and More

How do you control your smart TV? With a remote control of course, already quite capable since it allows voice commands to find a movie or TV show without needing all that fiddly button-based control and lookup. But there’s a range of things you can’t do that we take for granted on a tablet or phone screen. Point and click on an object, drag and drop, swipe, or draw freehand (maybe circle an actor to ID). This is what the emerging generation of smart automation for remotes will make possible for TVs, for gaming and for other applications such as interacting with images cast from a PC to a large monitor. LG Smart TVs with Magic Remote have already proven the appeal of an earlier generation of pointing control as LG has included this technology in its flagship and main line Smart TVs since 2010 and over 600 OEMs have licensed their underlying WebOS® providing pointing control enabled by Ceva’s MotionEngine™ sensor fusion software. Now Ceva has extended MotionEngine further, to bring true remote touchscreen-like possibilities to how we can interact with home entertainment and commercial presentation devices.

The opportunity

The global market for smart TVs alone was around a quarter of a trillion dollars in 2023/2024 and is expected to show a CAGR of between 11.5% and 12.8% through 2033. This is closing fast on and may ultimately surpass smart phone market sizes. I guess in a busy, always-on world we still need our passive entertainment down time at the end of the day, much though we like our phones.

Better ways to interact with home (and office) applications have always spurred innovation, as seen in the wide popularity of voice control. Looking for further innovation or parity, TV makers will need to encourage interaction in ways that mirror modern paradigms for touchscreens, without needing to touch the screen. Why not point and drag or wave controls off-screen with your remote? Or shake the remote or draw a symbol on the screen to prompt a user-selected action (jump to your favorite news channel perhaps). I expect that once a trend becomes apparent here it will quickly turn into a flood.

Behind a touchless touchscreen

These systems depend on capabilities in both the host (say a TV) and the remote. The remote uses sensors, such as an inertial measurement unit (IMU), fusing that information to determine pointing position on the target and communicating to the host side software.

Historically these systems have used 3DoF (3 degrees of freedom: pitch, yaw and roll) to sense where the remote is pointing, but more is possible with 6DoF systems combining a 3-axis accelerometer, 3-axis gyroscope and UWB-based position sensing. These can determine position and orientation determination with high spatial and temporal accuracy.

The Ceva-MotionEngine Hex software supports this capability with some unique advantages. First, as a software-only library it is designed to allow OEMs to build around their preferred processor and sensor choices. It also provides absolute cursor positioning, essential to support interactions like precise point and click sensitivity, on- and off-screen gestures and drag and drop.

Further, the library provides the must-have features essential in any application of this nature. Whatever sensors you may be using, Ceva-MotionEngine Hex handles automatic bias calibration, correcting for drift without need for manual calibration. For a hand-held remote, small tremors in holding the remote or pressing buttons could be damaging to accuracy. The library stabilizes against such movements. It supports superior slow-motion detection to select small targets. And it offers very low latency to ensure naturally responsive behavior.

Unsurprisingly given Ceva’s strength in low power applications, the embedded library is designed for systems with very constrained resources and can operate efficiently on low power MCUs or DSPs, just what you need for a TV or gaming remote or a stylus pen designed to write over a PC or projected presentation.  It can also reside in the TV operating system (Android®, Linux®, webOS®) simplifying deployment across many platforms and enabling additional advanced features.

The Ceva-MotionEngine Hex advantage

This software library is built on more than two decades of experience working with MEMS IMUs and partners building motion-aware systems. Among these LG Electronics built the library into their Smart TV and Magic Remote back in 2010 and adoption has grown across LG branded Smart TVs and many third-party brand partners since then.

It might seem that at first glance that an OEM might choose to build this capability themselves. According to Chad Lucien (VP and GM for the sensor and audio business unit at Ceva) this isn’t as easy as it might appear. Delivering a production solution is much more complex, including challenges like robustly handling sensor drift over time, understanding the many possible user experiences that could be modeled, and mitigating the impact of users’ hand tremors which can result in large pointing inaccuracies if not addressed.

Translating from a prototype to a robust production solution is where the promise of a remote touchscreen-like experience could fall apart, unless you work with a supplier which has significant and proven experience in this field. To learn more, check HERE.

Also Read:

WEBINAR: What It Really Takes to Build a Future-Proof AI Architecture?

WEBINAR: Edge AI Optimization: How to Design Future-Proof Architectures for Next-Gen Intelligent Devices

Podcast EP291: The Journey From One Micron to Edge AI at One Nanometer with Ceva’s Moshe Sheier