Linux for Medical Devices Q&A

As I have mentioned before SemiWiki gets to meet some very smart people and here is another one. Scot Morrison has an MS degree in Aerospace Engineering from MIT specializing in control systems. Today he is the general manager of the Embedded Platform Solutions Division at Mentor, a Siemens business. Scot oversees the Linux®, Nucleus®, and Mentor Embedded Hypervisor runtime product lines, as well as associated tools, middleware, and professional services.

Prior to joining Mentor in 2012, Scot served as GM and SVP of products at Wind River Systems, Inc. Before that he worked at Integrated Systems Inc., where he last served as the VP and GM of the design automation solutions business unit in 1999, responsible for MATRIXx, and the pOSEK embedded operating system.

In speaking with Scot, the top three points of interest/intrigue with Linux for medical devices is Safety, Security, and Size (of code) so keep that in mind as you continue reading:

Q: Can I use Linux in a medical device?
The answer to this question is, it depends. Linux has been deployed safely in a wide variety of medical devices, but in order to use Linux in a medical device that has a safety requirement you need to follow the process defined by the certification standard that you must comply with.

Q: We all know that safety and security are both necessary when we’re looking at medical devices. We hear that you can’t have safety without security, but why is that?
Security is something that can be looked at standalone; and even in medical devices, not all aspects of security are tied to safety. For example, when we talk about protecting people’s personal information, this is an aspect of security that does not overlap with safety. But, when we talk about safety, we talk about things that if they go wrong will impact the patient’s health. If the device is not secure, it makes it possible for bad actors to make these negative impacts happen either accidentally, or purposefully (which is much rarer).

Q. Does the use of Linux and other open source software help protect these devices?
Linux is the most heavily used operating system for devices in the world. It has a large, worldwide developer base that focuses on ensuring that it works as expected in all conditions, and is as secure as possible. That said, it is also the most studied operating system in the world, both by the vast majority of developers who are conscientiously working on improving Linux and other open source packages, and also by a small number of actors who are looking at ways to break into Linux for their own purposes. The security of applications using open source is a constant tug of war between these opposing forces. And as above, without security, you can’t have safety.

Q. How is it possible that Linux can have so many security flaws that we’re always finding more?
Linux (and other major open source packages like OpenSSL or SQLite) are large packages that have sometimes unpredictable interactions with other software that might be running in the system. This is combined with the fact that many flaws are hard to find in code reviews, normal testing, or by static analysis, and are undetectable unless software is combined with task switching and inter-process communication. Best practices will not identify every possible flaw or exploit, and much of the open source software that we rely upon was not originally developed with today’s best practices in place.

That said, the most important pieces of open source software used in devices world-wide are in a much more stable and secure place than they were 5 years ago, mainly due to the hard work and diligence of engineers all over the world in identifying avenues of exploitation, fixing those when they find them, and then the worldwide community looking for similar issues in their own projects. The work will never be complete, but it is becoming harder and harder to find exploitable flaws in this important infrastructure software.

Q. What happens when a security issue is found in Linux?
For the most part, security issues in Linux (and other important software like OpenSSL) are found by engineers either by happenstance (a bug that they uncover as part of their work), or through concerted efforts to find exploits (“white hat” hacking). Very occasionally, an exploit will be found as a result of a post-mortem analysis of an attack, but that is very uncommon. In either event, the discoverer of an exploit will notify the community of the offending open-source component, and in most cases, the discoverer or somebody in the community will notify the Common Vulnerability and Exposures (CVE) group, which is run by MITRE, and is closely related to US National Vulnerability Database (NVD), run by the National Institute of Standards and Technology (NIST).

Once a vulnerability is understood, and usually after a fix is available, the CVE is publicized by inclusion in these lists and if sufficiently serious, discussed by the security community worldwide. This is the point where devices are potentially most vulnerable; since most vulnerabilities are found by the “good guys”, the bad guys find out about them at the same time as the rest of the world, and can deploy exploitation that take advantage of the newly found vulnerability. That said, this publicity is very important, since it alerts the worldwide community of both the issue and the fix, so that an organization can determine if a particular exploit might affect their devices, and if it is, they can mitigate the issue before it may be attacked.

Of course, not everybody will be able to update their devices, which will leave them open to attacks, but since there are no real secrets in the world, this openness prevents more issues than it causes.

Q: Let’s bring it back to safety. Is Linux safe?
An operating system like Linux does not directly do anything to make a device safer. The Operating system doesn’t prevent a failure from occurring, nor does it make the system recover when a failure occurs; so in the terminology of safety an operating system is not a safety mechanism. This makes sense when you think about it. When you put Linux on a system with no other application and turn it on, Linux boots, but it just sits there at a login prompt; it’s not doing anything until applications run that leverage Linux, and it’s those applications that contribute to the overall safety of the device. When you think in those terms, while an operating system is not a safety mechanism, it does enable them; going back to the terminology of safety, the operating system is considered to be a safety element; something necessary but not sufficient to create a safety mechanism. So, the real question is “Can Linux be used as a safety mechanism?” The answer to that question is “Yes, but it’s complicated”.

Q. Why is it complicated?
To answer that, we need to look back 5 years or so. At that point, if Linux was used in a medical device, the architecture was designed to separate Linux from the safety critical part of the system to the greatest extent possible because it wasn’t trusted. The system would probably run Linux on a separate processor, and Linux would be responsible for activities that it was well suited for, like connectivity to the office or hospital network, running a display, taking user input, etc. Then, if information came in that would affect the safety function of the device (say, a dosage in an infusion pump), the new setting would be validated on another processor or in hardware before the change was administered to the patient. In many cases, these kinds of inputs that impact safety might not even come from Linux, but maybe from a dial or other hardware input. If an issue arose in Linux it would not impact the rest of the device, which would continue to execute safely.

Today, microprocessors are much more powerful and complex, with many cores, which are designed to support what we call heterogeneous multiprocessing; where there will be powerful general-purpose cores to support running an operating system like Linux, and then more specialized cores to handle other functions. Your cell phone probably has such a processor (or capabilities that provide for separation); the powerful processor is running Android or iOS and your apps, while something separate is managing your secure data. Safe devices today are taking advantage of this trend; the microprocessors are much cheaper than the multiple ones you might use in the past, and much more powerful. But, it means that there are more considerations to think about when you use them.

Q. How does a multiprocessor system affect safety?
Designing for safety is no longer just a hardware or software issue, it’s an integrated systems issue. To take full advantage of the lower board BOM costs and higher integration of components in an advanced multiprocessor chip for a safety sensitive design, applications must be kept separated – what’s known as mixed-safety criticality.

For example, you can now run the user interface portion of your system on a processor cluster that’s optimized for user applications with features like multi-level cache, and power islands for shutting off individual processors and memory, thus conserving power, say in a battery-operated medical device. Linux can also make optimal use of the application processor cluster with built-in features like Symmetric Multi-Processing, parsing tasks and threads to individual processors. Simultaneously, the safety critical portion of the system runs on a separate cluster that is dedicated to real-time processing with features like tightly-coupled data and instruction memory with extremely low fetch cycles, and highly deterministic performance; or lockstep mode to for error detection.

Advanced multiprocessing systems contain hardware-enforced isolation that keeps the application world and the safety-critical world separated, but the software designer will need to use middleware such as the Mentor Hypervisor or Mentor Multicore Framework to take advantage of those hardware features. These software packages make important system-level functions like secure Inter-processor Communication (IPC) between the processor clusters possible.

Q. Are there other hardware considerations Linux can take advantage of?
Definitely. One example is an advanced option in the Linux kernel known as the Power Framework. The software application can utilize the kernel to power down portions of the system when they’re not needed, and power them up when required, conserving system power. Advanced SoCs include peripheral interfaces in the processor clusters like I2C or USB that can be controlled in this manner, but that advantage can also be extended to implementing “soft” peripherals external to the processing cluster in RTL logic, for example a CAN interface; and even peripherals outside the chip, like an Ethernet PHY.

Taking full advantage of modern embedded systems means designing hardware with awareness of what the software is capable of; while at the same time, writing accompanying software drivers, firmware, and applications that can utilize all the features of the hardware.

Q. And I can get this advanced middleware and drivers from the silicon vendors?
The silicon vendors are primarily focused on what they do best, providing the best chips available to the market. All embedded silicon vendors provide some level of software support, but typically this is limited to bare-metal drivers, open-source builds of Linux, and some limited middleware. For advanced features like enabling mixed-safety criticality medical systems, the best option for the designer is to utilize a software vendor like Mentor that specializes in enabling multi-core applications. Thus your designers can focus on your own IP, and not having to code and debug the middleware.

Q. So, can Linux be pre-certified for use in a device?
Not really. Certain real-time operating systems (RTOSs) such as the Nucleus RTOS from Mentor, can be acquired pre-certified, as can other embedded software components from a number of vendors. To achieve this kind of pre-certification the vendor must be able to show that the complete software development process (requirements, design, development, testing and verification and all of the steps of development) have been performed to medical industry standards such as ISO 13485 and/or IEC 62304. Linux and other open source components are not developed to these standards, and thus cannot be pre-certified. As a note, there have been efforts to show conformance of Linux to the over-arching concepts of functional safety (usually mapping to IEC 61508, from which many industry standards are derived, including IEC 62304). While these have not been successful so far, the current effort (project ELISA) is showing promise in this area both in improving the processes used in open-source software development, and in mapping the higher-quality output to these standards. However, this promise is likely years away from being completely realized.

Instead of pre-certification, in medical devices, Linux is generally handled using a concept from IEC 62304 called Software of Unknown Providence, or SOUP. Under these guidelines the use of Linux is considered as part of the risk assessment of the overall device, and potential failures of Linux as used in the device must be considered, and mitigated if they might cause harm to a patient.

This risk assessment must meet the requirements of the FDA’s pre and post submission guidance, so on the front end it requires considerations on the use of Linux in the design, implementation, testing and verification of the device, and then after release, the use of Linux (and all open-source software) must consider the possibility that issues will be found after release. Certifiers are taking a very close look at this aspect of open-source, especially as far as security issues (i.e. CVEs) are concerned.

Q. How does using a commercial Linux provider help?
If you use open-source in your device, you are responsible for acquiring it, and maintaining it in your device both during development and after release. Acquiring Linux (and other open-source modules that you are likely to use) is easy; it often is available from your board or microprocessor vendor targeted to your development board. However, this Linux is usually made available on an as-is basis and is meant to get a system up and running; your board vendor will not be performing maintenance or timely updates of that Linux for you to use after release. As a result, this responsibility falls onto the device developer to manage, for the life-time of the device. This is not impossible, but by doing it yourself you are committing engineering resources to monitor and maintain the significant number of CVEs that are reported against open source components every day… As an example, in 2019 there were 170 CVEs issued just against the Linux kernel and 12,174 CVEs created in total. It is a significant effort to review each of these, determine which are important, round up the fixes, and verify that the operating system is still operating as you would expect. A commercial Linux provider does this work as part of their core competency, and while they will charge you for this kind of service, it will cost you less in the long run and you’ll end up with a higher-quality product with fewer integration issues as you maintain the product over its lifetime.

For additional information please check the Mentor Embedded Software and Medical Devices landing pages.