In the context of safety-critical systems, such as transportation systems, it is important to verify if such systems meet/do not meet safety requirements. ISO26262 expresses these requirements as a maximum number of failures per hour of operation. It is generally of a very small magnitude for the most critical components of a system, i.e., 10^-9 failures per hour of operation at most.

As there is a trend towards inter-connected safety-critical systems, there are higher risks that such systems become the target of remote attackers. In this context, remote attackers can eventually control the system remotely (e.g., CarShark attack or Jeep Cherokee remote control). Therefore, there are cyber-attacks that can impact the safety of the system (i.e., endanger the environment of the system).

Therefore, more and more works are dedicated to integrate security and safety such that the addition of security countermeasures inside a system can increase the system overall safety. In existing works, to the best of my knowledge, nobody takes into account the fact that security mechanisms can fail, just as any other components.

My question to you guys is the following: do you have any information about what could be the consequences of a security countermeasure failure on a component or system safety? Is it possible for example that, if an encryption mechanism fails, it flips a bit in a message which could provoke (eventually) a DoS (or anything else), that could lead to a catastrophic event (again, think of it as a worst case)? Is there any discussion/document about this?

  • 1
    I think the first (more important) question one should ask is whether safety-critical systems should be connected to the internet. – Jeroen Aug 16 '19 at 6:13
  • Yes, but it is the case anyway ^^ – Jean Oudot Aug 16 '19 at 6:16
  • There are other options to not expose such systems to the internet. The design in my opinion is flawed and you try to mitigate it while the root cause is not being evaluated. – Jeroen Aug 16 '19 at 6:21
  • Yes, there is a whole field of "vulnerability design", if that's what you're asking about. The problem is that digital systems have a complex and unpredictable universe of states that the system as a whole can have as a result of a failure in any one system. There is an answer to this question, but I'm afraid that it will either be far too long to post, or a pointer to some fields of study that will help you form an answer. – schroeder Aug 16 '19 at 6:39
  • Thanks for the answers. My question came up after reading many recent research papers about safety and security integration. Many researchers work on quantifying the impact that security has on security, in a quantitative or qualitative way. It seems that the impact of adding a countermeasure in a system is not/very rarely taken into account, except at a very high level of the risk analysis. – Jean Oudot Aug 17 '19 at 2:02

You are asking a quite open question, and I think you are in a too-theoretical situation, whereas in the outside world you would probably find that the critical system is using a Windows XP that hasn't been patched in the last 15 years.

Obviously more security checks will lead to “more complexity” in the code/design, which means a larger surface where things could go wrong. But actually, the extra checks should benefit the whole system.

If the countermeasure is properly designed, it should be clear what the status would be on failure. For instance, a system that was available through plain HTTP may have been updated (the security countermeasures) to use HTTPS. Then there was a failure on this countermeasure (the certificate expired). The result is that the system is no longer available (it fails-closed). It could instead have been designed to fail deadly. Different systems may have different goals. The important thing is that such goal is documented and a conscious decision. Not just that, "nobody thought that if the countermeasure we added failed, everyone would die".¹

Note that I don't think the case "if an encryption mechanism fails, it flips a bit in a message" is a suitable scenario. There may be bits that get erroneously flipped, and an encryption layer could amplify it (rather than getting one bit wrong, you get a whole block of garbage). However, if the security countermeasure only added encryption (confidentiality), I would consider it as being badly implemented, since you also need to authenticate that what you received is the right message (integrity), which is what you should be detecting in such failure and whose output situation should be defined in some way (stop the car, drop the internet connection...).

As a non-IT security countermeasure "failing", see the following real-life scenario:

  • Initial state: Terrorists can enter into the plane cockpit and hijack planes
  • Security countermeasure: a security door is added that can only be opened from the cockpit, in order to protect the pilots.
  • Final state: One of the pilots can hijack the plane with the "good" crew members not being able to stop him due to the security door.

Here it is not really a "failure" of the security countermeasure, as it is working as intended (not allowing external access to the cabin). Actually the issue is that it is working so well that it then creates a different problem in the system as a whole.

The problematic part would be to detect that in the design phase of the countermeasure. (And then figure out how to mitigate it)

¹ Even if it is done on purpose, the people that would be killed may still object to that design, though.

| improve this answer | |
  • Thanks for your comments. I knew my encryption mechanism example was not particularly realistic, but another good example is an IDS or with a number of false positive that is too high and unnecessarily trigger safety reactions. Indeed, all of this is highly dependent about how well the system is designed. I was asking because in the research world,many people try to integrate security and safety and analyze how one influence the other. However, at a very formal level (i.e., formal verification), the fact that adding a security mechanism could violate safety requirements seems ignored. – Jean Oudot Aug 17 '19 at 2:07

"what could be the consequences of a security countermeasure failure on a component or system safety?"

The impact would be proportionate to the square root of the damaged quality rate of the countermeasures, criticality rate, and the elevated number of controls that interact with the damaged countermeasures (which can open up more consequences).

Managing all the components, and do an independent risk analysis for each component especially for those who are connected to the internet.

| improve this answer | |

Your Answer

