# **Regulatory Data Science for Medical Devices**

Edited by Jeroen Bergmann Printed Edition of the Special Issue Published in *Prosthesis*

www.mdpi.com/journal/prosthesis

## **Regulatory Data Science for Medical Devices**

## **Regulatory Data Science for Medical Devices**

Editor

**Jeroen Bergmann**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editor* Jeroen Bergmann University of Oxford UK

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Prosthesis* (ISSN 2673-1592) (available at: https://www.mdpi.com/journal/prosthesis/special issues/medical devices prosthesis).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-3893-8 (Hbk) ISBN 978-3-0365-3894-5 (PDF)**

© 2022 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**

## **About the Editor**

**Jeroen Bergmann** is Associate Professor in the Department of Engineering Science at the University of Oxford. He is the Director of the Oxford Healthtech Labs and Official Fellow at Reuben College. He has a keen interest in Biomedical Entrepreneurship and has worked on a range of industry collaborations and business development projects. He became interested in using data science methods to tackle issues with navigating medical device regulations. He is now applying quantitative methods to the field of medical regulations to create new insights in the field of Medical Regulations.

## *Editorial* **The Emerging Field of Medical Regulatory Technology and Data Science**

**Jeroen H. M. Bergmann**

Department of Engineering Science, University of Oxford, Parks Road, Oxford OX13PJ, UK<EMAIL_ADDRESS>
Regulations contain rules setup by (governmental) authorities to control specific aspects of certain industries, which often influences the way companies operate. These rules affect how industries are managed, and the importance of regulations is such that many companies create specific divisions focused solely on the regulatory strategy. Regulatory frameworks encourage consumers to adopt innovations by ensuring that their safety and effectiveness has been evaluated. However, they also create barriers that can hold up the innovative process. For innovative firms, regulations are one of the most significant barriers of perceived environmental uncertainty [1], which is especially problematic for start-ups with constrained resources [2]. Entrepreneurs need more information to better identify the relevant regulation and understand the requirements for conformity based on these regulations [3], which are the initial steps in the regulatory navigation pathway. The exact impact of regulation on innovation varies between both industries and countries [2]. Certain sectors, such as finance, energy and medical products, are rigorously regulated. It has been suggested that the successful disruption of an industry from pioneering innovation is always followed by regulation. A strategy must be in place to react to the legislations introduced as a result of new innovation. An example of this is data protection laws, which resulted from an increased level of personal information being stored by organizations due to advances in medical care, telecommunications, transportation systems and financial transfers [4]. When used effectively, regulation drives the direction of innovation and can stimulate it within industries [5]. Environmental policies forced car manufacturers to improve gas mileage, resulting in improvements in engine technology. A better understanding of legislation can help to alleviate the barrier to innovation that regulation presents.

Technology and data science has become an integrated part of how many industries operate, and it often affects their regulatory strategy. The rapid expansion of digital technology has also started to impact regulations themselves. Not only is legal information now available in a digital form, but some of the data held by regulators have become freely available online. The particular intersection between regulations and technology is known as Regtech. The main focus of Regtech is to support the different processes that are related to regulations. RegTech was initially suggested for addressing regulatory challenges in the financial system, through the use of innovative technologies [6]. However, the term has evolved to capture any area of regulation, including medical regulation. Buckley et al. have stated that RegTech can help create more effective and efficient ways to comply with the regulations [6]. RegTech can be applied to obtain better regulatory compliance or give the same level of compliance at a lower-cost. It is easy to see that both these outcomes are valuable for those working in the medical technology sector.

As mentioned, regulations are a key part of the medical innovation roadmap. It provides a framework to ensure patient safety and aims to guarantee a beneficial clinical performance of novel solutions. Any medical device that wants to be brought onto a (regulated) market needs to adhere to the regulations that have been set out by governments. All major markets in the world are regulated, and thus manufactures need to think carefully about their regulatory strategy. At the moment, there is relatively little research on RegTech

**Citation:** Bergmann, J.H.M. The Emerging Field of Medical Regulatory Technology and Data Science. *Prosthesis* **2022**, *4*, 169–171. https://doi.org/10.3390/ prosthesis4020017

Received: 2 April 2022 Accepted: 6 April 2022 Published: 9 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

within the medical field. Nonetheless, this is very likely to change in the future, due to the potential benefits it can offer to the different stakeholders (such as manufacturers, regulators and healthcare professionals). The digitisation, data availability and need for more concise understanding of regulations are all driving this change. We have already seen a similar thing happening within the financial sector, where the label Fintech was coined for financial RegTech. However, FinTech encapsulates more than just regulatory technology, as it also covers the technology which replaced traditional financial services. The term FinTech is thus not limited to just regulations, and this has muddied the waters somewhat. Despite this limitation, value can still be derived from a common term that can capture relevant regulatory developments in a specific field.

Such a label is currently missing for medical regulations, whilst it could be captured by the introduction of a new term, such as "MedRegTech". Unfortunately, the term MedTech is not appropriate in this case, as it represents medical technology more generally. This issue mirrors the problem described with the label FinTech. Introducing a more specific term of MedRegTech should allow for an easier classification of (scientific) articles that apply or describe RegTech in the medical area. This will help the (research) community to find relevant studies quicker, whilst making it easier to identify new research trends.

One of the benefits of MedRegTech research is that it could inform policymakers in an objective and critical manner. For example, the exploration of complexity within the regulations can provide new insights into how regulations can be made more userfriendly. We have showed that complexity can vary between the different medical device regulations [7]. Creating less-complex regulations, without losing the legal context of the regulation, can increase overall adherence and understanding. Moreover, this research also found that there is a need for better metrics in terms of regulatory complexity in general. How we best define complexity within regulations is still an open question. These kinds of studies can therefore provide starting points in the debate of determining an appropriate level of regulatory complexity.

Understanding regulations is essential for medical innovators, as they will need to be able to navigate them. Developing new ways to help people navigate the regulations forms another interesting avenue of exploration. Decision trees that are rule-based can potentially help with this. They offer an approach to the digitisation of the regulation that is logic-based [8,9]. These techniques are not perfect, and a good understanding of the context is still needed in order to apply them correctly. Yet, at the same time, they can also bring to the surface potential issues regarding some of the logic behind these regulations. Mapping the rules using data science techniques can help to consider them more holistically.

The unfamiliarity with the regulations often makes it hard for innovators to engage with them during the early research and development (R&D) stages. Health service providers are particularly well placed to comment on the R&D routes that medical devices take when they enter the clinical setting. It seems that a lot of new medical technology reaches the UK health service provider through non-commercial studies [10]. This is a thought-provoking finding, as a commercial company normally brings these medical technologies into the market. Delays in translation might occur if these non-commercial studies are not or less aware of the regulations. It should also be noted that only a very small number of these clinical studies seem to relate to software as a medical device (defined as a device that is entirely composed of software without any additional hardware). This poses a fascinating question in terms of how fast the field of software as a medical device is really growing. Looking at the number of devices that are registered in Australia, as software as a medical device, we found that there is indeed an upward trend [11]. However, these data from a publicly accessible database also made clear that software as a medical device only made up 1.6% of the total number of registered devices. It indicates that the majority of medical devices that are entering the market in this region are not software-based. These outcomes shed a more quantitative light on how fast stand-alone software with a medical purpose is moving into the market. Much of the research on medical Artificial Intelligence (AI) or Machine Learning (ML) might not yet have translated into a real market entry. This is likely because these methods are still relatively new from a regulatory standpoint, as well as the fact that software poses a different set of safety and performance problems compared to hardware. Nonetheless, it is important to look ahead and see how medical regulations might influence these new developments.

Another obvious topic that is gaining momentum is the environmental impact of healthcare innovation. The environmental impact of regulations on the product life cycle should be researched more thoroughly. Single-use devices and equipment are often selected to prevent pathogen transmission, but this tactic does come at an environmental cost [12]. More recently, the rising dependence on digital health records and information technology is starting to be mentioned in relationship to the environmental impact. Digital solutions might reduce landfill waste, but the energy requirements might create new challenges. These aspects will need to be considered along the more obvious waste management approaches of hardware.

In general, there is a strong need to take a more multi-disciplinary, holistic and datadriven approach in order to tackle the interconnected problems that emerge at the interface of regulations and medical technology. MedRegTech research allows for a critical appraisal of our current situation and could assist in the planning for the future. It can disrupt the regulatory landscape and help push the boundaries of our understanding forward to create better regulations for all.

**Conflicts of Interest:** The author declares no conflict of interest.

#### **References**

## *Article* **The Complexity of Medical Device Regulations Has Increased, as Assessed through Data-Driven Techniques**

**Arthur Arnould, Rita Hendricusdottir and Jeroen Bergmann \***

Natural Interaction Lab, Department of Engineering Science, University of Oxford, Oxford OX1 3PJ, UK<EMAIL_ADDRESS>(A.A<EMAIL_ADDRESS>(R.H.) **\*** Correspondence<EMAIL_ADDRESS>Tel.: +44-1865-273000

**Abstract:** Medical device regulations are dynamic, as they need to cover an ever changing landscape. In Europe this has led to a new set of regulations (both for Medical Devices and In Vitro Diagnostics), which replaced the old rules. This study is interested in how the complexity of these medical regulations changed over time and if additional time-based metrics can be associated with any of the complexity metrics. Complexity is defined in terms of readability of the text and it is computed using established linguistic measures, as well as Halstead complexity scores. It was shown that the regulatory complexity of new EU medical device regulations was higher than their predecessors, especially when Halstead complexity measures were considered. The complexity metrics obtained for the new regulations were subsequently associated with the time it took to consider these regulations. Only very weak Pearson's correlation coefficients were found between the complexity scores and the obtained response times for the new regulations. This could indicate that there are issues with how complexity is perceived by those that need to apply these regulations. Taking the complexity of regulations into account can greatly help with the development of more user friendly regulations. The results from the data-driven methods that are applied in this research indicate that governments could benefit from focusing on making regulations more accessible and utilitarian. This would improve the stakeholder adherence and facilitate effective implementation. This work also highlighted the need to develop more suitable methods to analyse regulatory text to further inform the wider research community.

**Keywords:** data science; regulations; law; medical devices; regulatory data science; natural language processing; linguistic analysis; optimisation

#### **1. Introduction**

The medical device industry has been home to some of the most revolutionary innovations by mankind. The evolution in this field has been a cornerstone of the advances in global health. The term medical device itself encompasses a vast array of products that are used for diagnosing, treating and assisting patients. All aiming to improve the quality of life of patients. There is a great demand for new medical devices due to the aging population [1] and it is a growing global market, which was valued at \$425.5 Billion in 2018 [2]. The United Kingdom alone was estimated in 2015 to have 4060 medical device manufacturers [3] and globally there are many small and medium enterprises operating in this domain. All these medical device companies have to engage with the medical device regulations if they want to bring their ideas to market.

#### *Medical Device Regulations*

Regulations are primarily designed to protect the patients, with regulatory bodies ensuring that medical devices are safe and that they perform as intended [4], but the regulations are also in place to protect the manufacturers themselves [5]. In some circumstances, patients may misuse devices or ignore instructions, injuring themselves in the process.

**Citation:** Arnould, A.; Hendricusdottir, R.; Bergmann, J. The Complexity of Medical Device Regulations Has Increased, as Assessed through Data-Driven Techniques. *Prosthesis* **2021**, *3*, 314–330. https://doi.org/10.3390/ prosthesis3040029

Academic Editor: Marco Cicciu

Received: 31 August 2021 Accepted: 24 September 2021 Published: 28 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Compliance with the regulations acts as a safety net for the manufacturers in the case of a legal dispute. Manufacturers and designers can also use regulation to their advantage and innovate more efficiently, as they can follow clearer guidelines which in turn saves time and reduces some of the uncertainty regarding the process. Additionally, regulations enforce that devices offer a clinical benefit, which prevents markets from being flooded with devices that provide no gains for the user [6]. Finally, regulations are also intended to enhance post-market surveillance to ensure that devices have good longevity and prevents faulty devices from remaining in circulation [7].

While medical device regulations have been implemented for many years, much like other legislation, they do vary between different countries and continents. A major challenge faced by regulators is to ensure that the regulations remain relevant and fit for purpose. This is made more difficult by the sheer rate of innovation, as well as the ever changing needs within healthcare [8]. Consequently, regulations could become outdated and thus compromise patient safety. A prime example of this is the emergence of medical device software [9] and medical devices which have solely cosmetic purposes. These can range from coloured contact lenses to instruments used in cosmetic surgery for liposuction or tattoo removal. It was in fact a scandal related to leaky silicone breast implants from French company Poly Implant Prothèse that highlighted the short-comings in the Medical Device Directive which covered the regulations at that time [10]. Another well-documented example of where regulations did not adequately prevented harm was the discovery of blood poisoning in patients using metal-on-metal hip implants [11]. These kind of events led to an overhaul of the regulatory system in 2017 in Europe. Two new EU legislations were brought into force. Firstly the Medical Device Regulations (MDR) [12] replaced the Medical Devices Directive and Active Implantable Medical Devices Directive (MDD) [13,14]. Whilst, the In Vitro Diagnostic Regulations (IVDR) [15] replaced the EU [16,17]. For these new legislations, manufacturers had until May 2021 and May 2022 respectively to update their technical documents in order to conform with the new requirements. This clearly shows how manufacturers or suppliers of medical devices must continuously adhere to new regulations by ensuring that the correct device classification is achieved, general safety and performance requirements are met and conditions for clinical evidence are met. Manufacturers are responsible for regulatory compliance and that the specific requirements for importers, as well as distributors are respectively met where relevant [18].

#### **2. Background**

#### *2.1. Complexity of Regulations*

There are clear arguments for more stringent regulations in terms of patient safety, yet this form of legislation can only be effective if it is well understood and properly implemented. The MDR and IVDR are 175 and 157 page-long documents. They take the form of complex legal documents filled with lots of jargon. They can be difficult to navigate with the aim of locating specific information and subsequently the information itself can be hard to interpret once it has been found. Many medical devices, especially at the point of conceptualisation, are designed and manufactured by small or medium companies. The small nature of many companies means that their teams are often compact, with few employees. This means that they can struggle to secure the resources necessary to have sufficient legal competencies to navigate these regulations. Most of the innovators are academics or entrepreneurs with no or limited legal training or knowledge. Not understanding the regulations can cause misclassification and innovators might incorrectly interpret the regulations. This can have a compounding effect that leads to expensive redesign and/or retesting of certain devices. Not only does this hamper and discourage innovation, but companies could inadvertently develop faulty or inadequate devices. This raises concerns about patient safety, whilst simultaneously presenting risks to the manufacturers who have legal liabilities and also face significant losses of time and money. These outcomes are undesirable for all stakeholders and so it is vital to facilitate proper understanding of the

regulations by medical device companies and innovators. The complexity of the legal text will have a direct effect on the ability of the reader to understand it.

#### *2.2. Complexity of Text*

The difficulty presented by a particular piece of text can be attributed to its linguistic complexity which is a measure of the extent to which the type of language used makes communication more or less complicated. Though the complexity of text is inherently subjective, for research purposes and large scale projects, it is important to utilise objective metrics to gain a better understanding of the problems innovators face.

Natural Language Processing (NLP) is a branch of computer science, combined with linguistics and artificial intelligence to interpret, analyse and process human text and speech across many languages. Recent progress in the management of unstructured data, the type generated from conversations or human written text, has equipped machines to understand language in a cognitive way which facilitates the identification of particular nuances and features of language [19]. The automatic manipulation of natural language by software is a field which has benefited from a data-driven approach.

The use of NLP methods enables the evaluation of text to return a quantitative score for its complexity and is particularly useful for treating vast quantities of data efficiently. These techniques range from numerical methods that focus on text length and sentence structure, to readability metrics which estimate the number of years of education required to easily read the text and finally, more evolved complexity methods that investigate the types of words and language used in a text to determine the overall complexity. Each of these methods are used in a range of fields, but not all are applicable to legal texts.

Intuitively, it can be assumed that the longer a piece of text is, the bigger of a challenge it will present and therefore it will require more time to read and process. This line of thinking generates numerical techniques which count the words, characters or syllables and only focus on the overall length as a predictor of expected time taken. However, the most common method for assessing text complexity originates from techniques focused around the expected level of education required to read the text. Traditionally, text complexity has been equated to its readability and this is what a series of methods from the 20th century aim to measure.

#### 2.2.1. Common Readability Metrics

The Dale–Chall Readability formula was initially developed in 1948 under the name of "A Formula for Predicting Readability" [20], but has since been updated to reflect the changes undergone by language use [21]. The formula is based around a list of common words which are deemed "not difficult"; this list was initially 973 words long but has since been expanded to 3000 words. The formula combines the average length of sentences, in terms of the number of words, with the percentage of words which are not present on the list (these are known as difficult words). The method returns a grade estimate ranging from "Grade 4 and below" to "Grades 16 and above" which are equivalent to college graduates. The theory behind the use of familiar words as a metric, as opposed to letter or syllable count, is that tests have shown that readers typically find it easier to read, process and recall a passage if it is made up of familiar words [22,23]. However, this method has been criticised for failing to account for more complex structural relations within a text [24].

The Flesch-Kincaid Reading Ease formula follows a similar concept of determining the age of students who should be able to easily read a text [25]. This is achieved by attributing a score of 0–100 to the text; the higher the score, the easier the text. These scores are then associated with grades; scores above 90 indicate that the text is very easy and should be easily read by an average 5th grade student. The Flesch-Kincaid formula has become the chosen readability metric of many US Government Agencies such as the US Department of Defense. Also developed in 1948, the formula uses the ratios of words per sentence and syllables per word to calculate the score [26].

Another metric that assesses the U.S. grade level required to read sections of text is the Automated Readability Index (ARI) [27]. The index was designed in 1967 for real time readability on typewriters used by the military [28]. The origins of the formula dates back to the writing of manuals in the U.S. Navy, as the manuals used previously were written in a style which was above the reading capabilities of most of the staff. The ARI was validated as being more reliable and better suited to the technical nature of the text than other formulae such as Flesch-Kincaid [29]. Similarly, the ARI considers the words per sentence ratio, but it also includes the number of characters per word. This outputs a score that corresponds to the grade level required, but it exceeds the actual number of grades and reaches 14 (with this score corresponding to being above the level of a college student, for example a professor).

The Coleman–Liau index is the youngest of these formulae and was developed in 1975 to assess the readability of textbooks used in U.S. public schools [30]. Its creators deemed that counting syllables was too time-consuming and lacked accuracy. Therefore the Coleman–Liau Index uses the average number of letters per hundred words and also the number of sentences per hundred words [31]. This does have the drawback that it means that for shorter texts, these figures need to be extrapolated and thus may not be as representative. Once again, the numerical output is the estimated grade level required to read the text.

Whilst the Dale–Chal considers the familiarity of words, the other methods almost exclusively inspect the text as a collection of characters without considering the meaning of each word. Sentiment analysis is an area of machine learning which has seen exponentially more use in recent years. It aims to identify and extract subjective information from text which allows it to determine whether there is a positive, neutral or negative sentiment [32]. This process of perception is unconscious cognition within humans but until very recently was impossible for machines to achieve. Sentiment analysis techniques are incredibly powerful and are now being leveraged in fields such as market research, customer interactions and the analysis of social media activity [33]. Its entire premise is using computational linguistics to extract subjective meaning and information from text. In the regulatory context, all text is written in an objective manner to clearly outline regulations that must be adhered to and so sentiment analysis is redundant. Similarly, work done on phonetic analysis cannot be applied to this area either as it assesses speech rather than written text.

With this in mind, when exploring methods from other fields, it is important to consider the type of language that is used. Consequently, attention should be focused on similarly technical fields. The field of financial regulation was found to offer a wealth of previous work. This is largely due to an overhaul of financial regulation following the financial crisis of 2008. This increasingly stringent regulation has drawn attention, with a number of research papers investigating the change in complexity such as those by Gai et al. [34], Colliard & Georg [35] and Spatt [36]. Parallels can be drawn between this situation and the change in medical device regulation, therefore, the techniques should be transferable to a certain extent.

Historically, it was widely accepted in the field of financial research that the Gunning Fog Index was most suitable to measure the readability of documents and it was therefore almost universally used. First published in 1952 [37], the Gunning Fog Index is a metric that generates a grade level from 0 to 20 to indicate the level of education required to read the text, much like the other readability methods described earlier [38]. The formula combines the total number of words, number of sentences and also the number of complex words. Complex words are considered to be those consisting of three or more syllables, excluding common suffixes such as -es, -ed, or -ing. This list also excludes proper nouns, familiar jargon or compound words. A number of studies use the Gunning Fog Index, showing its popularity [39,40].

Despite being widely adopted, the method is not perfectly suited to for example financial text and this was highlighted by Loughran and Mcdonald [41]. At one point the Securities and Exchange Commission (SEC) considered using the Gunning Fog Index to gauge filings' compliance with the SEC's plain English initiatives, however, it was argued that it is ill-suited to analysing financial text, which inherently contains many longer words, despite these being well understood by analysts. Instead it was proposed that a focus on financial terminology and vocabulary that appears in a glossary and master dictionary is more pertinent to assess the readability. Though its efficacy may be questioned in a financial context, the Gunning Fog Index could be an interesting method to apply with the regulatory medical data field.

Whilst text containing intricate ways of describing concepts and elaborate language may once have been highly regarded and considered well written, there is now an ever growing desire for simplicity and effective communication in all fields. A prominent example of this is the Plain English Movement and other campaigns to limit the use of superfluous language and make technical text more accessible for everyone [42]. In some cases, these campaigns have published guidelines for individuals to refer to when drafting text to ensure that it is made as rudimentary as possible. Moreover, this drive for simple language extends further to regulatory agencies such as the SEC who provide very specific guidance in recommending that managers employ plain English attributes, by avoiding writing constructs like passive voice, weak or hidden verbs, superfluous words, legal and financial jargon, numerous defined terms, abstract words, unnecessary details, lengthy sentences, and unreadable design and layout in their financial disclosures [43]. The notion that the absence of such constructs makes text less complex (in a variety of fields that include medical, legal and military), is supported by many language experts [44]. It is with these considerations in mind that S.B. Bonsall IV et al. introduced a new readability metric by the name of the Bog Index [45]. The Bog Index aims to implement the concepts discussed above and one of its features is the way in which complexity is determined. The word complexity is derived from the principle of familiarity which is based on a proprietary list of over 200,000 words. This is in contrast to other techniques which assume that words are complex if they are multi-syllabic or contain many characters. Note that this is the primary criticism of the Gunning Fog Index from many language experts. The fundamentals of the Bog Index evaluate complexity using the trade off between Bog and Pep characteristics. Bog characteristics, as the name describes, bog the user down in unnecessary complexity such as jargon. Conversely, Pep identifies writing attributes that facilitate the understanding of texts by readers. The lower the Bog score, the easier the text is to read. 0–20 is considered excellent, most business and government writing scores 60–100 but some legal texts score over 1000 [46].

Colliard and Georg [35] also aimed to quantify the complexity of financial regulation by methods other than the mere length. In their work they attempted to achieve this by treating the regulation as an algorithm, using concepts from computer science literature to consider the rules for how an input leads to an output (the regulatory decision). The concept of operators and operands is the core feature of this analogy. This approach to complexity was pioneered by Maurice Howard Halstead [47]. The principle is that by segmenting a computer programme into its constituent parts, the relationships between these entities can be used to measure the algorithmic complexity. The two classes are known as the operators and operands and this logic is applied to financial regulation by [35]. Several techniques translate financial information such as balance sheets into pseudo-code to implement the algorithms developed. These methods will not be considered here due to the discrepancies in the format of the regulations. Instead, focus will be placed on methods designed for treating text. Words can be classified according to their function as either operands or operators, using the classification system proposed by Colliard and Georg. The focus is then to translate the algorithmic complexity from code to text-based analysis. The operators are words such as "and" or "excluding" which serve as logical connectors within an algorithm or, in this case, a sentence. "Operands" on the other hand are variables and parameters represented by values (e.g., "seven years" and "10 days"), concepts (e.g., "maturity" and "expiry") or entities (e.g., "manufacturers" and "council"). Words used for grammatical reasons (e.g., "by", "on", "the") can be ignored as they don't correspond

to either operators or operands. Instead they are classed as function words which serve to ensure coherence of the text. The concept of unique operators and operands refers to words from those categories that have not been previously used in the text up to that point. A series of formulae relating the quantities of operators, operands, unique operators and unique operands were first developed by Halstead, but have since been tailored and added to by other papers including research by David Flater [48]. A combination of these equations can be used to assess the complexity of the regulatory text.

#### 2.2.2. Complexity and Response Time

Time driven methods can provide a further insight into the complexity of the regulations in addition to metrics that are obtained directly from the available text. Text length is often positively associated with overall complexity and therefore response time. This theory also translates to question length, with longer questions requiring more cognitive resources and therefore being more likely to interfere with the mapping process. This suggests that question length is likely to be positively associated with the prevalence of both comprehension and mapping difficulties. However, if a question is long because the author has taken care to explain its intent fully, then comprehension will actually improve [49]. The medical device regulations can be posed as a set of questions [50], which allows for further exploration of this metric. One can then even incorporate a suggested minimum of 3 s that is considered needed to perform cognitive tasks and formulate responses to questions [51]. Applying this to the context of questions can provide a sense of expected response times once the reading time and answer selection time is accounted for.

Time-based data for medical regulations can be obtained from a rule-based classifier that is available online [52]. The questions have been ordered using a rule-based decision tree which leads the user through the nodes within the tree, ensuring that users only encounter questions that will aid the classification of their device, based on their previous responses. The technical text is made more accessible through the use of glossaries and examples to contextualise the information. The terms contained within the glossary are those which are defined individually in the definitions sections of the MDR and IVDR regulations. To accurately represent the classification guidelines, phrases of each question posed in the digital tool retain wording from the regulations, which provides a clear mapping back to the regulatory content published by the EU. Each user's interaction with the tool is recorded, which creates a unique database (that will be referred to as OGGD in this paper). Due to the importance of speed and simplicity in the classification process, the time taken to respond to each rule in the decision tree is used as a metric for the regulatory burden that each question places on the user.

#### 2.2.3. Research Aims

The first aim of this paper is to explore how the complexity of the medical device regulations has changed when the MDD/IVDD was replaced with the new MDR/IVDR. The aforementioned complexity metrics can be applied to objectively assess this and determine to what extend regulations within the EU might have increased in complexity. Secondly, an association between the time it takes to consider parts of the regulation and the complexity of these parts will be investigated. This would provide an idea on how the complexity of regulations maps onto the user experience. It also provides an additional metric with time itself acting as an surrugate for complexity.

#### **3. Materials and Methods**

#### *3.1. Linguistic Complexity of Regulatory Documentation*

A descriptive analysis will be performed on the available textual data. This will include the total number of pages, letters, words and syllables. In addition, the average word length will be determined for each relevant document.

Subsequently, a set of readability metric scores will be computed consisting of the Dale–Chall Readability score, Automated Readability Index (ARI), Coleman Liau Index, Gunning Fog, Flesch Kincade Grade and Bog Index. The formulae for these can be found in the Table 1. In the case of this paper, the output from each formula will be retained in its raw form for analysis as it is the variation of scores between questions that is of interest rather than the variation of estimates across individual techniques.

**Table 1.** Readability metrics. WC = Word Count, DWC = Difficult or Complex Word Count, SC = Sentence Count, SyC = Syllable Count, CC = Character Count.

The Bog Index will also be determined, including contrasts Bog and Pep features which ultimately determine the Bog score for the text [45]. The StyleWriter software [46] is used to process the text in order to obtain Bog metrics.

$$\text{Bog Index Formula} = \text{Sentence Bog} + \text{Word Bog} - \text{Pep} \tag{6}$$

$$Sentence\text{ }Bog = \frac{(average\text{ }sentence\text{ }length)^2}{\text{long }sentence\text{ }limit} \tag{7}$$

$$\text{Word } Bag = \frac{(style\text{ } problems + heavy\text{ }words + alkyl\text{ }widths\text{ }words) \times 250}{number\text{ }of\text{ }words} \tag{8}$$

$$Pep = \frac{(names + interest\,\,words + conversation\,\,al) \times 25}{number\,\,of\,\,words} + \text{sentence\,\,vortic} \tag{9}$$

For the Halstead-based methods, the first step was to classify words as either operators or operands. However, inferring the class of the elements of medical device regulations is novel and there is no preceding literature which can be used as a benchmark. Online scientific glossaries were used alongside the financial classification lists published by [35] and some case-by-case discretion to compile two distinct dictionary lists of words. Combined, these contained over 100,000 words, primarily medical in nature, which were utilised to assign classes. Words that were not contained within either the list of operators or that of operands were assigned to the "other words" category. These contained primarily function words that are included to make the text readable and coherent. The count of unique operators *n*1, total operators *N*1, unique operands *n*<sup>2</sup> and total operands *N*<sup>2</sup> formed the basis of the different approaches to quantifying complexity in the text (defined as "programme" for this approach). The metrics for complexity begin in a similar vein to the numerical techniques by associating psychological complexity to the length of the parts of the algorithm. Programme length, vocabulary size and programme volume, which actually measures the length of the binary encoding of the programme in software, all fall into this category. The remaining methods are in form of the ratio of total operators to total operand and the two programme level constructs, which take into consideration the most

efficient expression of the programme by considering unique words. The equations for these methods are outlined below.

$$\text{Programme Length, } N = N\_1 + N\_2 \tag{10}$$

$$Vocabulary\ Size,\ n = n\_1 + n\_2\tag{11}$$

$$\text{Programme Level, } L = \frac{(N\_1 + N\_2)}{(2 + n\_2)} \tag{12}$$

$$\text{Surrogate Programme Level, } \hat{L} = \frac{(n\_1 \times N\_2)}{(2 \times n\_2)} \tag{13}$$

$$\text{Classification Ratio, } \text{C} = \frac{N\_1}{N\_2} \tag{14}$$

$$\text{Proχramme Volumec, } V = N \log\_2 n \tag{15}$$

#### *3.2. Data*

Complexity of the text was determined for the European Union regulatory documentation that consisted of the new Medical Device Regulations (MDR) [12]/the In Vitro Diagnostic Regulations (IVDR) [15] and the older Medical Devices Directive (MDD) [13,14]/In Vitro Diagnostics Directive (IVDD) [16,17].

The time data was obtained from a online tool that mapped the MDR and IVDR on to a rule-based classifier [52]. Complexity of the text was determined for each node of the classifier. These nodes consisted of questions that cover specific parts of the regulations. The response time reflected the time it took to consider the relevant regulatory text and answer the question accordingly. This response data was generated using timestamps from users' interactions with the online regulatory text [52]. Upon submitting a response to a question on the corresponding web page, the user is then brought to a new web page corresponding to the next question to be answered according to the rule-based decision tree nodes. This submission generates an entry into a database which records the question number, the number of questions answered up to that point in the session, the answer submitted, the user's unique identification (ID) and the session ID. The time of answer submission is recorded as a timestamp and it is the difference between the timestamps for two successive questions that is used to calculate the time spent answering a particular question. In total the data set contains information for 903 unique user sessions, covering 112 questions. Data anonymization was performed, so there was no identifiable information about the users that generated the data.

#### *3.3. Data Processing and Analysis*

All readability metrics of the text were divided by taking the median of the full data set for a particular metric in order to aid the visual comparison. The same processing took place for both the data obtained from the legal documentation, as well as the online tool.

All outliers for the time data, in the form of response times less than 0.1 s in length and those greater than an hour, were removed. A total of 17,903 response data points were remaining. These were generated from 903 unique user session IDs, with a mean number of questions per session of 19.8. They covered responses to 112 nodes from the online tool. The time data had skewed distribution and thus the median was used for the analysis of the response times which were calculated for each node in the classifier. The readability metrics are computed for each of the 112 questions that are included in the online tool and represent the text in both the IVDR and MDR.

Pearson's Correlation Coefficients were computed between response times and the aforementioned complexity metrics. Data processing and analysis were done using Python (3.8.0, Python Software Foundation, Wilmington, DE, USA).

#### **4. Results**

*4.1. Linguistic Complexity of Regulatory Documentation*

The overall count of the MDR/IVDR compared to the MDD/IVDD has increased according to every metric, with some increase in the average word length as well (Figure 1).

**Figure 1.** Descriptive complexity scores for the MDR, MDD, IVDR and IVDD. The word length represents the average word length.

The outcomes of the readability scores applied to the overall documents are shown in Figure 2.

**Figure 2.** Readability scores for the MDR, MDD, IVDR and IVDD. Outcomes consist of the Dale–Chall Readability score, Automated Readability Index (ARI), Coleman Liau Index, Gunning Fog, Flesch Kincade Grade and Bog Index.

The Halstead-based complexity analysis is shown in Figure 3. All the metrics increased when new regulations (MDR/IVDR) were compared to the previous legislation.

**Figure 3.** Halstead-based metric scores for the MDR, MDD, IVDR and IVDD. The following metrics are used: Total operators (*N*1), unique operators count (*n*1), total operands (*N*2) and unique operands count (*n*2). Other metrics consist of the length, vocabulary size, classification ratio (*N*1/*N*2), level, surrogate level and volume of the "programme".

#### *4.2. Response Time and Linguistic Complexity*

Having removed the outliers, in the form of response times less than 0.1 s in length and those greater than an hour, there were 17,903 response times remaining. The majority (96.7%) of question/node response times were within a minute, with the median being 7 s for the overall dataset. Figure 4 shows a box and whisker plot for all of the nodes.

**Figure 4.** Box and whisker plot for the response times (vertical axis) of individual nodes (horizontal axis). Any data point which is greater than 1.5 times the interquartile range away from either the upper or lower quartile is denoted an outlier and marked by a circle. The data is displayed on a log axis for response time.

Figure 5 shows the median response time for each question plotted as a bar graph which is overlaid by the normalised values for three numerical complexity methods: Word length, letter count and word count. The variation in mean word length is smaller than the differences in overall question text length.

**Figure 5.** Normalised numerical complexity metrics and response times (vertical axis) of individual nodes (horizontal axis). The mean word length and median response times are shown. The central tendency metrics are selected based on the distribution of the data.

The readability scores for all 112 questions is shown in Figure 6. The histograms are overlaid on top of one another and there is a range of spread between the distribution of scores.

**Figure 6.** Frequency distributions for the readability metrics across all 112 nodes/questions of the digital tool.

The correlation coefficients for each of the readability methods are tabulated in Table 2. The Dale–Chall has the lowest absolute Pearson's correlation coefficient, whilst the strongest association is found between the word count and response time.

**Table 2.** Pearson's correlation coefficients for readability metrics and response times.

Four Halstead-based complexity metrics for each question are plotted with the response times overlaid upon it as a bar graph (see Figure 7).

**Figure 7.** Normalised Halstead-based complexity metrics and response times (vertical axis) of individual nodes (horizontal axis).

The Pearson's correlation coefficients for Halstead-based complexity metrics are shown in Table 3. The lowest correlation coefficient was found for operand count and the highest was for the classification ratio. However, there was only a difference of 0.035 between these coefficients.

**Table 3.** Pearson's correlation coefficients (p-values) for Halstead-based complexity metrics and the response times.

#### **5. Discussion**

In general, the majority of the complexity scores indicate that complexity increased with the implementation of the new legislation. The most important contributor to this seems to be the vast increase in length of the documents. This has undoubtedly made the regulations more difficult to negotiate for users.

#### *5.1. Linguistic Complexity of Regulatory Documentation*

The most complex words found in the regulations are in the form of technical medical references. While these words may be hard to understand, they are not particularly long. This is different to what is found for words in a non-scientific context where word length is indicative of complexity [53]. The findings of this study suggest that the conventional readability metrics may need to be further improved for use within a regulatory context and it echos the limitations summarised by Redish [54]. The techniques that were applied are actually used in many fields as the default measures of complexity. However, they are likely to fall short in similar ways as seen in this paper.

The Dale–Chall method, based on a set list of common words, did not yield a strong association with response time. This is understandable as the list of words was not designed with either scientific or technical objectives in mind and thus it would be constrained for this task. Perhaps a similar list, which had been tailored for for regulations could perform better. The Gunning Fog method also looks at a ratio of complex words and raises similar concerns as also seen in technical financial fields [45]. The Coleman Liau Index was designed to average metrics over 100 words, but several regulatory segments that need to be considered consisted of regulatory "questions" that were considerably shorter than this. Several approaches also heavily relied on the average word length, which might not capture complexity well in regulations. The ARI is the only method that was designed with technical content in mind, which should make it more suitable. However, it does not fully cover the regulatory context. Finally, there is the Bog Index which is far more recent and uses a more developed method to assess the complexity. This method gives importance to style and Pep, which is the way in which the language can serve the interest of the reader. Neither of these are considerations for writing regulation. Consequently, it was hard to differentiate between many of the regulatory text in this regard, which is reflected in the histograms of Bog Index scores. It should be noted that more sophisticated metrics need to be developed to capture the regulatory complexity more accurately. These metrics provide an initial assessment, but are limited in terms of measuring the nuances that are present within the regulatory text. Despite these limitations the metrics due seems to indicate a similar trend in terms of the complexity increase seen in the new regulations.

#### *5.2. Response Time*

The box plot with the individual nodes demonstrates how the spread of samples for each question varied (see Figure 4). These differences would be amplified without applying a log scale for the response time. Certain nodes have very tightly packed response times while others see a vast spread of times. This could be explained by the nature of each question. Some questions could be perceived as easy (or difficult) by most users and thus their response times will be very similar. However, other questions which are particularly technical or specific to a certain field may polarise the user cohort with some users having significant difficulties, which can explain the variation in ranges for response times. It is hard to account for this and perform robust corrections on the data. Yet, increasing the size of the data set could cover a more representative set of users.

The current data set consisted of over 17,000 unique response times. However, a larger data set can help further increase the external validity of the research findings. Furthermore, no in-depth information was gathered with regards to the response times that were generated, due to the fact that data was anonymised. Future studies can aim to collect information on the expertise of the user, device type considered and confidence of answering a particular question. This could create a better model, which can help to explain the response times that were observed.

#### *5.3. Response Time and Linguistic Complexity*

Only a (very) weak relationship was found between the response times and the complexity of the questions. This could be due to the lack of variation in the complexity measures. However, there are other factors that can also influence the strength of the association. Information was missing with regards to the specific intent of the device considered for the questions, the user's knowledge on regulations and a reliance on the outcome of the classifier. Therefore, the strength of these associations should not be considered to generalise easily. Response time data under more controlled settings can yield important additional information that can be used to build a more robust model to determine if there is not a stronger link between complexity and time. It should also be noted that the time spend on the questions might not be fully representative of the overall time spend on considering it. These are limitations that can be addressed in future studies.

#### *5.4. Considerations*

The techniques presented here offer a starting point to better understand the complexity of regulations. As shown there is no clear association between the time spent on a regulatory question and the associated complexity. However, it should be noted time only focuses the (instant) answer of the classification questions and therefore just captures the first steps in the regulatory process. More work can be done to objectively study the time consideration across the full application of the regulatory text. Using more data-driven methods can greatly increase our understanding of the regulations and allows to generate better questions that can help improve future regulations [9].

The complexity of a legal text can form a barrier to innovators. Understanding the complexity can therefore be essential in optimising the pathway for new devices. The increased complexity found in this study highlights the importance of improving education and guidance. Previous research already showed that more can be done to provide further support in terms of education [55] and this should be considered by stakeholders as complexity of regulations increases. Regulations that are currently being created can benefit from considering these aspects during the development process. These preliminary findings from this paper propose that the complexity of regulations should be reduced, with a focus on making them more accessible and utilitarian. This would improve the stakeholder adherence and facilitate effective implementation, which in the long term will improve patient welfare.

**Author Contributions:** Conceptualization, R.H. and J.B.; methodology, A.A. and J.B.; software, A.A. and J.B.; formal analysis, A.A.; investigation, A.A., R.H. and J.B.; resources, R.H. and J.B.; data curation, R.H. and J.B.; writing—original draft preparation, A.A. and J.B.; writing—review and editing, A.A., R.H. and J.B.; visualization, A.A.; project administration, R.H. and J.B.; funding acquisition, J.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the European Institute of Innovation and Technology (EIT) Health and supported by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC).

**Institutional Review Board Statement:** The study was approved by Central University Research Ethics Committee (CUREC): R63968/RE001.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** All data on complexity were obtained from public documents and references are provided in the manuscript whenever they are discussed.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

### *Article* **Using Rule-Based Decision Trees to Digitize Legislation**

**Henry R. F. Mingay, Rita Hendricusdottir, Aaron Ceross and Jeroen H. M. Bergmann \***

Department of Engineering Science, University of Oxford, Parks Road, Oxford OX1 3PJ, UK<EMAIL_ADDRESS>(H.R.F.M<EMAIL_ADDRESS>(R.H<EMAIL_ADDRESS>(A.C.)

**\*** Correspondence<EMAIL_ADDRESS>
**Abstract:** This article introduces a novel approach to digitize legislation using rule based-decision trees (RBDTs). As regulation is one of the major barriers to innovation, novel methods for helping stakeholders better understand, and conform to, legislation are becoming increasingly important. Newly introduced medical device regulation has resulted in an increased complexity of regulatory strategy for manufacturers, and the pressure on notified body resources to support this process is making this an increasing concern in industry. This paper explores a real-world classification problem that arises for medical device manufacturers when they want to be certified according to the In Vitro Diagnostic Regulation (IVDR). A modification to an existing RBDT algorithm is introduced (RBDT-1C) and a case study demonstrates how this method can be applied. The RBDT-1C algorithm is used to design a decision tree to classify IVD devices according to their risk-based classes: Class A, Class B, Class C and Class D. The applied RBDT-1C algorithm demonstrated accurate classification in-line with published ground-truth data. This approach should enable users to better understand the legislation, has informed policy makers about potential areas for future guidance, and allowed for the identification of errors in the regulations that have already been recognized and amended by the European Commission.

**Keywords:** classification; healthcare; innovation; regulation; medical devices; decision tree complexity

**Citation:** Mingay, H.R.F.;

**1. Introduction**

Regulations contain rules setup by (governmental) authorities to control specific aspects of certain industries, which often influences the way companies operate. These rules affect how industries are managed, and the importance of regulations is such that many companies create specific divisions focused solely on the regulatory strategy [1]. Regulatory frameworks encourage consumers to adopt innovations by ensuring that their safety and effectiveness has been evaluated. Regulations promote better utilization of technologies and encourage the identification of novel technologies within industries [2]. When used effectively, regulation drives the direction of innovation [3], and can stimulate innovative ecosystems [4]. However, regulations also create barriers that can hold up the innovative process [5]. For innovative firms, they are one of the most significant barriers of perceived environmental uncertainty [6]. Organizational characteristics can have a significant influence on the effect of the regulation with regards to innovation [7]. SMEs need more information to better identify the relevant regulation and understand the requirements for conformity based on these regulations [8], which are the first steps in the regulatory navigation pathway. A better understanding of legislation can help to alleviate the barrier to innovation that regulation presents. One useful approach to provide a better understanding of complex legislation is by applying engineering methods to represent legislations in a more quantitative manner. Digitizing legislation is proposed as a method to present the rules within legislation in a simpler and logic-based format for the stakeholders that need to conform to the regulation. There are two types of classification problems resulting from legislation: identification of relevant regulation, and classification

Hendricusdottir, R.; Ceross, A.; Bergmann, J.H.M. Using Rule-Based Decision Trees to Digitize Legislation. *Prosthesis* **2022**, *4*, 113–124. https:// doi.org/10.3390/prosthesis4010012

Academic Editor: Marco Cicciu

Received: 8 January 2022 Accepted: 1 March 2022 Published: 10 March 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

of certain use cases for conformity requirements within the legislation. While all products and services need to consider relevant legislation, a smaller subset of legislations contain risk-based classification problems within the legislative document. Examples of this include the classification of data types within data protection regulation (e.g., the General Data Protection Regulation) and the classification of devices within EU medical device regulation (e.g., the Medical Device Regulation and In Vitro Diagnostic Regulation). Legislations containing risk-based classes have different conformity requirements depending on the risk level of the use case in question. This article presents a novel method for using rulebased decision trees to digitize classification problems within legislation, using the In Vitro Diagnostic Regulation (IVDR) as a case study example. Rule-based decision trees were selected as classification is based on the rules within the legislation. It is not dependent on a specific training dataset. This use case will focus on the classification problem within the IVDR legislation, although it is also applicable for determining whether the devices are governed by the IVDR. These classifications problems appear in complex legislation with multiple conformity requirements. Stakeholders require greater understanding of these types of legislation as the conformity requirements are dependent on the risk level of their product or service.

#### **2. Digitizing Legislation**

Early attempts to create digital tools for classification problems within legislation were constrained by challenges arising from differences between legal and technical semantics. Where engineering lexicon is based on precision [9], legal language is characterized by its construction to remain as generalized as possible to the class of problems that the legislator is addressing [10]. Legal requirements present multiple problems for managing compliance including ambiguities, cross-referencing, acronyms, domain-specific definitions, and frequent amendments due to revisions of legislation and case law [11]. One of the first attempts to digitize legislation was undertaken by Sergot et al. (1986) [12], when the British Nationality Act was converted into a logic program. They concluded that representation of rule-based legislation using logic programming is optimal, as it is simple for both naive users and experts to understand. This method allowed users to infer meaning from the legislative rules and made modifications of the system straightforward. Translation of laws into a logic program was shown to provide a better understanding of the rules within the legislation and identified specific interpretation issues within legislation [13]. The use of logic programming for representation of legislation was met with a scathing rebuttal by Leith (1986) [14]. The professor of law argued that 'legislation could not be formalized in a truly logical format'. The use of a logic program was criticized for embedding assumptions into the logic program arising from vague concepts. Challenges regarding the interpretation and ambiguity within the legislation were highlighted. Other attempts to digitize legislation consisted of a proof of concept for modelling the Italian data protection legislation created by Massaccia et al. (2005) [15], and an evaluation tool for the compliance of websites to Canada's Personal Information Protection and Electronic Documents Act (PIPEDA) [16].

This article builds on the concept of using decision trees to better understand medical device regulation proposed by Bergmann et al. [17]. It will focus on the use of decision trees for representation of legislation, as they are suited to deal with issues arising from the differences between legal and technical semantics. Where engineers previously embedded assumptions into logic programs, decision trees allow users to directly evaluate the rules within the legislation at the level of attributes along the decision pathway.

#### **3. Decision Trees for Digitizing Legislation**

Bergmann et al. [17] proposed a data-driven methodology using the Iterative Dichotomiser 3 (ID3) decision tree learning methodology, introduced by Quinlan (1985) [18], to classify medical devices according to the rules set out in the Medical Device Directive (MDD). The alternative method to building a decision tree using a data-driven approach is the use of a rule-based approach. Knowledge is stored in the form of a set of rules, which are converted into a decision tree when a decision-making process is required [19]. Rule-based decision trees can be built from either static or dynamically changing decision rules [20]. If classification rules are stored as a set of declarative statements, then there are no constraints on the order in which the attributes can be evaluated, meaning that it is much easier to update the decision tree when new information and data are available. This is preferable for decision trees based on legislation, as there are often amendments added, revisions made in the legal text, and real-world examples published to further clarify ambiguities and issues with interpretation.

Decision trees also have advantages over 'black-box' classification models, including neural networks, as the logical rules in a decision tree are much easier to interpret than the complex relationship between hidden nodes in a neural network for the decision makers [21]. This allows stakeholders to obtain a one-to-one mapping between their product or service and the classification rules in the legislation, often required for conformity assessment. In addition to the ease of interpretability, a decision tree built from a pre-defined set of decision rules allows the classification of examples where there is no available data for training.

An important advantage that rule-based decision trees have over both data-driven decision trees and neural network approaches is the accuracy of the classifications from the legislative text. Classification is directly based on the rules that are set out in the legislation, and the accuracy of the classification is not dependent on training the decision tree using available data.

#### *Overview of Rule Based Decision Tree Algorithms*

The earliest paper identified that introduces the concept of a rule-based decision tree was Imam et al. [19]. The paper introduces a method called the AQDT-1 (AQ-derived Decision Tree-1), which builds a decision tree from decision rules generated using AQ-type inductive learning. The algorithm optimizes the order in which the rules are assessed in the decision tree structure. It is important to note that while this paper uses a set of rules that has been generated using inductive learning, the algorithm can be used to build a decision tree from any set of rules, including those in legislation. Rules at each node of the decision tree will subdivide the examples into smaller groups depending on whether the attribute at the node applies or not. The attributes are derived from the rules and placed at respective nodes along the decision tree. An attribute selection criterion is used to analyze the relationship between the attributes and the classification rules, and the order, in which they are evaluated, is optimized based on multiple selection criteria. This method for optimizing the ordering of the attributes in the decision tree is determined by an attribute utility ranking comprising of three criteria: disjointness, dominance, and extent. These, respectively, measure the effectiveness an attribute has in determining the final decision class for an example, the frequency in which this attribute is present in the rules and the number of values an attribute can take within the rules. The algorithm aims to maximize the disjointness and dominance of an attribute while minimizing its extent. Similar to greedy splitting decision trees, this method creates a large number of leaf nodes in the decision tree and decision rules are often pruned depending on the rule strength.

The AQDT-1 tree, whilst being novel in terms of the idea that it proposed, was far from optimal when evaluating the complexity of the decision trees created. This led to a publication shortly after by Michalski et al. [22], which outlined a refined method for this idea: the AQDT-2 algorithm. This method was shown to outperform the AQDT-1 in terms of both accuracy and decision tree complexity—determined by the number of decision tree leaf nodes. The notion of the cost of evaluating against certain attributes was introduced in this paper. It was deduced that the "inexpensive" attributes should be evaluated first, and thus should be assigned close to the root node. The "expensive" attributes should only be evaluated when necessary, therefore placed further away from the root node. This cost is quantified using an importance score. While this optimization was shown to improve the predictive accuracy and reduce the complexity of the decision tree, it was accompanied by the drawback that to calculate the cost of an attribute, a training data set is required.

The next rule-based approach was created when Abdelhalim et al. (2014) [20] introduced the concept of the Rule Based Decision Tree (RBDT-1) algorithm, which does not depend on a training dataset for optimization. This paper highlighted the fact that rule-based approaches are often used in situations with limited data, and commented that the previously developed AQDT-2 required a training set for optimization. The RBDT-1 algorithm depends on only the rules themselves and uses three parameters for optimization question sequencing: attribute effectiveness, attribute autonomy and minimum value distribution. Attribute effectiveness (AE) is the first of these to be considered. It depends on the influence that the attribute has in determining the class of an example. If an attribute does not contribute to the decision process of assigning a class to an example, then it has a lower AE score than one that does. The attribute with the highest AE score is chosen for the root node and then subsequent attributes are selected depending on the next highest AE score until a leaf node is reached. The attribute autonomy (AA) is considered when two attributes have the same AE score. This criterion selects attributes that create fewer subsequent nodes before the leaf node in an attempt to reduce the depth of the decision. The minimum value distribution (MVD) is only taken into account when two attributes have both the same AE and AA score. It also aims to minimize the complexity of the decision tree by favoring attributes that have fewer values in a given rule. An attribute with fewer values will in turn generate fewer branches, which will result in a smaller and less complex decision tree. Table 1 introduces an example problem, which is a modified version of a decision tree classification problem outlined in the literature [20]. Information is given in Table 1 for a range of different medical device companies, including the expertise level of the staff, size of the company (small <50 employees; medium 50–250 employees; large >250 employees) and investment attracted. The final column shows what level of (medical) risk the product developed by that specific company carries. More stringent regulations govern higher risk devices, which translates into additional costs and time for businesses.

**Table 1.** Device risk dataset used to compare the different decision tree algorithms. The "Expertise" column captures the human capital within a company with regards to medical devices. The "Size" indicates the number of employees within a company. The "Investment" shows how much investment that company had attracted, whilst the "Device risk" shows the risk of the medical devices they develop.

The RBDT-1 is compared to previous rule-based and data-driven decision trees across a variety of decision tree classification problems, and is shown to outperform the AQDT-1, AQDT-2 as well as the ID3 (entropy driven) algorithms in terms of complexity, as shown in Figures 1 and 2. This is exhibited by the smaller number of attribute nodes and leaf nodes generated by the RBDT-1 decision tree. The RBDT-1 algorithm generated three attribute nodes and five leaf nodes, whilst the other algorithms ended up with five attribute nodes and seven leaf nodes.

**Figure 1.** Decision tree generated for the device risk problem presented in Table 1 using the RBDT-1 algorithm [20]. It consists of three attribute nodes and five leaf nodes.

**Figure 2.** Decision tree generated for the device risk problem presented in Table 1 for the AQDT-1, AQDT-2 and ID3 algorithms [20]. It consists of five attribute nodes and seven leaf nodes.

While there is some literature introducing design of rule-based decision trees, there is a gap in the literature regarding the optimization of rule-based decision trees with additional case specific constraints. Currently, rule-based decision tree algorithms require modification to incorporate any additional constraints arising from the legislation. A rule-based decision tree that allows for constraints to be set (RBDT-1C) is introduced in this paper and the In Vitro Diagnostic Regulation (IVDR) is used as a case study example.

#### **4. IVDR Legislation as a Case Study Example of RBDT-1C**

Previous European medical device regulation promoted innovation by favoring faster regulatory approval and patient-access to novel technologies, which resulted in a series of patient outcome scandals—most notably regarding the Poly Implant Prosthesis (PIP). This led to a major overhaul of European medical device regulation in 2017, where the new regulation aims to increase the safety and effectiveness of medical devices by placing more emphasis on clinical evaluation and post market surveillance [23]. The new regulation has reacted to changes in the industry, including a rise in the use of medical device software [24]. However, Arnould et al. [5] have demonstrated that this has led to an increased complexity of Medical Device Regulations (both the MDR and IVDR). There is currently a shortage of resources within organizations responsible for certifying European medical devices due to the introduction of the new legislation [25], political factors [26], and other factors. The case study introduced in this paper is the use of a digital decision tree for classification of medical devices according to the new IVDR legislation. This can help to alleviate the strain for the regulatory stakeholders by providing resources to help medical device manufacturers classify their devices according to the new regulation. The proposed RBDT-1C algorithm was used to design a decision tree for the classification of in vitro diagnostic (IVD) devices into the risk-based classes outlined in the IVDR. The risk-based classes in the IVDR combine both patient and population risk as follows: Class A, low patient and population risk; Class B, moderate to low patient risk and low population risk; Class C, high patient risk and low population risk; and Class D, high patient and population risk. Population risk is important for IVD devices as a false negative (meaning the patient is positive) could endanger others if the disease is transmissible and life-threatening.

Decision tree attributes were created from the classification and implementing rules in Annex VIII of the IVDR [27]. Rules with multiple classification criteria were split into multiple attributes, as prior research has shown that complex questions lead to greater complexity of comprehension [28] and interfere with the mapping process [29], where the mapping process relates to the cognitive process, by which a user simultaneously recalls the question and answers based on the example they are testing. This additional complexity is likely due to the increased demand on the working memory of the user [30]. All complex and lengthy rules can be split into multiple attributes. Where rules contained many subrules, natural language processing (NLP) techniques were used to assess the semantic similarity of the sub-rules to cluster sub-rules into semantically meaningful groups. While long and complex questions are undesirable for a decision tree, excessive splitting of the sub-rules would lead to a deeper and more complex decision tree. Semantically similar subrules do not overcomplicate attributes, meaning multiple sub-rules can be asked at a single node of the decision tree if they relate to similar devices. The semantic properties of the sub-rules were assessed using word embedding methods, which convert words into vectorrepresentations, capturing and displaying their semantic properties in the vector space. The word embeddings for each sub-rule were determined using the pre-trained BioBERT sentence embedding algorithm [31]. BioBERT's pretrained and fine-tuned deep neural network allows it to create vector representations where both the complex characteristics of words such as syntax and semantics, as well as the linguistic context of the word to model polysemy, are considered and reflected in the sentence embedding. BioBERT is also pre-trained on all PubMed data, meaning it can capture the semantic properties of biomedical specific terminology.

#### *4.1. Building the RBDT-1C*

After the attributes were created from the classification and implementing rules in the IVDR, the decision tree was built using a modified version of the RBDT-1 algorithm proposed by Abdelhalim et al. (2014) [20]. The use of rule-based decision trees for the representation of legislation is possible when using the RBDT-1 algorithm, as no data is required to train the decision tree. Additional constraints proposed by legislation can be designed into the decision tree by introducing supplementary criteria to the RBDT-1 algorithm, creating the RBDT-1C methodology—detailed in Supplementary Materials.

For the IVDR, the classification of a device is determined by the rule with the highest risk-based classification. If multiple rules applied to one device, then the rule with the highest risk-based classification determines the class of the device (IVDR, Annex VII, Implementing rule 1.9), where Class D is the highest risk-based classification, and Class A is the lowest. The risk-based classification is based on the personal risk and public health risk associated with an IVD device. The conformity constraint for the classification of devices is that all classification and implementing rules had to be considered when classifying a device (IVDR, Annex VIII, Implementing rule 1.7). This means that even if a higher risk-based rule applies, the lower risk-based rules must be assessed despite not contributing towards the final classification due to the "hierarchy" of classification. The conformity constraint was adapted into the tree using a novel attribute hierarchy criterion (AH). This is defined for each attribute as the number of subsequent attributes that can be excluded owing to the attribute taking a specific value. This is calculated multiple times for each attribute, once for each of the potential values that the attribute can take. These exclusions were then designed into the final decision tree structure, where the ordering of the IVDR attributes was determined by the three criteria within the original RBDT-1 algorithm: attribute effectiveness, attribute autonomy and minimum value distribution.

If devices are not independent medical devices (e.g., calibrators) they are classified according to the risk-level of the 'Parent Device'. The 'Parent Device' is the device that they are used with. The dotted line in Figure 3 represents the choice that the medical device manufacturer is given to proceed to classify the 'Parent Device' when a device does not have an independent classification. Additionally, Implementing Rule 1.4 (Table 2) refers to medical device software, which is only classified in its own right if this is independent of any other medical devices. If the software drives or influences the use of another medical device, then it is classified according to the 'Parent Device' that it is used with.

**Table 2.** IVDR Classification Rules dataset for the IVDR decision tree.

**Figure 3.** IVDR Decision Tree with Rules, classification rules and device risks corresponding to the IVDR Classification Rules and criteria in Table 2. The final classification given to a device consisted of Class A, B, C or D.

#### *4.2. Classification Results from the IVDR Decision Tree, Build Using the RBDT-1C Algorithm*

The IVDR decision tree was tested using 55 example IVD devices with known classifications according to the IVDR. This list of devices, which included devices from all of the classification and implementing rules, was compiled from multiple sources including: Medicines and Healthcare products Regulatory Agency (MHRA) In Vitro Diagnostic Directive (IVDD) guidance document [32]; British Standards Institute (BSI) classification of IVD devices whitepaper and infographic [33,34]; Australian Therapeutic Goods Administration (TGA) classification of IVD devices [35]; Food and Drug Administration (FDA) IVD device examples for human genetic tests and companion diagnostics [36,37]; and an International Society of Blood Transfusion (ISBT) report on human blood grouping systems [38]. Each of the devices was classified using the IVDR decision tree, independently by three experts, which consisted of medical and regulatory specialists. The independent classifications were assigned using the IVDR decision tree without prior knowledge of the known classifications. The results were discussed collaboratively to highlight different interpretations of the rules within Annex VIII of the IVDR and human error within the results. Ethical approval was in place (R63968/RE001) when collecting this data. Classification of 52 of the 55 device examples were agreed by all members of the research team, and these results were compared to the ground truth classifications. The decision tree was shown to accurately classify all IVD devices where there was no ambiguity in interpretation of the rules in the IVDR. Classification was not agreed for three devices, which identified ambiguity and interpretation issues within the classification rules of the IVDR.

The process of design and testing of the RBDT-1C identified difficulties in interpretation of the legislation. These issues resulted from differences between legal and technical semantics, and consisted of: ambiguity in terms used to distinguish between different device classifications; interpretation issues within classification rules; vagueness in healthcare domain specific vocabulary; and a lack of coherence between cross-referenced regulatory texts.

#### **5. Discussion**

The approach of converting rules within legislation into a rule-based decision tree identified specific interpretation issues within the IVDR. These interpretation issues were identified when implemented in January 2020, many of which were independently clarified in guidance documents that were published in November 2020 [39] by the designated European Medical Device Coordination Group (MDCG). This demonstrates that the introduced RBDT-1C algorithm can help legislators identify areas where guidance and further clarification is required. As well as this, digitizing the IVDR legislation identified cross referencing errors in the IVDR (and Medical Device Regulation), which have already been recognized and amended by the European Commission.

The medical device regulation field is highly leveraged on experience, and this process gives knowledge and insight into the regulation that usually takes years to capture, which is especially important for new and recent legislations. It was shown that this kind of algorithm could help IVD device manufacturers conform to, and better understand the IVDR legislation as the logic-based evaluation allows understanding of the interaction between the classification rules within the legislation, and provides insight into how users navigate the complex regulation. This approach can be used for other legislation and is not limited to the medical device industry. The conversion of legislation into a digital decision-making tool can also aid engineering management decisions when considering regulatory strategy within organizations.

It should be noted that these algorithms should keep in line with any rule changes. Outdated representation of the rules within an algorithm can lead to incorrect outcomes. Applying these kinds of algorithms should therefore also include a clear process on how to keep them up-to-date. This is particularly important when these algorithms are integrated within larger digital platforms. The digitization of legal frameworks can also take into account these considerations in order to allow software engineers to build more robust systems.

In general, resources that aid regulatory strategy can increase the understanding of legislation for all stakeholders within an industry. This can help overcome the barrier to innovation, which is often presented by regulation. While this is not limited to the medical device industry, this research example showed that digital solutions can be built to help users navigate the regulations. Nonetheless, this rule-based methodology is limited to legislation with structured and well defined rules, as the attributes need to be clearly defined according to the legislation in order to allow for accurate classification based on a rule-based decision tree. An additional, limitation of this method is the interpretation issues within the legislation, as ambiguity within the legislation resulted in divergent classification when using the IVDR decision tree. However, we do believe that highlighting and identifying these challenges can also lead to further additional clarification and guidance from the legislators. This could in turn result in a better understanding of regulations themselves. The introduced RBDT-1C algorithm provides a valuable tool within the field of regulatory science and engineering management.

**Supplementary Materials:** The following supporting information can be downloaded at: https://www. mdpi.com/article/10.3390/prosthesis4010012/s1, RBDT-1C mathematical supporting documentation.

**Author Contributions:** Conceptualization, J.H.M.B.; methodology, H.R.F.M. and J.H.M.B.; data collection, H.R.F.M. and R.H.; formal analysis, H.R.F.M.; data curation, H.R.F.M. and R.H.; writing original draft preparation, H.R.F.M. and J.H.M.B.; writing—review and editing, H.R.F.M., A.C., R.H. and J.H.M.B.; visualization, H.R.F.M.; supervision, J.H.M.B.; project administration, H.R.F.M., R.H. and J.H.M.B.; funding acquisition, J.H.M.B. and R.H. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the European Institute of Innovation and Technology (EIT) Health [210692, EIT Health Education Portfolio, 2021] and supported by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC) [OxBRC4] and The Engineering and Physical Sciences Research Council (EPSRC) [EP/R511742/1].

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki, and approved by the Medical Sciences Interdivisional Research Ethics Committee (IDREC) of Oxford University (R63968/RE001), 27 June 2019.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors would like to acknowledge Azad Hussain and Eleanor Hughes for classifying example in vitro diagnostic (IVD) devices using the IVDR decision tree, and Oliver Lorton for implementing the designed tree for online testing.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**

## *Communication* **Medical Device Regulation from a Health Service Provider's Perspective**

**Man Ting Kwong 1, David Stell 1,\* and Emmanuel Akinluyi 1,2**

**Abstract:** Unfamiliarity with medical device regulations can sometimes be a barrier to deploying technology in a clinical setting for researchers and innovators. Health service providers recognise that innovation can happen within smaller organisations, where regulatory support may be limited. This article sets out to increase transparency and outline key considerations on medical device regulations from a UK healthcare provider's perspective. The framework used by Guy's and St Thomas' NHS Foundation Trust (GSTFT) for assessing research devices is presented to give an overview of the routes that R&D medical devices take to enter a clinical setting. Furthermore, current trends on research studies involving medical devices were extracted from the GSTFT internal R&D database and presented as the following categories (i) commercial vs. non-commercial, (ii) assessment type and (iii) software vs. non-software. New medical devices legislation will be introduced within the UK in July 2023. It is anticipated regulating software as a medical device may become more challenging for healthcare providers and device manufacturers alike. It is therefore important for different stakeholders involved to work together to ensure this does not become a barrier to innovation.

**Keywords:** medical device regulations; healthcare provider; software as medical device; technology translation; quality management system

#### **1. Introduction**

Innovation can often happen at the level of a research organization or by an early–stage innovator. Development of medical devices may require extensive clinical investigation and evaluation to ensure patient safety. Navigating the path towards clinical investigation and evaluation could pose a barrier to early–stage innovators and hence may impede innovation. The aim of this article is to give early–stage researchers and innovators an overview of the regulatory environment in the UK from a healthcare provider's perspective, and possible routes for R&D medical devices to enter testing in a clinical setting.

In the UK, medical devices are regulated under the Medical Devices Regulations 2002 (commonly referred to as the UK MDR 2002). This legislation implemented existing EU medical devices directives into UK law. The legislation required CE marking for all medical devices sold within the UK, to indicate compliance with EU legislation. The most widely applicable of the EU directives is the Medical Devices Directive (Directive 93/42/EEC, commonly referred to as the MDD) [1]. The legal requirements for medical device manufacturers are largely contained within the MDD, with some UK specific requirements contained within the UK Medical Device Regulations 2002 [2].

The MDD is no longer in force within the EU, having been replaced by the EU Medical Devices Regulation (2017/745, commonly referred to as the MDR) [3], on the 26 May 2021. New medical devices legislation will be introduced within the UK in July 2023, the contents of the new legislation are not known but may have similarities to the EU MDR.

Alongside the MDD, the UK Medical Devices Regulations 2002 implemented two other EU directives into UK law; the Active Implantable Medical Devices Directive

**Citation:** Kwong, M.T.; Stell, D.; Akinluyi, E. Medical Device Regulation from a Health Service Provider's Perspective. *Prosthesis* **2021**, *3*, 261–266. https://doi.org/ 10.3390/prosthesis3030025

Academic Editor: Marco Cicciu

Received: 3 August 2021 Accepted: 8 September 2021 Published: 14 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

(AIMDD, 90/385/EEC) [4] and the In Vitro Diagnostic Medical Devices Directive (IVDD, 98/79/EEC) [5]. Within the UK, the legal requirements for these devices remain largely contained within the EU directives. Within the EU the IVDD remains in force. The AIMDD has now been withdrawn, with the requirements for active implantable medical devices now contained within the MDR.

#### **2. Role as a Gatekeeper for Medical Devices**

Healthcare providers in the UK have a statutory responsibility to ensure patient safety. Most health providers have a process for approving and accepting medical devices for local deployment. Clinical engineers will often act as gatekeepers, assessing and approving new medical devices/devices being used for research. As an example to help improve transparency, the framework used by the Guy's and St Thomas' NHS Foundation Trust (GSTFT) for assessing research devices is presented below.

There are several possible routes for R&D devices to enter a health service, these are broadly categorized into: (i) Commercial R&D, (ii) Non-commercial R&D and (iii) Non-R&D. Each of these categories will be described in more detail in the rest of this section. It is important to note that ethics approval is required for medical device research involving patients.

#### *2.1. Commercial R&D*

The sponsor of commercial R&D studies is typically a commercial body, usually either a medical devices manufacturer or a pharmaceutical company. Clinical engineers are involved in the assessment of any study which involves medical devices, irrespective of who the sponsor is.

Studies run by pharmaceutical companies may involve medical devices. These studies do not typically assess the medical devices themselves; these devices are ancillary, for instance diagnostic devices may be used to assess treatment outcomes. In these cases, the devices used are post-market, with pre-existing regulatory approval. Clinical engineers' involvement in the assessment of these studies is typically limited to checking that devices do carry the necessary regulatory approval (i.e., UKCA or CE marking) and are used within their intended purpose.

Studies run by medical device manufacturers are more likely to assess the performance of medical devices themselves. These studies may involve pre-market or post-market devices. Even studies of pre-market devices typically involve devices whose designs are fairly mature.

Trusts typically charge more for commercial studies than non-commercial ones and retain less ownership of the data produced by the study.

#### *2.2. Non-Commercial R&D*

The sponsor of non-commercial studies is typically a non-commercial body, often the NHS Trust itself, or a university. These studies are very diverse. Non-commercial R&D studies which have a medical device as the subject of the study can involve early– stage technologies whose designs are less mature than those typical of commercial premarket evaluations.

Trusts typically charge less for non-commercial studies than commercial ones but will have stricter contractual limitations governing what data can be shared with the sponsor.

#### *2.3. Non R&D*

In some cases, new medical devices, medical device modifications, or systems of medical devices are not for research purposes, they are to meet an identified clinical need within the Trust. Devices we have seen within GSTFT include fixation devices to minimise patient movement during imaging, and incubator trolleys mounting multiple devices which require mitigations to assure electrical safety.

#### **3. Internal Assessment Considerations**

Clinical engineers involved in the assessment of medical devices for deployment within their local institution must consider several factors. Their ultimate responsibilities are to ensure that any deployment is safe and does not adversely affect the patient care delivered by their organisation. However, the assessment questions are also driven by the regulatory route used for the deployment, and local policies and norms.

Regulatory approval provides a high degree of assurance of device safety and performance. A correspondingly light assessment is therefore required for these devices.

Where a study has been notified to the Medicines and Healthcare products Regulatory Agency (MHRA), and they have indicated that they have no concerns, this too provides a high degree of assurance and reduces the scope of the assessment required from local clinical engineers.

No such assurance exists for medical devices which do not have regulatory approval, and where studies have not been notified to the MHRA. Deployment of these devices is permitted in a limited range of specific circumstances. Assessments of these deployments include detailed technical assessments, together with a consideration of the legal deployment route.

#### **4. Approval and Assessment Type**

Based on the above factors mentioned, the decision pathway to ensure patient safety could be categorised into the following assessment types: (i) Trust owned medical devices, (ii) UKCA/CE marked devices, (ii) registered clinical investigations, (iii) basic safety assessments and (iv) Health Institute Exemption.

#### *4.1. Trust Owned Medical Devices*

Some researchers wish to use medical devices which are already in clinical use at a Trust for their research. Provided that the devices are not being modified or used for a novel purpose, there are no regulatory barriers to this use.

#### *4.2. UKCA/CE Marked Medical Devices*

It is very common for research studies to involve use of medical devices which are UKCA/CE marked and available for purchase. There are no further regulatory requirements to this type of use. However, individual Trusts may have local governance arrangements which determine which medical devices may be introduced to the Trust. New devices may have to meet local governance requirements before they may be introduced to clinical areas.

#### *4.3. Registered Clinical Investigations*

To demonstrate that their device conforms to the regulatory requirements, manufacturers must sometimes run clinical studies known as *clinical investigations*. In many respects these are the medical devices equivalent of a clinical trial of an investigation medicinal product (CTIMP). Within the UK, the MHRA must be notified of any clinical investigation and have the opportunity to object before it may proceed.

Clinical investigations are typically run as commercial R&D projects within NHS Trusts. Once the MHRA have indicated that they do not object, Trusts are usually happy for the device to be used with their patients, subject to the usual R&D governance requirements.

This is a common but resource intensive route for devices to enter a NHS Trust, the manufacturer may act as the sponsor of the clinical investigation. This will commonly involve dedicated regulatory experts and clinical trials units. Securing funding for this step is a common barrier for devices undergoing the innovation translation process.

#### *4.4. Basic Safety Assessments*

In some cases it is possible for an investigational medical device to be used within a clinical study without being used for any medical purpose. For example, a diagnostic device may be used to make clinical measurements which are used for research purposes (e.g., for comparison with measurements made by a gold standard technology) but are not used for any clinical purpose (i.e., they are not used by clinicians to make decisions about patients' care).

Existing medical devices legislation applies only to devices intended by their manufacturer to be used for medical purposes. Investigational devices which are not intended to be used in this way do not need to comply with these legislative requirements. Healthcare providers are nonetheless required to ensure that such devices are safe for use with patients and will often wish to perform a "basic safety assessment" for these devices.

Clinical medical devices research conducted under a basic safety assessment can be a good option for studies involving medical devices whose design is not yet mature, and which can be meaningfully assessed without impacting patients' clinical care. However, data from these studies cannot typically be used to demonstrate devices' safety or performance as part of a formal conformity assessment process. A clinical investigation, notified to the MHRA, is required where research outputs are intended to be used in this way.

Basic safety assessments are also sometimes used to ensure the safety of non-medical devices which are introduced to clinical areas as part of a clinical study.

#### *4.5. Health Institute Exemption*

Within the EU, the MDR has introduced new requirements for healthcare institutions who wish to manufacture and use medical devices internally. These requirements are more demanding for healthcare providers than those under the previous legislation. There are requirements to be more structured and transparent, such as for health institution to draw up publicly available declaration that the devices meet the general safety and performance requirements. Most notably in the MDR there is now a specific requirement for an appropriate quality management systems. This will require that health institutions adapt their existing quality management system as they adopt new technology that is developed and manufactured in-house.

It is important to note that at the time of writing, the MDR does not apply in the UK. However, it is prudent for researchers developing medical devices to consider the quality management system requirements early. There is a need to make such systems more widely available to research institutions and harmonised with partnering healthcare providers.

#### **5. Current Trends of R&D Devices**

GSTFT is the largest NHS Trust in the UK. It has a well-established research partnership with King's College London, as part of the King's Health Partnership. It is therefore a very research active centre. The following summarises some key statistics extracted from our internal database of R&D device research activity between 2019 and 2021.

Table 1 presents the proportion of commercial and non-commercial clinical research and Table 2 presents the proportions of the different types of assessments conducted in 2019. Figures from January–December 2019 were chosen, as they represent the 12 months leading up to the start of the COVID-19 pandemic and may be informative of the amount of research activity that would occur when there were no disruption of the service.

**Table 1.** Percentage and number of studies GSTFT received in 2019 via commercial and noncommercial routes.

**Table 2.** Percentage and number of studies GSTFT received in 2019 under different assessment types.

The MDD included software under the "active device" category after an amendment made in 2007, software as a medical device is covered and can be classified. New legislation outlined in the MDR, however, prescribed a more robust and specific approach to regulating software as medical devices.

This change could present a particular challenge for healthcare providers because there are likely to be many non-R&D software algorithms (e.g., spreadsheet algorithms) which require more rigorous governance under the new legislation. This type of non-R&D software algorithm will likely fall in the HIE category. The MDR has many more requirements for healthcare providers who wish to use the HIE than does the MDD.

Table 3 presents the number of research projects that involved software vs. nonsoftware. The proportion of studies classified as software was low in 2019. If new medical devices legislation in the UK mirrors the EU MDR then the number of medical device software assessments are anticipated to rise as non-R&D algorithms are brought into compliance with the new legislation.

**Table 3.** Percentage and number of studies GSTFT received in 2019 grouped under software vs. non-software.

Figure 1 shows the number of studies received via the commercial and non-commercial routes since Jan-19. There is generally less commercial research activity than non-commercial, but non-commercial research activity is the more variable. All research activity stopped during the first peak of COVID-19 pandemic and levels have been very erratic since then, particularly for non-commercial research.

#### **6. Discussion**

Healthcare providers are aware that innovation is key to improving healthcare. Medical device innovation may occur at an innovator or research organization level, where routes to clinical deployment for research may pose a hurdle. This article has discussed the various pathways which exist within an NHS Trust and has discussed the range of research types supported by healthcare providers.

With the introduction of new medical device legislation in Europe, and new legislation also on the horizon for the UK, this article aims to increase transparency over healthcare providers' considerations when approving a piece of medical device for clinical studies. The largest change in terms of the device approval from a healthcare provider's perspective will likely be the increased requirements for deployments under the HIE. This imposes on the healthcare provider a similar responsibility to a commercial manufacturer, such as a more specific process governance with a quality management system. This may require adaptation of processes when introducing a newly developed in-house technology. From manufacturers' points of view, it is likely that the tighter classification rules and conformity assessment routes will pose a challenge. Transparency and cross-learning here is important as Trusts and healthcare providers will each have new responsibilities to ensure all software complies with the new legislation requirements.

**Figure 1.** Volume of research studies involving medical devices assessed by GSTFT received in 2019–2021.

The volume of clinical studies is likely to rise in the future as the healthcare demand created by the COVID-19 pandemic falls. Research activity is returning to pre-pandemic levels and may soon exceed historic levels as Trusts begin clearing the backlog of research studies delayed by the pandemic, and with the additional research opportunities created by the pandemic.

**Author Contributions:** Conceptualization, M.T.K., D.S. and E.A.; methodology, M.T.K. and D.S.; formal analysis, M.T.K. and D.S.; data curation, D.S.; writing—original draft preparation, M.T.K.; writing—review and editing, D.S. and E.A.; All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Review* **Evaluating the Presence of Software-as-a-Medical-Device in the Australian Therapeutic Goods Register**

**Aaron Ceross \* and Jeroen Bergmann**

**Abstract:** In recent years, medical device regulatory bodies have recognised software-as-a-medicaldevice (SaMD) as a distinct subgroup of devices. The field of SaMD has been rapidly evolving and encompasses a range of different digital solutions. Many organisations have now started to look into digital healthcare, as a way to solve key global challenges. However, there remains uncertainty regarding how many of these SaMD products are entering the market and to what extent these systems achieve a desired level of general safety once they are in the market. In this study, we utilise data collected from publicly available databases. The data are evaluated for trends and a descriptive analysis is performed of the recall and adverse events associated specifically with SaMD. We find that there is a significant positive trend (*p* < 0.05) of SaMD registrations, although the number of SaMD registrations remains relative low compared to non-SaMD. This rise in SaMD registrations coincides with increasing levels of recalls and adverse events. More importantly, it becomes apparent that adverse events notification is not yet fit for purpose with regards to SaMD.

**Keywords:** medical technology; digital healthcare; data science; regulations; machine learning; software

#### **Citation:** Ceross, A.; Bergmann, J. Evaluating the Presence of Software-as-a-Medical-Device in the Australian Therapeutic Goods Register. *Prosthesis* **2021**, *3*, 221–228. https://doi.org/10.3390/ prosthesis3030022

Academic Editor: Marco Cicciu

Received: 24 June 2021 Accepted: 20 August 2021 Published: 24 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

#### **1. Introduction**

Although the inclusion of software into medical devices is a long established practice, stand alone software offerings that provide diagnostic and therapeutic functions is relative new. Nonetheless, software applications are an increasingly fundamental feature of clinical practice and regulators around the world have responded to recognise this rise in interest within this area. Certain regulators, such as the one in Australia, have even made data available that can be used for further analysis. The use of medical devices in Australia is overseen by the Therapeutic Goods Authority (TGA). Consideration of software in medical devices has long been a concern for the TGA, which had evaluated means by which to address software through its essential principles as early as 2001 [1]. In 2020, the TGA released a study on the harms caused by software [2], outlining recalls and adverse events related to software through a literature review. The TGA report did not distinguish in its analysis between medical devices that have software and software as a stand alone medical device.

The distinction between devices that incorporate software and "software-as-a-medicaldevice" (SaMD) is important, as these now represent two distinct device types within regulation. SaMD are those entirely composed of software without any additional hardware. In other words, the software program itself is the medical device. Since 2013, there has been international consensus through the International Medical Device Regulators Forum (of which Australia is a member) on the definition of SaMD [3]. Although regulatory legislation now incorporates the definition, there remain unanswered questions about how the safety of such devices can be ensured. There has been work conducted related to recall and adverse events for medical devices that involve software using data from the FDA in the United States (e.g., [4,5]). However, we have been unable to find any quantitative analysis specific to SaMD in any jurisdiction after an extensive review of the available literature. Further to this, there has been no empirical treatment of reported risks associated

Department of Engineering Science, University of Oxford, Oxford OX1 3PJ, UK<EMAIL_ADDRESS>**\*** Correspondence<EMAIL_ADDRESS>Tel.: +44-1865-273000

with SaMD in Australia, or any other jurisdiction. To the best of our knowledge this work represents the first quantitative analysis of SaMD within any regulatory framework. In this work, we focus on the following three research questions; (1) Has SaMD registrations been growing in Australia and what is the origin (domestic or foreign)? (2) What types of recall events are associated with SaMD? (3) What are the types of adverse events associated with SaMD? We conclude with a discussion on results and the nature of regulatory actions for SaMD.

#### **2. Data and Methods**

#### *2.1. Data Collection*

For this study, we relied on the TGA's publicly accessible databases relating to (a) the registration of medical devices, (b) the recall of the devices, and (c) adverse events associated with the device. We utilised these databases in order to map SaMD development in Australia, with a particular focus on risk of harms that SaMD might pose. Below is a description of databases that were used in this study:

All the above databases are publicly available on the internet. In order to extract the data, a webscraper was developed for the ARTG and SARA. The webscraper interfaced with the search box provided by each of these databases. We searched only for medical devices, excluding medicines and other therapies. The webscraper downloaded the returned documents in the PDF format. The data were then extracted by transforming the PDF into plain text and extracting the information from the records..

#### *2.2. Analysis Methodology*

For the first research question, we undertook a trend analysis, evaluating whether SaMD has a trend and, if so, what the nature of that trend was. In order to make this determination, we utilise the Mann–Kendall test, which determines whether a time series exhibits a monotonic upward or downward trend. In relation to second question, a visualisation of recalls in SARA was created and the nature of those recalls related to SaMD was explored. Finally, the last question was investigated by obtaining the description of the types of adverse events associated with software generally, and the presence of SaMD in the dataset was determined. We developed the webscraper and conducted our analysis using version 4.1 of the R programming language.

#### **3. Results**

#### *3.1. Registration of Software-as-a-Medical-Device*

For all devices containing any kind of software, the ARTG dataset lists 971 devices, which represents 1.6% of the entire register of therapeutic goods. Of these, 736 were identified as SaMD with the remaining 235 being devices that incorporated software to some degree. We analysed the trend in SaMD registrations by finding the values related to the monthly registrations of SaMD. Figure 1 illustrates a pronounced increase in SaMD over time when compared to software that was incorporated into medical devices. The Mann– Kendall test showed that a significant positive trend for SaMD (*p* < 0.05) existed in the data. No significant trend was found for software that was incorporated into medical devices.

**Figure 1.** Times series comparison of software-as-a-medical-device (SaMD) and "software that incorporated into a medical device" (non-SaMD) based on the registrations in the ARTG over a period ranging from 2014–2019 (*n* = 558). The dotted red line represent the lowess regression smoothing of all the time series data.

Of the total amount of registered devices, foreign (non-Australian) products made up 92% with the remaining products being domestic (8%). Overall devices from either the USA and the EU constitute a total of 66.6% of the Australian market. However, there is a stronger domestic representation when SaMD is considered in its own right. Australian development of SaMD has a positive trend that is significant (*p* < 0.05) when tested with the Mann–Kendall test, which determines whether or not there is a linear monotonic trend, as illustrated in Figure 2.

The legislation provides a system of categorisation of medical devices based on the potential risk of harm to the user. Assessment for risk consideration include: (i) the intended use of the device, (ii) location on the body during use, (iii) invasiveness into the body, as well as (iv) duration of use. The Therapeutic Goods Act 1989, along with associated subsidiary legislation, regulates the use of medical devices in Australia. This legislation provides a classification system to assess the risk that devices can pose and provides the means to categorise these devices according to risk. There are three main categories for medical devices: (i) Class I for low risk, (ii) Class II for low to medium risk, (iii) Class IIB for medium to high risk, and (iv) Class III for the highest risk devices. The largest classification group for non-software devices and SaMDs is class I, while for other non-SaMD software, the largest group consist of those from class II. In Figure 3, we illustrate the division of risk classifications based on the type of device: (i) non-software; (ii) other software, meaning that the software is part of a physical device, and (iii) SaMD. The proportions remain consistent across all device types, with Class I representing around half of the classifications (57%, 52%, and 51%, respectively). SaMD has a higher proportion of low-medium risk class devices Class IIA (32% compared to 21% for non-software devices and 28% for devices that include software) and lower proportion of high risk devices, Class III (2% compared to 9% for non-software, and 7% for software included in another device).

**Figure 2.** Time series data of software-as-a-medical-device (SaMD) for registrations, based on the region of origin as reported in the ARTG. The data covers a period between 2014 and 2019. Dotted red lines represent the lowess regression for each of the time series.

**Figure 3.** Comparison of device classification between software-as-a-medical-device (SaMD), other software and non-software. Please note the difference in scales for each of the figures.

#### *3.2. Recall Actions*

The System for Australian Recall Action (SARA) is a searchable, public database containing product recall action notifications. These notifications are divided into (i) recalls; (ii) recalls for product correction; and (iii) hazard alerts. It is important to note that entries into SARA are commenced by the product manufacturer or product sponsor with the TGA monitoring the recall action itself. A total of 4746 notifications were extracted, from between 2 July 2012 to 24 June 2020. When cross-referencing the ARTG data set with SARA, it resulted in 3597 recall events, obtained from 1900 unique devices, as shown in Table 1.

SARA provides the "level" at which the device is recalled, which relates to hospital, retail, wholesale, or consumer. The hospital setting accounted for over 90% of the recalls actions (Table 2), which might relate to the nature of the devices.

**Table 1.** Recall actions obtained from the System for Australian Recall Action (SARA) database for the period from July 2012 to June 2020.

**Table 2.** The level at which recalls are made for all medical devices in System for Australian Recall Action (SARA).

There are 455 recall actions in SARA that mention software as the reason for the reported error, comprising 9.6% of all the reported recall events. A total of 72 software recalls were related to SaMD, constituting 16% of the software reported recalls and 2.3% of total recalls. All of these SaMD recalls come from 29 unique SaMD registrations, making up 3.9% of all registered SaMD that are in the ARTG. The connection between recall action, level and recall class is shown in Figure 4 using an alluvial plot to map them out for both SaMD and all others. The figure is set to a logarithmic scale in order to highlight the differences between SaMD and all other medical devices.

**Figure 4.** Alluvial diagram of recalls types, levels, and recall class in the SARA dataset. Data are plotted for SaMD and all other medical devices (Non-SaMD). The scale is logarithmic.

#### *3.3. Adverse Events*

The Database of Adverse Event Notifications (DAEN) records the types of harms caused by medical devices. We searched for the string 'software' in DAEN, which returned 68 events, of which there are 43 events that could be cross-referenced with data from the ARTG. This resulted in 17 devices, of which 7 were SaMD. Due to the nature of DAEN, we were unable to retrieve all reported events for all device types.

The event types in DAEN are categories according to the standard on medical device adverse event reporting [7]. The standard lists 20 different event types, 9 of which have been reported for software. According to the standard, all software errors fall under the broad event label of problems related to 'Computer Software', rather than providing specific software issue labels. As such, Figure 5 shows that the second-most frequent of the events in DAEN for software devices fall under this heading (*n* = 10, 26.3%) after usability issues (*n* = 11, 28.9%). Non-SaMD include a label describing software issues related to hardware control as well.

**Figure 5.** Number and type of adverse events for both SaMD and non-SaMD software.

The DAEN database provides injury outcomes of the reported events. Figure 6 shows that SaMD are associated with low levels of injury, but it should be noted that the "unknown" category is also high. This suggests that it is difficult to ascertain the exact level of injury from SaMD events.

**Figure 6.** Outcomes of adverse event related to software in DAEN for both SaMD and non-SaMD software.

#### **4. Discussion**

The results demonstrate that SaMD (and software more generally) are growing facets of the medical device landscape, as shown based on the Australian data. Caution needs to be taken in case of generalising too much, as there might regional differences. Nonetheless, the suggestion seems to be in-line with the popularity of machine learning and artificial intelligence for healthcare within the research arena.

The outcomes also highlight that there are some challenges in evaluating the risks associated with SaMD, as the limited regulatory vocabulary for software defects prevents further granular analysis. The blunt categorisation of the ISO code "Computer Software Problem" in DAEN makes it hard to identify clear areas of improvement for the SaMD. It acts as an obstacle for the regulator, and those assessing regulatory data, as it prevents a better understanding of the types of events that occur for SaMD and reduces the potential for better risk management. Within the recall data, there are no fields specifying the type of problem that prompted the recall. This is not because a lack of vocabulary to describe software defects. This does exist, in, for example, the IEEE Standard for Classification of Software Anomalies [8]. The inclusion of more detailed software-specific recall reports and adverse events may facilitate the policy prioritisation for the regulator, which, in turn, will ensure that devices on the market reach an appropriate level of safety and performance.

The issue of applying appropriate standards for clinical safety are more acute for software than for non-software. McHugh et al. [9] already suggested that the rapid, iterative approach to software development may be in tension with the regulatory requirements for medical devices. Non-software device manufactures require specific fabrication infrastructure. This is in stark contrast to what happens in software testing. The costs associated with changing software are different compared to those associated with hardware manufacturing. For example, a software developer could quickly push an update to customers upon the discovery of a problem, whilst this is impossible for a (hardware) manufacture to do at scale. This rapid response option is not available to non-software devices, which must collect all defective items and fix the issue physically. Increasing the regulatory support offered to software developers during the early phases of research and development, as well as integrating this kind of support into, e.g., computer science courses, will potentially create more robust solutions for these regulatory issues in the future [10].

The difference in how risks need to be addressed by the manufacturers of products also relates to the origin of the product itself. The Australian market for medical devices relies heavily on foreign suppliers of medical devices. Herpin et al. [11] reports that the funding opportunities within the Australian biomedical industry may motivate innovators within the country to seek to establish manufacturing operations in larger markets for profit maximisation as the distance to these markets has an impact on costs. These economic dynamics seem to affect SaMD development less as there is no need for a factory to manufacture these devices, just access to skilled software developers. The lower initial cost for a more scalable product might account for the higher levels of Australian representation in SaMD.

In their work on safety and security of products in supply chain management, Marucheck et al. [12] found that recalls and adverse events of products, rather than regulation, may limit innovation in medical device development. Although Thirumalai and Sinha [13] showed that within the medical device industry, recalls costs are expected and they normally do not influence the market too much. As mentioned software might be better suited for ongoing improvements and this is also reflected in the fact that the for SaMD corrections are taking place if defects are detected.