A tag (analogous to a barcode) is a unique entity that can be attached to an object or a person and thereby enables information environments to remotely distinguish objects and individuals, track their position, detect their status, etc. The RFID tag is a microchip with programmed identification plus an antenna. The distance between the tag and the tag reader (in fact the reader is the base station) should be short enough so that the signal could be coupled. In reality, there is no true antenna because no far-field transmission is employed. The tag communicates with the tag reader by electromagnetic coupling via radio frequencies. Parts of the tag and parts of the reader are coupled together in a way that is analogous to the transformer windings (inductive coupling) or as opposing plates in a capacitor (capacitive coupling). Generally, the information acquired by the tag is further processed by a more complex computer equipment. In fact, the tag is a kind of low-level network, which enables the transmission of sensor data.

The principle of operation is so that the tag behaves as an electrical load on the tag reader. Hence, the tag can transfer information to the reader by altering its own impedance. The RFID tag changes the value of the impedance via an electronic chip that is effectively an active switch. In result, the tag is not required to create a transmitted signal, and the impedance switching sample is utilized to encode the data in the tag. At any random moment, a tag reader can just read one tag in its locality and a tag must be read by one tag reader [32].

Tags might be either active or passive. Active tags have a dedicated power supply (a battery). They possess extended processing functionalities and have some capabilities for pressure or temperature sensing. Active tags are characterized with an operating perimeter of hundred meters and a relatively lower error rate.

On the contrary, passive tags have a limited operating perimeter of up to several meters and they are characterized with a pretty high error rate. Passive tags are cheaper and that is why they are most common in the RFID marketplace. They have no physical power source as they are powered by the near-field coupling between the reader (the radio waves caused by the reader) and the RFID tag. Passive tags have limited processing and communication capabilities but have no sensing capabilities for the information-carrying medium [33].

RFID technology has numerous applications such as tracking of assets and people, healthcare, agriculture, environment monitoring, etc. Many tracking RFID applications are based on the universal communication and computing technologies available [34–37].

A prospective area for development of applications is the integration of RFID systems and wireless sensor networks (WSNs). So far these are relatively separate areas of research and development. The combination of RFID and WSNs would open new scientific and industrial fields by utilizing the benefits of these technologies.

RFID systems are primarily used for identification of objects or tracking their location without delivering information about the object and its physical condition. In numerous applications the location or the identity of an object is not enough and extra information is needed – it can be extracted from other parameters characterizing the environmental conditions. Sensor networks could help in such cases. WSNs are systems consisting of small sensor nodes that can collect and deliver information by detecting environmental conditions, for example, temperature, humidity, light, sound, pressure, vibration, etc [38]. Nevertheless, the identity and location of an object is still vital information and it can be extracted by RFID techniques. In these situations, the ideal arrangement is to combine both technologies in order to ensure extended capabilities, portability, and scalability [39].

Sensors with integrated RFID tags can be classified in two categories: (1) tags communicating with RFID readers only and (2) tags communicating with each other and creating an ad hoc network [38]. RFID systems can be combined with wireless sensor networks by integrating the sensor nodes with RFID readers [40]. Another option for integration is the so-called mixed architecture where the sensor nodes and the RFID tags remain physically separate but they exist together and they operate separately in an integrated network. Accordingly, it is not necessary to design a separate hardware device in order to integrate the benefits of both technologies.

## 3 Monitoring and Assisted Living Systems for Elderly and Disabled People

#### Trend of the European Population

The proportion of the adult-population in the European Union is in a phase of rapid increase. The aging of the population is accompanied by increased occurrence and spread of chronic diseases, and hence a significant increase in healthcare costs. Staying at their own homes, or at places freely chosen by the elderly people, is one of the approaches already taken improve the quality of life and to reduce healthcare costs of the aging population.

The idea is to support elderly people to improve their quality of life and to create better conditions for their stay in the environment of their choice. To do this, it is necessary to develop modern equipment and systems for health status monitoring and to introduce comprehensive eHealth technologies. The use of such technologies at home or at home-like setting is still in its infancy, but this method is one of the most promising approaches to facilitate the independent living of the elderly people.

Combining health monitoring systems with smart home technologies (Fig. 3) makes it much easier for elderly people to access medical care without the need to leave their homes.

Contactless sensor systems provide 24-h surveillance of the elderly in their homes by collecting data from different sensors and fusing them from the so- "Data aggregators". Data aggregators can be devices that provide only simple offline storage and analysis features. However, in modern monitoring systems, they typically perform preprocessing and retransmission of online analysis data to systems of higher hierarchical level.

In addition to monitoring some medical-specific indicators, the main groups of indicators that can be monitored (Fig. 4) are related to:

Fig. 3. Combining health monitoring systems

Fig. 4. Main groups of indicators.

Since the end of the last century, global trends have seen a rapid increase in the share of elderly people. In 2035, one-third of the Europeans will be over the age of 65 [41]; for the USA this figure is expected to rise to 70 million in year 2030. This estimated figure is double than the one counted in 2000. In 2009 the average age, which allows daily activities can be carried out without difficulty, was about 67 for women and 63 for men.

The most developed countries are concerned about the aging of their population [42]. Quality of life is deteriorating with aging, which leads to worsening the skills and abilities of the people [43]. Statistics show that 30% of adults fall at least once per year and 75% of these events can even cause death. Much of the elderly people suffer from chronic illnesses that require medical treatment or periodic reviews.

Various initiatives have been taken to handle these issues. One approach is the called Assisted Living Systems (ALS). It acquires immense importance in helping elderly people who live alone in their own homes and need care [44]. The proposed assistance aims at increasing the autonomy and quality of life of the consumer and contributing to its social consolidation. The results in this area have a direct public impact. Many authors have discussed the requirements and engineering aspects of ALS.

Development of technology and research are directed towards systems for fall detection, detection of pressure to a chair or bed, video monitoring, motion and tilt sensors and devices, accelerometers, smart clock with gyroscopes or worn on the belt [45].

A European Union initiative [46] is being undertaken to increase the care of the aging through the penetration and use an information and technologies of communication. It aims to help elderly people to carry out their daily activities, thereby increasing their autonomy [47].

#### Assistive Systems: State-of-the-Art [47]

Ambient Assisted Living (AAL) includes concepts, devices, systems, methods, and services that ensure constant support without intruding user's system. Assisting everyday life depends specifically on the situation of the user. The technologies that are used in AAL are user-centric, that is, oriented to the potential needs of the particular consumer and integrated into the user's immediate personal environment. As a result, the technology adapts to the user, not vice versa. Internal and external monitoring is necessary especially for elderly people or people with disabilities (heavy hearing, deafness, limited mobility, etc.) The use of intelligent sensors is a desirable service that can potentially increase consumer autonomy and independence, while reducing the risk of life alone.

The services and systems developed aim being tailored to elderly people and their cognitive problems. By default, it is expected that the systems are able to integrate several subsystems that have been developed by different manufacturers [48]. Also, it is expected that every user could to adapt quickly and easily so that no constraints and difficulties arise. However, the real implementations are still fragmented and isolated.

One of the major projects in the field is Intel's fashionable smart home [43]. The aim is to help elderly people by making use of four technologies: sensors, networks, monitoring daily activities and environment visualizing. Sensors determine the location of people and objects. The networks integrate motion sensors, cameras and switches that define the activity and visualize the environment.

The idea of automating and introducing technology into people's homes is to build a positive home atmosphere. Numerous authors make analyses of experimental data from sensor systems used to monitor and demonstrate the functional capabilities of elderly people and analyze and produce statistics on how they change over time. The built-in system includes sensors in all rooms: kitchen, living room, vestibule, bedroom, bathroom, etc. There are high requirements for data visualization displays as it reflects to the end-user perception of the service [47].

The Intelligent Home Monitoring System at the University of Virginia [49] focuses on collecting data using a set of cheap, unobtrusive sensors. The information was recorded and analyzed in an integrated data system. It is managed through the Internet and collects the information in a passive way respecting the privacy of the older adults.

The Rochester University [50] developed a prototype smart medical home, consisting of computers, infrared sensors, video cameras and biosensors. The main service is used for a medical consultation through a conversation between the medical person and the patient. The activities and movements of the users are also monitored. The process supports decision making for the patient and caring personnel.

Hong Sun and others [51] show arguments that most of AAL's ongoing efforts to tackle older people's problems do not fully reflect the importance of social activities. Intelligent sensors and devices use has preference in comparison to the more important than human interaction. The AAL's assume that the older adults are passive and weak by default and it is not true in all cases as some people desire to support monitoring.

The SOPRANO Integrated Project [52] aims to extend the time that people can spend living alone in their own homes being independent in their activities and feeling safer. Required technologies include products and services that allow people to perform their everyday tasks.

Aviles-Lopez et al. [53] tested a lab platform for deployment at home for the older adults. This research aimed at helping people with physical and psychological abnormalities such as arthritis, Alzheimer's disease, diabetes, senile dementia, and cardiovascular diseases coming from the aging process. It is achieved by maintaining a certain degree of independence using new types of mobile embedded computing devices, wireless intelligent sensors and so on. The platform is contextual, mobile, invisible, and adaptable supposing that the users are traced and identified in space thanks for the wearable device as watch, bag, cups or other embedded accessory in clothing. If the older user has suffered a fall or an unpleasant event, the system should alert caring personnel without any interference. The communication way to the enduser and data on blood pressure, sugar levels, etc. have to be acquired, extracted and transmitted in a reliable and way.

Drug management applications have a special place in the daily schedule. customized approach towards this problem is presented in [47].

In this way, the supervising medical consultant could do a medical examination remotely and change the dose of any drug. Other components of the supervision include cameras and sensors with a built-in accelerometer. This provides an opportunity to track the motion of users and instantly record an event such as a fall. Patients who use electrically powered wheelchairs can move freely. However, very often they need help in opening or closing doors.

Various studies have found that large TVs and monitors are not the most appropriate means of monitoring. The whole system should be easy to use by the elderly. They find it difficult to adopt new technologies that they cannot understand. Homes are equipped with sensors that measure the state of the users and maintain communication with their friends and relatives.

Holtzinger and others [44] assessed the wrist unit that is well received by users. It is designed to monitor the vital signs and detect different situations such as loss of consciousness, detection of falling, etc. Healey and others [54] presented a monitoring prototype system that can record, transmit and analyze permanent echocardiogram data. The system is also designed to have the ability to record events, activities, and various medical symptoms.

Different researchers do experiments using systems to monitor daily activities, activity, exercises and medical tests. Madeira et al. [55] looked at the possible enhancement of the quality of life of the elderly people through telemedicine. The proposed system combines intelligent items such as wheelchairs and walkers with corresponding built-in sensors for remote measurement of mechanical and physiological parameters. In this way, the elderly will be monitored in different situations.

Some studies [56–58] focus on the development of a smart home where the elderly and disabled people can enjoy quality of life and greater independence. Smart monitors can constantly monitor patients and their vital parameters. Technologies that can track changes in activities and alert the care provider are: a smoke detector; flood detector; temperature sensor; gas detector; occupancy sensor for bed; occupancy sensor for the chair; a fall detector; hanging around the neck, on the wrist, or clinging to clothing; an epilepsy sensor located under the bed and more.

The publication of Wang et al. [59] describes the prototype of the so-called I-Living Assisted Living architecture, which includes various built-in devices such as sensors, actuators, displays, and Bluetooth-enabled medical device. This device may be a dedicated computer or black box equipped with one or more wireless interface cards. Independent devices can communicate with the appropriate server over the Internet that provides web-based interfaces to allow cares, healthcare providers and healthcare professionals to monitor the environment and analyze measured data.

De Florio and Blondia [60] do not believe that the expansion of the traditional approaches to social organization might be enough to provide effective support for the elderly [47].

All listed and described projects are only part of the AAL activities collected. Analyzing the results and new opportunities and trends shows that the topics discussed are up-to-date and will continue to develop significantly in the coming years. Various projects are aimed at solving many problems of some groups (adults, adults with special needs and people with diseases). Applied approaches and applications are specific, which limits the dissemination of results.

Flexibility of systems is a prerequisite for universality, as far as possible, of hardware that would lead to rapid production of the product at a reduced price. Similar to telemetric monitoring systems for high-risk patients, relatively simple AAL systems can perform two main tasks:

In addition to the constantly available communication interface between the observer and the user, it is also possible to automatically detect falls as well as momentary observation of many vital parameters.

## 4 Risks and Accidents Detection for Elderly Care

In order to improve people's lives, it is very important to reduce threats when people get older, such as detect and prevent falls. The research areas as fall detection (FD) and fall prevention (FP) have been developed for over a decade trying to improve people's lives through the use of pervasive sensing and computing. The most common reasons that can cause falling are obstacles at home and the aging. Getting older, people's bodies pass through some physical changes making them more fragile, and more prone to falls.

Why is so important to detect falls? Falls can result in critical injuries, especially for the elderly. The longer someone stays unassisted, the less chance he/she has to make a full recovery. Unfortunately, a fall detection system does not detect all fall cases. The most common injuries are to the head and lips which results in long-term complications. So, the faster help is very important in these cases [61].

Fall sensors use advanced technology to detect your movements and the position of human's body. They are able to give the difference between an emergency case and everyday movements, for example, can detect if the person is just laying down or there is a sudden change in the position, which means a fall.

Most falls happen at home because there are a lot of hazards there, such as slippery floors, clutter, poor lighting, unstable furniture, obstructed ways and pets, etc [61].

The first measure to be taken into account is to conduct a detailed analysis of the house and to identify the possible reasons which may lead to injuries. Then a preventive checklist can be developed to minimize the risk of fall.

Figure 5 demonstrates an example of a wearable elderly care system. The technologies are used for detection of accurate positioning, tracking the physical activity and monitoring the body signs data.

In order to track precisely the position of elderly people, a precise positioning sensor network should be developed in real time. Also, a software system should be designed with modules for data processing, data extraction, vital signs detection to support human activity recognition (Fig. 6). Biomechanical sensors are needed to monitor the physical state of elderly care which in essence are multiple sensors incorporated into the clothing, for example.

Overall, a wearable system consists of interconnected modules that can be placed at different body areas. Each module consists of sensors, Analog to Digital Converter (ADC), Radio Frequency, computing elements, circuitry and hybrid power supplies. When designing such systems, it should be taken into account the so called "wearability", which means weight, form, heat generation, flexibility and other properties. In technical point of view, the main considerations include the power consumption and overall system size in order to achieve good "wearability" [62].

Fig. 5. Wearable elderly system

Fig. 6. Monitoring system for data processing

One fall detection (Fig. 7) and prevention systems consist of either external sensors or wearable sensors. External sensors depend on subject of interest (SOI), and wearable sensors are attached to the SOI.

The most common types of external sensors are the camera sensors. They are placed in fixed locations where the person daily activities will be performed. The main

Fig. 7. Fall detecting sensing components

disadvantage of these sensors is that the person can fall out of the visibility area and the system can be unable to track the user.

Another type of external sensors is the proximity sensors, which are used in fall detection systems. They are commonly attached to walking-aid devices (cane, walker, etc.). When the user suddenly falls, the sensor detects the change of the position of the SOI. The disadvantages involve the price of such sensors and the short proximity range.

An alternative to the external sensors is the wearable sensor, which is employed into fall detection and prevention systems. They are attached to user's body and are cheaper than the external sensors.

Widely used in fall detection systems are the accelerometers because of their price and the fact that can be placed on different parts of the body. They can also be embedded in other devices as shoes, belts, watches, etc. The advantage of accelerometers is that with a single sensor a lot of movement characteristics can be successfully detected, especially falls.

## Wearable Sensors for Fall Detection

Due to the rapid development of Micro-electro-mechanical systems technologies, such as accelerators, gyroscopes, magnetic sensors, particularly wearable sensor-based human activity recognition technologies, such devices become more and more attractive for use in ambient assisted living systems, especially in monitoring elderly people. Because of the advances in these technologies, MEMS sensors become cheaper, lighter and small enough to carry. These systems do not require the use of base station, as cameras which have to be installed on particular area. These systems collect the data in passive mode and do not create electromagnetic pollution. Accelerometers and gyroscopes are easy to wear but also have less power consumption and also less sensitivity to body movements, which may cause false alarms. But from a commercial point of view, this technology is the most utilized one for commercial devices and can take the form of a belt or watch, for example. In order to minimize false alarms using such kind of sensors, researchers propose different methods, such as placing the sensor into human's head or ear [63–65].

The advanced wearable sensors incorporated multiple sensor technologies, for example in [66] proposed a system of gyroscopes and accelerometers, another approach has barometric sensors in additions for high variations sensors.

Interesting solutions for fall detections and prevention away from home became attractive after phone technology developments. In [67] a very promising solution is proposed that reports 100% fall detection prevention. The system is based on accelerometers which are used in mobile phones.

There are also systems that can not only detect a fall but also can specify the fall type. Such systems are proposed from [68, 69] and incorporate a tri-axis accelerometer, gyroscope and magnetometer, as well as the data processing, fall detection and messaging.

## Ambient Devices for Fall Detection

Ambient devices detect the environment of a person under protection. The technologies are used in commercial fall detection devices and the most common one is the infra-red technology, but there is also vibration sensing, noise sensing, etc. In order to cover the whole area, the system has to be installed in all needed rooms which is one of the drawbacks of such systems.

One example of infrared ceiling sensor network (Fig. 8) is proposed by [70]. They are using the "values of pixels" as features, 8 activities recognition, which are performed by 5 subjects at an average recognition rate of 80.65%. They have obtained a performance of 95.14%, the false alarm recognition of 7.5% and the FRR of 2.0%. This accuracy is not sufficient in general but high according to with such low-level information.

It is explained that such system has the potential to be used at home providing personalized services and detecting abnormalities of elders who live alone.

Another way for fall detection is through vibration sensors, which are incorporated into the flooring. In source [71] a system with 100% success is reported, which detects movement through vibrations. Electromagnetic sensors are present [72] which are again incorporated into the flooring, which can generate images of objects touched to the floor.

There are systems for fall detection based on lasers. A laser is used which interacts with light-sensitive device, which generate together a network of theoretical crosssections, which detect stable objects [73].

#### Vision Based Devices for Fall Detection

Systems based on object monitoring have the same disadvantage as ambient devices and must be installed in all necessary rooms in order to cover the required range. Another issue is privacy, working with photo material from everyday life. There are cases in which pictures are sent only when a fall is detected. It is also easy to process these photos; the person's face can be faded.

Camera-based monitor the posture and shape of the subject during and after a fall, which happens in fractions of seconds.

Fig. 8. Example of infrared ceiling sensor network

A considerable amount of processing power is consumed by the image dealing. To compensate the computational cost, images are usually compacted and with smaller pixels in a pre-processing stage.

In camera-based sensing systems there are different ways in detecting falls. Some systems are based on human skeleton, but their computational cost is not valuable for real time situations. Other types calculate and transform some parameters, as falling angle, vertical projection histogram etc., but their disadvantage is the false alarm rates.

Nevertheless, regardless of the number of cameras installed, continuous monitoring is still restricted to the camera locations. Another disadvantage is that such systems are influenced by light variability, which leads to lower recognition from laboratory environment to outdoor environment. Due to such limitations, vision-based human activity recognition systems are not so well suited to most elderly care applications.

A system architecture (Fig. 9), for example, includes a wearable device which is placed on human's waist [74]. The system uses acceleration analysis for fall detection. Then it gets the geographic position of the SOI and send short message for fall alarm.

The system has low power consumed hardware design and highly efficient algorithm which could extend the service time of the wearable device.

Overall, a wearable system is comprised of interconnected modules, which can be installed at different parts of the body. Each module consists of sensors, Analog to Digital Converter, computing elements, RF circuitry and hybrid power supplies (batteries and energy scavenging generator).

Fig. 9. Example of system architecture for fall detection with wearable sensors

## Vital Signs Monitoring

Most elderly people suffer from age-related diseases, as diabetes, hypertension, hypertension, etc. So, it is of great importance to design reliable real-time health monitoring system for elderly care. Most common used are wearable and non-invasive biosensors, which can be put on the body or near the body which can successfully measure a variety of vital signs. In Table 1 a summary of several vital signs is presented.

Table 1. Summarize of most common vital signs.

## Body Temperature

One of the most important vital signs to be monitored is the body temperature. Another important issue that should be taken into account is the location at which the temperature will be measured because it is different at the different locations. There are several means that can be used for measurements, such as thermistors, thermoelectric effect, optical means, etc. The most common technique for non-invasive measurement using wearable sensors is the thermistor. There are methods proposed in [75], where negative temperature coefficient resistors a temperature sensing element is used and the textile wires incorporated into the sensor element are integrated into wearable system for monitoring, in this case baby jacket. Other methods propose textile-based temperature sensor which is incorporated into knitted structure [76]. There are a lot of wearable temperature sensors available at the market that can be directly attached to the skin, as LM35 [77].

## Heart Rate

The heart rate is one of the most important signs, especially when we talk for elderly care, which should be precisely monitored. The heart should be in perfect working condition in order to consider that the patient is healthy. The heart rate of a healthy adult in resting position ranges from 60 to 100 beats per minute. Nevertheless, depending on person's activity and physiological state, these values can vary. During the night, a healthy person's heartbeat may vary from 40 to 50 beats per minute, which also should be considered. This parameter can be used in order to diagnose a lot of cardiovascular diseases. Heart rate can be measured through various technologies, as electrical, optical or strain sensors. The electrical measurements include electrocardiography through electrodes. There are some methods for such measurements proposed, for example in [78] chest electrodes are investigated which are silver coated, without the need to use gel or paste during the measurements. Other approaches use soft micro fluidics and adhesive surfaces to achieve highly stretchable state-of-the art systems [79]. Other researches describe magnetic sensitive sensors which are able to measure quasi noncontact pulse rate. These sensors can measure magneto-cardiogram in non-shielded conditions [80, 81].

#### Respiration Rate

Respiration rate is very indicative parameter for distinguishing diseases as asthma, sleeping apnea, anemia, etc. A healthy resting person respiration rate is typically one breath in every 6.4 s, the amount of inhaled air is approximately 500 mL [77]. Elderly people often have difficulties in breathing normal because the lungs expansion and contraction rates decreases. The methods for respiration rate measurement can be divided into two types, the first one detects directly the airflow during the breathing, the second one measure indirectly responding to chest and its expansion and contraction. For directly measurements sensors can be placed near the nose or mouth and respond to changes in the temperature of the air, the pressure, humidity, the concentration of carbon dioxide, etc [82]. The indirect measures involve physical parameters that need to be monitored, as changes in the lung volume and movement. With the advanced developed textile-based technologies nowadays, there are a lot of sensors available, which can be directly incorporated into the clothing which accurately detect the breath levels without interfering person's comfort. In [83] a garment-based sensing system with piezoresistive sensor is represented, which is able to determine a 10 s pause in breathing.

## Blood Pressure

The blood pressure gives the force inside an artery and is typically 120/80 mm of mercury for healthy persons, the systole is 120 mm Hg (maxima) and the diastole is 80 mm Hg (minima) [77]. Blood pressure is typically detected using sphygmomanometers, but they need stationary setup, not cost effective and do not have the possibility of monitoring. Nowadays the state-of-the-art sensors are capacitive sensitive strain sensors [84], which are compressible and piezoelectric. The difference between both of them is that compressible capacitive strain sensors are composed of elastic dielectric, while the piezoresistive sensors are composed of robust dielectric placed between 2 flexible electrodes. When an external pressure is applied to the dielectric, it will lead to change in the capacitance of the device. In the same way, if the piezoelectric material is strained, this will generate an induced voltage in the device. For example, in [85] a conformable lead zirconate titanate sensors are presented, which have piezoelectric response. It is reported that these sensors have 0.005 Pa sensitivity and 0.1 ms response time. Such kind of performance ensures that the sensor can be used for blood pressure measurements. Another approach that can be used for blood pressure measurements is the RFID (radio-frequency identification) technique, but such device require implantation under skin, such as presented in [86].

## Pulse Oxygenation

Oxygenation is the oxygen saturated hemoglobin compared to total hemoglobin in the blood, which is saturated and unsaturated. The normal state for the human organism is considered as 95% to 100% blood oxygen level. When this level is below 90% it can cause hypoxemia (more particularly tissue hypoxemia). The oxygenation may be separated into three groups: tissue, venous and peripheral oxygenation. The measurement technique is non-invasive in fresh pulsatile arterial blood. The most common method for measurements is using optic-based device, such as a pulse oximeter. The working principle is based on generated light by light emitting diodes through parts of the body as earlobe, forehead, wrist, fingertips, etc. Nowadays with the advances in organic electronics, the production of OLED (organic light emitting diode) and organic photo-detectors became prime devices for use in pulse oxygenation measurement due to their comfort in use [77]. Such sensors are described in details in [87].

## Blood Glucose

The measurements for blood glucose involve the glucose amount in human blood which concentration is usually lower in the morning and increases after every meal. If the blood glucose is out of its normal range, this may indicate health problems as hyperglycemia (low levels) or diabetes (high levels). In recent years, the number of people with diabetes has increased. The World Health Organization reports that 9% of adults worldwide suffer from diabetes [77]. It has been found that frequent (possibly continuous) measurement of blood glucose levels is essential for conducting insulin therapy and minimizing the harmful effects on the body. Modern methods of testing include periodic tests in specialized laboratories or analysis of daily profiles (periodically over several hours), using a portable blood analyzer at home. For this purpose, after a pinch, usually on the fingertip, a certain amount (drop) of blood is delivered to a special test strip which is placed in the analyzer and within a few seconds the current blood glucose level is indicated. These persistent pricks cause discomfort, especially in young children, and rarely can lead to infections. New developments in the art are directed to alternative methods for measuring glucose concentration, e.g. bloodless, by measuring glucose levels in body fluids (sweat, tears, urine) as indicated in [88]. Saliva nano-bio-sensor is presented there for noninvasive glucose monitoring which provide low-cost, accurate and disposable bio-sensor. Another method for non-invasive method is proposed in [89]. The described methods are still not applicable in mass practice. Another part of the research is directed to the development of invasive methods for the delivery and analysis of blood micro-bleeds. At this stage, there are no data on the implementation and applications in the mass practice of nanobiobs for determining blood glucose levels by analyzing blood micro beats in the absence of pain sensations for the patient.

## 5 Activity Recognition for Sports

Research of human activity is becoming a most popular and relevant topic for multiple scientific areas. Human activity recognition includes mobile computing [90], surveillance-based security [91], context-aware computing [92] and ambient assistive living [93]. The sensor technologies and data processing techniques have achieved much progress. Work on these supporting technologies has led to developments in the area of data collection and transfer and information integration. Many of the solutions to real problems related to human life are increasingly dependent on the human activity recognition. Recognizing human activity as a topic of work can contribute to many important activities related to security and monitoring, preservation of the environment, help in maintaining independent living and aging, etc. To develop such a system, it is crucial to work on four main tasks. The tasks include selection and deployment of sensors designed to collect information about and capture a specific user's behaviour while simultaneously monitoring respective changes within the environment. Another task is related to the application data analysis techniques which are used for/while processing and storing the accumulated information, to create computational activity models which, when incorporated within complicated software (packages/products), are designed to select algorithms to provoke responsive activities from sensor data through reasoning and manipulation. There is a variety of tools, methods and technologies available to implement each task.

Sensor-based activity is used for activity monitoring. The approaches involve computer surveillance, structural modelling, characteristic elements extraction, action extraction and movement tracking with the main purpose being to make analysis aiming to recognize certain pattern based on collected visual information. Another category is based on the application of recently developed sensor network technologies for activity monitoring [94].

Sensors are attached to the monitored person. This approach is applicable in order to follow physical movements such as workouts. There are multiple types of sensors available for activity monitoring (contact sensors, accelerometers, audio and motion detectors etc.). The sensors are divided according to their purpose – there are different types based on particular output signals, involving theoretical principles and defined by technical infrastructure. They are represented within two basic categories according to the way they are positioned during the activity monitoring process. Activity monitoring based on Wearable sensor. This type of sensor is attached directly or indirectly to the observed person. While the monitored object performs any type of action, the sensors generate signals. In this way we are able to monitor features which describe the human state of mind and respective motion patterns. The sensors can be put into clothing, in shoe soles or heels, inside cell phones, watches and other mobile devices etc. They can be located directly on the body as well. From them we get the necessary indicators about the position and movement of the test object at a given moment, the pulse, temperature, and so on. There are different types of relevant sensor information applicable for various types of activities. Accelerometer sensors are sensors for activity monitoring. They are used to monitor actions such as body movements such as walking, running, jumping and more. In a paper [95] a network of three-axis accelerometers has been reported. These accelerometers are fixed on different parts of the object's body and provide movement and orientation data for the part of the body that is selected. In [96] are used body worn microphones and accelerometers to measure acceleration and angular velocity through accelerometers and gyroscopes. In the paper [97] provides a method for determining the user's location. With this method, the behavior of sitting, standing and walking can be recognized. Another used wearable sensor are GPS sensors. These types of sensors are mostly applied when monitoring activities involving location changes or open air and mobile environments [98].

The state of the art can be divided into two sub-topics, the recognition of human activity and Human Activity Prediction. Activity recognition is a complex process. The basic tasks include:

The first one is based on surveillance tools, such as video cameras to monitor the object's behaviour and environmental fluctuations. The provided data can be a series of video or digitally presented visual image. Common are computer vision techniques for action extraction, feature extraction, structural modelling, motion detection, and motion tracking to specify pattern recognition.

The second one is based on sensor-based activity recognition using the newly developed sensor network systems for motion monitoring. The acquired sensor data is presented as time series of state changes. They can be used as parameters for data integration, probabilistic or statistical analysis. Beside the already described attachment techniques, the sensors can be also placed within the object's environment as long as the position allows the tracking activity. These types of sensors use inertial measurement units to capture the object's behavior. This method is used for registering motion. The use of multiple multi-modal miniature sensors enables a robust capture of activities to be accomplished by monitoring interactions between a person and an object. The activity information can be acquired through motion monitoring models. The motion monitoring models can be built to recognize activity models from previous experiments including rich database about monitored persons' behaviors. For this purpose, they can be used data mining and machine learning techniques through creation of statistical activity models. This method is based on data. The actions that follow are based on probabilistic or statistical classification. This approach has its advantages as handling uncertainty and timing information. On the other hand, requires large datasets for training and learning, and also suffers from the problems of scalability and re-usability. Another method involves the use of predefined models with a large database and research results directly using knowledge engineering and management technologies. The models in this method are used for activity recognition or prediction through logical reasoning. Knowledge-driven approaches are semantically clear and easy to get started. The drawback of this method comes from handling uncertainty and temporal information.

The field of vision-based activity recognition is focused on surveillance, improvement of robots and counter-terrorism, and this field includes a wide variety of options.

Human body structure extraction data from images, action recognition and tracking across frames [99], survey on the approaches based on the movement recognition as opposed to structured approaches [100], research focused on monitoring human movement using 2D or 3D models and the other recognition techniques [101] etc.

In the 2000s, a new approach based on sensors utilizing other sensors fixed to objects was development. This approach has been named "dense sensing" approach with the main purpose of performing activity recognition through user-object interactions. Over the past few years, there have been numerous impressive developments in sensor-based activity recognition [95].

There are two approaches:

Smart sensors are devices able to acquire, process and display data to users. The interconnection between two or more sensors present in the same system is called sensor fusion. This provides a more complex analysis which is not achievable using a single or multiple separate sensor. Sensor fusion combines this data with strategies to provide consistent and effective responses.

In the sensor fusion, two cases are possible. In the first case, sensors with different signals are merged, while in the second case, data is merged from sensors operating in different situations. The sensor fusion has three levels: acquisition and data merger, fusion of characteristics, and merger of decisions.

Figure 10 shows the three levels of a sensor fusion system. Signal types can be physical, chemical or biological quantities or images. Below is description of processing obtained signals. Smart sensors are used when the accuracy of signal processing complexity is not as significant, but different points should be interconnected. The smart sensors must contain a discrete communication system. In this way the sensors are integrated into a sensor network [102].

Fig. 10. Sensor fusion system [102].

In Fig. 11 a single module which includes the acquisition of all physical quantities by the sensor(s) is presented. Generated signals are electronically conditioned by filters, A/D converters, etc., and then they are processed by microcontrollers and/or microprocessors. The stage of communication, using different means in a system with other sensors for post processing elements and analysis of data is followed. The full system can be configured remotely or on the device itself.

Fig. 11. Flowchart of a smart sensor [102]

There is a great interaction between biomedical and sports applications. The physiological, physical and technical types of data can be analyzed in the development of athletes in sport. The physiological variables can be described as power, oxygen consumption, and others, to the physical variables: detach speed, acceleration, and fatigue index, and technical variables: starting time, correct execution of movements, correct gait, posture, etc.

Sensor fusion contains the following sensors: Accelerometer, Gyroscope and Magnetometer. The concepts of sensor fusion and smart sensor can be used in combination.

Athletics is the basis of many sports. For many types of sports, the main thing is running, jumping and throwing. The main thing is to perform an analysis during training and to have an interaction between the trainer and the sportsperson thus giving more help in a competitive environment and in sport. Inertial sensors are widely used in athletics. This category includes an accelerometer, a gyroscope and a magnetometer. They provide data for quantities such as acceleration, angular velocity and magnetic field and provide orientation analysis. When we fusion the data with video signals, we can compare and analyze the performance of two athletes. In order for the coach to provide corrections and instructions to the competitor, it is important to investigate the inertial behavior of the sensors, depending on the time of displacement of two athletes. In this way can be corrected positioning at starting time, starting time and others. It is also important to analyze the gait and the correct execution of movements in training and racing in real time.

In almost all kinds of sport (athletics, figure skating, short track speed skating, hockey, soccer, basketball, etc.), gait analysis is important. The gait depends on the correct position and movement. This analysis can be done by means of sensors for force and inertia in the athlete's footwear. In order to achieve the necessary analysis of the athlete's pace, three sensors measure force, acceleration and angular velocity.

Fig. 12. Smart sensors body placement: (a) ankle; (b) thigh and tibia; and (c) lumbar [102].

In Fig. 12 some examples of area to attached smart sensors are shown: (a) sensor on each ankle, (b) two sensors that are positioned on the thigh and tibia and (c) a sensor placed in the lumbar. Below we will describe the application of the types of sensors in different sports [102].

In the different swimming styles, the turn type and speed and resistance are measured. Two variables in the development of the sport person are important, the resistance and propulsion of the body in water, and also the efficiency of the arms during the movement. Based on these results, an analysis of the style of swimming and assessment of the necessary adjustments is made. In team sports (hockey and football), protection of the athlete is required [103]. Using the same type of sensor application, an analysis of the impacts experienced by the athlete can be made. In [103, 104] concussions and other injuries in the head area caused by impacts, especially in hockey and football are described. The impact monitoring system is built of smart inertial sensors. The sensors transmit impact acceleration, impact time, impact locale, impact direction, and the amount of impacts in sequence. A research impact on the head is important, such as the protective equipment is the helmet. The vest can also be instrumented. There are smart sensors in the hockey stick as well. These sensors analyze athlete movements, force and position of the hands. To analyze the stick movement in the hands during movement, the fusion of three sensors installed on the stick can be used. Inertial sensors at the top of the stick analyze the movements of the stick in the hands and linear potentiometers can be used to analyze the position of the hands on the stick and deflection of the stick at the time of the strike [102].

Figure skating (Fig. 13) is an individual's, duos, or groups sport. The skaters perform on figure skates on ice. This sport includes the disciplines men's singles, ladies' singles, pair skating, and ice dancing. UD biomechanics analysis is used to improve the potential in figure skaters. A behavioral analysis of 60 figure skaters was made. Richards and his research team used an array of 10 cameras that capture data from reflective markers. The markers are placed on the skaters. The cameras capture their precise positions, the speed of their rotation, and their time in the air. The movement in the air are monitored. The figure skaters have to get into their tightest position within a specific time period [105].

Figure skating is an extremely precise sport. Improving each jump requires a lot of work and hours on ice. Minimal displacement of a part of the body is sufficient for an inaccuracy in the performance of the jump and can cause the competitor to fall. Working in this area saves the contestants a lot of falls and makes it possible to see the error and adjust it in time, also to improve and reach a level of triple and quadruple jumps in combinations.

Additional comparison parameters for different environments could be found in [106]. The sensor use in the AAL/ELE platform is shown also in [107] and the position in the use-case scenarios is demonstrated in [108].

Fig. 13. Position: preparation of the competitor for the Lutz

## 6 Conclusion

The integration of RFID and WSNs will bring a higher level of synergy and more technological advances. These integrated networks will extend traditional RFID and sensor systems giving an advantage to control the environment.

An important step toward the wider adoption of identification and sensing technologies would be the implementation of techniques, methodologies, and approaches that are mature enough to be used in a wide range of applications. Nevertheless, it is important to take into account the restrictions posed by the available resources when deploying these tools methods and standards. In addition, it is desirable that the developed solutions would allow their evolvement into technical standards and future integrated platforms.

Acknowledgement. Our thanks to ICT COST Action IC1303: Algorithms, Architectures and Platforms for Enhanced Living Environments (AAPELE).

## References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Semantic Middleware Architectures for IoT Healthcare Applications**

Rita Zgheib1(B) , Emmanuel Conchon2(B) , and R´emi Bastide3(B)

<sup>1</sup> Universit´e de Pau & Pays Adour, E2S-UPPA, LIUPPA, 64600 Anglet, France<EMAIL_ADDRESS>
<sup>2</sup> Universit´e de Limoges, XLIM UMR CNRS 7252, 87060 Limoges Cedex, France<EMAIL_ADDRESS>
<sup>3</sup> Universit´e de Toulouse, ISIS-IRIT, Campus Universitaire, 81104 Castres, France<EMAIL_ADDRESS>
**Abstract.** The adoption of the Internet of Things (IoT) in healthcare has received considerable interest in the past decade. Indeed, IoT-based solutions are poised to transform how we keep people safe and healthy especially as the demand for solutions to lower healthcare costs increases in the coming years. However, the heterogeneity of the *things* that can be connected in such environments makes interoperability among them a challenging problem. Moreover, the observations produced by these things are made available with various vocabularies and data formats. This heterogeneity prevents generic solutions from being adopted on a global scale and makes difficult to share and reuse data for other purposes than those for which they were initially set up. In this book chapter, we provide an overview of the different solutions from both technical and semantic perspectives that have been used recently to tackle the interoperability issue in such IoT environments and especially in healthcare domain. We also present an overview of semantic middleware solutions that have combined the technical and semantic techniques for a complete interoperable solution.

**Keywords:** Semantic architecture *·* Ontology *·* Middleware Internet of Things

## **1 Introduction**

With the evolution of smart connected devices in the last decades, the world keeps asking "How smart will the Internet of Things be?". We all know that sensors are already embedded in all sorts of objects, machines, and things and that many of those sensors are communicating with other machines over the Internet. IoT is a reality today, and its power on improving the quality of life and business is quite remarkable. In this context, building IoT-based healthcare applications provides the possibility to improve people lives. However, IoT in

c The Author(s) 2019

This chapter is an extended version of Chap. 2 of Rita Zgheib's Ph.D.

I. Ganchev et al. (Eds.): Enhanced Living Environments, LNCS 11369, pp. 263–294, 2019. https://doi.org/10.1007/978-3-030-10752-9\_11

healthcare domain presents several challenges related to the fact that almost every week a significant vendor announces a new IoT strategy or division. The proliferation of ad-hoc and specific healthcare products, sensors and applications pose significant challenges about the complexity of designing and managing such systems, to the heterogeneity of the generated data, to the scalability, and the flexibility of the system to support the integration of highly distributed and heterogeneous data and knowledge sources.

In the research area, several Middleware architectures have been proposed in order to manage IoT system components and mediate between sensors and IoT applications. From the other side and beyond traditional management of sensors, some projects have addressed interoperability and scalability challenges in IoT from a semantic perspective. In this trend, ontologies [101] for describing sensors data and metadata as well as describing domain description, have been widely used and found as a reliable technique due to its significant efficiency in ensuring interoperability and clarification of knowledge structure. Middleware and ontologies have mostly been proposed individually. However, IoT environments are evolving rapidly, and the needs for a complete semantic middleware architecture is becoming an essential requirement in IoT environments. Especially in healthcare where the technical with the semantic interoperability promote the reduction of the number of sensors in the person's environment and the development of smarter applications by exchanging analyzed and processed data between all components.

Recently, some projects have tried to combine middleware solutions with semantic techniques in order to have a complete interoperable IoT architecture. Few developers and researchers rely on this combination due to the complexity of designing and integrating their concepts. Our objective is to promote research in combining semantic and middleware solutions while most researches focus on advancing one topic. For that aim, we provide in this chapter an overview of several middleware approaches in Sect. 2. Then, we outline in Sect. 3 the ontologies used to describe sensors and the domain knowledge in IoT. In Sect. 4, we study projects that have combined both approaches in their proposals, we show their characteristics and their advantage regarding traditional architectures for IoT management. Finally, we present the ongoing challenges in providing the most relevant semantic middleware for IoT healthcare applications.

## **2 Middleware Architectures for IoT Systems**

Middleware [11] solutions provide a technical infrastructure that mediates between two or more systems. Their historical role is to ensure the transport of a message from one subsystem to another with a more or less important level of coupling. Researchers find middleware as a suitable solution to fill the gap of heterogeneity and manageability of sensors because it provides an abstract layer positioned between the network layer and the application layer as it is shown in Fig. 1. Moreover, It aims to hide the technological details of communication technologies to enable the application developers to focus on the development of the IoT applications.

## **2.1 Middleware Challenges**

Several middleware architectures have been proposed in the literature trying to address the infrastructure challenges and the applications challenges in IoT. They are regarded as essential problems to be solved before presenting an approach as a final solution.

*The infrastructure challenges* refer to (1) the **interoperability** issue as heterogeneous devices will communicate and exchange information together; (2) the **scalability** issue since a large number of devices are expected to be supported by IoT application; (3) the **spontaneous interactions** of objects and devices; (4) the **diversity of infrastructure** since IoT devices have a specific infrastructure (mobile, wirelessly connected, etc.), and resource constrained which gives the IoT network a **dynamic behavior**; and (5) the need for **abstraction** at many levels such as the physical layer, interfaces, data stream, and development process.

*The application challenges* consist of guaranteeing (1) the **availability** of services and information at all time; (2) a high level of system **reliability** that should remain operational even in the presence of failures; (3) a **real-time** information delivery especially for critical domain such as healthcare; (4) and finally the **security and privacy** as much information is shared between IoT components. This information can be private and even personal such as with information about daily life.

The common goal of all these middleware development initiatives is to develop a framework which can enable an adaptation layer in a plug-n-play mode. Despite that, each middleware architecture [94] focuses on some of the challenges cited before and takes into account some application's requirements which differ from each other's regards to IoT domains they target. In light of the presented challenges, we provide an overview of these approaches emphasizing their characteristics.

### **2.2 Overview of Middleware Approaches for IoT**

Several studies on middleware approaches for IoT have been proposed like the classifications presented in [92,93]. A specific study is presented by Razzaque et al. in [94]. In this paper, authors have analyzed in depth several existing solutions and have presented a recent overview of the different kinds of middleware architecture. Thanks to this study, we have identified the different types of middleware (see Fig. 1) for intelligent environments that have been classified as follows: Application-specific, Agent-based, VM-based, tuple-spaces, databaseoriented, Service-oriented architecture and Message-oriented Middleware.

**Application-Specific:** This approach is based on the needs of a specific application and focuses on resource management. Thus, a strong coupling exists between application and data providers which leads to specialized middleware.

MidFusion [8] represents an example of this middleware approach. It discovers and selects the best set of sensors or sensor agents on behalf of applications.

**Fig. 1.** The Relationship between IoT Data sources and IoT applications via the middleware mediator

It provides a sensor selection algorithm to select the best set of sensors using the principles of Bayesian and Decision theories. Its limitation is that it provides a (Quality of Service (QoS) support only for networks based on Bayesian algorithms. Other middleware in the same category can be found in [60] which focuses on applications in smart homes. It uses application-specific functions to choose a specific context. Given multiple alternatives, one alternative at any time provides the context for all applications, whilst maximizing the applications total satisfaction with the quality of context from the chosen provider. In [55], authors propose MiLAN, a middleware based on tight coupling between the components. It allows applications to specify a policy for managing network and sensors. They argued that the needs of the application should be integrated with the management of the network into a single unified middleware system.

This kind of architecture does not satisfy all IoT middleware requirements due to its tight coupling nature between applications and data providers. It does not address the heterogeneity characteristic of the IoT environment.

**Agent-Based:** Agent or modular based approach [73] consists in applications division into modular programs to promote distribution and injection through the network using mobile agents. Impala [71], Smarty messages [66] and Agilla [46] are examples of this approach. They can be highlighted by providing decentralized systems capable of tackling the availability, reliability, and resource management requirements of middleware. This approach can reduce the complexity of a middleware architecture. However, it presents some limitations related to its inability to perform code management tasks and the unpredictability of agents in the system at runtime.

**VM-Based:** This approach is flexible and contains virtual machines (VMs), interpreters, and mobile agents. The middleware is then composed of two layers. Each physical device is deployed as a VM in the first layer of the middleware. In the second layer, a general VM interprets the modules and delivers data to the application that expresses its needs with a query. This approach addresses architectural requirements such as high-level programming abstractions, selfmanagement, and adaptivity while supporting transparency in distributed heterogeneous IoT infrastructures [58,74]. However, this approach suffers from the overhead that the exchanged instructions introduce.

**Tuple-Spaces:** A tuple space [82] is a data repository that can be accessed concurrently. Each device from the physical layer is represented as a tuple-space in the middleware. All tuple-spaces form a federated tuple space on the gateway. This approach suits mobile devices in an IoT infrastructure as they can transiently share data within gateway connectivity constraints. TinyLime [34] and TeenyLIME [33] are tuple-space middleware solutions for mobile ad hoc networks and sensor networks. Although they have a flexible architecture that allows middleware to be used in different environments, they address frequent disconnections and asynchronous communications problems. However, they offer limitations in resource management, scalability, security, and privacy.

**Database-Oriented:** This middleware approach considers the whole sensor network as a distributed and virtual database. It uses SQL like queries to collect data over the network. GSN [4] is a database-oriented middleware that has been integrated into other projects such as OpenIoT [68]. While this approach offers good programming abstraction and proper data management support, the remaining IoT requirements are not necessarily addressed like scalability, real-time and spontaneous interactions. Moreover, its centralized nature makes it difficult to handle dynamic and heterogeneity characteristics of the IoT network.

The architectures as mentioned above have been used in many specific IoT projects and research area. However, the important use of Service Oriented Architecture (SOA) and Message Oriented Middleware (MoM) solutions is remarkable in IoT projects in the past decade.

**Service-Oriented Architecture (SOA):** Two major trends in the world of IoT have been witnessed in the past years [49]. First, the hardware is becoming smaller, cheaper and more powerful. Second, the software industry is moving towards service-oriented integration technologies. Service-Oriented Architecture (SOA) is a way of thinking and designing the Information System and has been traditionally used in corporate IT systems. The key concept of SOA is the service which is a distributed software invocable that can that can be quickly modified or redeployed in new contexts, allowing applications to respond to changing consumer needs quickly. In IoT approaches based on SOA, intelligent sensors are depicted as services for consumer applications. The key point with these services is their encapsulated nature (i.e., the service interface is independent of the implementation). Service providers describe their services (sensors characteristics) and expose them to consumers. Web Services Description Language (WSDL) is the standard used for such a description [38].

A state of the art of SOA-based middleware solutions for wireless sensor network can be found in [81]. It refers to *SStreaMWare*, *USEME*, *SensorWeb 2.0*, *OASiS*, *B-VIS*, *MiSense*, *SOMDM (SI)*, *SOA-MM* as middleware proposals for wireless sensor networks. Their main features are supporting real-time monitoring, management of heterogeneous devices, data collection and filtering. However, the survey demonstrates that none of these solutions covers all the requirements of the management of sensors network in intelligent environments.

More recent approaches have tried to enhance the SOA features and adapted it to IoT. For example, SenseWrap [45] provides a standardized communication interface to hide the sensor-specific details from applications. It introduces the concept of the virtual sensor to offer transparent discovery of sensors. However, virtualization is applied only to sensors and not to actuators or computing resources which makes it not fully suitable for IoT environments. TinySOA [10] uses simple and deterministic mechanisms for WSN resource (e.g., sensor nodes) registration and discovery. It supports only a few basic functional requirements (e.g., abstraction, resource discovery, and management). SensorsMW [9] has been found as an adaptable and flexible middleware for sensors management. It allows easy and efficient configuration of wireless sensor networks for information gathering. However, the reconfiguration may fail in critical applications as they define strict QoS rules. MOSDEN [87] supports sensing as a service model built on top of GSN. It is based on a plugin architecture which improves the scalability and user-friendliness. However, the predefined resource/service discovery and service composition mechanisms features presented in this approach may be challenging in a dynamic IoT environment.

**Message Oriented Middleware (MoM):** A Message Oriented Middleware has been used for a long time in network communications especially in industrial networks such as integrated manufacturing systems [13,36]. It offers an eventbased architecture and a publish/subscribe communication model. In eventbased architecture, components, applications, and all other participants interact through events. Events are propagated from the sending application components to the receiving application components following the publish/subscribe paradigm. The publish/subscribe model is an interaction model that consists both of publishers and subscribers. Data sources (publishers) and destinations (subscribers) are decoupled from each other and data objects (messages) are filtered and delivered to destinations based on predefined topics expressed as subscriptions thanks to a dedicated component called the message broker as depicted in Fig. 2. The broker can be seen as a mediator responsible for the management of the distribution of messages to serve the right information to the right consumer. The strength of this middleware mainly lies in its support for asynchronous communication allowing a loose coupling between the sender and the receiver.

Since the apparition of the Internet of Things, the publish/subscribe mechanism has been put into the light for its effectiveness in offering loose coupling.

**Fig. 2.** General design model for an event-based middleware.

Compared to the SOA architecture which is also widely proposed for IoT solutions, a MOM follows a message-based distribution model focusing on the information. It differs from the classical client/server paradigm in that neither the source nor the destination of the message has to be known from each other before communication.

Few IoT projects have proposed publish/subscribe solutions in the literature. For instance, CenceMe project [78] aims to automatically infer people's activity (e.g. dancing in the party) based on a sensor-enabled smartphone in order to share this activity through social media like Facebook. Another example supporting easy access to sensor data on mobile phones is Pogo [21], a publish/subscribe middleware infrastructure for mobile phone sensing. It uses simple topic-based subscriptions to manage access to sensor data and reports significant energy gains due to topic-based filtering of sensed data on mobile devices.

On the other hand, a number of Cloud-based services dedicated to storing sensor-based data are nowadays available. Few examples that could be mentioned are Xively [3], ThingSpeak [2] and iDigi [1] which support connections using MQTT (Message Queuing Telemetry Transport) protocol. They represent a scalable infrastructure that enables users to build IoT products and services and to store, share and discover real-time sensor. To sum up, a comparative table is presented in Table 1. In the following section, we outline middleware solutions developed in the healthcare context.

#### **2.3 Middleware Solutions in Healthcare**

Healthcare is an active research topic in IoT where applications can be classified into Medical applications and Ambient Assisted Living (AAL) applications.

**Table 1.** Comparative table of Middleware approaches

Medical applications are mainly installed at hospitals and retirement homes to prevent, detect diseases or even monitor patients health condition. AAL applications are mainly installed at a patient's home to monitor and follow his/her health and activity at home. In this context, middleware solutions have been proposed to fulfill the requirement of IoT and healthcare domains.

– **Middleware for Medical applications:** In this strand we find the *Sphere* project [120] that follows a clustered-sensor approach. It aims to build a generic platform that fuses corresponding sensor data to generate rich datasets that support the detection and management of various health conditions. LinkSmart [85] within the REACTION European project is another SOA-based middleware with the aim of monitoring and managing the diabetes of patients as well as their therapy in operational healthcare environments. It has also been used recently for a smart home application [107]. A middleware for bedsore detection can be found in [86] where authors defined a SOA architecture for bedsores detection and sleep monitoring. The main contribution of this work is about collecting information from wireless and wearable sensors. Other monitoring system proposal can be found in [77] where body sensors are connected to a smartphone via Bluetooth to get information like heart rate and body temperature. MyHealthAssistant [103] is an Event-driven Middleware for Multiple Medical Applications. Its objective is to merge information from several wireless sensors on a Smartphone-mediated Body Sensor Network.

Uranus [31] is a SOA-based middleware architecture for dependable AAL and vital signs monitoring applications. It provides a rapid-prototype for monitoring the level of oxygen in the blood of a chronically ill patient. It also describes another prototype for monitoring patients in the context of a smart hospital. MyHealthAssistant [103] presents an event-driven middleware targeted for medical applications on a smartphone to enable flexible coupling with changing sets of wireless sensor units. Waluyo et al. [114] propose a middleware for medical Body sensor network that supports multiple sensors and applications, plug and play features, and resource management. In that project, however, parts of the middleware and the applications are hosted on a single PC.

– **Middleware for AAL applications:** SM4ALL [115] has been developed and build on top of the OSGi/UPnP standards for building smart homes in the context of European Framework 7 project and Smart Homes for All (SM4ALL) project. It targets Ambient Assisted Living (AAL) environments to monitor disabled elders. VIRTUS [12] is an event-driven middleware based on the standard XMPP (eXtensible Messaging and Presence Protocol). It guarantees a (near) real-time, secure and reliable communication channel among heterogeneous devices. In [43,54], the authors present a platform based on cloud computing to manage mobile and wearable healthcare sensors that demonstrates that the IoT paradigm can be applied to pervasive healthcare. In [30], a SOA based solution offering a distributed telemonitoring system that aims at improving healthcare and assistance for dependent people in their homes. In [110], authors present a pervasive health system integrating patient monitoring, status logging, and social sharing enabling self-management of chronic patients in their environment. Junnila et al. propose in [64] in-Home health monitoring platform with a *common sensor interface* architecture that supports a large number of Zigbee sensors. The platform is tested with two health related case studies: one with an senior woman living in sheltered housing and the other with a hip-surgery patient during his rehabilitation phase.

## **2.4 Discussion**

Middleware proposals based on Application-specific, Agent-based, VM-based, tuple-spaces and database-oriented, have tried to tackle the requirements of manageability and heterogeneity of IoT environments. These approaches have been used in specific IoT applications, and their limitations stated in each case make them not so popular in this environment. Moreover, it is clear that the most used approach in the IoT world is SOA-based architecture, in particular, most healthcare solutions rely on SOA-based middleware. However, we recognized that even if SOA presents a good solution for IoT healthcare applications, it can not meet all IoT requirements especially related to heterogeneity, scalability, real-time processing and flexibility. From the other side, we recognized that although a few recent studies have considered the MoM approach, it has been recognized as a flexible and suitable communication model in IoT. Based on the before mentioned middleware solutions, a comparison between SOA and MoM shows that:

– In regards to SOA architecture, MoM follows a message-based model focusing on the information itself and supports sending and receiving of messages between distributed systems. While in the SOA approach, data providers and data consumers agree on a service contract before data flow sharing as it is evident in Fig. 3, this approach makes the communication not completely decoupled between system components.

– Despite its historical role in sensors management and its potential benefits in IoT middleware solutions, SOA-based solutions focusing on services provided by the system do not scale well in ultra large and dynamic IoT environments. While on the other side, publish-subscribe approaches are gaining more momentum due to their ability to promote scalable, flexible and fully decoupled communication in intelligent environments. As a result, we believe that a MoM with the publish/subscribe paradigm is the most suitable middleware solution for IoT healthcare applications.

To sum up, middleware solutions have been found to address technical interoperability issues and communication requirements in an IoT environment. However, the heterogeneity of connected devices and the variety of data sent over the network introduce the requirement of providing a unified and coherent manner of representing data among system components. Moreover, in healthcare, it is necessary to take into account the patient comfort by reducing the number of sensor around him. This can be done by improving data sharing between IoT application. Although the benefits of middleware solutions in virtualizing the technological details and infrastructure, providing a semantic description would enhance the interoperability and provide a full abstraction of the technical level. In the following section, we present how semantic technologies have been integrated into IoT.

**Fig. 3.** Service Oriented Architecture vs Message Oriented Middleware Architecture

## **3 Semantic Description in IoT**

Different data providers send sensor data in IoT environments and received by many data consumers. Querying sensors and information sharing present a challenge for IoT applications due to the massive number of heterogeneous sensors with specific characteristics. To address this issue, research initiatives and standardization activities have mainly focused on modeling sensors observations and on sending the semantic observations over the network. In [35], different data models have been presented as a coherent and reliable way to share information across an IoT system: XML, Web Service Definition Language (WSDL), JavaScript Object Notation (JSON), Resource Description Framework (RDF) and Web Ontology Language (OWL). RDF and its new version, the OWL format, has become the most used technique for sensor data description due to their ability to ease both reasoning over sensor data and semantic interoperability among devices [51,109].

The Web Ontology Language (OWL) built upon a W3C XML standard is a Semantic Web language designed to represent rich and complex knowledge about things, groups of things, and relations between things. It can easily be integrated with many programming languages like Java through APIs (Jena or OWL-API). Ontologies in IoT have an essential role in representing sensors networks (physical aspect and infrastructure) and also in describing complex domain knowledge and concepts. Furthermore, the main advantage of using ontologies is the possibility to generate OWL representations which provide the opportunity to exchange formal messages at the technical layer in OWL format. Describing data in a formal language such as OWL guarantees the reliability of information and improves the interoperability of the system. Finally, it fills the need to fuse heterogeneous sensor data and potentially inaccurate data. In this section, we present an overview of the several ontologies used to describe sensors in IoT, and we also outline ontologies that describe the healthcare domain.

#### **3.1 Sensors Modeling and Description**

Sensors description has been proposed from one hand to promote reusability and integration of sensors and from the other hand to help to solve the difficulties of installing, querying [7] and maintaining complex, heterogeneous sensor networks. In this field, XML schema has been used for sensors description such as in sensor model language (SensorML) [18] that provides metadata model to describe sensor capabilities and measurement process. They are also used by SOA-based middleware solutions to describe sensors characteristics as a service contract to data consumers. Also, ontologies [101] for sensors description have been widely used as a reliable technique due to their efficiency in ensuring both interoperability and clarification of knowledge structure.

The main advantage of ontologies is their ability to describe three perspectives of a sensor [5]. First, it describes sensors metadata, sensor type, components, configuration, process, and properties. Second, it allows to describe what a sensor can measure after a stimulus is detected or to describe the observation of the sensor in term of frequency, accuracy and measurement capabilities. Third, it provides the concepts to describe data stream and the observation result, allowing semantic reasoning, generating new knowledge and detecting inconsistency in IoT environments.

Key standardization efforts that have sought to establish sensor data models for sensors to be accessible and controllable via the Web include the OGC Sensor Web Enablement (SWE). The SWE efforts established by the Open Geospatial Consortium include the following essential specifications: Observation & Measurement (O&M), Sensor Model Language (SensorML) and Sensor Observation Service (SOS) [19]. The O&M and SensorML contain standard model and XML schema for observations/measurements and sensors/processes respectively. The SOS is a standard service model, which provides a mechanism for querying observation and sensor metadata. Based on these standards, several ontology approaches have been developed to describe sensors. Compton et al. [29] and Bendadouche et al. [15] have surveyed the different attempts aiming at establishing a sensor ontology covering the description of all sensor topics.

In 2008, four potential ontologies have been proposed CESN, SWAMO, A3ME, and ISTAR. The Coastal Environmental Sensing Networks (CESN) [22] project for sensor networks for coastal observing has built an ontology to describe relationships between sensors and their measurements. It also provides logic programming rules reasoner to validate sensor observation and to test anomalous sensor observation by a decision maker. SWAMO [116] describes physical devices and process models and tasks in distributed and intelligent software agents environment. Also, A3ME ontology [57] was developed to classify devices and their capabilities in a heterogeneous network. The ISTAR [47] ontology was developed as part of a system to automatically select sensors for tasks based on their fitness for the task description.

In 2009, OOSTETHYS [16], MMI [100] and CSIRO [84] ontologies were proposed. The OOSTETHYS ontology has been designed to describe sensors and get observation and capabilities for oceanographic observing. Likewise, The Marine Metadata Interoperability (MMI) Device Ontologies Working Group has developed the MMI ontology of oceanographic devices, sensors and samplers. The CSIRO ontology is a generic ontology for describing sensors and deployments. It is intended to be used in data integration, searches, and classification features. It can express complex compositions and finds details of the function and results of sensors and processes. The ontology can also encode much of the information in SensorML documents.

Ontologies developed until 2009 are either specific to a domain (oceanographic, ecology, etc.) or discontinued. Therefore these efforts did not lead to a mature and general solution applied for ongoing projects. None of the ontologies can express all the properties required for a full description of sensors, and a standard is still not available until 2011.

In 2011, the W3C organism proposed SSN, the Semantic Sensor Network Ontology [28]. It has been initiated by the CSIRO, Wright State University, and the OGC as a forum for the development of an OWL ontology for sensors and to further investigate annotation of existing concepts, and links to existing standards. SSN has been designed as a generic ontology which has become one of the most popular and efficient ontology to describe sensors and observations. It has been conceived as a domain-independent ontology where extensions can be made to add domain-specific knowledge.

Extensions of SSN have been developed in the last years by adding concepts related to time, space, communication and concepts related to domain characteristics. We review in this section a broad range of these solutions considering that the presented ontologies are just a small example of existing ontologies that are built on top of SSN. It helps to show the impact of SSN on the semantic description of sensors over the years.

BFO [61] is a spatio-temporal extension that make a distinction between describing identities that happen at a finite time and events like storm and routing. It is a hierarchical system approach where sensors collect data from the real world and send it to clusterhead-node.

In Bandadouche et al. in [15], authors present the limitation of SSN in describing the communication process of a wireless sensor network. They propose a new ontology design pattern Stimulus-WSNnode-Communication, an extension of SSN, that addresses the communication limitation by integrating new concepts that describe the communication process of a wireless sensor network. Another extension for SSN based on fuzzy logic is proposed in [14] to support fault tolerance and for large-scale Wireless Sensor Network. It is a service-oriented approach to build diagnosis and test services for wireless sensors.

Authors in [96] present another SSN extension ontology for WSN, and it is presented as an alignment of many ontologies SSN, SWRLTO, TAO, and DOLCE. This solution aims to improve time descriptions limitation in SSN by representing temporal abstraction to analyze data in real time. SSN is used for sensors measurements; SWRLTO is used for temporal modeling and reasoning; and TAO designed by the authors of this paper to capture the semantic TA (Temporal Abstractions). This framework uses temporal reasoning to search and classify temporal patterns that help to infer the processed data. For the alignment of the three ontologies, the authors use DOLCE, a known upper ontology.

Recently, in [104], authors have analyzed general IoT ontologies: SSN, Smart Appliance REFerence (SAREF)<sup>1</sup>, IoT-ontology<sup>2</sup>, IoT-lite<sup>3</sup>, Spitfire<sup>4</sup>, IoT-S<sup>5</sup>, SA<sup>6</sup> and the oneM2M ontology<sup>7</sup>. They did not consider specific domains impacted by IoT (domotics, agriculture, smart cities...) but they studied ontologies that they found on the web. Based on their comparative study, none of the presented ontologies can describe actuators. Actuators are devices that transform an input signal into a physical output, making them the exact opposite of sensors. They have proposed the IoT-O ontology in order to describe actuators, services and energy consumption.

In October 2017, the last version of SSN ontology was recommended by the World Wide Web Consortium (W3C). The main innovation of this version of SSN has been the introduction of the Sensor, Observation, Sample, and Actuator (SOSA) ontology, which provides a lightweight core for SSN. SOSA aims

<sup>1</sup> http://sites.google.com/site/smartappliancesproject/ontologies.

<sup>2</sup> http://ai-group.ds.unipi.gr/kotis/ontologies/IoT-ontology.

<sup>3</sup> http://iot.ee.surrey.ac.uk/fiware/ontologies/iot-lite.

<sup>4</sup> http://sensormeasurement.appspot.com/ont/sensor/spitfire.owl.

<sup>5</sup> http://personal.ee.surrey.ac.uk/Personal/P.Barnaghi/ontology/OWL-IoT-S.owl.

<sup>6</sup> http://sensormeasurement.appspot.com/ont/sensor/hachemonto.owl.

<sup>7</sup> http://www.onem2m.org/ontology/BaseOntology/.

at broadening the target audience and application areas that can make use of Semantic Web ontologies. With the new version of SSN, actuators and virtual sensors can be easily described which was not possible with the last version.

SSN is commonly used by many projects and is still the most appropriate ontology that can describe an IoT sensor system. Describing physical sensors with SSN promotes the reusability of sensors and makes IoT systems sustainable. Furthermore, the new SOSA ontology integrated with SSN allows having a complete description of physical and virtual sensors that can be deployed in an IoT environment. We recognize virtual sensors as any application that generates information that can not be gathered from the physical world. For example, an application that uses a motion sensor in a living room data and outputs the presence of the person can be seen as a virtual sensor in the IoT world (Fig. 4).

**Fig. 4.** Overview of the SOSA ODP

## **3.2 Ontologies for Healthcare Description**

Domain-specific IoT applications are becoming increasingly popular and, domain knowledge description is gaining even more popularity. Domain knowledge is already defined in more than 200 ontologies and sensor-based projects [52]. These ontologies provide expert knowledge description as well as the conceptualization of the different domains like transportation or diseases. Specifically, domain applications use ontologies and semantic description as a way to define types, properties, and interrelationships of the entities that exist for a specific domain.

As an example, authors in [56] have proposed an ontology to define logistic terminologies. It is intended to assist a logistics expert in identifying and precisely specifying the logistic problem for solving a passenger train optimization problem. In the same context, in [44], an ontology is proposed for monitoring and improving public transportation. Another example of ontologies domain is related to energy concepts as presented in [102] where authors created ontologies to describe energy sources: wind power, biomass, and fossil fuels.  In agriculture, we distinguish AgroPortal ontology for agriculture and nutrition data [63].

In healthcare, many ontologies have been developed to describe both medical and AAL applications.

Other researchers have illustrated the limitations of ontology-based solutions in describing uncertainty. For instance, Authors in [41,97] rely on fuzzy concepts to describe real-world context and uncertainty in activities where several solutions can be deduced based on the same sensors data. In the same context, an ontology for gym addict and lifestyle profiling is presented in [40].

From the other side, there is a set of ontologies that try to tack the concurrent activities. For example, the Knowledge-driven approach for Concurrent Activity Recognition (KCAR) [118] consists on segmentation of sensors data into fragments, each of which corresponds to one ongoing activity.

With respect to other solutions, knowledge-based methods are semantically clear in modeling and representation and highly effective in inference and reasoning. They create a completely accurate model of ADL [95] that enhances the semantic interoperability of an IoT system without a prior learning phase. However, there are still some issues and challenges to tackle in this field such as defining an ontology that take into account the description of actions, activities with uncertainty, concurrent and overlapping activities that have been identified in [59].

## **3.3 Discussion**

In this section, we have discussed the several perspectives of using the semantic description in IoT. We have found that ontologies are the most common semantic technology that describes the three main concepts: sensors, data, and domain.

A well designed ontology enhances IoT environments and architectures by ensuring some features like abstraction, scalability, integration, smart reasoning and interoperability as summarized in Fig. 5. Moreover, an IoT ontology model relies on several design principles: it should be modular to facilitate its evolution, extension and integration with external ontologies; and it should be lightweight to be widely adopted and reusable. Furthermore, an ontology should be a complete reflection of a full description of an IoT environment which reduces the need to import other ontologies. The compatibility principle is related to the needs to be consistent with existing ontologies.

We have shown that SSN is a suitable ontology for sensors description in IoT and is able to generate data in RDF/OWL format. Presenting data in a formal way using RDF/OWL enhances the interoperability and homogeneity of information between IoT systems. In the context of healthcare application, several ontologies have been proposed to describe diseases and activities of daily living. We also found that requirements and challenges are still ongoing in this field as it is an active and recent topic. Moreover, the alignment of sensors ontologies and domain ontologies could be a research aspect of improving.

As we discussed before, integrating middleware approaches with semantic technologies creates a suitable platform and environment for IoT applications. It provides manageability and technical and semantic interoperability in the overall system. For that, it was cumbersome to state the several projects that used these two paradigms.

## **4 Semantic Middleware Approaches**

## **4.1 Overview on Existing Semantic Middleware Architectures**

Existing approaches have considered semantic description in proposing a middleware architecture for IoT applications. SOA and MoM approaches are prominent IoT architectures where semantic enhancements have been added.

**Fig. 5.** Main characteristics of an IoT Ontology

**Semantic SOA-Based Architecture:** Authors in [106] propose an application layer solution for interoperability for IoT aiming to resolve the interoperability issue between different kinds of protocols (Bluetooth and UPnP). The key idea is to use device semantics provided by existing specifications and dynamically wrap them in their SOA middleware into semantic services.

SENSEI [111] is a SOA middleware for IoT. It includes a context model, context services, actuation tasks, and dynamic service composition of services. The main component of this middleware is the resource layer, which stands between the application layer and communication services layer. Resources in SENSEI use ontologies for their semantic modeling.

ubiSOAP [24] is a lightweight service oriented middleware that offers resource management and network level interoperability by supporting heterogeneous networking devices and technologies. Dynamic composition and instantiation of new services are facilitated by the semantically rich models and XML descriptions of sensors, actuators, and processing elements. The lack of context-awareness in ubiSOAP could be an issue, as this is key in the adaptive and autonomous behavior of the things.

SMArc [99] is an acronym for Semantic Middleware Architecture based on SOA. It focuses on smart city energy management for smart grid environments. A light ontology and a data representation under a specific format have been used in the implementation in order to guarantee that the interchanged data uses a representation format which will be common in the system, and therefore information will be easier to extract.

OM2M [6] is an advanced semantic middleware based on SOA architecture. It is a Machine-to-Machine service based on autonomic computing and semantic annotation to provide an inter-operable system to connect billions of devices, but they do not consider real-time analysis and full loosely coupled architecture. Authors proposed the IoT Ontology (IoT-O) [104] for the autonomic management of M2M systems. They extended SSN with four main modules: Acting module to describe actuators, Lifecycle module to model state machines, Service module to describe services and finally the Energy module to represent the power consumption for appliances. However, this ontology does not consider the description of virtual sensors and cognitive processing of information.

Another example is the European project OpenIoT [68] that has developed an open-source middleware platform providing a "cloud-of-things". OpenIoT aims to propose on-demand access to cloud-based IoT services for Internet-connected objects. Trying to use sensing as a service, OpenIoT architecture embeds the CUPUS middleware as a cloud-Based publish/subscribe processing engine and relies on SSN for sensors description. The observed data is stored as linked data and processed based on SPARQL queries which are continuously executed as data arrive. It can be viewed as a federation of several middleware projects interconnected with each other targeting applications for smart cities or campus and agriculture domain. OpenIoT could have been a good candidate to target IoT health applications, but its complexity due to the variety of middleware solutions under use can be a major drawback for the programmer.

The aforementioned middleware projects seek to tackle general IoT requirements without taking into account specific IoT application domains. They evaluated their proposals by deploying IoT sensors and testing the performance in some context however, they present their work as general and applicable to IoT applications. From the other side, Semantic middleware approaches have been proposed in a specific domain such as **healthcare**.

In this context LinkSmart and KASOM are SOA-based middleware for **medical applications**. LinkSmart is SOA middleware [108] developed for diabetes monitoring and therapy in the REACTION project. IT relies on a semantic model-driven architecture and enables the use of devices as services. The semantic description of devices is based on ontologies. KASOM [32] is a knowledge-Aware Service Oriented Middleware proposal for pervasive environments. Its architecture consists of offering services through registration, discovery, composition, and orchestration of services. Most of these services are established on complex reasoning mechanisms and protocols based on a contextual model, which represents a semantic description of low- and high-level resources of the WSN. Real-life implementations in hospital and health management show its potential in terms of response time, efficiency, and reliability. However, KASOM does not provide dynamic service composition in mobile and resource constrained IoT infrastructures because of predefined service composition rules provided by in-network agents.

From **AAL applications** perspective, openAAL [117] is an open source middleware for AAL applications, it relies on SOA architecture as a communication paradigm and on ontology description enabling service discovery. It defines a framework on top of the OSGi specification to facilitate integration and communication among services, including the context manager, procedural manager, and composer. CHOReOS [53] enables large scale choreographies or compositions of adaptable, QoS-aware, and heterogeneous services in IoT. It provides a scalable probabilistic thing-based service registration and discovery to address scalability, interoperability, mobility, and adaptability. It uses ontologies for semantic enrichment over the architecture. The semantic thing based service compositions are transparently and automatically executable with no involvement from end-users, which is highly desirable in IoT, especially in M2M communications.

**Semantic MoM-Based Architecture:** Semantic MoM-based proposals have not been widely discussed in the literature regarding SOA-based proposals. However, new solutions have considered MoM and semantic description in their research work. Namely, SITRUS, EnTimid and xAAL middleware proposals. Bispo et al. propose SITRUS (Semantic Infrastructure for Wireless Sensor Networks) in [17]. Beside a MOM communication model, authors rely on ontology as semantic description processing module whose purpose is to generate a semantic database that provides the basis to decide whether a WSN node needs to be reconfigured or not.

EnTimid and xAAL have been proposed for **AAL applications**. EnTimid [83] is a MOM middleware for smart home monitoring. It relies on a component model for sensors description but it does not address messages or domain description in this contribution. xAAL [72] is another example of MOM middleware, it has been designed in the context of the PRECIOUS<sup>8</sup> European project for AAL. Sensors schema model has been proposed in this work for sensors description and system components exchange Json messages between them.

## **4.2 Overview on Existing Semantic Middleware Platforms**

The previously mentioned approaches have been presented in the literature as semantic middleware solutions. However, other middleware solutions have been introduced through IoT platforms. In [75,79,80] for instance, authors present a large review of 38 contemporary IoT platforms. They described each platform and classified them based on seven characteristics: support of heterogeneous devices, type of the platform, architecture's design, proprietary or open source, support of REST, data access control and service discovery.

As presented in Table 2, the second column shows if the platform supports heterogeneous devices or otherwise what kind of sensors it handles. We can see that most of the described projects support heterogeneous devices. Some solutions like Everyware needs a gateway to manage devices and there are solutions that support only one kind of sensors like Fosstrack that supports only RFID sensors.

We also deduce that in most cases, the platforms are provisioned from a cloud, either in the form of a Platform-as-a-Service (PaaS) or a Software-as-a-Service (SaaS). This type of platforms provides storage facilities, devices management, device connectivity, backup mechanisms or online support.

Table 2 also includes information about the openness of the platforms. Opensource platforms are considered more promising compared with the proprietary

<sup>8</sup> http://www.thepreciousproject.eu/.

#### **Table 2.** Comparative table (Recap) of IoT Platforms [80]

alternatives because they are expected to enable the faster integration of new IoT solutions across the application domains. Among the platforms presented in the table, only 11 are open-source.

Moreover, it may be deduced from the table that only a few platforms do not have a REST API. This demonstrates that the current IoT services will tend to adopt the web of things paradigm [89]. We also deduce that only a few platforms have integrated some type of service discovery mechanisms. A comprehensive survey on discovery protocols for M2M communications can be found in [113].

The presented solutions have not been studied from a semantic perspective. For that aim, we added the last column in the table to show if a platform integrates semantic description or not. We also present the semantic data format when the semantic description is applied. As it is depicted in the table, few platforms have integrated semantic descriptions in their solution. JSON and XML are predominant used formats in both open and proprietary platforms.

Besides the before mentioned platforms, there are some open source solutions that have been developed in the context of home automation and AAL environments. We have studied these solutions and classified them in Table 2 taking into consideration how the semantic description in each solution is addressed.

<sup>9</sup> https://www.kaaproject.org.

<sup>10</sup> https://github.com/paulbellamy/mango.

Commercial IoT platforms have been put into light by technology companies such as Amazon and Microsoft. We summarize some of the industrial IoT platforms in this section.

<sup>11</sup> https://www.nimbits.com/.

<sup>12</sup> https://www.ge.com/digital/predix/industries.

<sup>13</sup> https://www.ibm.com/watson/.

<sup>14</sup> https://aws.amazon.com/.

<sup>15</sup> https://azure.microsoft.com/en-us/suites/iot-suite/.

## **4.3 Discussion**

To sum up, a comparative table is presented in Table 3 with respect to the different perspectives of semantic modeling in IoT: sensors, messages and domain. This table summarizes the open source semantic middleware solutions cited in this section with respect to the two middleware architectures SOA and MOM. Finally, we provide the application domain that each solution has been proposed for.

**Table 3.** Comparative table (Recap) of Semantic Middleware solutions for IoT

As depicted in this table, semantic IoT middleware proposals have been integrated in different domain applications, namely in AAL and home automation, monitoring of elderly and disabled people, smart cities, agriculture, and monitoring of vital and health signs. It is noticeable the predominant choice of SOA architecture as a middleware solution. Moreover, when semantic approaches are proposed they are mainly integrated for semantic descriptions of sensors. But there is no full description including sensors, sensor data and domain.

Moreover, the semantic description in SOA architectures has been proposed in order to describe services which refer to sensors specifications in IoT systems. For example, XML schema has been proposed in SM4ALL [25] for sensors description and used as a contract for applications service description.

Other semantic description techniques can also be noticed like model driven architecture for devices description in [62] and Linksmart<sup>16</sup>. Lately, ontologies have gained sufficient importance for knowledge and sensors description as it is presented in SOPRANO [105], Sensei [91] OpenIoT, SMArc, UBIWARE and OM2M. Another important aspect of this comparison study is the lack of messages and domain description in these propositions, we can see that two ontologybased solutions OM2M and XGSN/OpenIoT have considered OWL format for exchanging messages.

From the other side, only a few attempts address the semantic topic in their proposals and most of these attempts rely on existing SOA-based solutions. So we end up with more semantic SOA-based then MOM-based architectures for IoT. But as previously mentioned, to provide a loose coupling and information centric solution, SOA-based solutions are not so relevant. Therefore, MOM in the past few years has been a topic of interest for IoT researcher and some semantic MOM solutions have been proposed. Entimid, XAAL and SITRUS are MOM-based middleware solutions that have addressed semantic representation in their solution. However, these solutions do not offer a full description model for IoT as it is shown in the table.

## **5 Conclusion and Ongoing Challenges**

IoT has been gradually bringing changes in the healthcare domain. There is innumerable usefulness of IoT applications in enhancing the quality of life of elderly and in helping medical staff. Though IoT has abundant benefits, there are some challenges related to the heterogeneous components in an IoT system that should communicate and exchange data. In this book chapter we have presented several paradigms used to answer these challenges. Some projects have proposed to tackle the interoperability issue from a technological perspective by relying on middleware solutions that promote the interoperability and the manageability of sensors in an IoT system. From a semantic perspective, ontologies have been used to promote the semantic interoperability in IoT systems and to ensure the homogeneity of data formats. The coupling of both technologies has been found as a promising solution for IoT environments. Hence, semantic middleware architectures have been proposed as a complete IoT solution.

Although the existing semantic middleware proposals address many challenges and requirements regarding the interoperability in IoT systems, there are still some open research challenges related to the scalability and real-time reasoning. Using ontologies affects these two requirements as parsing, storing, inferencing and querying of/over OWL/RDF data as well as communicating such data take longer time than working with simple native data. Furthermore, ontologies present a high computational cost and require domain knowledge and expertise leading developers to avoid its use. Furthermore, the complexity related to the diversity of libraries to use and the complexity of the programming environment, make its use more complex. Providing simple software API usable in

<sup>16</sup> http://hydramiddleware.eu/news.php.

various application domains, to alleviate the tasks of software developers who decide to rely on semantic middleware architecture would be an open challenge to investigate. Another challenge that could be addressed in the research area is related to ontologies for sensors and domain description. As pointed out in most projects, these two aspects are described separately. Providing a complete ontology that is able to describe both domain and sensors in IoT is still a challenge to tackle. From the other side, using MoM approaches with semantic description in IoT is still in an early stage. Extensive researches in this context help invention of promising interoperable and scalable IoT architecture that adopt the Web of Things concepts as described in [119].

Web of Thing (WoT) [50] paradigm has been introduced as one of the major solutions for interoperability issue in IoT. It is an improvement of IoT allowing an easier way for IoT applications to build upon smart things. The WoT concept relies on the connectivity service of IoT and easy access to sensors in order to create applications exploiting the IoT data [48]. It can be seen as an evolution of the Internet of things where all components share their information and collaborate to generate advanced knowledge such as wisdom. Moreover, using semantic description allows data contextualization for optimized data stream discovery, indexing and querying. Moreover, new research projects are shifting to the semantic web of things [90] where data can be integrated with data and services available in other information systems. This flexibility eases the production of novel applications and services that are based on the state of the real world. It also supports autonomous semantic reasoning and decision making mechanisms to provide higher-level actionable knowledge from low-level sensor data [39]. Hence, performing semantic reasoning is linked to the ability to define a description model of sensors observations. The WoT is an open research topic to improve and investigate by testing in several IoT environments such as Healthcare.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## The Role of Drones in Ambient Assisted Living Systems for the Elderly

Radosveta Sokullu , Abdullah Balcı , and Eren Demir(&)

Department of Electrical and Electronics Engineering, Ege University, Izmir, Turkey<EMAIL_ADDRESS><EMAIL_ADDRESS>
Abstract. Following years of restricted military applications, recently unmanned aerial vehicles (UAV) also known as drones have become the new disruptive force in many industrial and everyday life applications. Very obvious usage areas are transportation and parcel delivery. However, recent research reveals that there are a number of ways that drones can be employed to help elderly people sustain a better independent lifestyle.

This paper introduces some of the most recent and interesting applications that drones can find in creating ambient assisted living environments for the elderly. Advantages and disadvantages, possible healthcare models and challenges are discussed. Even though these are some very interesting and original applications there are a lot of challenges involved in accepting drones as "flying assistants" to extend the independent living environment of the elderly.

Keywords: Ambient assisted living systems Unmanned aerial vehicles Elderly Drones Daily activities monitoring

## 1 Introduction

Extending lifespan and increased aging population in many countries around the world present a considerable challenge for healthcare services and systems. The two main options – personal care at home or in care in a nursing facility – are very human labor demanding and place a huge financial burden both for the elderly and their relatives, and the healthcare system. Recent advances in networking, computing and sensor technologies have helped address these issues in an efficient and cost-effective way while extending the independent life of the elderly. In the past two decades numerous research projects have focused on various aspects of ambient assisted living systems. (AALS) – from sensor networks to context aware data services – trying to provide endto-end solutions. Today there are many IoT based applications which ensure continuous and often real-time monitoring of the occupant and his surrounding environment, providing assessment and triggering assistance when necessary. AALS is a growing area of research where new enabling technologies help add new features and solve new challenges. Unmanned aerial vehicles, commonly known as drones, are one such technology. For years they have been indispensable for many military applications, but recently they have claimed their place in our everyday lives as well. From weather probing and gaming, to photography and transportation, drones have quickly become the new disruptive force in our society. Could they also add something new to AALS? How will the conservative elderly accept these flying mini robots? There are numerous questions that can come to one's mind.

In this chapter we first introduce the drones – their structure, features, general capabilities and limitations. Then we discuss the main characteristics of AALS, technologies and protocols. The third section presents an overview of existing drone based applications for ambient assisted living followed by a critical analysis and evaluation of drone related technologies and their relationship with ambient assisted living for the elderly.

## 2 Unmanned Aerial Vehicles – Structure and Main Features

## 2.1 Structure and Functionality of Unmanned Aerial Vehicles

Unmanned Aerial Vehicles (UAV), simply known as "drones" can be defined as small size vehicles capable of flying by utilizing air currents and driving forces, capable of autonomous flight, capable of carrying loads and can be controlled remotely. In general, an UAV flight system consists of five basic components as seen in Fig. 1. The first basic component is the set of sensors such as a barometer, a pitot tube, a current/voltage sensor, an IMU (inertia measurement unit), distance, temperature and magnetometer sensors. The barometer sensor provides altitude information by sensing pressure, while the pilot tube is generally incorporated in fixed wing aircrafts and is useful in measuring air velocity. This information is further used for flying at stall speed. Stall speed is the minimum speed the aircraft must fly without losing its altitude. The IMU sensor provides three or six axis gyro stabilization information to the flight controller and it is one of the most important and essential parts of an UAV flight control system. Distance sensor measures the distance between UAV and surrounding obstacles, thus providing information to avoid collision with other objects. Localization and navigation is very important for an UAV. Therefore, in most cases UAVs use magneto sensor and GPS satellites for navigation and localization. However, in closed places like inside buildings and sometimes in very rugged terrain, it is impossible or very difficult for an UAV to navigate because of non-existing or weak GPS signal. Thus, in some application scenarios more robust location system design might be required. One of the most important parameters affecting the wing lifting power of airplanes is pressure and temperature. Therefore, the temperature sensor provides the flight controller with the necessary information for the lift force account.

The second main component is the autopilot which interprets the information from all the other components, keeps the aircraft in balance and allows it to fly. For the motors to reach the desired speed it sends commands to the electronic speed controller (ESC). In addition, the autopilot receives commands from the remote controller and according to the information received it controls the servos and other motors. The third component is the communication unit which is also divided into three sub-components; remote controller, telemetric module and video transmitter. The Remote Controller (RC) unit is capable of transmitting control commands to the UAV. The communication link of the RC unit uses 433 MHz, 2.4 GHz or 5.8 GHz. Each frequency can be

Fig. 1. Basic components of an UAV

used to meet different application requirements. UAVs can fly autonomously or with pilot (human) remote control navigation. In order to ensure safety, UAVs are obliged to be continuously in the RC coverage area, a rule regulated by law. The telemetric module sends the information from the sensors to the ground control station for evaluation. Also, if desired, sensor information can be superimposed on the camera image via the OSD (on screen display) module and sent to the GCS (ground control station) via the video transmitter. The fourth component is the GCS which displays real time data for the UAVs location and performance. It is also used to montitor live stream data generated by the UAV camera. The fifth and last main component is the battery. Lithium polymer (LiPo) is used almost entirely in batteries for UAVs, offering high capacity with low weight and high discharge rates. However, it has some disadvantages such as higher cost and ongoing safety issues.

## 2.2 Classification of UAVs

UAVs are categorized either based on their flight capabilities or on their structure. The categorization is given in Fig. 2. As far as flight capabilities are considered there are three main UAVs which are HALE, MALE and VTOL. HALE (high altitude long endurance) can fly over 9000 m and has long flight endurance whereas MALE (medium altitude long endurance) can fly up to 9000 m. VTOL (vertical take-off and landing) has the ability of vertical take-off and landing. In addition, after rising up to a specific altitude above sea level, VTOL can move to horizontal flight by the action of the propeller. Considering the configuration, the UAVs can be divided in three categories "fixed-wing", "multi-rotor" and "flap wing". Fixed wing UAVs have greater range and endurance when compared to multi-rotor ones, however multi-rotor UAV have the ability to hover and are easier to use. On the flip side, multi-rotor UAVs have greater drain on the battery. Flap wing UAVs are known to experience difficulties in the autopilot design because the wings have to support the control of movement in pitch, yaw and roll direction.

Fig. 2. Unmanned aerial vehicle categorization

In the literature there are a lot of examples related to the applications of UAV and general usage fields can be summarized as: civil transport and transportation, scientific monitoring, geological and ocean surveys, atmospheric research and weather forecasting, surveillance and monitoring (hurricane, volcanic, oil spills, earthquakes, landslides, coast protection, pipelines, water resources, water pollution), international border patrol and drug traffic control, emergency situations (search and rescue, firefighting, disaster operation management, post-disaster site scanning) control of high voltage lines, aerial photogrammetry, detection and analysis of archaeological sites, Earth mapping and 3D city/terrain modeling. Adding to this long list of applications we have also found that very recently drones are finding their place in applications that allow the elderly to live longer full, active lives – both physically and mentally.

## 3 Ambient Assisted Living – Technological and System Architecture Issues

## 3.1 Technologies for AALS

Assisted living technologies have become a very popular research area in the recent years because of providing more helpful and life enhancing approaches to the challenges in daily life of the elderly. As a rule, each assisted technology is specialized according to the application it addresses to perform a specific task. For instance, wearable technology such as eyeglasses, [1], or watches aim to improve the quality of life for people interested in fitness and physical exercises. In a similar way, ambient assisted living technologies address the challenges of an aging population.

A very in depth review is presented in [2], where the evolution of AAL technologies is divided into three categories; first, second and third generation technologies. First generation technologies are usually concentrated on emergency response mechanisms. Solutions consist of an alarm system which can be wearable detection sensors like pressure sensor, gyroscope, etc. to detect mainly falls and other life threatening emergency situations. Thanks to such systems and the notifications they send, the uneasiness of families and caregivers about the security and health of their elderly is greatly reduced. However, the wearable devices can be difficult to accept and cause high stress level for the patient himself. Because the older generation in general is not at ease with AAL, they may take off the wearable devices or damage them. In such cases, the alarm may not be triggered even if the patient is stuck in a difficult situation. The second-generation technologies differ from first-generation in terms of the functionalities. The automated systems, which involve home sensors, automatic response to emergencies, aim to reduce the negative effects of first-generation technology by giving less responsibility to the elderly. For instance, if there is a water leakage or flood in the house, the sensors may detect the hazards and automatically trigger the alarm. The third-generation technologies are the integration of first and second-generation technologies which not only monitor the home, but also monitor the physical activities of the person and possibly his environment. In general they are integrated systems which consist of the home automated system and wearable devices and provide continuous monitoring of the vital signs and activities of the elderly patient, reporting the situation to families and caregivers. Actuators play an important role in this technology to improve the quality of life. Actuators can be medical devices such as smart pillbox, or controlling mechanism such as cooker control. Such technologies highly increase the independence of the elderly patient.

AALS rely greatly on IoT-based technologies that support the connection between different devices over the Internet to enable monitoring and remote assisting. IoT has emerged as the new paradigm describing the world of connected physical or virtual things, devices, or machines equipped with sensors and actuators.

#### 3.2 IoT-Based AAL Architecture

IoT based architectures are generally designed to ensure the connection between the sensing devices and end-user via gateways. Major standardization institutes, such as 3GPP and IEEE, focus on a general architecture based on comprising a three tier IoT structure: sensing layer, network layer, and application layer as shown in Fig. 3 [3]. The lowest layer, comprising different sensor units is responsible for the collection of the data from the environment. All data is transmitted through the network via specific gateways to the application layer/server. However, the evaluation of the data is only done at the application layer where the application server makes a decision and extracts the required information from the collected data.

Fig. 3. IoT-based AAL architecture

## Sensing Layer

Sensing equipment is responsible for gathering data about the environment and its inhabitants. This layer plays a big role in the system and constitutes the lower layer of the architecture. Because it is the immediate connection with the environment, lot of attention should be paid to the accuracy of sensors. Further, the sensors should be aggregated to generate more reliable data. In [4], the sensors for the AAL are divided into two main groups: ambient sensors used in smart environments and wearable and mobile sensors. Ambient sensors are generally used for monitoring the environment in order to track the activity of elderly people which are located in a predefined area. Because the ambient sensors are located in the home area, their main advantage is that they do not affect the mobility and comfort of elderly people. Without being exhaustive a list of such sensors is given in Tables 1 and 2.

Wearable and mobile sensors monitor the vital signs of elderly people which are generally incorporated in another electronic equipment such as mobile phones, smart watches, etc. or attached on human body. This type of sensors is used in Body Area Networks (BAN). While most of the smart phones include an accelerometer, gyroscope sensors, there are also some specific models which are even equipped with pulse sensors to measure the hearth rate. Instead of sticking or attaching the sensors to the human body, researchers concentrate on optical sensors to measure the blood pressure [5], heart rate [6], etc. Such an approach will provide a more comfortable sensor platform than most of today's wearable sensors.

Table 1. Ambient sensors

Table 2. Wearable and mobile sensors

#### Network Layer