higher productivity, better responsiveness, more efficiency and better sustainability. All of these are expressions of the same question, viz. how to improve the performance of health services and health workers? However, measuring healthcare performance presents difficulties because of its multidimensional nature, which can easily lead to conceptual and methodological confusion. As a consequence, there is a scarcity of models which fully analyse performance at healthcare system level. Unsurprisingly, virtually all current performance frameworks include quality of care as a key element, with effectiveness, productivity and efficiency also being recurrent themes. Examples include the World Health Organisation's (WHO) World Health Report 2000, the Organisation for Economic Cooperation and Development's (OECD) framework (2004), and the Nuti's framework (2008). In contrast, social outcomes of healthcare and equity are missing or little developed in most frameworks, with Australian and Canadian national frameworks being notable exceptions. Given this situation, Sicotte et al. (1999) developed the comprehensive Evaluation Globale et Intégrée de la Performance des Systèmes de Santé (EGIPPS) framework for the assessment of the performance of Health Care Organizations (HCOs). Therefore, the main aims of this paper are:

# 2. Key features of the EGIPSS model

In the healthcare sector, one framework stands out: the EGIPSS framework developed by Sicotte et al. (cit.), which is a comprehensive approach to the assessment of performance of HCOs. It includes goal achievement, service production and adaptation to the environment as core dimensions of performance, and usefully adds a focus on values and culture. EGIPSS is geared towards North American settings and has been mainly used in OECD countries. For example, it acts as the basis of WHO-Europe's framework for assessing hospitals, to assess accreditation schemes, to analyse how actors and stakeholders of an HCO define performance and to explore how HCOs learn.

In this paper, the authors present a practical, simplified version of the EGIPSS framework. Keeping the key strengths of this framework, some elements were redefined based on

Pietro Renzi, University of San Marino, San Marino<EMAIL_ADDRESS>0000-0001-6200-7265

153 Alberto Franci, University of Urbino Carlo Bo, Italy<EMAIL_ADDRESS>0000-0002-8157-9792

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Pietro Renzi, Alberto Franci, *EGIPSS model for the evaluation of performance in healthcare*, pp. 167-172, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.32, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

concepts of integrated healthcare systems and public service. Inspiration was found in Parsons' social system action theory to develop an integrative framework of performance, with the performance of a HCO considered to be multi-dimensional. More specifically, it is the result of the interaction between four organisational functions (see Figure 1). Consequently, the success of an organisation depends not only on how each of these functions is organised, but also on how they are aligned with each other. Performance is therefore understood as something more comprehensive than merely efficiently producing desired outputs. Furthermore, it incorporates the managerial approach of the New Public Management (NPM). The framework also describes six equilibriums or alignments between these four functions, which can be best understood as tensions that may arise between the functions as a result of a change in one of them (Figure 1).

The tactical alignment links the Goal Achievement and Service Production function. This deals first with the appropriateness of the service provision in relation to the goals: "To what extent do the service production processes contribute to attaining the goals? Are they effectively producing the output needed to reach the goals?".

The allocative alignment links the Interaction with the environment and the Service Production function. It first deals with resource acquisition. Questions that can be used to assess this include: "Are the obtained resources adequate to organise the service production function? Is the service production function optimal in relation to available resources?".

The strategic alignment examines the link between the Goals that the HCO is pursuing and its Environment. Here, questions include whether the organisational goals correspond with the needs of the population and other key actors.

The legitimating alignment is about the congruence of the Goal Attainment function with the Culture and Values Maintaining function, and questions how the strategic choice of goals influences and shapes the organisational values.

The operational alignment covers the congruence of the Culture and Values Maintaining function with the Service Production modalities, and the impact of the Service Production system on the organisational culture and values.

Finally, the contextual alignment between Culture and Values Maintaining function and Adaptation to the environment deals with how the social, political and cultural dimensions of the environment influence the organisational culture and its core operational values.

Fig. 1: EGIPSS model

Figure 1 below sets out the model:

# 3. Materials and methods

The research made use of various statistical sources (Istituto Nazionale di Statistica (ISTAT), Centro Studi Investimenti Sociali, Osservasalute (CENSIS), Istituto Superiore di Sanità and an array of survey methods. The indicators used in the performance evaluation model were determined through an in-depth study of the existing literature (Sicotte, cit.) and in collaboration with experts from the two locations involved in the study.

Data relating to the Republic of San Marino was provided by its Health Authority, its Istituto per la Sicurezza Sociale (ISS RSM) and by the Office of Statistics of the Republic. The data for the "AreaVasta" of the Marche Region were local and regional statistical sources, plus some internal information sources.

Once the list of indicators was identified, a "balise" (according to French terminology) or benchmarking (according to English terminology) of excellence was determined for each indicator. This represented a norm/guide/method against which results could be compared, thereby enabling opinions and judgements to be formed. The EGIPSS model and methodology incorporates performance indices that enable comparisons to be made based on excellence, which can then be weighted relative to each other within a set of dimensional and subdimensional categories. For example, the Adaptation function covers the dimension of 'Availability of resources' which has two subdimensions of 'Healthcare expenditure and financing' and 'Health workforce' (see tab.2 below). The weights used were based on the original weights provided by the model which were in turn validated by a panel of experts representing the various stakeholders of the healthcare systems studied, chosen according to their skills. This validation process incorporated the DELPHI method (Fabbris et al 2007).

One analytical issue involves establishing the relationship between an indicator and performance. The approach adopted was to determine a balise of excellence for each indicator, that involved values considered to be 'high performing'. The sense of variation in the relationship between an indicator and its associated performance can be positive, negative or parabolic. An overall performance achievement index for a subdimension and dimension is calculated by applying the assigned weights to the calculated percentage of achievement of the balise for each indicator and then aggregating the results. This can be done at each level, on the basis that if the weights are expressed as a percentage their sum within a subdimension, dimension and function must be equal to 100. The process can be repeated for the four functions provided by the model.

Once the percentage of achievement of the balise has been calculated, it is possible to assign a qualitative scale of performance. This serves to add precision, with the values used in this study shown in table 1:

Tab.1: Levels of performance

# 4. Results

This section presents a synthesis and summary of the results derived from the application of the EGIPSS model in the "Area Vasta" of the Marche region and the Republic of San Marino (the latter involves a more reduced version, due to a lack of certain data in local information systems).

Comparisons were made using the indicators relating to the Adaptation function, the Service Production function, Goal Attainment, and the Culture and Values Maintaining function, which are set out in tabs. 2, 3, 4 and 5, respectively. In addition to the above tables the authors sought a helpful diagrammatical presentation that enables the reader to judge the relative performance of different healthcare organisations in terms of 'Strategic equilibrium'. Figure 2 below illustrates this for the relationship between Infant Mortality and Healthcare Expenditure and Financing for the three healthcare organisations studied.

Tab. 3: Service Production function

# Tab. 4: Goal Attainment function

# Tab. 5: Culture and Values Maintaining function

# 5. Weaknesses and strengths of the results obtained

The model provides an overview of the performance, especially for the "Area Vasta" of the Marche, through a prism of 107 indicators. These indicators can present warning signs regarding accessibility, technical quality, efficiency and fairness of a system.

Among the inevitable weaknesses that can affect any frameworks of this nature it should be pointed out that the evaluation of performance is equivalent to comparing the result of an indicator, of a dimension or of a sub-dimension, to a given standard. Where such standards exist they have been used. Otherwise, the performance evaluation was set against external objectives (such as those determined by Canada, the WHO and OECD) or comparisons with the similar results from other countries. The authors' choice on Canada was justified by the fact that this country, by establishing its own empirical standards of excellence, utilised its own comparisons with the EU15 countries (Vrijens et al., 2016). This approach, the only practicable one available, made it possible to position the areas studied in relation to those states that have similar healthcare systems. However, it should be noted that the interpretation of relative performance between micro-areas and States appears very delicate due to the methodological and contextual differences that can compromise the validity of the comparisons. Further constraints were the absence of relevant indicators or the lack of data in the information systems of the two areas.

# 6. Conclusions

Performance evaluation is a process that enables the holistic analysis of healthcare systems utilising measurable indicators. Its role, therefore, is to improve the quality of the decisions being taken by all the staff in the healthcare arena and those services that can impact on people's health. The principles and orientation of the EGIPPS model are useful for assessing healthcare systems at any level, whether it be a country, a province or even a local community.

# 7. References

Nuti S. (2008), La valutazione della performance in sanità, Il Mulino Editore.

Vrijens F., Renard F., Camberlin C. et al. (2016), Performance of the Belgian health system – Report 2015, Belgian Health Care Knowledge Centre (KCE), KCE Reports 259C.

### Yuri Calleoa , Simone Di Zioa , **Unsupervised spatial data mining for the development of future scenarios: a Covid-19 application**

**Unsupervised spatial data mining for the development of future scenarios: a Covid-19 application**

a Department of Legal and Social Sciences, University "G. d'Annunzio", Chieti-Pescara, Pescara, Italy. Yuri Calleo, Simone Di Zio

# **1. Introduction**

In the framework of Future Studies, the development of future scenarios can contribute within the social context by providing inputs at the decision-making level in order to take action in the present. However, this implies an effort and a long-time frame in the first two phases of the scenario development (typically *Framing* and *Scanning*) which require a long desk research, such as reading of documents, research of the scientific literature or the consultation of experts for identifying the key factors (Bishop et al., 2007). In particular, the goal of the scanning phase is to define a number of basic driving forces which constitute the base for the construction of alternative futures scenarios. Some scholars (Kayser and Shala, 2020) estimated an average time of two weeks, in which typically the research team compose a panel aimed at understanding the object of study. Recently, with the exponential growth of social networks, users are constantly in connection with each other, disseminating textual, multimedia, and geographical content on a daily basis. It therefore follows that given the enormous increase in data sources within them and given the communication with which users share ideas, thoughts, and information, all this could be exploited in the context of scenario building.

From the premises made so far, we have developed a new approach that uses unsupervised classification models aimed at speeding up the first two phases of scenario development and optimizing the entire process. To capture the topics and the relevant key factors we used Machine Learning methods, including text-mining (Kayser and Blind, 2017) and Spatial Data Mining techniques. The goal of this work is to provide an answer to the following questions: "Is it possible to obtain information on the object of study by extracting key factors from Twitter?", "Does this approach speed up the Scanning phase?". And, above all, "What contribution can spatial data mining offer to the process of development of future scenarios?". To apply the method, we extracted a dataset from Twitter containing textual and geo-spatial content relating to Covid-19.

# **2. Materials and Methods**

The approach used here applies unsupervised classification models belonging to Machine Learning and aims to extract the major topics within a dataset of tweets, in order to use them as key factors in the scenarios' development process. During the month of November 2020, a dataset of 60.000 tweets was extracted through the use of the Streaming API System using 95 keywords and hashtags related to the discussions on Covid-19 (Uhl and Schiebel, 2017). After extracted the matrix, we proceeded to import it into Python to clean and manipulate it, and then we applied the techniques useful for our analysis (after this phase, the remaining tweets resulted in 29.949). The first step carried out saw the conversion into numbers, better defined as "number vectors" (Atenstaedt, 2012) of the data matrix, through the "lemmatisation" and "tokenization". In the processing of a specific language, the vectors of numbers are determined by textual data, in orderto reflect various linguistic properties of the text, where a coding of the characteristics is necessary (Goldberg, 2017). First of all, we tried to have a qualitative general view of our dataset by applying the text-mining technique using the bag-of-words model that extracts and flexibly represents the data of a given text describing

159 Yuri Calleo, Gabriele d'Annunzio University, Italy<EMAIL_ADDRESS>0000-0002-0190-6061 Simone Di Zio, Gabriele d'Annunzio University, Italy<EMAIL_ADDRESS>0000-0002-9139-1451

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Yuri Calleo, Simone Di Zio, *Unsupervised spatial data mining for the development of future scenarios: a Covid-19 application*, pp. 173-178, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.33, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

the occurrence of words within a document or corpus of documents. The same extracts in a document only the words known and therefore present in a vocabulary assigned to it, while any other information is discarded a priori. We then applied a Sentiment Analysis to understand the degree of polarity of the terms found within the dataset, using two distinct algorithms, called *Vader* and *Afinn,* in order to have a comparison between the two results obtained. We decided to use these algorithms since they are two of the most used in Sentiment Analysis for social networks (cfr. Narasamma et al. 2021; Mayor & Bietti, 2021; Tan & Guann, 2021), and compared to the others, they are able to specifically decipher abbreviations and emojis in the corpus of documents.

They are tools based on lexical rules in relation to what is published mainly on social networks (Hutto & Gilbert, 2014) using a vocabulary of words generally labelled a priori (manually) and subsequently acquired by the model based on their semantic orientation (for example they can be labelled as positive, negative, or neutral). Both algorithms assign a final score based on a sum of the valence scores of the terms in the text and normalized usually between the negative (-1) and the positive (+1) extremes (Huang et al. 2019).

After having a general view of the dataset, in order to understand the most cited terms and their polarity, we used topic modelling to extract possible topics and keywords from the tweets. In this case, we used the Latent Dirichlet Allocation (LDA) (Tong and Zhang, 2016) with the following term frequency function of term :

$$tf(t,d) = \frac{f\_{t,d}}{\sum\_{t' \in d} f\_{t',d}}$$

where , represents the raw count of the term in document .

It is based on a distributive hypothesis of statistical measurement, through the extraction of a series of topics from a corpus of documents. This process is carried out through the mapping of every single document with a good part of the words present (Wang & Grimson, 2007), and the model assigns to each topic a word arrangement determining the key factors.

The LDA assumes that the topics follow a Dirichlet distribution (Minka, 2000), in fact the similarity of documents and topics is controlled by hyperparameters known as and ; if is low it will assign fewer topics to each document, while when is high, we will have the opposite. A low value will use fewer words in the topic modelling process, while a high value will use more words, thus making the topics more similar to each other. The LDA, in fact, does not know a priori the number of topics or terms to be extracted. The model produces a vector that contains the coverage of each topic for the document to be modelled: = (1, 2, … ) where 1 is the coverage of the first argument and so on.

To answer the research questions, we propose an analysis of georeferenced data that will optimize all process by adding important spatial information. Here we use the expression "georeferenced data" in a broad meaning, including any kind of information useful to link a tweet to a geographic object, where the object can be a unit of a vector shapefile layer (like for example a country). Numerous studies have been conducted on Twitter using text-mining or open-mining techniques (Pang & Lee 2008; Taboada et al., 2011; Liu 2012; Poria et al., 2014). Few studies, on the other hand, have focused on the construction of future scenarios starting from the extraction of georeferenced data from social networks. The spatial aspect, in our case, becomes of fundamental importance, as having a geographical view of the subject would benefit the development process.

In the scientific literature, some studies (including Haining, 2010) have highlighted the importance of georeferenced data and therefore the presence of such information in the data (if any) is worth to explore. Actually, through web mining it is not easy to extract spatial information, given that the geographic coordinates (latitude and longitude) are rarely available. The social networks themselves, while previously freely providing quantities of data relating to the positions of users, recently they try to protect themselves by not having such data extracted in a substantial way. In our case, having used a streaming API system extraction, it was possible to model them.

First, we replaced the missing values with the wording "data\_2 NA", after this first step, we have obtained 20.372 tweets with a geographic information included. Subsequently, we linked each tweet to the corresponding country from which it was written by means of information on the location (e.g. village or city), so that, for example, a tweet from Paris is assigned to France. From that, we obtained the frequency distribution of the number of tweets for each country, for each topic and for each key factor permitting to calculate the discussion rate in a single topic and in a single country. These relative frequencies are then reported in a GIS software (Q-GIS Development Team, Open Source Geospatial Foundation Project. http://qgis.osgeo.org.) to create cartograms for the topics, in order to have a representation of the spatial distributions of the same.

# **3. Results and discussion**

The results obtained fully answer the research questions. In fact, it was possible, through the use of text-mining and spatial data mining techniques to extract the influencing factors from our dataset for future scenario development. From sentiment analysis it was possible to measure the polarity of the terms within our matrix, identifying more positive words than negative ones in both algorithms. In support of this analysis, the results are shown in Table 1.

The key factors were extracted through topic modelling which highlighted 5 topics with 6 related keywords (Table 2). The first topic focuses on the *health* aspects, important to understand how the existing pandemic has brought health problems, causing discomfort and death. But beyond the physical aspect, the psychological aspect has also been affected, in fact we can find the presence of the term "anxiety" within the keywords that compose our topic. However, the vaccination uncertainty that persisted in November 2020 should not be underestimated, this aspect is of fundamental importance precisely because it fuelled – and feeds – discussions and conspiracy theories (see topic 3). The second topic describes the *political* aspects, and it is worth noting that the dataset, having English as language and having been extracted during the American elections, was affected by a strong influence of the same. This perspective can be observed from the terms: "government" and "trump". The third topic is reserved for denial and *conspiracy*, as we can see from the words "forced", "reality", "protest" and "planning".

The fourth topic refers us to the *economic* field, in fact governments (see keyword "recession") were suffering for the pandemic. Citizens too (see keyword "employee") – forced to close shops, companies, etc. to prevent the spread of infections, they found themselves in a difficult grip to overcome with consequences at work and personal level. Finally, the fifth topic regards the *social* context, and shows how the pandemic issue has had implications in the social structure. Social distancing adopted by governments prevented the normal development of social activities. Not only that, the spread by governments of applications aimed at tracking movements has also had a debate on social networks, specifically on possible complications and on the possible violation of citizens' privacy. The results are shown in Table 2. After analysing the keywords for each topic, we constructed a cartogram for each of them (Figures 1-5).

Specifically, topic 1 (fig. 1), concerning health, it was carried out in Austria, Brazil, Canada, Greece, Philippines and Turkey. These rates of discussion may be higher than in other countries because during the period studied these countries were experiencing more infections and deaths from Covid-19. As for topic 2 (fig. 2) which analyzed political discussions on Twitter, it was more discussed in American countries, Australia, Germany, South Korea and New Zeland, probably due to the political involvement of the american elections carried out in November 2020. Topic 3 (fig. 3) – which analyzed the conspiracy aspect – sees a multitude of countries involved, take for example Spain, Sri Lanka, New Zeland, China and Pakistan. China, first saw the virus appear in its territory, and subsequently it had to interface with conspiracy theories about the nature of the virus trying to disprove them rapidly. Topic 4 (fig. 4), depicting the discussion rates of the economic topic, was most discussed in Iran, China, Japan, Malaysia and United Kingdom, probably because they were particularly affected by the economic damage that has occurred resulting in a strong response from central governments. The last topic (fig. 5), depicting the discussion of topics of a social nature, finds its foundation in Singapore, Switzerland, Uganda and Sweden. A specific note must be addressed in the analysis of African territories, in fact it is possible to find a strong rate of discussion in some countries such as Nigeria, Uganda, Gambia and Kenya compared to other continents, probably due to the social problems added by the pandemic issue to those already existing.

Since the world scale does not allow to highlight all the details, especially for the smaller countries, in Figure 6 we report the five cartograms with a focus on the European region.

The analysis of georeferenced data has fully answered our research questions, given that the results can be used in the context of futures studies in order to implement the initial process of constructing futures scenarios. This approach provides an effective tool for the development of future scenarios greatly reducing the timing of the *Framing* and *Scanning* phases. Furthermore, it provides a contribution to the future research from a statistical-spatial field and, in particular, in the field of spatial scenarios.

Starting from these results, the scenario planning process will continue with the forecasting phase (Bishop et al., 2007; Hines and Bishop, 2015), which consists of the generation of a sufficient number of alternative futures.

# **4. Concluding remarks**

The approach developed above confirms the possibility of introducing the use of text-mining and spatial data mining within the first two phases of the scenario development (Framing and Scanning). It was therefore possible to extract the influencing factors in a short time frame without any literature review of the object studied and without the consultation of experts. Our study, in addition to providing elements for speeding up the process, enrich the analysis through the spatial component that offers important insights, when it is possible to observe the dynamics on geographical distributions. Understanding in which situations and in which parts of the globe a certain key factor is spoken of, means that much more information is provided. The analysis of Twitter data is only a starting point, in fact, in future studies additional social networks could also be taken into consideration (e.g., Reddit, Facebook, Instagram etc.). Furthermore, it will be possible to analyse much larger datasets in order to have a more complete vision of a given subject.

We recommend that subsequent studies focus on the spatial analysis, too often underestimated in futures studies, but capable of providing important information and, if combined with text-mining techniques, it could lead to an important turning point in the process of scenario and/or spatial scenario development.

It is worth noting that the method proposed in this paper produces spatial data that can be analyzed with the typical tools of spatial statistics. For example, a spatial autocorrelation analysis could reveal similarities between adjacent countries, even if in this case study it was not possible given the very low contiguity of the nations included in the dataset.

# **References**

### Massimo Aria a, Corrado Cuccurullo b, Agostino Gnasso a a Department of Economics and Statistics, University of Naples Federico II, Italy b Department of Economics, University of Campania Lugi Vanvitelli, Italy **Supporting decision-makers in healthcare domain. A comparative study of two interpretative proposals for Random Forests**

**Supporting decision-makers in healthcare domain. A comparative study of two interpretative proposals for Random Forests**

Massimo Aria, Corrado Cuccurullo, Agostino Gnasso

# **1. Introduction**

Today, the availability of data is growing exponentially in all sectors, especially in the healthcare sector. Machine Learning (ML) techniques allow to analyze big data to exctrat knowledge and support healthcare activities (Miotto et al., 2018), such as models for the diagnosis of complex diseases (Dhillon and Singh, 2019), (Aria et al., 2020). Despite the use of ML is spreading in many applications, it is characterized by some limitations and disadvantages.

ML main drawback corresponds to its lack of interpretability which does not allow users to represent causal relationships and interactions between predictors and response. This leads to the inability to learn how particular decisions are made. From this problem derives the definition of the Black Box model, a highly accurate model with a large complexity that cannot be represented by a relational structure. In other words, it is not possible to visualize how it internally works.

Furthermore, the opaque nature of these models hinders application in various sectors, especially in critical ones such as healthcare. To undertake a decision-making process, having faith in a machine learning model is essential, to feel reassured when analyzing and using it.

Ribeiro et al. (2016) identify a different but at the same time-related definitions of trust: trust in a prediction and trust in a model. Trusting a prediction implies that the user will take a certain action based on it; it is important to determine this confidence given that the model will be used to make decisions think for example of the use of a decisionmaking process in the clinical field, the consequence of acting with absolute confidence on the predictions obtained without being able to understand how they are obtained. Having faith in a model is equivalent to evaluating the model as a whole and testing its ability to generalize with appropriate evaluation metrics. A problem that recurs in using data from real contexts is that they are often significantly different and the chosen metric may not be adequate, therefore an inspection procedure of individual predictions and their interpretations may be the optimal choice.

In this work, we pay attention to one of the most used, accurate, and performing models in Machine Learning, the Random Forest model (RF) (Breiman, 2001).

Random Forest is an evolution of Bagging which aims to reduce the variance of a statistical model, simulates the variability of data through the random extraction of bootstrap samples from a single training set, and aggregates predictions on a new record (see Breiman, 1996). Being an evolution of Bagging, Random Forest aims to obtain even more different and unrelated trees. It is known as an efficient ensemble learning model, as it ensures high predictive accuracy, flexibility, and immediacy; it is recognized as an intuitive and understandable approach to the construction process, but is also considered a Black Box model due to the large number of deep decision trees produced within it (Haddouchi and Berrado, 2019).

Massimo Aria, University of Naples Federico II, Italy<EMAIL_ADDRESS>0000-0002-8517-9411

165 Corrado Cuccurullo, University of Campania Luigi Vanvitelli, Italy<EMAIL_ADDRESS>0000-0002-7401-8575 Agostino Gnasso, University of Naples Federico II, Italy<EMAIL_ADDRESS>0000-0002-9220-9754

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Massimo Aria, Corrado Cuccurullo, Agostino Gnasso, *Supporting decision-makers in healthcare domain. A comparative study of two interpretative proposals for Random Forests*, pp. 179-184, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88- 5518-461-8.34, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

The results deriving from the use of the Random Forest are valuable. Various studies have confirmed RF effectiveness in many sectors, such as biomedical for genetic selection (D´ıaz-Uriarte and De Andres, 2006). Breiman et al. (2001) states that Random Forest has A + performance but, having a prediction process that is difficult to understand, evaluates an F on interpretability. This leads to Occam's dilemma (Domingos, 1998) (Domingos, 1999).

The poor interpretability has prevented the adoption of the model in some sectors where there is little or no tolerance for errors, such as healthcare and clinical context (Ahmad et al., 2018). Having set the common goal of interpretability, in recent years the scientific community has fueled considerable interest in Interpretable Machine Learning, which today is an extremely open and active research field with numerous approaches that continually emerge every year (Adadi and Berrada, 2018) (Du et al., 2019) (Guidotti et al., 2018).

This research focuses on the comparison between two approaches proposed in the literature that attempt to overcome the interpretative problem. These approaches, Node Harvest by Meinshausen (2010) and inTrees by Deng (2019), are based on a post-processing interpretation method. They are also defined as Rule Extraction (Haddouchi and Berrado, 2019) approaches as they are focused on the extraction of rule sets. Both proposals use an understandable model based on the rules extracted from a Random Forest. The general idea is to identify a representative weak model to provide the interpretation. This one is selected from the sequence of weak models generated by the ensemble procedure. In particular, Node Harvest selects the set of rules through weights that are assigned based on quadratic programming with linear inequality constraints. Performing this task manages to coincide with two objectives, such as interpretability and accuracy in prediction.

Similarly, inTrees obtain interpretable information through the extraction and processing of rules deriving from a tree ensemble sequence. The extracted rules are used for the realization of a learner, which serves to make predictions on new data.

inTrees works through a series of algorithms that, at first, extract the rules and classify them; subsequently, they carry out a pruning phase on each rule, eliminating the rules that produce background noise or that are irrelevant. Subsequently, these algorithms select a compact set of rules considered relevant and not redundant. Frequent interactions are extracted and finally, everything is summarized in a learner that will be used to make predictions on new data.

# **2. Comparison Study**

We compare Node Harvest and inTrees on four health datasets.

Comparison analysis is performed in an empirical context, where their performance is evaluated using performance metrics. These are obtained from the output and are compared to a reference standard (Aria et al., 2021).

The metrics that evaluate the performance of predictive models, when used for classification, are based on the confusion matrix, which contains the expected and observed class labels, as well as the predicted target category and the source category, as can be seen from Table 1 which represents the structure of a 2x2 confusion matrix.

Regarding comparison, the goal is to compare these approaches through the use of different health datasets. The analysis is conducted on four binary classification health datasets. These datasets are available in the UCI Machine Learning repository. They have different characteristics (see Table 2).

Table 1: Confusion Matrix

Table 2: Main characteristics of the selected health datasets.

The analysis follows the following structure: we proceed with carrying out the random forest for each of the four datasets to obtain the performance of the standard model, in terms of the confusion matrix and prediction of the target variable; the extraction of the set of rules is carried out to investigate the paths taken by each observation, of which the most important and frequent rules of the set itself will also be shown.

Finally, the comparison of the various sets of rules obtained from the two investigated methodologies is performed. The final performance evaluation is conducted through nine parameters obtained from the confusion matrices: Accuracy, Precision, Sensitivity, Specificity, G-Mean, F1 Score, Youden's Index, Balanced Accuracy, Kappa (see Sokolova et al., Garc´ıa et al., Akosa).

Examples are provided of the outputs obtained from the Node Harvest and inTrees approaches. These examples derive from the analysis conducted on Pima Indians Diabetes data: Node Harvest allows you to view the set of rules through an explanatory plot, provided in figure 1, while inTrees allows easy reading through summary tables that show the most frequent rule sets, such as in the table 3.

Table 3: inTrees (STEL) on Pima Indians Diabetes: set of decision rules that are easily applicable to new data. The impRRF value measures the relative percentage decrease in the Gini index for each rule derived from the random forest. The impRRF consider the length of each rule as a proxy of its complexity.

Table 4 shows the nine performance metrics calculated on the four health datasets. The highest score, for each metric, is marked in bold. First of all, the interpretative solutions

Figure 1: Rule set plot obtained from Node Harvest on Pima Indians Diabetes.

proposed by Node Harvest (NH) and inTrees (STEL) represent an understandable approximation that provides an accurate summary of Random forest structure. All datasets show accurate measures very close to the reference value, provided by RF.

Focusing on the comparison, inTrees obtained higher scores in all the analyzed datasets. In particular, for EEG Eye State and Diabetic Retinopathy Debreceen, it shows much higher classification performances. It worth to noting, Node Harvest reports higher scores of sensitivity for all datasets. Maybe, it depends on the fact that this classifier can better recognize positive observations.

# **3. Conclusion**

InTrees represents an excellent strategy for obtaining interpretative learners from Random Forest models.

The results deriving from this methodology are just as good, considering that the simplified rules based on the STEL classifier can be implemented in any programming language.

This work is a starting point for understanding the potential of Interpretable Machine Learning, which requires the development of innovative approaches that can meet the interpretative needs of each application context, such as the healthcare framework. A more complete comparative analysis should focus on analyzing data characterized by unbalanced responses and the presence of missing data (D'Ambrosio et al., 2012), and multiclass responses.

Precision 0.78 0.74 **0.78** G-mean 0.66 0.62 **0.68** F1 0.79 **0.82** 0.79

Youden's Index 0.35 0.35 **0.38**

Table 4: Summary tables on the performance metrics performed on the four health datasets.

# **References**

Precision 0.71 0.64 **0.70** G-mean 0.73 0.68 **0.71** F1 0.74 0.72 **0.73**

Youden's Index 0.47 0.38 **0.43**

pp. 37–43.

*pattern recognition and image analysis*, pp. 441–448. Springer.

Meinshausen, N. (2010). Node harvest. *The Annals of Applied Statistics*, pp. 2049–2072.

Miotto, R., Wang, F., Wang, S., Jiang, X., and Dudley, J. T. (2018). Deep learning for healthcare: review, opportunities and challenges. *Briefings in bioinformatics*, **19**(6).

Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). " why should i trust you?" explaining the predictions of any classifier. In *Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining*, pp. 1135–1144.

Sokolova, M., Japkowicz, N., and Szpakowicz, S. (2006). Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation. In *Australasian joint conference on artificial intelligence*, pp. 1015–1021. Springer.

### **toward misinformation in European countries** Mauro Ferrantea , Anna Maria Parrocob **Media and fake news: An analysis of citizens' attitudes toward misinformation in European countries**

**Media and fake news: An analysis of citizens' attitudes**

a Department of Culture and Society, University of Palermo, Italy b Department of Psychology, Educational. Science and Human Movement, University of Mauro Ferrante, Anna Maria Parroco

Palermo, Italy.

# **1. Introduction**

The rapid changes determined by the rise of Internet and by the recent development of social media in everyday life have led to profound consequences on the quantity and quality of information made available and on the mechanisms of their dissemination. Today, information is increasingly shared through decentralized mechanisms in which social media play a role as a distribution channel, thanks to tools and platforms that enable peer-to-peer sharing mechanisms (Baldacci & Pelagalli, 2017). The rapid spread of on-line misinformation is one of the most-discussed issue today and has been identified as one of the top-trends in modern societies by the World Economic Forum (2013), partly because of the link between these processes and political communication. Among the reasons behind the relevance of this phenomenon, in addition to the already mentioned process of decentralization of the information, it is possible to identify also: the loss of control by the media on the dissemination process, now increasingly determined by algorithms that decide what, when and to whom to show in an unpredictable way; the growing power of Internet giants, such as Google, Facebook, and Twitter; to mention but a few, in deciding who to allow to publish news, what news to show, to whom to show it and how to earn from this process. This because among the scope of on-line disinformation, it is possible to identify the intention of generating interaction on social media, to gain profits from advertising or to discredit someone image (Figueira & Oliveira, 2017). It is therefore important to better understand citizens' attitude and trust toward media, and eventually to identify the potential determinants of different attitudes.

Starting from these premises, the present work aims at analysing the attitude of European citizens toward fake news and disinformation. After briefly discussing the growing literature on fake news and disinformation, by virtue of the availability of micro-data from the Flash Eurobarometer survey on "Fake news and disinformation online" (European Commission, 2018), a segmentation of users is proposed according to their attitude towards different types of media. Secondly, clusters are characterized both in terms of socio-demographic characteristics and in relation to users' behaviour and opinions regarding misinformation. In consideration of the social and political relevance of misinformation, potential strategies to face with fake news and online misinformation are discussed.

# **2. Background**

Fake news and misinformation are not new phenomena. However, starting from U.S. Presidential election in November 2016 a rapid increase in the use of the term "fake news" has been observed (Rose, 2020). Also, terms such as "post-fact" and "alternative facts" emerged in new media communications. These terms are referred to deliberate distortion of the news with the aim of having an influence in public opinion and to exasperate the internal divisions in the society (Martens et al., 2018). This determined a rise in preoccupation for fake news and for their capability in generating confusion among the public. When the term fake news is used, the reference is generally to deliberate fraudulent media products. It is indeed a more severe judgement compared to "biased news". Also, fake news is something different from on-line satire. However, except for striking situations, in many cases it is not easy to identify the border between satire and discredit

Mauro Ferrante, University of Palermo, Italy<EMAIL_ADDRESS>0000-0003-1287-5851

171 Anna Maria Parroco, University of Palermo, Italy<EMAIL_ADDRESS>0000-0003-3213-7805

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Mauro Ferrante, Anna Maria Parroco, *Media and fake news: An analysis of citizens' attitudes toward misinformation in European countries*, pp. 185-190, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.35, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

intention. Allcott and Gentzkow (2017) define fake news as "news articles that are intentionally and verifiably false and could mislead readers" (Allcott and Gentzkow, 2017, p. 213) with facts entirely false. Dentith (2016) points out that fake news is an "allegation that some story is misleading – it contains significant omissions – or even false – it is a lie – designed to deceive its intended audience" (Dentith, 2016, p. 66), with facts that may be entirely false, contain partial truths, or omissions that would undermine the real fact. Fake news and misinformation have attracted the interest of researchers and institutions in identifying the mechanisms of dissemination of fake news and, eventually, potential strategies for their identification (Shao, 2018). The use of social media to get information has even more amplified the fake news issue: the news, due to its bounce rate, is likely to be contaminated, that is to undergo considerable changes until it becomes itself a fake news. It is undisputed that nowadays mainstream media have been progressively displaced by social media as a source of information. Consequently, individuals must be able to select reliable or unreliable information.

Some studies focused on the principal factors causing fake news and they found that both micro and contextual variables act (Kim & Kim, 2020).  They agree that age, education, tech-profile, and cultural and ideological differences among users are relevant variables in shaping the attitudes towards fake news and disinformation. Reuter et al. (2018), referring to a survey conducted in Germany, find that people who are younger or more educated show more ability to identify fake news, and liberal or left-wing persons are more critical; Borges-Tiago et al. (2020) show that citizens attitude towards fake news is different among European countries and report that younger and tech savvy users recognize fake news most likely than others. Quan‐Haase et al. (2018), highlight the importance of information literacy characteristics and information technology skills and Dinev et al. (2009) focus on cultural dimension. Finally, Fletcher et al. (2018), in presenting the results on Italian and French attitude towards fake news, stress the relevance of policy makers, private and public companies in acting to regulate information sources. Nonetheless, few studies have assessed whether populations can be segmented according to their attitude toward media. The present work aims at filling this gap and to assess whether these segments exhibit specific characteristics, both in terms of socio-demographic profile and according to media use.

# **3. Data and Methods**

This study uses micro-data from the European Commission Flash Eurobarometer 464 on "fake news and disinformation online" (European Commission, 2018). The survey carried out in 28 Member States in 2018 on a sample of about 26 thousand respondents interviewed via telephone, aims at exploring EU citizens awareness and attitude toward fake news and disinformation online. Detailed information on the survey, as well as the questionnaire and micro-data are made available by the European Commission through the official portal for European data: https://data.europa.eu.

With the aim of identifying the main determinants of consumer attitudes towards misinformation and fake news, in a first step clusters users have been identified in relation to their attitude toward media, for the six different media types considered (i.e. Printed newspapers and news magazines; Online newspapers and news magazines; Online social networks and messaging apps; Television; Radio; Video hosting websites and podcasts). Secondly, the degree of association of socio-demographic characteristics and of media usage with the proposed cluster is explored in order to characterize different profiles of users across European countries.

In consideration of the categorical nature of data concerning the level of trust in media *k*-mode clustering (Huang, 1997) has been implemented. According to this approach, let A1, A2, …, A6 the set of attributes, describing the categorical space Ω, representing the users' opinion on the different media types considered, where the domain DOM(Aj) of each categorical attribute Aj is given by the three answers' categories, namely: *trust*, *don't trust*, *don't use that media type*. A categorical object X ∈ Ω is represented by the set of attribute-value pair for each of the set of attributes considered, and it can be represented as a vector [*x1, x2, …,x6*]. Let X = {X1, X2, …, Xn} be the set of *n* categorical objects observed in the *n* sample units. We write Xi = Xz if *xi,j = xz,j*, for 1 ≤ *j* ≤ 6, i.e. if two generic sample units, *i* and *z*, have the same value for any of the 6 considered attributes.

The *k*-mode algorithm is an extension of the *k*-means clustering procedure to categorical variables (Chaturvedi et al., 2001) and it aims to partition the objects into *k* groups such that the distance from objects to the assigned cluster modes is minimized. A mode of X is a vector Q = [*q1, q2, …, q6*] that minimises a dissimilarity measure *d*, which is computed by counting the number of mismatches in all variables (simple-matching distance). The *k*-mode algorithm works iteratively by selecting initial *k*-modes of each cluster, allocating each unit to the cluster with the nearest mode according to *d*, then retesting the dissimilarity of units against the current mode and eventually reallocating the units to the cluster with the nearest mode iteratively, until no unit has changed cluster after a full cycle test of the whole dataset (Huang, 1997). In the present work the package *KlaR* implemented in *R* software has been used for the analysis.

Having been identified the clusters which characterise our sample according to their attitude toward different types of media, a regression modeling approach was undertaken to quantify the degree of association of socio-demographic characteristics and of users' behaviour and opinions regarding misinformation with cluster membership. The covariates included in the model were: 1) *Gender,* 2) *Age,* 3) *Occupation,* 4) *Social network use (How often do you use online social networks?),* 5) *Reading and sharing attitude on social network (Do you read or share things when using social network?)* 6) *Presence of fake news and misinformation in the media (Do you come across news which misrepresent reality or are even false?)* 7) *Confidence in the ability to detect fake news (Are you confident that you are able to identify news or information that misrepresent reality or is even false?)* 8) *Perception on the danger of misinformation and fake news (Is the existence of news or information that misrepresent reality a problem in your country or for democracy in general?).* By considering the categorical nature of the response variable, a multinomial logistic regression model was implemented.

# **4. Results**

The dataset under analysis is constituted by 26,576 respondents residing in one of the EU28 countries. As also reported by the EU Commission, at an aggregate level, most respondents tend to trust news and information they receive through radio (70%), television (66%) and printed media (63%). However, less than half (47%) trust online newspapers and magazines, and lower proportions trust video hosting websites and podcasts (27%) and online social networks and messaging apps (26%). Also, these results are consistent across all the Member States (European Commission, 2018). Before implementing the clustering algorithm, those cases containing missing data for at least one of the covariates examined for the analyses were removed. This reduced the dataset to 22,384 cases. Then, to implement *k*-mode algorithm, according to the elbow method, a number of clusters equal to 5 was fixed. Modes of each item corresponding to the attitude for the different types of media considered are reported in table 1.

The results of the *k*-mode clustering procedure highlight different segments of users according to their attitude toward different type of media. Those here called *Impatients* are constituted by users which tend to trust on online social network, radio, and television, whereas they tend to do not trust on printed or online newspapers or magazines. On the contrary, it is possible to define as *Traditionalists* those who trust mainly on traditional sources of information, such as printed or online newspapers and radio. Also, they tend not to trust on social network, television, and they do not use video-hosting websites or podcasts. A particular group of users constitute those who can be defined *Sceptics*, which tend not to trust to any type of media. A fourth group are here named as the *News buff*. They trust on media coming from printed or online newspapers, radio, and television. They are very similar to the *Traditionalists*, except for a trust on television compared to *Traditionalists* who do not. Finally, the last group can be labelled as *Credulous*. They believe in almost any type of media; the only exception being represented by video hosting websites which generally are not used by this type of users.

**Table 1.** Cluster modes according to the attitude toward different types of media.

Table 2 summarizes beta-coefficients, odds ratios(OR) and related *p*-values of the multinomial logistic regression model. From an analysis of the results in Table 2, all the considered factors appear significant, although differences emerge in their effects in relation to the various clusters considered. Conditionally to the other variables, and considering the cluster of *Traditionalists* as baseline, *Gender* is significantly associated to the cluster of *Sceptics*, with a risk of being "Male" of about 1.30 higher compared to the baseline, whereas being "Female" is associated with the cluster of *Credulous* (OR=1.11). In terms of *Age*, being of an age comprised between "25 and 39 years old" and "older than 55 years old", decreases the 'risk' of belonging to the *Impatients*, compared to the other categories. Also being older than 55 years old decreases the risk of belonging to the *News buff*, which tend to be younger compared to the *Traditionalists.* Different occupation profiles characterize the various clusters. Being "manual worker" or "not worker" increases the 'risk' of belonging to the *Impatients*(OR=1.46 and 1.23, respectively); similarly, "not workers" are more likely to belong to the *Sceptics* (OR=1.23), compared to the other occupation categories. Whereas being "self employed" is negatively associated with the cluster of *Credulous* (OR=0.75).  A more active behaviour in terms of *reading or sharing things on social media* characterizes the *Impatients,* the *Sceptics* and the *News buff*, compared to *Traditionalists*. On the other hand, *Credulous* tend not to read or share things on social media, thus indicating a more passive behaviour. *Sceptics,* as expected, tend to come across news which they think misrepresent reality or are even false. The other clusters perceive less this risk, compared to the *Traditionalists.* Nonetheless, the *Sceptics* are less confident on their capability in identifying fake news; the same holds for the *Impatients*, whereas *News buff*  and *Credulous* are more confident from this perspective. Finally, a perception of fake news and disinformation as a problem in the country or for democracy in general characterizes mainly the *News buff*. On the contrary, those who do not perceive this problem are more likely to belong to the *Credulous.*

In summary, the analysis of the segmentation results reported in Table 1, jointly with the results of the logistic regression, suggest that *Impatients* and *Credulous* seems to be those at more risk for fake news and misinformation on-line. Also, they do not perceive misinformation as a problem for democracy in general. In the case of the *Impatients* an active behaviour in terms of on-line sharing emerged, thus potentially determining an active role in the spreading of on-line misinformation, also in consideration that both groups are constituted by regular social network users.

**Table 2.** Multinomial regression coefficients, odds ratios (*Exp(β)*), and p-values. The baseline is the cluster of "Traditionalists" for the response variable.

# **5. Conclusion**

The results of the present work show different attitudes of European citizens towards the media, and this is related not only to socio-demographic characteristics, but also to their behavior and opinions regarding misinformation. In considering the relevance of misinformation and fake news in contemporary times, it is important to identify potential strategies for tackling misinformation. Indeed, the role of countering misinformation is the responsibility of a variety of actors. Policymakers could promote a climate of calm discussion around decision that have to be made. The media could make greater efforts to promote unbiased reporting and ensure high standards of quality. It is incumbent on public institutions to provide support and monitoring misinformation, just as social media should pay more attention to the content disseminated through their platforms, playing a role that increasingly resembles that of a publisher. But a key role is represented by education and training, to act on the side of the final recipients of information and make the effects of misinformation less dangerous.

Reflecting on the limitations of this study and future research, it was not possible to include other potentially relevant information, such as the ones regarding tech-profile and cultural background of users since no information are provided from the Eurobarometer survey. It is likely that these aspects markedly affect users' attitude toward media. Finally, the proposed clusters have not been validated in other contexts, a deeper analysis through other data sources and in relation to different geographical areas is required to investigate the validity of the proposed users' segments.

# **References**

### Matteo Di Masoa , Monica Ferraronia , Pasquale Ferranteb , Serena Delbueb , Federico Ambrogia **Longitudinal profile of a set of biomarkers in predicting Covid-19 mortality using joint models**

**Longitudinal profile of a set of biomarkers in predicting Covid-19 mortality using joint models**

 Department of Clinical Sciences and Community Health, Branch of Medical Statistics, Biometry and Epidemiology "G.A. Maccacaro", University of Milan, Milan, Italy Department of Biomedical, Surgical & Dental Sciences, Università degli Studi di Matteo Di Maso, Monica Ferraroni, Pasquale Ferrante, Serena Delbue, Federico Ambrogi

Milano, Milan, Italy

# **1. Introduction**

a

b

In survival analysis, time-varying covariates (i.e., covariates that are repeatedly measured over time) are endogenous when (i) their measurements are directly related to the event status and (ii) when incomplete information occur at random points during the follow-up because subjects may skip schedule visits and dropout from the study (Rizopoulos, 2012). Consequently, the classical time-dependent Cox model (Therneau and Grambsch, 2000) leads to biased estimates.

In order to correctly estimate the association between a time-to-event outcome and endogenous covariates, two approaches become in widespread use. The first is the joint model (JM) for the simultaneously analysis of longitudinal and time-to-event data (Rizopoulos, 2012). In this approach, the survival sub-model (used to predict hazards for a set of time-invariant covariates) and longitudinal sub-model (used to predict timevarying covariates) are interdependent by means of a set of random effects (i.e., shared parameters). Random effects are individual-specific model terms, and their inclusion in JM provides a way of producing overall predictions. The second approach is the landmarking analysis (van Houwelingen and Putter, 2012), a more pragmatic method that avoids modelling the time-varying covariates. In this approach, the estimated effect of the time-varying covariates is based on the value at the landmark time point, after which values of time-varying covariates may change.

During the first wave of Covid-19 pandemic, physicians at Istituto Clinico di Città Studi in Milan collected a set of inflammatory biomarkers in order to understand what might be used as prognostic factors in progression and mortality of Covid-19 disease. Biomarkers were collected repeatedly over the follow-up. Furthermore, particularly in the first epidemic outbreak, physicians did not have standard clinical protocols for management of Covid-19 disease and for this reason, measurements of biomarkers were highly incomplete especially at the baseline.

The aim of the present study is twice. Using data on Covid-19 patients, we firstly evaluate the association of a single biomarker on Covid-19 mortality using JM, landmarking, and time-dependent Cox model in order to compare estimates. Second, we present JM estimates for the whole set of biomarkers collected on Covid-19 patients to evaluate their association on mortality risk.

177 Matteo Di Maso, University of Milan, Italy<EMAIL_ADDRESS>0000-0002-6481-990X Monica Ferraroni, University of Milan, Italy<EMAIL_ADDRESS>0000-0002-4542-4996 Pasquale Ferrante, University of Milan, Italy<EMAIL_ADDRESS>Serena Delbue, University of Milan, Italy<EMAIL_ADDRESS>0000-0002-3199-9369

Federico Ambrogi, University of Milan, Italy<EMAIL_ADDRESS>0000-0001-9358-011X

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Matteo Di Maso, Monica Ferraroni, Pasquale Ferrante, Serena Delbue, Federico Ambrogi, *Longitudinal profile of a set of biomarkers in predicting Covid-19 mortality using joint models*, pp. 191-196, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88- 5518-461-8.36, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

# **2. Methods**

# *Theoretical framework of JM*

According to the shared parameter approach, the JM consists of two sub-models: one to model the time-to-event outcome (survival sub-model) and the other to model the timevarying covariates (longitudinal sub-model).

The survival sub-model is a typical semi-parametric (or parametric) model for timeto-event outcome. Let \* *Ti* be the true event time for the *th i* subject (with *i N* 1, , ), *Ti* be the observed event time, defined as the minimum of the potential right-censoring time *Ci* and \* *Ti* , i.e., \* min , *T TC i i i* , and let \* *i ii IT C* be the event indicator. Furthermore, let *m t i* be the true and unobserved value of a single time-varying covariate at time *t* . The (proportional) hazards model is:

$$h\_i\left(t^\*\right) = h\_0\left(t^\*\right) \cdot \exp\left\{\beta\_j X\_{ij} + \alpha m\_i\left(t\right)\right\}$$

where *h*0 denotes the baseline risk function, *Xij* is the set of *j* time-invariant covariates measured at baseline for the *th i* subject, *j* is the corresponding vector of regression coefficients, and is the regression coefficient for the time-varying covariate, quantifying the effect of such variable to the event risk.

The longitudinal sub-model is a typical linear mixed model for longitudinal outcome. As information on the time-varying covariate are collected intermittently and with error at a set of few time points for each subject, the aim of longitudinal sub-model is to predict the complete longitudinal history (also called trajectory) of the time-varying covariate (the outcome of the longitudinal sub-model) for a set of time-invariant covariates. In particular, longitudinal sub-model is:

$$\mathcal{Y}\_i\left(t\right) = m\_i\left(t\right) + \varepsilon\_i\left(t\right)$$

where *m t X gZ t i j ij i i* , with *gND i* ~ 0, and 2 ~ 0, *i t N* . The quantity *y t i* is the observed longitudinal outcome for the *th i* subject at time *t* , *j* denote the estimates for the fixed effects *Xij* and *i g* denote the estimates for the random effects *Z t i* . In the shared parameter approach, the random effects are common for the longitudinal and survival sub-models.

Recently, a Bayesian approach for fitting JM was introduced. In particular, estimation of JM's parameters proceeds using Markov chain Monte Carlo (MCMC) algorithm. The posterior distribution of the model parameters is derived under the assumptions that given the shared parameter, both longitudinal and survival sub-models are assumed independent, and the longitudinal outcomes of each subject are assumed independent. In this approach, non-informative priors can be used for explorative purposes.

# *Landmarking analysis*

The idea behind landmarking analysis is to select, for a given time point *LM t s* , all subjects alive and under follow-up at time *s* . In particular, landmarking involves to set *s* and using the value of the time-varying covariate at *s* as fixed covariate in a timedependent Cox model from *s* onwards, in a subset of subjects at risk at *s* . For a generic subject *i*, the objective is to use part of the information of the time-varying covariate of the subject to estimate the conditional probability that the subject is still alive after a predefined time window *w*. More specifically, at a prediction time point *s* , the conditional probability that the subject is still alive at time *w s* conditionally on being alive at time *s* and conditional the history of the time-varying covariate up to *s* is given by:

$$\pi\_i\left(\mathbf{s} + \boldsymbol{\omega} \mid \mathbf{s}\right) = P\left\{T\_i > \mathbf{s} + \boldsymbol{\omega} \mid T\_i \ge \mathbf{s}, m\_i\left(\mathbf{s}\right)\right\}$$

with *m s i* denoting the history of time-varying covariate up to *s* .

# *Data collection*

Between 21 February and 19 March 2020, a total of 403 Covid-19 patients were admitted at Istituto Clinico Città Studi in Milan. Patients aged 21-100 years and 58.3% were men. Person-time at risk was computed as the time elapsed from the day of hospital admission to the day of Covid-19 death (event time), to the day of hospital discharge, or to the day of moving in other structure (right-censoring time), whichever came first. Baseline characteristics included sex and age of patients, whereas biomarkers measurements included ferritin (ng/ml), lymphocytes count, neutrophil granulocytes count, D-dimer (ng/ml), C-reactive protein (ml/l), glucose (mg/dl) and lactate dehydrogenase (LDH; U/l).

# *Statistical analysis*

In order to compare JM, landmarking, and time-dependent Cox model estimates, ferritin was considered. In particular, logarithm of ferritin (log-ferritin) levels was used to account for the skewedness of the measurements. According to the Bayesian approach, independent and non-informative priors for the fixed effects of the longitudinal and survival sub-models (i.e., age and sex) and for the shared parameter (i.e. subject-specific predicted trajectories of log-ferritin level) were used in the JM. In addition, a natural cubic spline with 2 knots was used to model the subject specific log-ferritin trajectories through time and to model age. Two knots are generally sufficient to detect mild non-linear effects and to avoid over-parametrization of the model considering the available sample size.

In landmarking analysis, a set of landmarking time point for log-ferritin time of measurements was considered. In particular, data were analysed with *s* running from 3 to 20 days which corresponded to the median and the 75th centile of log-ferritin time of the first and last measurements, respectively. Prediction windows *w* were set at 7, 14, 21, and 28 days. Age was modelled in the same way as the JM.

In the time-dependent Cox model, observed log-ferritin levels was incorporated as time-varying covariate using a natural cubic spline with 2 knots as well as age.

In order to provide associations between biomarkers and Covid-19 mortality, a JM of each biomarker one at time with the occurrence of Covid-19 death was performed (univariable JM). Logarithmic transformation was considered for ferritin, lymphocytes (log-lymphocytes), neutrophil granulocytes (log-neutrophil granulocytes), D-dimer (log-D-dimer), and C-reactive protein (log-C-reactive protein). In the multivariable JM including all biomarkers (multivariable JM), D-dimer was excluded due to the high number (78; 19%) of patients with missing values. Assumptions for priors, biomarkers trajectories and age were the same as the JM for log-ferritin previously described.

Analyses were performed using JMbayes (Rizopoulos, 2016) and dynpred (Putter, 2015) packages in R Statistical Software, version 4.0.5 (R Core Team 2021).

# **3. Results**

Among 403 Covid-19 patients admitted at Istituto Clinico Città Studi, 140 patients died during the follow-up. Among 263 patients survived, 99 were discharged and 164 were moved in other structures. The median of follow-up was 14 days (range: 0-78 days).

Hazard ratios (HR) and corresponding 95% confidence intervals (CI) from the (biased) time-dependent Cox model and JM for log-ferritin levels (ng/ml) were 2.10 (1.67-2.64) and 1.73 (1.38-2.20), respectively. According to landmarking analysis, the HR was 1.73 (1.25-2.38) for a prediction window of 7 days. With regards to 14, 21, and 28 prediction windows, HRs were 1.86 (1.36-2.54), 1.91 (1.40-2.60), and 1.91 (1.40- 2.61), respectively.

The estimates obtained from univariable JM showed decreased level through time for expected log-ferritin according to the negative coefficients for the splines of time at measurements (table 1). Conversely, the expected level of log-ferritin increased with increasing age and men showed higher expected levels than women. The expected loglymphocytes count increased through time, whereas it decreased with age. No association emerged between log-lymphocytes and sex. The expected log-neutrophil granulocytes count decreased through time, whereas it increased with age and men showed higher levels. Likewise, expected log-D-dimer levels decreased through time, increased with age and men had higher levels. For log-C-reactive protein, expected levels showed a mixed trend through time. In particular, levels initially decreased according to the negative coefficient for the first part of follow-up and increased thereafter. The expected log-Creactive protein levels increased with age and men showed higher levels than women. Expected levels of glucose and LDH decreased through time, while increasing with age and men had higher levels.

In univariable JM, all biomarkers were significantly associated with Covid-19 mortality. An increase in the levels of biomarkers was associated with an increased in the mortality risk, except for lymphocytes. In particular, doubling of levels for loglymphocytes count was associated with approximately halving mortality risk (HR=0.58; 95% CI: 0.46-0.73). The strongest associations were observed for log-neutrophil granulocytes (HR=2.87; 95% CI: 2.30-3.51 for doubling of levels), for log-C-reactive protein (HR=2.44; 95% CI: 2.01-2.97 for doubling of levels) and glucose (HR=2.89; 95% CI: 1.92-4.26 for an increase of 100 mg/dl).

The multivariable JM was estimated using data on 320 patients with 96 (30%) events (after exclusion of patients with missing values for D-dimer). For ferritin and lymphocytes there were no more evidence of association with mortality. The strength of the association was attenuated with respect to the univariable JM for log-neutrophil granulocytes (HR=1.78; 95% CI: 1.16-2.69 for doubling of levels), log-C-reactive protein (HR=1.44; 95% CI: 1.13-1.83 for doubling of levels), LDH (HR=1.28; 95% CI: 1.09- 1.49 for an increase of 100 UI/l), and glucose (HR of 2.44; 95% CI: 1.28-4.26 for an increase of 100 mg/dl).

However, the strongest effect in both univariable and multivariable JM was observed for age with a HR starting to rapidly increase approximately at 60 years.

# **4. Conclusion**

In the present work, we firstly compared HR estimates of a single time-varying covariate (log-ferritin) using different approaches. The HRs from JM and landmarking approaches were lower than that of the time-dependent Cox model. In addition, landmarking estimate for a 7-day prediction window was similar to the estimate of the JM, but it tended to increase increasing prediction window. However, landmarking estimates were lower than the time-dependent Cox model one.

Finally, the multivariable JM model showed associations between some biomarkers and Covid-19 mortality but the strong association between age and mortality risk persisted after adjusted for biomarkers considered.

# **References**

Rizopoulos D. (2012). *Joint Models for Longitudinal and Time-to-Event Data. With Application in R.* Boca Raton: Chapman & Hall/CRC.

**Table 1.** Univariable and multivariable joint model estimates.

### level: A stochastic frontier approach Alessandro Magrini **Assessment of agricultural productivity change at country level: A stochastic frontier approach**

Assessment of agricultural productivity change at country

Department of Statistics, Computer Science, Applications – University of Florence, Italy. E-mail<EMAIL_ADDRESS>Alessandro Magrini

# 1. Introduction

Productivity growth of agriculture is widely recognized as a key resource to meet food demand of the rapidly increasing world population, thus monitoring agricultural productivity change at country level is of core importance for international decision makers. The United States Department of Agriculture (USDA) represents the reference source for agricultural productivity change estimates at country level, covering almost all countries in the world for a long and updated period (from 1961 to 2016). USDA estimates consist of yearly changes in Total Factor Productivity (TFP) based on the growth accounting method, i.e., they are obtained as the ratio between the aggregated output and the sum of the input quantities weighted by their cost shares (Caves *et al.*, 1982). Growth accounting is a widely adopted methodology to assess TFP change due to several advantages, in particular it does not require assumptions on the characteristics of the production processes and allows to consider one decision making unit at a time. However, input cost shares are often partially available, and thus they should be approximated or imputed based on several different sources, like in the case of USDA estimates (see Fuglie, 2015, Table A2), with uncontrollable consequences on the accuracy of estimates. In addition, the growth accounting method has the limitation of assuming that the decision making units operate at their optimal conditions, thus it may overestimate TFP change in presence of technical inefficiency. Frontier-based methods like Data Envelopment Analysis (DEA, Charnes *et al.*, 1978) and Stochastic Frontier Models (SFMs, Schmidt & Sickles, 1984) represent valid alternatives because, by estimating the production frontier from the sample of decision making units, they can distinguish between change in technology and change in technical efficiency, and do not require input cost shares. The main difference between DEA and SFMs is that DEA does not make any assumption on the production frontier, but it is unable to account for random shocks independent of production and, as a consequence, all the deviations from the frontier are attributed to technical inefficiency. Instead, SFMs can disentangle technical inefficiency from external shocks, but they require parametric assumptions on the production frontier. Despite their appealing properties, SFMs and DEA have been employed only in some scattered studies (see the review in Kryszak *et al.*, 2021) and, as such, the available estimates are not comparable with USDA ones.

In this paper, we apply a SFM with translog specification to the same data on agricultural output and inputs employed by USDA, and exploit the generalized Malmquist index proposed by Orea (2002) to derive country level measures of agricultural TFP change, which are then compared with USDA estimates. Our preference for SFMs over DEA relies in the opportunity to account for external shocks and to assess differences in technology across countries, interactions among inputs and the trend of returns to scale.

# 2. Data and methodology

In this study, we employ the same data on which USDA estimates of agricultural TFP change are based (USDA, 2019). These data are sourced to Food and Agriculture Organization (FAO)

Alessandro Magrini, University of Florence, Italy<EMAIL_ADDRESS>0000-0002-7278-5332 FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

183 Alessandro Magrini, *Assessment of agricultural productivity change at country level: A stochastic frontier approach*, pp. 197-202, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.37, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

and to International Labour Organization (ILO), and integrated by modeled estimates. The output variable is the gross agricultural production (Y , thousand US dollars, 2004–2006 average international prices), while the input variables consist of six measures: land use (X1, rainfed cropland equivalents), labour force (X2, economically active adults), livestock (X3, cattle equivalents), machinery stock (X4, 40 CV tractor equivalents), fertilizer use (X5, tonnes of nutrients), animal feed (X6, megacalories of metabolizable energy). The data have annual frequency in the period 1961–2016 and cover the almost totality of countries in the world. In particular, the considered countries account for more than 99.7% of FAO's global gross agricultural output. Some national data have been aggregated to create consistent political units over time (e.g., former Yugoslavia, former Czechoslovakia, Ethiopia plus Eritrea, former Soviet Union) or to avoid very small measurements (e.g., Lesser Antilles, Micronesia), for a total of 170 countries. Futher details and descriptive statistics can be found in Fuglie (2015).

Let i = 1,...,n denote the decision making units (countries) and t = 1,...,T the time points (years). Also, let yi,t be the output level produced by unit i at time t and xi,j,t the level of the j-th input (j = 1,...,p) employed by unit i at time t. A Stochastic Frontier Model (SFM) has the following general form (Schmidt & Sickles, 1984):

$$y\_{i,t} = f(\mathbf{x}\_{i,t}; \Theta) \exp(v\_{i,t} - u\_{i,t}) \qquad i = 1, \ldots, n; \ t = 1, \ldots, T \tag{1}$$

where f is the production frontier, representing the maximum output level technically feasible based on a given combination of the inputs xi,t = (xi,1,t,...,xi,j,t,...,xi,p,t) and a given technology Θ, while vi,t ∈ R and ui,t ∈ R+ are two random errors representing the deviation from the production frontier f due to shocks, respectively, independent of the producer and related to the production. As such, the maximum feasible output may differ from the maximum output level technically feasible due to the occurrence of either favourable or unfavourable events beyond the control of producers. Specifically, the maximum feasible output for unit i at time t is equal to y∗ i,t = f(xi,t; Θ) exp(vi,t), thus technical efficiency is TEi,t = yi,t/y∗ i,t = exp(−ui,t). We employ the following translog specification for f:

$$\begin{split} f(\mathbf{z}\_{i,t};\Theta) = \exp\left(\alpha\_i + \delta t + \gamma t^2 + \sum\_{j=1}^p \beta\_j \log x\_{i,j,t} + \sum\_{j=1}^p \sum\_{k=i}^p \beta\_{j,k} \log x\_{i,j,t} \log x\_{i,k,t} + \delta t\right) \\ + \sum\_{j=1}^p \lambda\_j t \log x\_{i,j,t} + \sum\_{j=1}^p \eta\_j t^2 \log x\_{i,j,t} \end{split} \tag{2}$$

This formulation is identical to the most commonly adopted one in the literature (see the review in Laureti, 2006, Chapter 3, and in Magrini, 2021), with the difference that we added parameters η1,...,ηp to allow output elasticities to vary in time according to a quadratic trend, rather than to a linear one. The frontier specification in (2) leads to the following SFM:

$$\begin{aligned} \log y\_{i,t} &= \alpha\_i + \delta t + \gamma t^2 + \sum\_{j=1}^p \beta\_j \log x\_{i,j,t} + \sum\_{j=1}^p \sum\_{k=i}^p \beta\_{j,k} \log x\_{i,j,t} \log x\_{i,k,t} + \\ &+ \sum\_{j=1}^p \lambda\_j t \log x\_{i,j,t} + \sum\_{j=1}^p \eta\_j t^2 \log x\_{i,j,t} + v\_{i,t} - u\_{i,t} \end{aligned} \tag{3}$$

with εi,t = vi,t − ui,t. We complete the specification of the SFM by assuming:

$$\begin{aligned} v\_{i,t} &\sim\_{i.i.d.} \mathbf{N}(0, \sigma\_V^2) \\ u\_{i,t} &= \phi\_i t + \psi\_i t^2 + U\_i & U\_i &\sim\_{i.i.d.} \mathbf{N}^+(0, \sigma^2) \\ \mathbf{Cov}(v\_{i,t}, U\_i) &= 0 \; \forall i, t \end{aligned} \tag{4}$$

where parameters φi and ψi regulate the second order polynomial trend of the logarithmic technical efficiency of country i, 'i.i.d.' stands for 'independent and identically distributed', N(·) and N+(·) denote the Normal and the half Normal distribution, respectively. This specification for ui,t is the same as in Battese & Coelli (1995) with the addition of the quadratic term.

In order to account for technological gaps among countries with different level of development, we specify four separate models according to the WESP 2020 classification (United Nations, 2020): 'industrialized' (28), 'transition' (22), 'developing' (42), 'least developed' (78). Before estimating the parameters, the time variable is coded as the year minus 1961, thus t = 0, 1,..., 55, and the input variables are divided by their respective sample mean. This allows first order coefficients β1,...,βp to be interpreted as the output elasticity of each input evaluated at the sample mean and at the first time point (year 1961), and makes the output elasticity of the j-th input evaluated at the sample mean and at year s equal to βj+λj (s−1961)+ηj (s−1961)2.

TFP change is assessed through the generalized Malmquist index proposed by Orea (2002), which allows to account for variable returns to scale. Based on this index, the TFP change between two time points s and t (TFPCs,t) is decomposed into technological change (TCs,t), technical efficiency change (ECs,t), and scale change (SCs,t):

$$\text{TFPC}\_{s,t} = \text{TC}\_{s,t} \cdot \text{EC}\_{s,t} \cdot \text{SC}\_{s,t} \tag{5}$$

Orea (2002) showed that, under a translog production frontier, these three terms equate to:

$$\begin{split} \mathbf{TC}\_{s,t} &= \exp\left[\frac{1}{2} \left( \frac{\partial \log y\_{i,s}}{\partial s} + \frac{\partial \log y\_{i,t}}{\partial t} \right) \right] \\ \mathbf{HC}\_{s,t} &= \exp[\mathbb{E}(u\_{i,s} \mid \varepsilon\_{i,s}) - \mathbb{E}(u\_{i,t} \mid \varepsilon\_{i,t})] \\ \mathbf{SC}\_{s,t} &= \exp\left[\frac{1}{2} \sum\_{j=1}^{p} \left( \frac{\sum\_{j=1}^{p} e\_{i,j,s} - 1}{\sum\_{j=1}^{p} e\_{i,j,s}} e\_{i,j,s} + \frac{\sum\_{j=1}^{p} e\_{i,j,t} - 1}{\sum\_{j=1}^{p} e\_{i,j,t}} e\_{i,j,t} \right) \frac{\log x\_{i,j,t}}{\log x\_{i,j,s}} \right] \\ e\_{i,j,s} &= \frac{\partial \log y\_{i,s}}{\partial \log x\_{i,j,s}} \qquad e\_{i,j,t} = \frac{\partial \log y\_{i,t}}{\partial \log x\_{i,j,t}} \end{split} \tag{6}$$

# 3. Results

We performed maximum likelihood estimation of model (3) for each group of countries using the R package frontier (Coelli & Henningsen, 2020). Parameter estimates imply significant and positive output elasticities at the sample mean for the almost totality of time points in all the four models, suggesting consistency with the economic theory. Also, the quadratic component of the trend of output elasticities (parameters ηj , j = 1,...,p) and of logarithmic technical inefficiencies (parameters ψi, i = 1,...,n) are significant, respectively, for most inputs and countries, supporting the adequacy of our model formulation.

Figure 1 displays the time series of the estimated overall elasticity at the sample mean, equal to the sum of all output elasticities at the sample mean by time point. Since the overall elasticity is almost always significantly lower than one for all groups of countries, we deduce that returns to scale are decreasing (and not constant) in the considered period. Based on this result, the assumption of constant returns to scale made by many authors appears just a simplification and not a real property of the production processes of the various countries.

Based on the estimated models, we computed TFPC and its components (TC, EC and SC) with s = t−1 (chained index numbers) and with s = 1961 (index numbers with base year 1961). Table 1 reports average annual percentage changes averaged by group of countries, while Figure 2 displays the time series of index numbers with base year 1961 for a selection of countries. We see that USDA estimates of TFP change are greater in absolute value than ours for most

Figure 1: Time series of the estimated overall elasticity at the sample mean. Shaded areas indicate 95% confidence intervals.

Table 1: Average annual percentage variation of our and USDA's estimates of TFP change averaged by geographical region. The region 'Africa, sub-Saharan' does not include South Africa, while the region 'Oceania' does not include Australia and New Zealand.

geographical regions and periods. Exceptions include North America, Sub-Saharan Africa, East Europe, Central Asia, North Africa and Australia-New Zealand, where our estimates are greater in absolute value than USDA ones, or even discordant, for at least half the periods shown in Table 1. From TFP changes at country level, we note that our and USDA's estimates are in substantial agreement for United States, France, United Kingdom, Australia, South Africa and India, while USDA estimates are very higher than ours for Germany, Italy, Japan, China and Brazil, and moderately higher for Russian Federation and former Yugoslavia. Instead, our estimates are fairly higher than USDA ones for Canada, Afghanistan and Somalia.

The difference between our and USDA's estimates may be due to the presence of techni-

Figure 2: Our and USDA's estimates of TFP change for a selection of countries (indices, 1961=1). The time series of TFPC is shown in blue (TC, EC and SC denoted respectively by straight, dashed and dash-dotted black lines), while USDA estimates are shown in red.

cal inefficiency, that can be taken into account only by stochastic frontier models, but also to inaccuracies in USDA's input cost shares and/or in our model specification. Furthermore, our model is able to detect changes in input use through the term SC, which appears generally non-negligible, coherently with the evidence found in favour of decreasing returns to scale.

To provide an overall assessment on the agreement between our and USDA's estimates, we computed the Person correlation by country and found a median equal to 0.857, with first and third quartile equal to 0.554 and 0.943, respectively. These correlations emphasize that our and USDA's methodology provide different results but in substantial agreement, thus confirming the different theoretical foundations and suggesting the empirical validity of both of them. Full results are available at https://github.com/alessandromagrini/agrTFP.

# 4. Concluding remarks

We have estimated agricultural TFP change at country level based on the same data employed by the United States Department of Agriculture (USDA) using a stochastic frontier model instead of the growth accounting method. This work has the value to provide, for the first time in the literature, a comparison between agricultural TFP changes estimated with different methodologies, and an additional data source that can be employed in a large variety of longitudinal economic analyses at country level.

Our methodology overcomes the limitation of USDA estimates which rely on approximated and imputed input cost shares, and of the growth accounting method in general, which ignores technical inefficiency. However, the accuracy of estimates based on a stochastic frontier are sensitive to model specification. For this reason, we employed a more flexible specification than those adopted in the literature, but, since it is based on deterministic trends, it may be inadequate for long periods, like the one considered in this study. We also paid attention to account for heterogeneity in technology among the various countries by specifying four separate models based on the level of development.

In the future, we plan to improve our methodology by introducing autoregressive coefficients to represent stochastic trends, and by specifying latent classes to account for heterogeneity in technology among the various countries.

# References

### a Bicocca-Applied Statistics Center, University of Milano-Bicocca, Milan, Italy. b Dermatology Unit and Genodermatosis Unit, Genetics and Rare Diseases Research Division, Bambino Gesù Children's Hospital, Rome, Italy c Fondazione REB Onlus, Milan, Italy **Patient-generated evidence in Epidermolysis Bullosa (EB): Development of a questionnaire to assess the Quality of Life**

Pilo c, Gianluca Tadini d

, Carlotta Galeone a, Paolo Mariani a, Cinzia

**Patient-generated evidence in Epidermolysis Bullosa (EB): Development of a questionnaire to assess the Quality of Life**

d Centro Malattie Cutanee Ereditarie, UOC Dermatologia Pediatrica ospedale Policlinico e Università degli Studi di Milano Laura Benedan, May El Hachem, Carlotta Galeone, Paolo Mariani, Cinzia Pilo, Gianluca Tadini

# **1. Introduction**

Laura Benedan a, May El Hachem b

Epidermolysis Bullosa (EB) is a genetic disorder characterised by skin fragility and blistering from mild mechanical trauma. There are four major classical EB types: EB simplex, junctional EB, dystrophic EB, and Kindler EB (Has et al., 2020). All types and subtypes of EB are rare. The overall prevalence of inherited EB in the US is about 11 cases per 1 million live births, and the incidence about 20 per 1 million population (Fine, 2016). Similar results have been obtained in some European countries, Italy included (Tadini et al., 2005).

The clinical manifestations and the severity are very heterogeneous. Physical symptoms include fragile skin that blisters easily, causing pain, itch, and odour; dental problems and blisters inside the mouth and throat, dysphagia, and hair loss. This disease may also present muscle, heart, brain, gastrointestinal, bone, or kidney issues. The physical symptoms significantly impact on daily life and everyday activities and are associated with functional limitations and time-consuming medications that can severely affect the Quality of Life (QoL) of patients and their families. Besides, the disfiguring nature of these symptoms causes an additional burden at the psychological and social level, and the overall EB management may have detrimental financial consequences. The rarity of the disease is an additional issue because there is a lack of awareness and understanding by both laypeople and non-specialist healthcare professionals. Dures and colleagues (Dures, Morris, Gleeson, & Rumsey, 2011) underlined how EB patients' unmet needs were above the medical support. Informational needs, self-management, peer support, social skills and one-to-one therapy emerged as critical themes to be improved.

Considering all the implications of living with EB, a valid and reliable scale to assess the QoL of these patients is essential in patient care and management. The most used instrument available to assess EB patients' QoL, which has proven to be valid and reliable, is the QoLEB questionnaire (Frew, Martin, Nijsten, & Murrell, 2009). It was initially developed in English with an Australian sample, and it was successively translated and validated in other languages (Cestari et al., 2016; Dănescu et al., 2019; Frew, Cepeda Valdes, Fortuna, Murrell, & Salas Alanis, 2013; Yuen et al., 2014).

Even though the translation of this existing tool would have been a valid option, in the present study it was decided to conduct a Delphi study to fully understand the patients' point of view, make their voices heard, and capture possible peculiarities of the Italian context.

A three-stage online Delphi consensus procedure was conducted to identify the key domains and specific statements to assess crucial areas of EB patients' QoL.

1

189

Gianluca Tadini, University of Milan, Italy<EMAIL_ADDRESS>0000-0003-1164-2802

Laura Benedan, University of Milano-Bicocca, Italy<EMAIL_ADDRESS>0000-0003-0427-2487 May El Hachem, Bambino Gesù Children's Hospital, Italy<EMAIL_ADDRESS>0000-0002-8145-4797 Carlotta Galeone, University of Milano-Bicocca, Italy<EMAIL_ADDRESS>0000-0003-1934-5167 Paolo Mariani, University of Milano-Bicocca, Italy<EMAIL_ADDRESS>0000-0002-8848-8893 Cinzia Pilo, REB Onlus Foundation, Italy<EMAIL_ADDRESS>
FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Laura Benedan, May El Hachem, Carlotta Galeone, Paolo Mariani, Cinzia Pilo, Gianluca Tadini, *Patient-generated evidence in Epidermolysis Bullosa (EB): Development of a questionnaire to assess the Quality of Life*, pp. 203-207, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.38, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

# **2. Methodology**

The project started from the request of the Italian no-profit association for EB research and the Italian Registry for EB Foundation (REB) to develop for the first time a patientcentred questionnaire to assess the QoL of patients affected by EB. The methodological process to develop the questionnaire consisted of two phases: firstly, a critical review of scientific literature was performed; secondly, an online pseudo-Delphi study was carried out. The Delphi method is an iterative process where several rounds are organised to identify a shared solution, with useful applications also in health research (Trevelyan & Robinson, 2015). It is a flexible method to determine the gist of the discussed problem when it is not entirely known and when it may be challenging to apply a specific statistical model. The Delphi method consists in envisaging one or more topics to a determined group of experts to provide subsequent evaluations in an iterative process aimed to reach a consensus, which will represent the final expression of the group opinion (Marbach, Mazziotta, & Rizzi, 1991). In this case, the Delphi procedure may be considered "Pseudo-Delphi" because, even though each questionnaire was anonymously analysed and summarised to be presented to the group, the discussions were open, and each participant contributed to the group discussion.

A literature review was conducted to understand what was already known about this pathology and what instruments are used at both a national and international level.

After the problem definition, the expert panel was identified. A multidisciplinary panel including patients, caregivers, and clinicians actively participated in round tables.

The team comprised:

Then, a first group meeting was organised to discuss every step of the project, the main topics to cover, and the primary aim to be achieved. Successively, the patients and clinicians were asked to provide a list of spontaneously generated items to describe different areas of the EB patient's QoL. They worked separately, and all the answers were collected in an anonymous way, allowing every person to freely express their opinions and personal state of mind without any social pressure or external influence. As a result, some powerful statements appeared (e.g. *"Sometimes I think it would be better if I died"*). A total of more than 160 items were created. All answers were carefully considered and grouped within a specific domain. Accurate analysis and harmonisation of all the statements were carried out, in a first attempt to summarise the questionnaire, combine the items with the same meaning, and obtain statements that had a clear value generalisable for the entire reference population. The results were presented in the first Delphi table, i.e., a roundtable session to discuss all the implications of daily living with the disease openly. This group meeting was essential to skim the scopes and find the most salient and relevant assessment in daily practice. On this occasion, great care was taken to ensure a comprehensive and accurate understanding of the experts' points of view.

Hence, the first questionnaire (Q1) was created. This questionnaire also included some items from the literature that were not originally reported during the Delphi roundtable. The questionnaire comprised seven core domains (see Table 1) for a total of 80 items. Each participant was asked to read every statement and assess their degree of importance. They were also required to comment on the clarity and specificity of each item and write any missing information that might have been included. Each expert responded anonymously to the questionnaire and returned it to be discussed in the second Delphi round.

All the answers were carefully examined, and a ranking was created for every item within each domain according to the degree of importance indicated by the participants. The results of this analysis were discussed in the group, and further refinement of the questionnaire was made. Some items were changed or rephrased for greater clarity; others were merged or removed because of their lesser importance.

A new questionnaire (Q2) was defined, considering all suggestions that emerged from the group meeting. The previously identified core domains remained unchanged, but some new items were suggested and inserted. Overall, Q2 was composed of 86 items. At this stage, each participant was asked to rate both the degree of agreement and the degree of importance of each item on a four-point Likert scale (*"Not at all"*, *"A little"*, *"Quite"*, *"Very"*). This step is necessary to remove some irrelevant statements and evaluate the order in which the items are presented. The agreement and importance measures were constructed as satisfactionimportance measures, in line with the widely used Customer Satisfaction techniques.

In addition to the abovementioned seven domains, some specific questions were inserted about the type of EB diagnosed, some socio-demographic information (e.g., age group, the Italian region of residence, the perceived need for psychological support, the perceived satisfaction of the quality of care, etc.). Finally, an overall QoL satisfaction question was asked ("On a scale from 1 to 10, how do you rate your quality of life?").

The results of this phase were presented to the group to define the questionnaire structure further and prepare the new version (Q3) with 85 items, which each participant anonymously filled in. Only one sentence was removed, and some others were modified to be more easily understandable and clear.

It should be noted that, in some cases, a different view emerged between clinicians and patients, and some information learned by the literature were then rejected or modified to be adapted to the language and the experience of the patients (e.g., the terms used to talk about some physical symptoms).

The final version of the questionnaire will be administered to a larger sample to assess its validity and reliability.

# **3. Conclusions**

The present study is part of a more extensive research project aimed at developing a valid and reliable questionnaire to assess the QoL of EB patients. This tool is meant to grasp the point of view and the patient's subjective experience beyond clinical classifications and take into account the patient's overall experience. Starting from an initial set of areas and through the three-round pseudo-Delphi methodology, a gradual refinement of the statements was carried out, and a list of items was defined to be included in an easy-to-use but meaningful patient-centred questionnaire. Each participant had the opportunity to read and fulfil the questionnaire in private, having anonymity assured, allowing free expression of opinions without any social pressure or compliance effect that may conversely arise during the group discussions. On the other hand, knowing all information gathered from the questionnaires and discussing it in the group offered them the opportunity to critically analyse and re-consider all items and areas composing the questionnaire and achieve a final agreement among participants. From a methodological point of view, this approach is worthy in analysing realworld data pertaining to a subjective topic such as QoL, especially in rare diseases. The final patient-centred questionnaire is thus able to measure the QoL beyond the physical symptoms and the clinical evolution of the disease, encompassing functional autonomy, psychoemotional state, social relations inside and outside the family context, the working field and several aspects of the medical care and assistance. The experts approved the final version of the questionnaire after three iterations of anonymous online questionnaire completion and related presentation and discussion of results within the group. The future steps of this research will provide for the assessment of the psychometric properties of the questionnaire to prove its reliability and validity in measuring the QoL of EB patients. This new tool may be a valid aid for clinicians to understand patients better and identify the areas that need more attention; moreover, it may allow them to follow the patients over time and evaluate the impact of any treatments.

# **Acknowledgements**

We are grateful to those who provided valuable contributions to this study, particularly to Simona Buonanno, Claudia Campus, Valeria Manca, Sandra Micich, and Valeria Romiti.

# **References**

# Fabrizio Culottaa **A Prospective Sustainability Indicator for Pension Systems**

**A Prospective Sustainability Indicator for Pension Systems**

a Department of Political Science, University of Genoa, Genoa, Italy. Fabrizio Culotta

# **1. Introduction**

Nowadays, population ageing is a central topic in the political agenda of many OECD countries. It is well known that when the demographic structure of population gradually shifts towards older ages, the pressure on the financial sustainability of the welfare system raises. Health care and pension are those public systems affected the most.

Focusing on PAYG (unfunded) public pension systems, i.e. the first of the multi-pillars architecture for pension systems (Holzmann, 2005), the pressure from an older population is twofold. On the payment side, it increases the proportion of recipients and the duration. On the contribution side, instead, it decreases relative portion of contributors. This pressure can be captured by an indicator for old-age dependency ratio tracing the ratio between pension recipients and pension contributors.

From a pension system perspective, the old-age dependency ratio can be conceived as an indicator for the *extensive margin* of financial sustainability since it traces how many individuals are involved in the pension payments and contributions flow. Clearly, solely considering the extensive margin is not exhaustive. In fact, within the set of indicators for *sustainable pensions* (i.e. the second objective of the Pension strand of the Open Method of Coordination), Eurostat considers indicators for two other types of margins. In particular, the extensive margin is combined with the *intensive margin*, i.e. how much workers contribute to and pensioners receive from the public pension system (as a share of GDP). Note that, from the contribution side, the intensive margin can be further decomposed into the product between pensionable wage and pension contribution rate. Thirdly, Eurostat adopts a durational indicator for the length of postretirement period as well as for the working life. The former traces the effective duration of pension payment, interrupted at individual level because of pensioner's death. The latter proxies the effective duration of pension contribution, often suspended in the case of unemployment (see Bravo and Herce, 2020) or even interrupted in the case of disability. These two indicators are important since they allow to trace a third dimension of the financial sustainability: for how long workers contribute and pensioners benefit. As such, they can identify a *durational margin*.

Considering all three margins not only enriches the set of dimensions to be evaluated, but it also stresses the importance of integration between labour market and pension statistics to analyse more narrowly the sustainability of public pension systems. In this sense, no sustainability indicators for pension systems follows this approach. This work tries to fill this gap by proposing a *Prospective Indicator for the Sustainability of Pension systems* (hereafter, PISP) coherent with the informative system proposed by Eurostat. A pool of European countries is considered as an application, and their ranking assessed. Finally, PISP is compared with an alternative formulation, stressing the contribution of the durational margin, as well as with a benchmark indicator.

# **2. Data and Methods**

Data to construct PISP span over five years, namely the period 2015-2019, and they are extracted from Eurostat database. The pool of European countries is represented by Austria, Germany, Finland, France, Italy, and the Netherlands. The following statistics are selected to construct PISP for each country in the pool (Table 1).

195 FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Fabrizio Culotta, *A Prospective Sustainability Indicator for Pension Systems*, pp. 209-214, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.39, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

Fabrizio Culotta, University of Genoa, Italy<EMAIL_ADDRESS>0000-0002-3910-3088

*Table 1: Statistics used for the construction of ISPS. Source: Eurostat.*

Once statistics are collected, the PISP is constructed as the difference of the product among margins for each flow. Let *i* and *t* index country and time, the PISPi,t can be defined as:

$$PISP\_{i,t} = PC\_{i,t} \cdot WL\_{i,t} - PP\_{i,t} \cdot LE\_{i,t} \tag{1}$$

where PCi,t and PPi,t refer to the pension contributions and payments, as share of GDP, respectively. WLi,t and LEi,t represent the duration, in years of working life and life expectancy at retirement (age 65) respectively. Values of statistics for each country and year are reported in Appendix A (Table 3). Note that to construct PISP there is no need to explicitly consider countryspecific pension parameters of pension systems (e.g. contribution rates, pension formulas).

# **3. Results**

The computed scores of PISP are depicted below and reported for each country (Fig. 1).

*Figure 1: PISP scores across countries and time. Source: author's own elaborations on Eurostat data.*

The Netherlands shows the highest profile in terms of PISP, despite it has decreased in the last years. Germany and Austria report an increasing profile, while those of France and Finland are decreasing. Furthermore, France profile is penalized by a decreasing GDP share of pension contributions (PCi,t). Italy shows the lowest profile, despite it slightly increases after 2016.

The PISP, as defined by equation 1, is now compared to an alternative version that excludes the durational margins from PISP (CISP, standing for Current ISP). Thus, CISP is defined as:

$$\text{CISP}\_{i,t} = \text{PC}\_{i,t} - \text{PP}\_{i,t} \tag{2}$$

For each country in the pool, the time profile of CISPi,t is depicted in figure 3.

*Figure 3: CISP scores across countries and time. Source: author's own elaborations on Eurostat data.*

CISP profiles, being simply defined as the current balance of pension system expressed in terms of share of GDP, show the direct impact of pension contribution and payment flows. All countries but Italy and Finland report positive values, with Finland having the lowest. Overall, three features emerge. Firstly, Italy and Finland maintain the lowest profile across indicators. Secondly, the Netherlands remains at the top positions followed by Germany. Thirdly, from 2017 onwards the financial sustainability of the French public pension system decreases. Explanations rely on the dynamics of each component forming the contribution and payment side.

Finally, the Mercer-Melbourne Global Pension Index - Sustainability (GPIi**,t**) is taken as a benchmark (Mercer-Melbourne, 2015-2019). Countries profiles are reported in Figure 4.

*Figure 4: GPI scores across countries and time. Source: Mercer-Melbourne (2015, 2016, 2017, 2018, 2019).*

 Despite GPI is an index for the sustainability of the whole (i.e. all pillars of) pension systems, it is possible to note that the Netherlands are confirmed to be ranked first throughout the considered period. Finland, ranked second, shows a stable pattern. Germany and France, placed third and fourth respectively, evolve along the same trend. Lastly, Austria and Italy show a positive trend, with the latter country having the lowest profile for the whole period.

 PISP and CISP are then compared each other as well as to the benchmark indicator GPI. For each pair of indicators, their (Pearson) correlation coefficient ρ is measured. Results are reported below (Table 2).

*Table 2: Correlation (Pearson) across pair of indicators: PISP, CISP and GPI. Source: author's own elaboration.*

 The first column shows that the correlation between PISP and CISP is quite high for all countries (on average 0.95), with the lowest dispersion across countries (0.06) if compared to the other two columns. This result can be interesting if one considers that countries have different pension regimes, namely different rules determining pensions contribution and payment flows. The last two columns of Table 2 report the correlation between PISP and CISP with the benchmark GPI. Firstly, note that in the case of France it is negative in both cases. This means that GPI may underestimate the impact of current imbalances on the overall sustainability of public pension systems. Secondly, correlation is very high for countries like Germany, Austria, Finland and Italy. On the other side, the proposed indicators are not informative about the overall sustainability of France and the Netherlands as measured by GPI. Overall, PISP reveals some desirable properties. In fact, not only the average correlation with current imbalances is higher for PISP (0.95) than for GPI (0.51), but it is also the least dispersed across countries (0.06 and 0.64, respectively).

# **4. Conclusions**

This work proposes an indicator for the financial sustainability of public pension systems. The novel lies in the consideration of durational margin for pension contribution and payment sides, namely the duration of working life and the life expectancy at retirement. In doing so, it explicitly combines both labour market and pension statistics in a unifying indicator, thus stressing their interplay. These novels are coherent with the recent focus posed by Eurostat within the set of indicators selected to monitor the sustainability of pension systems.

The proposed indicator PISP satisfies some properties which are desirable in the context of financial sustainability of public pension systems: highly correlates with current imbalances and the benchmark GPI and it is the least dispersed across countries. On the contrary, GPI shows a weaker correlation with current imbalances and negatively correlates in the case of France. This consideration opens the need towards a solid and reliable indicator which meaningfully track the financial performances of pension systems. The structure of PISP can be conceived as a first step to be further developed towards an informative system reliable and comprehensive for policymakers (Whitehouse, 2012). The challenge becomes of utmost importance in a context of an ageing society, where the proportion of retired people increases and accordingly the importance of pensions.

# **References**

Mercer, Melbourne (2015). *Melbourne Mercer Global Pension Index*.

Mercer, Melbourne (2016). *Melbourne Mercer Global Pension Index*.

Mercer, Melbourne (2017). *Melbourne Mercer Global Pension Index*.

Mercer, Melbourne (2018). *Melbourne Mercer Global Pension Index*.

Mercer, Melbourne (2019). *Melbourne Mercer Global Pension Index*.

Whitehouse, E. (2012). Pension indicators: reliable statistics to improve pension policymaking (No. 70347, pp. 1-12). *World Bank Publications.*

# **APPENDIX A: Dataset**

*Table 3: Dataset for the construct PISP and CISP. Source: Eurostat.*

### Illya Bakurova , Fabrizio Culottab **Unemployment dynamics in Italy: a counterfactual analysis at Covid time**

**Unemployment dynamics in Italy: a counterfactual analysis at Covid time**

a Information Management School, NOVA University of Lisbon, Lisbon, Portugal. b Department of Political Science, University of Genoa, Genoa, Italy. Illya Bakurov, Fabrizio Culotta

# **1. Introduction**

The experience of the Covid pandemic has revealed the importance of statistical monitoring systems. When the phenomenon of interest is evolving, information about past and future dynamics becomes fundamental to assess both effects and future trajectories. This is true especially during the recovery and post-recovery phases.

The various measures undertaken in each country to contain the spread of the virus moved towards the reduction of physical interaction and, a fortiori, gathering of people. The underlying uncertainty, in the Knightian sense, forced governments of almost all countries to impose harsh remedies. The freedom of movement has been suspended for a while. Undoubtedly, the onset of Covid was an unprecedented and unexpected shock for the world population and, thus, for the whole economy. In this, Italy can be considered a case of study.

Focusing on labour markets, from one side, a substantial drop in unemployment has been observed in Italy during the year 2020 (Fig. 1). The conditions of active search and (immediate) availability to work, whose simultaneous fulfilment identifies an unemployed individual, were not met. Accordingly, a consequent rise in inactivity occurred. From the other side, the Italian government introduced a ban on dismissal operating throughout the year 2020. The goal was to preserve the employment level avoiding firms to fire workers massively. In doing so, the level of employment was forced not to drop. Overall, this was the extraordinary regime under which the observed dynamics of unemployment evolved in Italy during the year 2020.

The aim of this work is to compare the observed dynamics of unemployment during the year 2020 in Italy with a counterfactual outcome to assess the (broad) impact of Covid in Italy in terms of unemployed individuals. In doing so, counterfactual outcomes are generated by Seasonal ARIMA models (SARIMA). Results are presented for the whole population of unemployed individuals and disaggregated by socioeconomic dimension as gender, age, education.

*Figure 1: Unemployed individuals (thousands) in Italy over the years 2014-2020 quarters. Source: Italian Labour Force Survey.*

201 Illya Bakurov, Nova University of Lisbon, Portugal<EMAIL_ADDRESS>0000-0002-6458-942X Fabrizio Culotta, University of Genoa, Italy<EMAIL_ADDRESS>0000-0002-3910-3088

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Illya Bakurov, Fabrizio Culotta, *Unemployment dynamics in Italy: a counterfactual analysis at Covid time*, pp. 215-220, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.40, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www. fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

# **2. Data and Methods**

This work adopts data from the Italian Labour Force Survey (Rilevazione sulle Forze di Lavoro, hereafter ILFS, for the years 2014-2020 at quarterly frequency focusing on working age population, i.e. aged 15-64 years, is considered. Raw (not smoothed) data covering the period 2014-2019 are used to train SARIMA models to forecast the four quarters of year 2020 (see Hyndman and Athanasopoulos, 2018, as a reference book). It is implicitly assumed that the first quarter of 2020 is the first period affecting unemployment dynamics, that is training data are not affected by the treatment. Estimated projections are then compared with observed values. The causal impact of Covid will be then defined as the difference between what is observed during the 2020 quarters (under the influence of Covid measures) and what would have been observed (in the absence of Covid measures). This empirical exercise is performed not only for the total population but also for eleven socioeconomic groups: two by gender (males and females), five by age (15-24, 25-34, 35-44, 45-54, 55-64), four by educational level (primary, lower and upper secondary, tertiary). The analysis is performed in *R* with help of the package *fpp2* (Hyndman et al., 2020).

Diagnostics analyses are also performed and available upon requests. In particular, trend-cycle decompositions visually suggest that each series exhibits strong seasonality which however is stable in variance over time. The visual inspection suggests that both seasonal and first differencing could take place. Therefore, we run a sequence of Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test (Kwiatkowski et al., 1992) for the null hypothesis of stationarity in the data, and we look for the evidence of rejection. Results from KPSS tests confirm that both seasonal and first differencing should take place. Model orders are selected by inspecting PACF and ACF. The winner model has been selected based on common information criteria AIC and BIC. Estimated models are reported below (Table 1).

# *Table 1: Estimated Seasonal ARIMA (SARIMA) models. Drift refers to the time-invariant intercept component.*

The resulting error components are distributed as white noise. Results from Ljung-Box suggest accepting null hypothesis of serially uncorrelated errors. Accordingly, we use such model specifications to produce the counterfactual trajectories for the 2020 quarters.

# **3. Results**

Results are reported in figures 2-5. For sake of graphical clarity, confidence intervals are provided only for the total profile (Fig. 2). For the other profiles, they are available upon requests.

*Figure 2: Total profile. Observed (FITTED=F) vs Forecast (FITTED=T) at Covid time (COVID19=T). Source: authors' own elaborations on ILFS data.*

*Figure 3: Gender Profiles. Observed (FITTED=F) vs Forecast (FITTED=T) at Covid time (COVID19=T). Source: authors' own elaborations on ILFS data.*

*Figure 4: Age profiles. Observed (FITTED=F) vs Forecast (FITTED=T) at Covid time (COVID19=T). Source: authors' own elaborations on ILFS data.*

*Figure 5: Education profiles. Observed (FITTED=F) vs Forecast (FITTED=T) at Covid time (COVID19=T). Source: authors' own elaborations on ILFS data.*

All profiles depicted above share some general features. The difference between observed and forecast values, i.e. the impact of Covid on the number of unemployed, is negative during the first quarter of 2020. The difference is even larger (in absolute value) during the second quarter of 2020, where the impact of Covid in 2020 led a relevant drop in the number of unemployed. The difference is positive at the third quarter, coinciding with the summer season, and becomes negative during the fourth.

Results are also displayed in a tabular format (Table 2). The largest impact of Covid on unemployed workers corresponds to 2020-Q2 (-652000). This drop is concentrated among females (-385000, about 60% of the total drop). Across age classes, 45-54 reports the largest reduction (-237000, around 36%). Whereas unemployed with lower secondary education are the educational group (-295000, 45%). During the third quarter of 2020, the raise in unemployed occurred (410000). Of such a raise, men were the majority (242000, 60%). The age class 25-34 shows the largest share (153000, 37%). Similarly, the largest increase is observed for the upper secondary educational group (238, 58%). Overall quarters, it results that the drop in unemployment caused by Covid during the year 2020 regarded women more than men especially from the second quarter. Among the age groups, Covid had an impact especially on unemployed individuals aged 45-54. Across educational levels, following this logic, individuals without a tertiary education were affected the most.

# **4. Conclusions**

This work studies the impact of Covid pandemic, and related measures, on the number of unemployed workers during the 2020 quarters in Italy. Observed and counterfactual outcomes are compared to identify the causal impact of the onset of Covid since the first quarter of 2020.

In doing so, counterfactual outcomes are produced by means of SARIMA models applied to different socioeconomic groups in the population of unemployed. The causal impact is then measured as difference between observed and forecast values. Results confirms that the drop in unemployment caused by Covid was heterogenous, i.e. not homogenously distributed in the population. Females, individuals aged 45-54 and those with secondary educational levels were those groups associated with the highest drop.

In general, the counterfactual analysis is used as a tool to identify causal mechanism. In the case of this work, the (macro-)econometric model is also offered as a (simple) policy statistical tool. It can be used to identify future patterns and to reason on possible thresholds or rebounds. It can offer an informative, yet statistical, support to face important decisions under uncertainty. Possibly, it can reveal insights for future planning.

*Table 2: Counterfactual analysis on unemployment dynamics (thousands of individuals) in Italy at Covid time (2020 quarters). Covid impact is the difference between observed and forecast values. Source: authors' own elaborations on Italian Labour Force Survey data.*

# **References**

Kwiatkowski, D., Phillips, P. C., Schmidt, P., & Shin, Y. (1992). Testing the null hypothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root? *Journal of econometrics*, **54**(1-3), 159-178.

Hyndman, R. J., Athanasopoulos, G. (2018). *Forecasting: principles and practice*. OTexts.

Hyndman, R. J., Athanasopoulos, G., Bergmeir, C., Caceres, G., Chhay, L., O'Hara-Wild, M., ... , Wang, E. (2020). *Package 'forecast'*. https://cran. r-project.org/web/packages/forecast/forecast.pdf. SESSION

# TOURISM AND GASTRONOMY

### Alfonso Piscitellia , Roberto Fasanellib , Elena Cuomoc and Ida Gallib a Department of Agricultural Sciences, Federico II University of Naples, Naples, Italy b Department of Social Sciences, Federico II University of Naples, Naples, Italy c Department of Political Sciences, Federico II University of Naples, Naples, Italy **Understanding the sensory characteristics of edible insects to promote entomophagy: A projective sensory experience among consumers**

**Understanding the sensory characteristics of edible insects to promote entomophagy: A projective sensory experience among consumers.**