Background
==========

Recent studies on care quality improvement and patient safety show that health care is still not safe and that the number of adverse events is underestimated \[[@B1]-[@B3]\]. Health care across organization levels is often poorly organized, complex, and uncoordinated. Furthermore, not all patients receive consistent, high-quality medical care. The organization of care can be more effective when multidisciplinary teams are involved in its organization and if care is organized around medical conditions and care processes \[[@B4],[@B5]\].

Care processes contain key interventions that support the diagnosis or treatment of patients. These key interventions contain unique bundles of products and services, and temporary firms can effectively achieve their delivery. Temporary firms are health care providers who operate together when a specific well-defined patient group is admitted to a care facility or institution. These teams contain members from different health care professions, have a shared clinical purpose, and have direct care responsibilities \[[@B6],[@B7]\]. They work in a complex environment, under interactive and dynamic conditions, and their membership frequently changes. Therefore, these teams are action teams \[[@B7],[@B8]\]. The challenges for these teams are effective communication, coordination, and control over the care process \[[@B7]\].

A care process has five key characteristics that affect the organization of care: coordination of the care process, patient-focused organization, communication with patients and family, collaboration with primary care, and follow-up of the care process \[[@B9]\]. The Care Process Self-Evaluation Tool (CPSET) assesses these five key characteristics by using a 29-item Likert scale. It is based on the perceptions of team members involved in organizing a care process. The primary study assessing the validity and reliability of this tool was performed in 2007, and statistical analysis of the five factors produced Cronbach's alpha values between 0.776 and 0.928 \[[@B9]\]. In 2011, an international Delphi study was performed to identify indicators that affect multidisciplinary teamwork in care processes. This study showed that CPSET is a good tool for following up improvements in multidisciplinary teamwork \[[@B10]\].

Since its original validation, CPSET has been used in different organizations and teams. The first aim of this study was to evaluate the stability of the psychometric properties of CPSET. The second aim was to calculate cutoff scores for the subscales and overall score, with the goal of helping health care managers rank the CPSET scores of their teams.

Methods
=======

Study design and sample
-----------------------

The present study was a cross-sectional, multicenter study involving 114 organizations in Belgium and the Netherlands. Data were collected between November 2007 and October 2011. The participating organizations were members of the Belgian-Dutch Clinical Pathway Network (<http://www.nkp.be>) \[[@B11]-[@B13]\]. These organizations were offered the opportunity to use the CPSET to evaluate the organization level of care processes after its original validation in 2007. Team leaders decided which team members would complete the CPSET. Teams received feedback on their scores after they sent their data to the central database at Leuven University. We received a total of 3378 questionnaires from team members. Questionnaires with more than three missing values on the 29-item CPSET were excluded from analysis. Thus, 3139 questionnaires were included for secondary analysis in this study. Informed consent was obtained from all participants. The secondary analysis was performed as part of a larger study with the ethical approval of the University Hospitals of Leuven for Belgium (identifiers: B32220096036 and B32330096038) \[[@B14]\].

Measures
--------

A two-section questionnaire form was used as a data collection instrument. The first part of the questionnaire collected data on demographic characteristics, which included age, gender, profession, and which kind of care process was evaluated. The second part of the questionnaire consisted of the 29-item CPSET. Each item was scored on a 10-point ordered, categorical scale, ranging from totally disagree (1) to totally agree (10) \[[@B9]\]. An average score per subscale (%) and an overall score (%) were calculated.

Statistical analysis
--------------------

We performed secondary data analysis on 3139 questionnaires. Descriptive statistics were used to analyze the demographic characteristics. SPSS version 19.0 was used (SPSS Inc., Chicago, IL, USA). Confirmatory factor analysis (CFA) was used to test the structure of the scale. Normed fit index (NFI), comparative fit index (CFI), and adjusted goodness-of-fit index (AGFI) values ≥0.90, and root mean square error of approximation (RMSEA) values \< 0.08 were considered to indicate a good fit \[[@B15],[@B16]\]. Cronbach's alpha analysis was used to measure internal consistency. Cronbach's alpha values range between 0 and 1, and a high Cronbach's alpha means that the items are strongly correlated \[[@B17]\]. CFA, NFI, CFI, AGFI, RMSEA, and Cronbach's alpha were determined using SAS® software (SAS Institute Inc., Cary, NC, USA).

The degree of within-cluster dependence, interclass correlation coefficients (ICCs), and confidence intervals were calculated using a formula derived by Donner and Klar based on analysis of variance \[[@B18]\]. Team- and hospital-level ICCs were calculated to determine to what extent scores at these levels correlate with each other. The Kruskal-Wallis test was used to compare CPSET scores of more than two independent groups. We used this test to compare differences between profession and age and differences between the overall CPSET and its five subscales. The Mann--Whitney test, a test used to assess differences between two independent groups, was used to assess differences between gender, profession, and age of all categories. Kruskal-Wallis tests and Mann--Whitney tests were extracted from StatsDirect 2.7.8 (StatsDirect Ltd, Altrincham, UK) and SPSS version 19.0 (SPSS Inc., Chicago, IL, USA). The level of significance was set to p  \< 0.05.

Results
=======

Respondents
-----------

This study included participants from 92 organizations in Belgium and 22 in the Netherlands. These organizations can be classified as acute hospitals (n = 88), psychiatric hospitals (n = 2), specialized hospitals (n = 9), and primary care organizations (n = 15). Overall, 48.3% of the participants were 40 years or older, 60.31% were female, and more than half (54.76%) were nurses. In total, 283 teams participated. Between one and 28 teams per organization participated. Some teams included a coordinator. Coordinators could be either team leaders or care process coordinators. Some teams included, for example, care logistics in their team; these were classified as others (Table [1](#T1){ref-type="table"}).

###### 

Demographic characteristics

  **Characteristics**                    **N**   **%**
  --------------------- ---------------- ------- -------
  Age (y)                                         
                        20-29            677     21.57
                        30-39            870     27.72
                        40-49            885     28.19
                        50-59            591     18.83
                        \>60             41      1.30
                        Unknown          75      2.39
  Gender                                          
                        Male             831     26.47
                        Female           1893    60.31
                        Unknown          415     13.22
  Profession                                      
                        Nurse            1719    54.76
                        Medical doctor   543     17.30
                        Paramedic        524     16.69
                        Coordinator      134     4.27
                        Others           219     6.98

Confirmatory factor analysis
----------------------------

CFA was performed on the original five-factor solution with 29 items \[[@B9]\]. The data of the 3139 participants were used. The four fit indices revealed a good fit for CPSET (AGFI = 0.87; RMSEA = .06; CFI = 0.94; and NFI = 0.93).

Internal consistency
--------------------

Internal consistency was measured by calculating Cronbach's alpha reliability coefficients for each of the five factors. The Cronbach's alphas for the five factors or subscales were between 0.869 and 0.950 (patient-focused organization, alpha = 0.919; coordination of care, alpha = 0.900; communication with patient and family, alpha = 0.897; collaboration with primary care, alpha = 0.869; follow-up of care, alpha = 0.950).

Interclass correlation coefficients
-----------------------------------

We calculated ICCs at team (n = 283) and hospital levels (n = 114). ICCs for each of the five dimensions are shown in Table [2](#T2){ref-type="table"}. A team was defined as more than three health care providers who work together around a care process. At the team level, subscale ICCs ranged from 0.127 (collaboration with primary care) to 0.232 (coordination of care). At the hospital level, subscale ICCs ranged from 0.071 (collaboration with primary care) to 0.151 (communication with patient and family). The ICC for the overall CPSET score at the team level was 0.221, whereas the ICC at the hospital level was 0.147. These results showed that there was poor agreement at team and hospital levels and that there was less variation within teams than within hospitals.

###### 

ICCs and 95% confidence intervals

                                           **Team level**  **Hospital level**           
  --------------------------------------- ---------------- -------------------- ------- -------------
  Subscale scores                               ICC        95% CI                 ICC   95% CI
  Patient-focused organization                 0.197       0.150-0.243           0.146  0.085-0.208
  Coordination of care                         0.232       0.182-0.282           0.131  0.074-0.188
  Communication with patient and family        0.210       0.163-0.258           0.151  0.088-0.214
  Collaboration with primary care              0.127       0.090-0.164           0.071  0.034-0.108
  Follow-up of care                            0.215       0.167-0.264           0.134  0.076-0.191
  Overall CPSET score                          0.221       0.172-0.270           0.147  0.055-0.209

*Abbreviations:ICC* interclass correlation coefficient, *CI* confidence interval.

CPSET subscales
---------------

Gender, age, and profession of team members had a significant impact on the five subscales and the overall CPSET score. Men scored significantly higher on the subscales 'communication with patient and family' and 'collaboration with primary care,' and on the overall CPSET. Team members younger than 30 years old scored lower on the subscales 'patient-focused organization,' 'communication with patient and family,' and 'collaboration with primary care' and on the overall CPSET. Medical doctors scored significantly higher on the subscales 'patient-focused organization,' 'coordination of care,' 'communication with patient and family,' and 'collaboration with primary care' and on the overall CPSET. Paramedics scored significantly higher on the subscale 'follow-up of care.' These results are summarized in Table [3](#T3){ref-type="table"}.

###### 

Differences in team member demographics according to CPSET subscales

                                                       **Patient-focused organization**             **Coordination of care**                                                      **Communication with patient and family**               **Collaboration with primary care**          **Follow-up of care**                                   **Overall CPSET score**
  ---------------------------------------------------- -------------------------------------------- ----------------------------------------------------------------------------- ------------------------------------------------------- -------------------------------------------- ------------------------------------------------------- -------------------------------------------------------
  Differences between genders                                                                                                                                                                                                                                                                                                                   
    Group A: men   (n = 831)                           74.28 (14.48)                                69.80 (13.83)                                                                 64.74 (17.35)                                           68.36 (16.71)                                60.86 (18.06)                                           67.60 (12.71)
    Group B: women   (n = 1893)                        73.58 (13.78)                                69.33 (13.42)                                                                 61.94 (17.70)                                           66.11 (15.39)                                62.44 (16.56)                                           66.59 (12.21)
    Test                                                                                                                                                                          A ≠ B^†^                                                A ≠ B^†^                                                                                             A ≠ B^†^
  Differences between age groups                                                                                                                                                                                                                                                                                                                
    Group A: 20--29 y   (n = 677)                      72.29 (14.21)                                68.69 (13.87)                                                                 59.44 (17.86)                                           65.51 (15.33)                                63.64 (15.47)                                           65.85 (12.73)
    Group B: 30--39 y   (n = 870)                      73.33 (14.03)                                63.13 (13.78)                                                                 61.37 (18.45)                                           66.66 (13.35)                                61.25 (17.45)                                           66.31 (12.62)
    Group C: 40--49 y   (n = 885)                      74.13 (14.91)                                70.01 (13.61)                                                                 63.41 (17.78)                                           67.58 (16.09)                                61.20 (18.05)                                           67.20 (12.79)
    Group D: 50--59 y   (n = 591)                      75.45 (13.93)                                70.54 (14.07)                                                                 64.84 (17.06)                                           67.94 (16.02)                                62.64 (16.91)                                           68.21 (12.34)
    Group E: \> 60 y   (n = 41)                        73.93 (14.11)                                73.42 (11.54)                                                                 67.62 (17.15)                                           66.50 (19.57)                                67.07 (16.26)                                           69.73 (13.30)
    Test                                               A ≠ B ≠ C ≠ D ≠ E\*                          A ≠ B ≠ C ≠ D ≠ E\*                                                           A ≠ B ≠ C ≠ D ≠ E\*                                     A ≠ B ≠ C ≠ D ≠ E\*                          A ≠ B ≠ C ≠ D ≠ E \*                                    A ≠ B ≠ C ≠ D ≠ E \*
  C \> A^†^, D \> A^†^, D \> B^†^                      D \> A^†^, E \> A^†^, D \> B^†^, E \> B^†^   B \> A^†^, C \> A^†^, D \> A^†^, E \> A^†^, C \> B^†^, D \> B^†^, E \> B^†^   C \> A^†^, D \> A^†^                                    A \> B^†^, A \> C^†^                         C \> A^†^, D \> A^†^, C \> B^†^, D \> B^†^              
  Differences between professions                                                                                                                                                                                                                                                                                                               
    Group A: medical   doctor (n = 543)                76.10 (14.17)                                72.64 (13.42)                                                                 69.80 (16.34)                                           72.62 (16.26)                                60.76 (18.88)                                           70.69 (12.35)
    Group B: nurses   (n = 1719)                       73.03 (14.80)                                69.17 (13.84)                                                                 58.80 (18.00)                                           64.83 (16.00)                                61.97 (16.70)                                           65.49 (12.83)
    Group C: coordinator   of care process (n = 134)   73.60 (11.7)                                 65.17 (12.17)                                                                 61.09 (14.73)                                           65.51 (12.52)                                56.45 (16.27)                                           64.24 (9.63)
    Group D: paramedic   (n = 524)                     72.89 (13.66)                                68.82 (14.19)                                                                 64.50 (17.72)                                           67.69 (15.38)                                64.88 (16.42)                                           67.74 (12.57)
    Test                                               A ≠ B ≠ C ≠ D\*                              A ≠ B ≠ C ≠ D\*                                                               A ≠ B ≠ C ≠ D\*                                         A ≠ B ≠ C ≠ D\*                              A ≠ B ≠ C ≠ D\*                                         A ≠ B ≠ C ≠ D\*
                                                       A \> B^†^, A \> C^†^, A \> D^†^              A \> B^†^, A \> C^†^, A \> D^†^, B \> C^†^, D \> C^†^                         A \> B^†^, A \> C^†^, A \> D^†^, D \> C^†^, D \> B^†^   A \> B^†^, A \> C^†^, A \> D^†^, D \> C^†^   D \> A^†^, A \> C^†^, D \> B^†^, B \> C^†^, D \> C^†^   A \> B^†^, A \> C^†^, A \> D^†^, D \> B^†^, D \> C^†^

*Abbreviations:SD* standard deviation.

\*Kruskal Wallis Test, P  \< 0.05.

^†^Mann--Whitney test, P  \< 0.05.

CPSET cutoff scores
-------------------

On the basis of the individual scores of the 3139 participating health care providers, we calculated cutoff scores for the five subscales and the overall CPSET (Table [4](#T4){ref-type="table"}). Percentiles were used in the cutoff table based on a normal distribution of the subscale scores. The subscales 'communication with patient and family' and 'follow-up of care' had a broader range of scores than the other subscales and the overall CPSET.

###### 

Cutoff table

  **Percentile**   **Patient-focused organization**   **Coordination of care**   **Communication with patient and family**   **Collaboration with primary care**   **Follow-up of care**   **Overall CPSET score**
  ---------------- ---------------------------------- -------------------------- ------------------------------------------- ------------------------------------- ----------------------- -------------------------
  P10              55.00                              51.43                      37.50                                       46.67                                 38.75                   50.57
  P20              63.33                              60.00                      47.50                                       53.33                                 48.75                   57.00
  P30              68.33                              64.29                      55.00                                       60.00                                 55.00                   61.46
  P40              71.67                              67.14                      60.00                                       63.33                                 60.00                   64.68
  P50              75.00                              71.43                      65                                          70.00                                 63.75                   67.79
  P60              78.33                              74.29                      67.50                                       70.00                                 68.75                   70.54
  P70              80.00                              77.14                      72.50                                       76.67                                 71.25                   73.52
  P80              85.00                              81.43                      77.50                                       80.00                                 76.25                   76.90
  P90              90.00                              85.71                      85.00                                       86.67                                 81.25                   82.41

Discussion
==========

This paper describes the psychometric properties of the CPSET and defines CPSET cutoff scores. The psychometric properties of the CPSET showed that this tool is valid and reliable for evaluating the organization of a care process as perceived by team members. The original five-factor structure with 29 items was confirmed by CFA. The reliability of the CPSET was measured by Cronbach's alpha. The Cronbach's alpha results in this study varied between 0.869 and 0.950 and were higher than those reported in the original 2007 study of Vanhaecht et al. (Cronbach's alpha = 0.776-0.928) \[[@B9]\]. This indicates that the scale is still reliable.

A multilevel analysis was performed at team and hospital levels. Our results showed that the ICCs of scores at the team level were higher than those at the hospital level. This means that there was less variance in CPSET scores within teams than within hospitals, which was expected. The ICCs in our study were low, and most of the variation could be explained by team and hospital variations. One possible reason for the low ICCs in our study was that teams were composed of professionals from different disciplines, with each team member having different perceptions about the organization of care. As shown in Table [3](#T3){ref-type="table"}, profession, age, and gender significantly influenced CPSET scores. Medical doctors scored significantly higher than other health professionals on the overall CPSET scale and on the following subscales: 'patient focused organization,' 'coordination of care,' 'communication with patient and family,' and 'collaboration with primary care.' Paramedics scored significantly higher on the subscale 'follow-up of care.' Men scored significantly higher on the overall CPSET scale and the subscales 'communication with patient and family' and 'collaboration with primary care.' Team members between 20 and 39 years old scored significantly lower on the subscale 'communication with patient and family' than those in other age categories. Significant differences were found between younger ( \< 40 years) and older (\> 50 years) health care professionals on the subscale 'coordination of care.' Coordinators of care pathways scored lower on the subscales 'coordination of care' and 'follow-up of care,' perhaps because they tend to be more critical of the organization of care.

The differences we observed in the perceptions of medical doctors and nurses are consistent with those observed in previous research. In the present study, physicians perceived teams to be more organized than nurses in terms of teamwork, collaboration, and communication with nurses, which is consistent with the finding that physicians generally perceive teamwork to be better coordinated \[[@B19]-[@B21]\]. A negative correlation exists between professional autonomy and the level of nurse-physicians collaboration \[[@B22]\]. Different perspectives in communication can be caused by hierarchical factors, gender, different patient care responsibilities, different perceptions of requisite communication standards, and differences in training methods for nurses and doctors \[[@B19]\]. Communication skills training can improve patient-nurse communication but not patient-doctor communication. Skills training that contains patient-centered communication can increase information exchange and continuity of care for patients \[[@B23]\].

The lowest CPSET scores were observed on the subscale 'communication with patient and family' and 'follow-up of care.' Organizations need to improve on priorities, communication, and coordination of care, as suggested by Bates et al. \[[@B24]\]. Relational coordination can be used to improve 'follow-up of care.' This framework can lead to better quality of care for patients, and health care providers reported fewer adverse events \[[@B25]\]. Effective and safe hospital care depends on good teamwork. Greater teamwork results in higher patient satisfaction rates, higher nurse retention, and lower hospital costs \[[@B26]\]. Multidisciplinary teamwork is essential for quality health care.

The 2012 review of Deneckere et al. showed that multidisciplinary teamwork can be supported by using care pathways \[[@B7]\]. Care pathways can improve the work environment and organization of care, and can have a positive impact on the well-being of health care providers \[[@B7],[@B27],[@B28]\]. Further research is needed on using the CPSET to study the effect of coordinating mechanisms, such as care pathways, on how health care providers perceive the organization of care.

Although the CPSET has been used for several years, health care managers have problems interpreting the CPSET scores of their team members. Therefore, we compiled a table of cutoff scores that will permit health care managers and team members to compare how they perceive the organization of their care process relative to other team members. By using the cutoff table as a starting point, team members can discuss how they can improve the organization of care. When different teams of the same care facility or institution complete the CPSET, the cutoff scores will help health care managers rank the teams in that facility or institution. However, this should be done carefully. The primary aim of the cutoff scores is to initiate discussions within teams. For example, they can look for possible reasons why they perceive the organization of care to be low and what they expect from other team members. They can also learn from actions taken by other teams. We hypothesize that teams that use care bundles, care pathways, or evidence-based protocols will have CPSET scores in the higher percentiles compared with teams that do not use quality improvement initiatives. But further research is needed to determine whether the structured care associated with quality improvement initiatives does indeed change the perception of health care providers according to the actual organization of care.

Some limitations of our study should be considered. One limitation is the risk of social desirability and selection bias. Team leaders and coordinators of care processes decided which specific team members would complete the CPSET. Hence, it is possible that not all team members or health care professionals involved in a specific care process were surveyed. Another concern is that the results of this study are based on data from two countries: Belgium and the Netherlands. Therefore, a comparable study should be conducted in additional countries. The validity of the CPSET is currently being tested in French, Norwegian, Italian, Portuguese, English, and German languages.

Conclusions
===========

The CPSET is a valid and reliable tool for measuring the organization of care as perceived by involved health care providers. Some of the CPSET scores depend on age, gender, and profession. Team leaders can use the CPSET to evaluate how their team members perceive the organization of care. The cutoff scores presented in this study will aid health care managers rank their team, identify differences between teams within a care facility or institution, and analyze the needs of teams in their collaborative search for excellence.

Competing interests
===================

The authors declare that they have no competing interests.

Authors' contributions
======================

KV, WS, and MP defined the design of the study and organized the data collection. SD, EVG, and DS coordinated the data cleaning and supported the participating organizations. LB, TM, RCB, and SK performed the statistical analysis. KV, SD, and DS prepared the first draft of the manuscript. All authors discussed the results and approved the final version of the manuscript.

Pre-publication history
=======================

The pre-publication history for this paper can be accessed here:

<http://www.biomedcentral.com/1472-6963/13/325/prepub>
