generic mortality scoring systems for children admitted to intensive care units ( icus ) have been developed for use at specific time points in the icu stay .
two systems have been validated in paediatric icus ( picus ) : the paediatric risk of mortality ( prism ) and the paediatric index of mortality ( pim ) .
the prism , which is used in picus worldwide , requires an observation period of 24 hours , and the updated prism iii score   measures severity at two time points ( 12 and 24 hours ) during the picu stay .
the pim and the recently updated pim2 scores are calculated 1 hour after admission . the 1224 hour period of observation has been a criticism levelled at the prism scoring system , and it has been speculated that it may diagnose rather than predict death . with the pim and pim2 scores ,
the single measurement of values shortly after admission is susceptible to random variation   or may reflect a transient state resulting from interventions during transport .
severity models have been used for time periods different from those for which the scores were developed . in children with meningococcal septic shock ,
castellanos - ortega and coworkers   recorded the worst values for each variable included in the glasgow meningococcal septicaemia prognostic score , the malley score , and the pim score over the first 2 hours in the picu .
indeed , early identification of patients who could benefit from therapeutic interventions may be useful .
we hypothesized that an intermediate observation period ( we arbitrarily chose a time point of 4 hours after picu admission ) would be a good compromise between two objectives  to take into account a short period of stabilization after a patient 's admission to the picu and to obtain an accurate measure of illness severity in the picu . to our knowledge
, no study has ever evaluated the accuracy of generic paediatric scoring systems in predicting death for the whole picu population , and for time periods different from those for which the scores were developed .
the aim of the present study was to externally validate the prism , prism iii and pim scores at their intended time points , and to compare their accuracy in predicting mortality at those times with their accuracy at a different time period , namely 4 hours after admission .
all consecutive patients admitted to our university hospital picu from june 1998 through to may 2000 were included unless they met the following exclusion criteria : admission in a state requiring cardiopulmonary resuscitation without achieving stable vital signs for at least 2 hours ; admission for scheduled procedures normally done in other hospital wards ; prematurity ; and age more than 18 years .
data were prospectively collected to generate scores and predictions for the time periods for which the scores were developed ( i.e. pim at 1 hour , prism at 24 hours , prism iii at 12 hours , and prism iii at 24 hours ) and to generate scores and predictions for a different time point ( i.e. 4 hours after admission ) . the pim2 score was not evaluated because it had not yet been reported when we began the study .
the probabilities of death were calculated at different time points ( table 1 ) . to generate a prediction for the prism iii 4-hour score
, we used the prism iii 12-hour equation ( 1996 version ) . in order to compare observed with expected mortality and to estimate the calibration of the scores , a hosmer  lemeshow goodness - of - fit test with five degrees of freedom ( df
; we considered five classes of mortality probability : 0% to < 1% , 1% to < 5% , 5% to < 15% , 15% to < 30% , and 30% ) was performed . according to this test ,
the p value is greater than 0.05 if the model is well calibrated ; the greater the p value , the better the model fits .
the areas under the receiver operating characteristic curve ( aucs ) and their standard errors were calculated to estimate the discrimination of the scores .
an auc 0.7 is generally considered acceptable , 0.8 as good , and 0.9 as excellent .
standardized mortality ratios ( smrs ) and their comparison to 1 were calculated . to study the effect of length of
stay on calibration and discrimination of the three scores , calibration was calculated each day and discrimination at days 5 , 10 and 20 after admission . for a patient who had died on dayx ,
the picu outcome - dependent variable was considered as survival at dayx - i ( i = 1 , 2 , 3 ... ) .
statistical analyses were performed using the statistical program for social science ( spss inc . , chicago , il , usa ) .
the median age was 26 months ( interquartile range 892 months ) and the median length of stay was 2 days ( interquartile range 16 days ) .
the frequencies of the additional variables in the prism iii score were as follows : nonoperative cardiovascular disease 4.2% , cancer 5.9% , chromosomal anomalies 2% , previous picu admission 6.5% , pre - picu cardiopulmonary resuscitation 4.2% , postoperative surgical procedure 19% , acute diabetes 0.7% , and admission from routine care area 13% .
for the time periods for which the scores were developed , the three scores had poor calibration ( p < 0.01 for each ) , with large differences between the goodness - of - fit test values ( table 2 ) .
we observed underestimations of mortality in the low mortality risk groups ( risk 1% to < 5% and risk 5% to < 15% ) , and an overestimation in the group with very high risk for mortality ( risk 30% ) .
smrs varied from 1.03 to 1.39 , but only the smr for the pim 1-hour assessment was significantly greater than 1 ( table 3 ) . all scores exhibited good discrimination ( table 2 ) . at 4 hours pim had good calibration ( p = 0.44 )
conversely , both prism and prism iii had poor calibration at 4 hours ( p < 0.01 ) , with significant differences between observed and predicted mortality ( tables 4 and 5 ) . expected mortality with prism and prism iii underestimated the observed mortality in the groups at low risk for mortality .
smrs varied from 1.17 to 1.57 , and were significantly greater than 1 except for the pim 4-hour assessment ( table 5 ) .
all scores exhibited good discrimination ( table 4 ) . for the time points for which the scores were developed , study of the length of stay showed good calibration for the pim 1-hour assessment between days 3 and 28 , for the prism 24-hour assessment between days 11 and 22 , for the prism iii 12-hour assessment between days 51 and 58 , and for the prism iii 24-hour assessment between days 10 and 11 ( fig .
for the different time point examined ( i.e. 4 hours ) , study of the length of stay showed good calibration for pim from day 4 until discharge , for prism between days 2 and 15 , and for prism iii between days 3 and 10 ( fig .
for both time periods , study of the length of stay showed that the auc for all scores , both for the time points for which the scores were developed ( fig .
with regard to the poor calibration identified in some of the assessments , retrospective analysis of patients who died was performed for the classes of mortality probability for which the value exceeded 2.5 .
a value of 11 was needed to obtain statistical calibration with the five classes of mortality probability . for these deceased patients we analyzed length of stay and comorbidities ( cancer , prematurity , and chronic cardiac , respiratory , neurological and digestive diseases ) .
chronic organ disease was defined as disease with or without organ failure , requiring multiple admissions ( to paediatric department or day care center ) and requiring supervision by a subspecialist in paediatrics .
a value over 2.5 , which indicates a significant difference between observed and predicted probability of death in a mortality class , was observed for 55 deceased patients . in this subpopulation ,
the median length of stay was significantly different from that in the other 25 deceased patients ( 7 days versus 1 day , respectively ; p < 0.001 ) , and only seven ( 13% ) had a pre - icu cardiac massage versus 18 ( 72% ) in the other deceased patients ( p < 0.000001 ) . in these 55 patients ,
only 611% of the above mentioned comorbidities were taken into account in the probability of death calculated with the different scores .
in this single unit study , discrimination of the pim , prism and prism iii scores was good whereas calibration was poor for the time points for which the scores were developed . at 4 hours , only the pim score had good discrimination and calibration . both discrimination and calibration must be considered when evaluating the performance of scoring systems .
discrimination measures the predictive performance of scoring systems , and when the outcome is dichotomous it is usually described by a receiver operating characteristic curve . in the studies that compared the original pim , prism and prism iii scores , the aucs were as follows : 0.7 for the pim and prism iii scores ; 0.8 for the pim score , and 0.9 for the prism and prism iii scores ; and between 0.83 and 0.87 for the pre - icu prism , pim and prism scores .
however , for zhu and coworkers   auc was not as sensitive to differences in icu care as the hosmer  lemeshow goodness - of - fit test .
gemke and van vught   externally validated the prism iii and pim scores ( n = 303 patients , 20 deaths ) ; the goodness - of - fit test values with 10 deciles of mortality risk were 10.8 ( p = 0.21 , df = 8) for the prism iii 12-hour assessment , 13.3 ( p = 0.10 [ not p = 0.21 , as was published ] , df = 8) for the prism iii 24-hour assessment , and 4.92 ( p = 0.77 , df = 8) for the pim score .
however , the p values that we calculated from these data using the five conventional mortality risk categories were 0.14 for the prism iii 12-hour assessment , 0.04 ( indicating no statistical calibration ) for the prism iii 24-hour assessment , and 0.07 for the pim score .
pearson and coworkers   tested the pim score in a picu population of 7253 children ; the goodness - of - fit test value calculated from their data was 37.4 ( p < 0.0001 , df = 10 ) , which suggests no statistical calibration , as indicated by others [ 19 - 21 ] .
tibby and coworkers   compared the pre - icu prism , pim and prism scores in 928 patients .
they concluded that all scoring systems exhibited suboptimal calibration ( p = 0.08 for the prism , and p < 0.0001 for the pre - icu prism and pim ) .
slater and coworkers   observed 20 picu deaths in their study , including 598 children from one unit ( 21 with inclusion criteria for the prism iii , which considers patients who die within 24 hours of picu discharge ) , whereas expected deaths were 21.3 , 32.3 and 23.4 for the pim , prism and prism iii scores , respectively .
although goodness - of - fit test values were not calculated in their study , calibration of the prism score could be expected to be poor .
the previously reported miscalibration of the prism score   led tilford and coworkers   to use a different set of coefficient estimates . when interpreting the calibration of the prism iii score , the version selected must be considered . in the present study
the prism iii score was calculated using the 1996 version and not the 1999 one , which includes other variables that are not described in the first prism iii report and , to our knowledge , have not been reported elsewhere . in our study , as in that by gemke and van vught , the expected mortality underestimated the observed mortality in the group at low risk for mortality and overestimated it in the group at very high risk for mortality ( > 30% ) .
such discrepancies have been reported with both paediatric   and adult   generic scoring systems .
the length of stay was studied by bertolini and coworkers   because the prism score could not correctly predict outcome .
those authors found a good calibration for patients with a length of stay of 4 days or less and a poor calibration in those patients who stayed for longer than 4 days .
the present study showed that , for the time periods for which the scores were developed , the pim score provided the earliest ( from day 3 ) and longest ( to day 28 ) calibration . for a different time point ( i.e. 4 hours ) , the three scores were calibrated after a few days : day 2 for the prism 4-hour assessment , day 3 for the prism iii 4-hour assessment , and day 4 for pim 4-hour assessment ; only the pim 4-hour assessment was calibrated until discharge .
moreover , patient mortality is affected by demographical , physiological and diagnostic data , but it also depends on many other factors such as comorbidities , which did not appear to be accounted for sufficiently in our population . in the recently reported pim2 , the numbers of diagnostic criteria ( high risk and low risk diagnosis ) and comorbidities have been increased .
in fact , prism score , acute physiology and chronic health evaluation ( apache ) score , mortality probability model ( mpm ) score and simplified acute physiology score ( saps ) were reported in several studies to exhibit good discrimination but poor calibration [ 23,25 - 29 ] .
unsatisfactory calibration of scores can be attributed to various factors , including poor performance of the medical system ( if observed mortality is greater than predicted mortality ) , differences in case mix   and mortality rate , as well as failure of the score equation to model the actual situation accurately .
the above mentioned paediatric studies did not give any information on the childrens ' characteristics ( case mix ) , which potentially could explain discrepancies between discrimination and calibration .
indeed , the two studies using the additional variables of the prism iii score   did not provide a clear description of their population .
important differences in case mix data are represented by mortality rates , which were different between picus ( e.g. 4.8% for pollack and coworkers , 6.6% for gemke and van vught   and 10.0% in the present study ) .
the further the hospital mortality rate diverged from the original rate , the worse the performance of the model .
goodness - of - fit tests are more sensitive than aucs , and it has been suggested that , in the presence of good discrimination , bad calibration due to the source is correctable by using customization
. however , diamond   demonstrated that perfect calibration and perfect discrimination can not coexist ; a perfectly calibrated model is not perfectly discriminatory because it has an auc of only 0.83 rather than 1 .
customization of a score is justified when the database on which it was developed is old and when the score is used in a specific population .
however , customization by a unit could lead to inability to evaluate ( or compare ) performance between units .
is a score with poor calibration useful ? if scores are used to assess quality of care , as estimated by smr , then calibration , rather than discrimination , is the best measure of performance .
it is also recognized that there are no formal means of directly comparing the values derived from the goodness - of - fit test . our data and those reported by livingston and coworkers   showed large differences in goodness - of - fit test values between several scores .
thus , one can consider that a way to describe calibration of a score is to detail the goodness - of - fit test values for different classes of mortality probability , which reflects exact prediction across the full range of severity ( tables 3 and 5 ) .
scoring systems are used to compare or control for severity of illness in clinical trials and have been integrated into guidelines .
the question is , what kind of scoring system do we need if we are to include children in clinical trials ? we probably need a score that represents well the patient 's condition early after admission to the picu . with this aim in mind
pim score takes into account the condition of the patient directly on arrival in the picu ( i.e. when the patient 's condition is least affected by therapeutic intervention ) .
prism score require an observation period of 24 hours , which represents a limitation of its use as an inclusion criterion in clinical trials . to date
, no consensus has been reached as to which approach represents the ' gold standard ' . in order to minimize inclusion delay , pollack and coworkers   proposed estimation of the probability of death using the prism iii calculated 12 hours after admission . however
, this delay is too long for serious diseases ( e.g. meningococcal septic shock ) . in the present
thus , a 4-hour observation period seems to be a good compromise , allowing evaluation of the patient 's clinical condition and permitting stabilization , without delaying inclusion in a therapeutic trial .
calculation of the scores at 3 or 5 hours would probably have yielded similar results . to our knowledge ,
no study has compared the performance of generic paediatric mortality scores calculated within a few hours of admission to the picu .
castellanos - ortega and coworkers   used a similar approach in a specific population of children with meningococcal septic shock by calculating one generic ( pim ) and two specific scores 2 hours after picu admission ; the pim 2-hour score was as discriminant ( auc 0.82 ) as their new score ( auc 0.92 ; p = 0.10 ) but exhibited poor calibration .
the present study indicates that , among generic scores calculated at 4 hours after admission and with good discriminatory capacity ( i.e. auc > 0.80 ) , only the pim 4-hour score was well calibrated .
the updated pim2 , which takes into account new primary reasons for icu admission and comorbidities , must be validated for the time point for which it was developed and at a different time point .
further studies are required before the pim ( or pim2 ) 4-hour score can be used as an inclusion criterion for clinical trials .
among generic paediatric mortality scores calculated at 4 hours after admission in 802 consecutive children , only the pim score was both discriminant and calibrated
auc = area under the receiver operating characteristic curve ; df = degrees of freedom ; icu = intensive care unit ; picu = paediatric intensive care unit ; pim = paediatric index of mortality ; prism = paediatric risk of mortality ; smr = standardized mortality ratio .
the authors ' contributions were as follows : study conception and design : francis leclerc and stphane leteurtre ; acquisition of data : stphane leteurtre , jessica wirth , odile noizet , eric magnenant , ahmed sadik , catherine fourier and robin cremer ; analysis and interpretation of data : stphane leteurtre , francis leclerc and jessica wirth ; draft of the article : stphane leteurtre , francis leclerc and eric magnenant ; critical revision of the manuscript for important intellectual content : all investigators read and commented regarding important intellectual content ; statistical expertise : stphane leteurtre and jessica wirth ; admistrative , technical , or materiel support : stphane leteurtre , ahmed sadik , odile noizet , catherine fourier and robin cremer ; supervision : francis leclerc .
effect of the length of stay on calibration of the paediatric index of mortality ( pim ) , paediatric risk of mortality ( prism ) and prism iii scores   for ( a ) the time points for which the scores were developed ( 1 hour [ h1 ] for pim , 24 hours [ h24 ] for prism , and 12 hours [ h12 ] and h24 for prism iii ) and ( b ) a different time period , namely 4 hours ( h4 ) . effect of the length of stay on discrimination of the paediatric index of mortality ( pim ) , paediatric risk of mortality ( prism ) and prism iii scores   for ( a ) the time points for which the scores were developed ( 1 hour [ h1 ] for pim , 24 hours [ h24 ] for prism , and 12 hours [ h12 ] and h24 for prism iii ) and ( b ) a different time period , namely 4 hours ( h4 ) .
lemeshow goodness - of - fit test values and aucs : time point for which the scores were developed auc , area under the receiver operating characteristic curve ; df , degrees of freedom ; pim , paediatric index of mortality ; prism , paediatric risk of mortality . hosmer 
lemeshow goodness - of - fit test values : time point for which the scores were developed * significantly greater than 1 ( p = 0.002 ) .
ci , confidence interval ; pim , paediatric index of mortality ; prism , paediatric risk of mortality ; smr , standardized mortality ratio .
lemeshow goodness - of - fit test values and aucs : 4 hours auc , area under the receiver operating characteristic curve ; df , degrees of freedom ; pim , paediatric index of mortality ; prism , paediatric risk of mortality
. hosmer - lemeshow goodness of fit test values : 4 hours * significantly greater than 1 ( p < 0.0001 for prism at 4 hours and p = 0.0025 for prism iii at 4 hours ) .
ci , confidence interval ; pim , paediatric index of mortality ; prism , paediatric risk of mortality ; smr , standardized mortality ratio .
 introductiontwo generic paediatric mortality scoring systems have been validated in the paediatric intensive care unit ( picu ) . 
 paediatric risk of mortality ( prism ) requires an observation period of 24 hours , and prism iii measures severity at two time points ( at 12 hours and 24 hours ) after admission , which represents a limitation for clinical trials that require earlier inclusion . 
 the paediatric index of mortality ( pim ) is calculated 1 hour after admission but does not take into account the stabilization period following admission . to avoid these limitations , we chose to conduct assessments 4 hours after picu admission . 
 the aim of the present study was to validate prism , prism iii and pim at the time points for which they were developed , and to compare their accuracy in predicting mortality at those times with their accuracy at 4 hours.methodsall children admitted from june 1998 to may 2000 in one tertiary picu were prospectively included . 
 data were collected to generate scores and predictions using prism , prism iii and pim.resultsthere were 802 consecutive admissions with 80 deaths . for the time points for which the scores were developed , observed and predicted mortality rates were significantly different for the three scores ( p < 0.01 ) whereas all exhibited good discrimination ( area under the receiver operating characteristic curve 0.83 ) . at 4 hours after admission only the pim had good calibration ( p = 0.44 ) , but all three scores exhibited good discrimination ( area under the receiver operating characteristic curve 0.82).conclusionsamong the three scores calculated at 4 hours after admission , all had good discriminatory capacity but only the pim score was well calibrated . 
 further studies are required before the pim score at 4 hours can be used as an inclusion criterion in clinical trials . 
Introduction
Methods
Results
Discussion
Conclusion
Key messages
Competing interests
Abbreviations
Acknowledgements
Figures and Tables