###### Strengths and limitations of this study

-   The large and representative sample (n=2229) of lower secondary school students who responded to the science achievement test improves the external validity of our findings.

-   Estimating students' proficiencies and task difficulties using Rasch modelling, we could compare students' proficiency in science with the difficulty of identifying and appraising a health claim in a fictitious brief news report.

-   All achievement test items were piloted twice to ensure a valid and reliable measure of scientific literacy, and the use of a digitalised assessment tool reduced sources of errors.

-   We did a secondary analysis of test data collected in 2013, thus a shift in proficiency in subsequent student cohorts may have occurred.

-   Using raters to code responses to the open-constructed 'news report' item, there is a potential of misclassifying responses owing to rater subjectivity.

Background {#s1}
==========

News media is a leading source of health and scientific information for the public,[@R1] including adolescents and young people, who frequently encounter and share news and information through digital media.[@R3] According to Eurostat, more than two-thirds of young people access online news media regularly.[@R4] More than half also deliberately search for health information online, indicating health-related topics to be important for youth, especially for those aged 15 years and above.[@R5]

Media reports of health research often address preliminary and poorly executed studies as sensational 'breakthroughs', leading to large discrepancies between the claims made and the underlying strength of the evidence.[@R6] The result is confusing and conflicting claims, for instance, about what to eat and drink to maintain good health---claims that influence peoples' perceptions and actions of health.[@R9] Knowledge about scientific methods and scientific concepts is assumed as a necessity for appraising the reliability of health claims.[@R11] Health literacy initiatives at schools might help develop students' skills in apprising claims, and some suggest that these skills may empower students to make informed decisions about health and well-being over the life course.[@R13]

Some claim that a minimum level of scientific literacy is a prerequisite for developing health literacy.[@R13] The aim of compulsory science education is to develop students' scientific literacy, including the proficiency to design and evaluate scientific inquiry, and gain knowledge about how the procedures of science support or disprove claims.[@R16] School science may therefore be a key learning area for developing adolescents' proficiency to critically appraise health claims in the media. Importantly, educational frameworks promote media reports of research as important real-life contexts for advancing and assessing students' scientific literacy in terms of evidence appraisal.[@R16]

Without appropriate training, adolescents find it difficult to engage critically with media reports containing scientific content, and this challenge continues as they move from compulsory to higher education.[@R19] Studies indicate that students tend to overestimate the certainty of scientific claims and accept them at face value.[@R19] Moreover, they rely on substitute credibility indicators such as expertise (eg, researchers, journalists) and authors' use of scientific statements and prompts (eg, 'evidence-based' or 'scientifically proven') without any in-depth conceptual understanding.[@R19]

The majority of these studies reside within the body of research on scientific literacy, not health literacy. This reflects that critical thinking around science-related claims in media, including the proficiency to appraise the science behind health claims, are underscored themes in models and definitions of health literacy.[@R29] Accordingly, these issues are hardly emphasised in measures and empirical studies of adolescents' and young peoples' health literacy.[@R31]

There has been a call for studies that explore how people's scientific literacy correspond to their proficiency in accomplishing specific tasks associated with their health literacy, such as identifying and appraising health claims (National Academies of Sciences, p. 107 \[[@R35]\]). A relevant question concerns 'what someone who scores in the upper quartile on a science literacy measure can do that someone who scores in the lowest quartile cannot?' (National Academies of Sciences, p. 107 \[[@R35]\]). Our study aims to address this question, using data from a national science achievement test of Norwegian 10th grade students. We explore responses to an item designed as a brief news report of a fictitious scientific study that assessed students' proficiency to identify and appraise a health claim.

In Norway, grade 10 is the final year of compulsory education and most students are 15 years of age. According to the Programme for International Student Assessment (PISA) studies of those aged 15 years, Norwegian students perform slightly above the OECD average in science (OECD, p. 44 \[[@R36]\]) and approximately 80% and 30% perform at or above PISA proficiency level 2 and 4 in science, respectively (OECD, p. 320 \[[@R36]\]). At level 2, students can *typically* 'use common scientific knowledge to identify a valid conclusion from a simple data set' (OECD, p. 68 \[[@R36]\]) and hence identify scientific claims---a prerequisite for appraising claims.[@R20] At level 4, students can *typically* 'identify the evidence supporting a scientific claim' and draw on knowledge about scientific procedures (eg, experimental designs) to justify conclusions (OECD, pp. 72--4 [@R36]). Hence, they can most likely request further evidence when encountering unsupported science-based claims, a hallmark of critical appraisal.[@R38] Previous studies suggest that students, *if* they request information, usually emphasise methodological aspects of the reported research, the findings as such, and theoretical explanations of the findings.[@R24]

Building on knowledge from prior research and applying the national science achievement test of Norwegian 10th grade students as a measure of scientific literacy, we hypothesised that:

1.  Students who score at or above the lower quartile on the scientific literacy measure are proficient in identifying a health claim among other competing textual information.

2.  Students who score at or above the upper quartile on the scientific literacy measure are proficient in *both* identifying a health claim *and* formulate at least one request for further information relevant for appraising that claim, predominately information about either the research methods applied, the data collected or the underlying mechanisms causing an outcome.

Methods {#s2}
=======

Design {#s2-1}
------

We did a secondary analysis of existing data from a large-scale cross-sectional, web-based science achievement test assessing a random sample of the 2013 cohort of 10th grade students in Norway.

Participants {#s2-2}
------------

In 2013, the cohort of 10th grade students comprised about 64 000 individuals[@R41] distributed across 1238 schools.[@R42] Using random sampling, excluding special schools and international schools, 200 public schools were contacted for consensus of participating in the voluntary student assessment. Eligible schools were selected with a probability-proportional-to-size sampling. No schools selected themselves into the study. All schools were contacted by email and telephone between 20 December 2012 and 6 February 2013. One class at each of 97 schools---a total of 2229 students (50% females), completed the digitalised assessment during April--May 2013. We estimated the school/class average participation rate as 86%. Owing to technical shortcomings beyond our control, no data on students' socioeconomic status or ethnicity was recorded. The mean final assessment grade in science at each school was available from the Norwegian Directorate for Education and Training. On a scale from 1 to 6, where 6 is best, the sample average grade in science was 4.0---identical to the eligible population average. No experimental manipulations or interventions were part of our study.

Participant and public involvement {#s2-3}
----------------------------------

Participants were not involved in the development of any part of this study.

Study context {#s2-4}
-------------

In Norway, the integrated subject 'natural science' is a mandatory subject throughout compulsory education. At the time of the survey (spring 2013), the natural science curriculum was structured into six subject domains: 'body and health', 'diversity in nature', 'the universe', 'phenomena and substances', 'technology and design' and 'the budding researcher'.[@R43] The latter is a cross-cutting domain to ensure that *knowledge about science as a process* is integrated more systematically throughout science domains.

The national science achievement test assessed students' proficiency in science based on the competence aims in the science curriculum for grade 8--10, with assessment items distributed across the cognitive domains 'knowing' (knowledge of scientific facts, concepts and procedures), 'applying' (apply knowledge to explain phenomena and solve problems) and 'reasoning' (evaluating scientific enquiry and alike). The items were distributed across the science domains and cognitive domains as described in [online supplementary file 1](#SP1){ref-type="supplementary-material"}, the 2013 assessment emphasised the science domains 'body and health' and 'diversity in nature'.

10.1136/bmjopen-2018-028781.supp1

Test items and the administration procedures {#s2-5}
--------------------------------------------

The 54 test items constituted a sufficiently valid and reliable scale for measuring scientific literacy as defined by the Norwegian curriculum. All but the one open-constructed news item, positioned at the end of the assessment test and scored 0--4 points, were dichotomously scored selected-response items. Accordingly, the science test data were analysed against the partial credit parameterisation of the unidimensional Rasch model.[@R44] By sampling items from a bank of prior field-tested items, it was possible to construct a scale with difficulty well-targeted at the population of interest. The 'test reliability' was acceptable (Cronbach's alpha based on completely scored data=0.93; person separation index based on person proficiency estimates=0.92). Measured up against the applied Rasch model all but one item discriminated sufficiently well between students with low and high standing on the latent trait (scientific literacy), and no significant differential item functioning, violations of unidimensionality or local independence were observed. The one poorly discriminating item was discarded and the analysis was re-run. The students completed the test within 80 min at school using a digitalised assessment tool.

The open-constructed 'news item' was designed to evaluate students' proficiency in identifying and critically appraising a health claim. The item's stem was designed as a brief news report (70 words) that referred to a fictitious study concluding that eating corn regularly reduces the risk of type II diabetes. The content and format of the item was similar to the news brief items in an instrument developed by Korpan *et al,* [@R38] with no details about the study except being conducted by American scientists. In addition, there was a brief background statement about the rising global prevalence of type II diabetes along with a declaration from a diabetes interest group promoting the study findings. Students were first asked to identify the health claim in the news report, more specifically the conclusion from the fictitious study (the word 'conclusion' was used in the item's question), that is, a regular intake of corn reduces the risk of type II diabetes. Second, they were asked to generate requests for information about the study that they would need to appraise the reliability of the health claim. Students were instructed to write a maximum of one and two sentences for the health claim and requests, respectively. A 250-character limit on students' responses was imposed by the electronic assessment system (beyond our control). Responses to the news item allowed us to assess aspects of students' *functional* and *critical* health literacy,[@R46] more specifically their comprehension of health information and claims, and their ability to critically appraise claims. The item has been retained for continuous test use and is thus unavailable for publication.

Analysis {#s2-6}
--------

We coded responses to the news item using a coding guide of assessment criteria that reflected both credited and non-credited responses with regard to identifying the health claim (first part of the item) and requesting information about the study referred to in the item's stem (second part of the item). The process of coding students' information requests was based on a taxonomy for classifying questions and knowledge about scientific research.[@R38] See [table 1](#T1){ref-type="table"} for an overview of the coding guide, including the taxonomy's main scientific research categories (eg, methods). We continually improved the coding guide during field tests and clarified it by including examples of authentic student responses (see [online supplementary file 2](#SP2){ref-type="supplementary-material"} for a complete version of the guide).

10.1136/bmjopen-2018-028781.supp2

###### 

Overview of coding guide for the news item

  Part 1: health clam                
  ---------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **Credited response**              *A response...*
   Complete                          providing a complete account of the claim (ie, that a regular intake of corn reduces the risk of type II diabetes).
   Mostly complete                   providing a mostly complete account of the claim, with some significant words lacking (ie, 'regular' and/or 'type II'), and/or referring to amount of intake (eg, 'much' or 'more' corn).
  **Non-credited response**          *A response...*
   Wrong                             where the claim relates to the topic of the news report, but is otherwise wrong.
   Vague                             with no reference to *corn* and/or *type II diabetes.*
   Other                             which is irrelevant or a 'do not know' response.
   Blank                              
  **Part 2: information requests**   
  **Credited response**              *A response relating to...*
   Methods                           *how* the study was conducted, including study design, subjects, procedures and measurements.
   Data/statistics                   *what* was observed in the reported study, or about statistical tests used to analyse the data.
   Theory/agent                      *why* the reported effects might have occurred, including questions about the properties of the presumed causal agent and/or possible underlying mechanisms.
   Social context                    the credentials and bias related to who did the study or funded it and where it was conducted or published.
   Relevance                         the importance or applicability of the study findings, or the impact of the study.
   Related research                  whether the findings have been replicated or fit results from previous research.
   Ambiguous                         the study described in the news report that is ambiguous because it fits under two or more scientific categories.
  **Non-credited response**          A *response*...
   Future studies                    indicating the need for one or more *future* studies, either in general, or relating specifically to one of the scientific research categories (methods, data, etc).
   Disbelief                         indicating that the student does not believe that the study has been conducted.
   Wrong                             relating to the topic of the news report, but is otherwise wrong.
   Vague                             only vaguely referring to the scientific categories (methods, data, etc).
   Other                             which is irrelevant or is a 'do not know' response.
   Blank                              

When coding part 1, the raters applied one variable and used values starting with '1' and '0' to indicate whether the response included an acceptable account of the health claim or not. For part 2, the raters applied eight variables. Seven of these were labelled to reflect the scientific research categories (methods, etc), raters used the values '1' and '0' to indicate whether the response included an acceptable request for information within the specific category. For the eight variable, non-credited responses to part 2, values starting with '0' were used to indicate type of response. Blank responses (part 1 or part 2) were coded with the value '99'.

One rater coded all student responses and consistency was evaluated by using an additional rater who coded 25% of the responses. Inter-rater agreement (ØG and KSP) for the health claim was 94% and improved to 96% after discussion. Inter-rater agreement (LVN and KSP) for the information requests was 86% and improved to 98% after discussion. The lower initial agreement rate was mainly owing to interpretation of responses that concerned the need for future studies (see specifications in [table 1](#T1){ref-type="table"}). As the item's stem explicitly asked students to relate their requests to the specific study presented in the news report, our final decision was not to credit responses that concerned 'future studies'.

Overall, we credited students' responses to the news item according to a 'full credit' (4 points), 'partial credit' and 'no credit' (0 point) system as specified in the scoring guide ([table 2](#T2){ref-type="table"}). This cumulative scoring guide made it possible to identify a student's skill simply by knowing that student's item score. We considered it unlikely that students who failed to identify the health claim were able to request information needed to establish the reliability of that claim. Thus, an acceptable account of the claim, as specified in [table 1](#T1){ref-type="table"}, was a premise for being *credited* on the item.

###### 

News item scoring guide

  Credit           Score categories   Type of response
  ---------------- ------------------ -----------------------------------------------------------------------------------------------------------------------------------------------------------
  No credit        0                  Wrong or vague health claim, irrelevant or blank response.
  Partial credit   1                  Acceptable account of the health claim.
                   2                  Acceptable account of the health claim and requests relating to one scientific research category (eg, methods).
                   3                  Acceptable account of the health claim and requests relating to two unique scientific research categories (eg, methods and data).
  Full credit      4\*                Acceptable account of the health claim and requests relating to three or more unique scientific research categories (eg, methods, data and theory/agent).

\*No student made more than three unique requests for scientific information. Accordingly, full credit on the item was set to 4 score points.

The software package RUMM2030 was used for Rasch modelling.[@R47] Using unidimensional Rasch modelling, one may construct a scale and locate each item's threshold(s) on that scale. A dichotomously scored item has one threshold reflecting the difficulty of the item, and a polytomously scored item has *k*−1 thresholds reflecting the difficulty of its *k* score categories.[@R48] The news item had five score categories ([table 2](#T2){ref-type="table"}), and four ordered thresholds reflecting the difficulties of each score. We located the four thresholds on the left side of the scale in [figure 1](#F1){ref-type="fig"}. The scale was made up of observable behaviours---the specific achievements associated with each threshold of the news item described in [table 2](#T2){ref-type="table"}. These observed achievements were governed by the students' proficiency in science (scientific literacy)---the underlying but unobservable latent trait. On the right side of the scale ([figure 1](#F1){ref-type="fig"}), we located the person (student) proficiencies associated with the 10th percentile, the quartiles and the 90th percentile. The possibility of locating item thresholds (difficulties) and person proficiencies on the same logit-scale, is a benefit of using Rasch modelling. We used the information in [figure 1](#F1){ref-type="fig"} to test both our hypotheses.

![Difficulty of each score on the news item (left side) and the proficiencies associated with different percentiles (right side) on the scientific literacy scale.](bmjopen-2018-028781f01){#F1}

Missing data were handled using pairwise maximum likelihood estimation for the item location estimates---a so-called full information method. During field trials, items displaying 'differential item functioning' (DIF) for central person factors were revised or discarded. DIF means that for example males and females or minority and majority students with the same proficiency estimate have different probabilities of responding correctly. Hence, items displaying DIF are biased as gender and/or cultural background significantly influences students' responses. An example is an item assessing how hormones influence the menstruation cycle. This item probably uniformly favours females at all proficiencies along the latent trait scale.

Results {#s3}
=======

Two-thirds (64%) of the students identified the health claim, of whom only half gave a complete account of it ([table 3](#T3){ref-type="table"}). [Figure 1](#F1){ref-type="fig"} shows that the difficulty associated with identifying the health claim was 0.0 logits (score point 1), that is, it equals the mean difficulty of the test items in the national achievement test, which was set to 0.0. Accordingly, the average scientific literate student was likely able to identify the health claim in the brief news story. Students who reached the lower quartile on the scientific literacy scale were *not* likely to identify the claim, as their proficiency (−0.5 logits) was 0.5 logits below the difficulty estimate of score point 1. Hence, hypothesis 1 was weakened as students' skills were much poorer than we expected based on our interpretation of PISA results.

###### 

Proportion of students who identified the health claim of the (news item part 1) and the proportions who requested different types of information for appraising that claim (part 2)

                                      N (of 2229)   \%
  ----------------------------------- ------------- -----
  **Part 1:** **h** **ealth claim**                 
  **Credited responses**              1420          64
   Complete                           710           32
   Mostly complete                    710           32
  **Non-credited responses**          809           36
   Wrong, vague or other              415           18
   Blank                              394           18
  **Part 2: information requests**                  
  **Credited responses**              652\*†        29†
   Methods                            376           17
   Data/statistics                    189           
   Theory/agent                       146           7
   Social context                     57            3
   Relevance                          12            \<1
   Related research                   9             \<1
   Ambiguous                          31            1
  **Non-credited responses**          1577          71
   Future studies                     365           16
   Disbelief                          79            4
   Wrong, vague or other              618           28
   Blank                              515           23

\*Comprise all students who made one or more information requests, including the 50 students who were not credited on the news item part 1 (health claim).

†The total sum of requests will exceed 652 as 154 students made requests relating to more than one unique scientific category.

Less than one-third of the students (29%) made one or more information requests about the reported study relevant to appraise the health claim ([table 3](#T3){ref-type="table"}). [Figure 1](#F1){ref-type="fig"} indicates that the difficulty associated with score point 2 (1.5 logits)---identifying the claim and making one request for information, was rather close to the proficiency associated with the upper quartile (1.4 logits). Therefore, hypothesis 2 was strengthened. However, students located even at or below the 90th percentile (2 logits) were not likely to score \>2 points on the news item, that is, identifying the health claim and making more than one request for information.

A few responses (n=115) exceeded the 250-character limit and were thus truncated by the assessment system. By coding and analysing these responses, we concluded that the technical deficit might have constrained the opportunities to achieve a higher score (ie, to be credited for further information requests) for 31 students only.

Characteristics of students' information requests {#s3-1}
-------------------------------------------------

As shown in [table 3](#T3){ref-type="table"}, and in line with hypothesis 2, the most frequent requests were related to how the study had been conducted (methods), the data collected (data/statistics) and the theoretical explanations of the results (theory/agent). The requests across these topics varied in level of detail ([table 4](#T4){ref-type="table"}). More than half of the requests about data/statistics were rudimentary. In comparison, all requests about theory/agent were specific, for example, concerning what active ingredient in corn actually caused the preventive effect. Methods was the only topic where several students made more than one request about specific features of the topic. Nearly half of these requests concerned the study participants, primarily the sample size (126 of 230 requests). Less frequent were requests about design, including the control of confounding variables and use of control groups (33 of 60 requests). As these requests belonged to the same unique scientific research category (methods), they were credited only once ([table 2](#T2){ref-type="table"}).

###### 

Characteristics of the requests related to methods, data and theory (specifications of [table 3](#T3){ref-type="table"})

  Type of request                 Examples of students' responses                                                              No. of requests   \%
  ------------------------------- -------------------------------------------------------------------------------------------- ----------------- ---------
  **Methods**                                                                                                                  **471**           **100**
   Rudimentary                    How was the study conducted?                                                                 80                17
   Design                         Did they use a control group? How long did the study last?                                   60                13
   Agent delivery (procedure)     How much corn is necessary to eat? How often?                                                100               21
   Participants                   How many were tested? Who participated in the study? What was their eating habits?           230               49
   Measures                       Did they measure the participants' blood sugar?                                              1                 \>1
  **Data/** **s** **tatistics**                                                                                                **191**           **100**
   Rudimentary                    We need the results.                                                                         111               58
   Absolute nature of data\*      For how many did this (not) work?                                                            32                17
   Comparative nature of data\*   What is the reduction in risk? How large was the effect?                                     43                23
   Duration of effect             Does it work over time?                                                                      5                 3
  **Theory/** **a** **gent**                                                                                                   **149**           **100**
   Identification                 We need to know what corn is composed of (nutrients).                                        2                 1
   Agent mechanisms               What ingredient in corn prevents type II diabetes? Why does corn prevent type II diabetes?   117               79
   (Side) effects                 Is there any side effect of eating corn?                                                     23                15
   Alternative agents             Do other corn products have this effect?                                                     7                 5

\*For the dependent variable (here: diabetes risk).

Seventy-one per cent of students provided a response that was either blank or otherwise disapproved, and were thus assigned a 'non-credit' category. The average proficiency estimate for this group was 0.0 logits, which equals the difficulty of identifying the health claim. Students who made suggestions for the conduct of future studies, rather than making requests for information related to the study reported in the news story (and thus were not credited), performed somewhat better on the achievement test (average proficiency 0.64 logits).

Discussion {#s4}
==========

We assessed how 10th grade students' levels of scientific literacy corresponded to their proficiency in identifying and critically appraising health claims in the news media---two essential aspects of health literacy. The findings weakened our first hypothesis, as only the average scientific literate students aged 15 years leaving compulsory school was able to identify a clearly stated health claim in a rather simple news report. Students performing at the upper quartile of the scientific literacy measure were barely proficient at identifying and appraising the claim, namely making a request for evidence needed to determine the reliability of the claim. Accordingly, our second hypothesis was strengthened.

About half of the invited schools participated, and the average student participation rate at these schools was at an acceptable 86%. Our analyses indicated that the sample school average grade and gender distribution matched the population distributions at grade 10, thus indicating generalisability of our findings. Although data on socioeconomic status and ethnicity was unavailable for this study, previous studies have found these factors to predict science proficiency in Norwegian students aged 15 years.[@R36] For socioeconomic status, however, the relationship with proficiency is relatively weak compared with most other countries.

Previous studies of students' evaluations of scientific claims have mostly been conducted at upper secondary school level and above, and on smaller, mostly self-selected samples of students.[@R24] Thus, to our knowledge, this is the first study investigating students' critical appraisal of science-based health claims in the context of a large student sample at lower secondary school level. Furthermore, while students' scientific health knowledge in important areas such as chronic and infectious diseases and their knowledge of sources to science-based health information has previously been explored,[@R51] we have addressed a call for research on the utility of scientific literacy for critical appraisal of health claims.[@R35] The analysis of responses to the news item provided useful information about *how* proficient students were (ie, the levels of scientific literacy necessary to involve in identifying and appraising health claims in news reports of science), and *what* kind of knowledge they possessed and applied when approaching such claims (ie, which responses earned credits and which did not). Such in-depth knowledge of students' thinking around a topic or task is an important outcome of secondary analyses of individual test items used in large-scale surveys, as it may formatively inform and develop teachers' practices.[@R53]

There were some limitations to this study. First, the test data were collected 6 years ago (2013), thus we acknowledge that a possible shift in students' knowledge might have occurred. Nevertheless, our study is timely due to a major revision of the curriculum that will be implemented from autumn 2020. Second, to avoid response dependence between similar items, and accordingly violations of local independence in the data, the test comprised only one of the news item developed and field-tested. This prohibited us from evaluating whether variations in text dimensions (eg, the claim's plausibility and how familiar the students were with the health topic) could have influenced students' information request---dimensions previously reported to impact on students' critical engagement with news reports.[@R37] For the same reason, and because the news item did not include any embedded attitudinal items, we were unable to assess whether important personal factors (eg, interest in the health topic, belief in the claim, scientific attitude)[@R24] could have affected students' requests. Third, to make students respond shortly and 'on task', they were encouraged to write only two brief sentences. This might have constrained their opportunities to make several requests for evidence regarding the claim, although our analysis of incomplete responses due to the limit of 250 characters indicated this was probably not the case. Moreover, our findings resemble previous studies with regard to both the number and type of requests made.[@R24] Finally, while it is common practice to retain test items for re-use, this implies a lack of transparency in the test data used for this study.

Being able to correctly identify the nature of information included in media reports of science is a prerequisite for critical appraisal. High school and university students exposed to news reports containing a variety of scientific features, often confuse conclusion statements with statements about the results (data) and explanations (theory).[@R20] In comparison, the news item in our study was less complex, including only a few statements beside the health claim (study's conclusion). Still, only students being proficient at or above the average in science managed to identify the claim, often providing an incomplete account of it. Therefore, the underlying problem seems to be the same as noted in previous studies, namely a lack of training in *reading* media reports of science.[@R20] As previously noted, we were unable to explore whether the students perceived the claim as plausible or not. The former might be the case as uncertainty about a claim's plausibility has been found to provoke more methods questions.[@R37] For instance, students' occasional requests about procedures indicate that they believed in the claim, and thus simply wanted information about how much, how often and how long the intake of corn should be to see the reported effect on diabetes risk. A further observation was the few requests about the social context of the research (eg, the American scientists' affiliation), perhaps suggesting that students regarded science and scientists as authoritative and accordingly that the claim was plausible. This has also been noted in previous studies of students across educational levels.[@R19]

Our findings are of concern as they illustrate students' functional and critical health literacy at the end of compulsory school. Almost three-quarters of the 10th grade students were unable to identify *and* critically appraise the health claim in a brief news report. Even the highest performing students mostly requested only one scientific research category of an optimum of six broad categories. Despite curricular mandates to develop scientific literacy and critical appraisal skills important for health literacy, it seems like students' actual skills are underdeveloped and not taught in a way that improve their appraisal of health claims as assessed by the news item. This is consistent with findings from a qualitative study where science teachers reported that opportunities for teaching critical appraisal during inquiry-based activities, such as online health information seeking or small-scale experiments, are lost in the need to emphasise factual knowledge on health topics.[@R54] Importantly, teachers do not acknowledge the relevance of teaching critical appraisal, or they lack methods to teach them. Thus, there is clearly a prospect for cross-sector collaboration between healthcare and education professionals and researchers to work together to enable laypersons to appraise health claims, as pointed out by Sharples *et al*.[@R14] Research is still scarce as to which interventions best improve students' ability to appraise health claims.[@R55] However, a recent cluster-randomised controlled trial shows promising effects of a cross-disciplinary developed intervention aimed at teaching primary school children to appraise claims about treatment effects.[@R57]

Our study has identified specific areas that require attention in further development and evaluation of interventions---areas that align with important key concepts lay people need to know to assess health claims.[@R11] It was encouraging that the students in our study*---when* they employed scientific criteria---were sensitive to methodological information, which often is lacking in media reports of health research.[@R7] However, students requested only a limited range of methodological evidence, with little attendance to details about the study design, such as the use of control groups or control of confounding variables. This was noteworthy given the news report's assertiveness in claiming a causal relationship between the intake of corn and the reduced risk of type II diabetes, mirroring the many misleading media reports that fail to differentiate association from causation.[@R6] Science instruction should therefore develop students' knowledge of good and weak designs for establishing a cause-effect relationship, including the design of controlled studies, the importance of fair comparisons, the principles of randomisation and blinding and proper and improper ways of reporting outcomes (eg, absolute vs relative risk). Existing evidence suggests that such knowledge is better gained through teacher-guided investigations that allow students to reflect on adequate and inadequate experimental strategies, rather than through student-led hands-on or virtual experiments.[@R58] Importantly, teachers need to make explicit the link between experiments, critical reading, and appraisal of health claims in the new(s) media. In our study, several students suggested the conduct of a future study rather than requesting information about the reported study, often involving themselves in doing so (eg, "we have to test a number of people"). This perhaps supports the notion that teachers enforce experimentation and hands-on activities without linking relevant learning outcomes to reading and critically appraising science presented in out-of-school contexts, including media reports.[@R59] Finally, students hardly requested related research supporting or disproving the claim. Accordingly, teaching could sensitise students to the limitations of single studies, introducing the idea of systematic reviews.

Supplementary Material
======================

###### Reviewer comments

###### Author\'s manuscript

The authors would like to thank Jorån Østerholt Dalane who, together with KSP, developed the original version of the news brief item. The authors would also like to thank Bjørn Vidnes, Anders Isnes and Kirsten Fiskum who, together with ØG and LVN, developed all the other science achievement test items.

**Contributors:** LVN, KSP and ØG developed a revised version of the news brief item and the coding guide, and coded the student responses. ØG administered the survey, managed the data handling, conducted the Rasch analysis and constructed the achievement scale. LVN wrote the preliminary draft of the paper with contributions from ØG. The paper was critically reviewed by KSP, BE and SAF for important intellectual content.

**Funding:** The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

**Competing interests:** None declared.

**Patient consent for publication:** Not required.

**Ethics approval:** The data set used in this paper did not include information that identified individuals, so ethical approval was not required under Norwegian regulations.

**Provenance and peer review:** Not commissioned; externally peer reviewed.

**Data availability statement:** No additional information is available. A data file with a sample of the coded student responses (in Norwegian) is available on reasonable request. Please contact Øystein Guttersrud (oystein.guttersrud\@naturfagsenteret.no).
