Input:
"
Here is some messy data parsed from a pdf. These files may contain redacted information.
Using a json format, ouput the who, what, when, and where of the document, and required security clearence
Use the following json format:
{"Who": "key names or groups", "What": "keywords or ideas from document", "When": "day/month/year", "Where": ["referenced locations"], "Security Clearance": "CONFIDENTIAL, SECRET, or TOP SECRET"
} Text: # OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Integrity and Standards Evaluation Program:

Key Results from FY2015 (ONI & IC)

Analytic Integrity and Standards

Office of the Deputy Director of National Intelligence for Intelligence Integration

August 2016

20-cv-02020 (DF-2020-00235) 000035

SECRET
---
# Rationale for ODNI Analytic Evaluation Efforts

Reviews of finished intelligence product or other analytic products" (IRTPA Section 1019)

ODNI's Analytic Integrity & Standards (AIS) group "shall perform, on a regular basis, detailed evaluation criteria: ICD 203 IC Analytic Standards

|Criteria|Evaluation|
|---|---|
|A. Objectivity|Properly describes quality; reliability of sources|
|B. Independent of Political Considerations|Properly caveats and expresses uncertainties or confidence in analytic judgments|
|C. Timeliness|Properly distinguishes between underlying intelligence and analysts' assumptions, judgments|
|D. Based on All Available Sources of Intelligence|Incorporates Alternative Analysis|
|E. Exhibits Proper Standards of Analytic Tradecraft|Relevance to US national security|
| |Logical argumentation|
| |Consistency, or highlights change|
| |Accurate judgments, assessments|
| |Effective visual information (New; wasn't rated in FY15)|

# Components:

- Product evaluations
- Analyst surveys
- Consumer interviews
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Distribution of ONI Product Evaluations

|Products Evaluated|30 FY15 ONI products evaluated|33 FY14 products evaluated|
|---|---|---|
|Evaluations focus on ONI Today, Baseline Assessments, and Maritime Intelligence Reports|FY15 Product lines|(30 Products)|
|Production dates range from October 2014 to May 2015| | |

| |FY15 Product lines|
|---|---|
|DONI Today|27% (8)|
|Baseline Assessment|43% (13)|
|Maritime Intelligence Report|30% (9)|

| |FY14 Product lines (33 Products)|
|---|---|
|DONI Today|9% (3)|
|Baseline Assessment|18% (6)|
|Maritime Intelligence Report|30% (10)|
|Custom|3% (1)|
---
|Evaluations Profile|35|
|---|---|
|65% of all ratings "Good" or "Excellent"|30|
|"Excellent" ratings|25|
|Five for Sourcing|20|
|One for Distinguishing|5|
|11 "Poor" ratings|15|
|Two each for Sourcing and Five for Alternatives|15|
|Relevance|10|
|One each for Uncertainty and Logic|5|

Median scores: "Good" on four standards, "Fair" on Alternatives and Relevance

| |Sourcing|Uncertainty|Distinguishing|Alternatives|Relevance|Logic|
|---|---|---|---|---|---|---|
|Standard| |Poor|Fair|Good|Excellent|Median|
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

ONI: Rating Comparisons (FY15/FY14/FY15 IC-Wide)

| |Sourcing|Uncertainty|Distinguishing|Alternatives|Relevance|Logic|
|---|---|---|---|---|---|---|
|Overall Ratings|No significant change to| | | | | |
|Looking at individual standards:| | | | | | |
|Distinguishing improve|Sourcing, Uncertainty| | | | | |
|Ratings for Relevance and| | | | | | |
|Logic decline| | | | | | |
|Larger share of| | | | | | |
|ratings for Alternatives| | | | | | |
|Compared to IC| | | | | | |
|ONI surpasses the IC in all| | | | | | |
|standards except|Relevance| | | | | |
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# FY 15 Product Ratings

|Evaluated FY15 ONI Products|Evaluated FY14 ONI Products|All IC Evaluated FY15 Products|
|---|---|---|
|All Good or Excellent|At least 1 Poor|All Good or Excellent|
|13% (4)|23% (8)|12% (b)(3)|
|All Fair; Good Excellent|All Fair; Good Excellent|All Fair; Good Excellent|
|64% (19)|70% (237)|66% (b)(3)|

Percent of Products with All "Good" or "Excellent" Ratings: IC All: 150

With 1 being the best and 20 being the worst, ONI is tied for 11th

Percent of Products with at least One "Poor" Rating: IC All: 190

With 1 being the best and 20 being the worst, ONI is tied for 12th
---
| |Percent of All Judgments, Unclear|
|---|---|
|Unconditioned|33%|
|Conditioned|62%|
|Single Source|5%|
|IC All Source|45%|
|Services|48%|
|Domestic|0%|

| |Percent of Future Judgments, Tested Accurate|
|---|---|
|IC-wide Future Judgments|44%|
|Tested Accurate|56%|
|Tested Inaccurate|13%|
|IC All Single Source|87%|
|Services|3%|
|ONI Domestic|0%|
---
|Category|Standard|Percent Good or Excellent|Top Performers|
|---|---|---|---|
| |Distinguishes|90% ONI 87% IC|DOE, IICT, OSE, OIA, USCG|
|Strengths|Sourcing|83% ONI 47% IC|DOE, NGA, ONI, OSE, QIA|
| |Uncertainties|67% ONI 63% IC|NGA, OSE, OIA|
| |Logic|57% ONI 55% IC|NGA, NIC, OSE|
|Areas for Improvement|Alternatives|47% ONI 28% IC|DOE, FBI, NIC|
| |Relevance|40% ONI 64% IC|FBI, IICT, NIC|

Recommendations: [0b)(3), (6)(5)

20-cv-02020 (DF-2020-00235) 000042
---
# Analytic Objectivity & Process Survey: Overview of FY15 Program and Results

Background: Approximately ((b)(3) analysts and managers responded to the 2015 survey. Of those respondents, 149 were analysts and managers from ONI; significantly less than the number who participated in 2014 (301 analysts and managers) and 2013 (194 analysts and managers). The work role distribution of all respondents in 2015 was similar to historical distributions, but the tenure composition of survey participants changed slightly from previous years. Comparing the tenure composition of respondents, ONI respondents in 2015 and over the past five years exhibited similar tenure levels to those of the IC as a whole.

| |ONI|All|
|---|---|---|
|Work Role|2015|2011|2015 Avg|2015|2011|2015 Avg|
|Analysts|86%|81%|80%|84%|
|Managers|14%|19%|20%|16%|
|0-5 years|32%|36%|25%|39%|
|6-10 years|25%|29%|21%|24%|
|10+ years|43%|34%|53%|38%|

Key finding: Most respondents (85% All, 84% ONI) reported workplace conditions that enabled them to produce analysis that was objective and free from political distortion.

Footnote: 20-cv-02020 (DF-2020-00235) 000043
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Objectivity & Process Survey: Analysts' Perceptions of Attempted Distortion

| |Percent indicating "Yes"|2015|2011 - 2015 Avg|Syr Trend|2015|2011|2015|Syr Trend|
|---|---|---|---|---|---|---|---|---|
|During the past year; has anyone attempted to distort or suppress analysis on which you were working in the face of persuasive evidence?|16%|11%|10%|9%|15%|13%|10%|9%|
| |(20 of 126)| |596| |((b)(3)| | | |
| |MOE+ -8%| |8888| |MOE : +/- 2%| | |88888|

The figure below represents the percentage of analysts answering Yes to the question above in 18 different IC elements. With 1 being the best and 18 being the worst; ONI ranks 12th.

0%
5%
10%
15%
20%
25%
30%

Key: 20-cv-02020 (DF-2020-00235) 000044
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

E A0 ! # N T : L L [6 E [ N T E 6 R A T [ 0 N

# Analytic Objectivity & Process Survey: Analysts' Perceptions of Attempted Distortion

| |ONI 2015|2013 - 2015 Average|3yr Trend|AII 2015|2013 - 2015 Average|3yr Trend|
|---|---|---|---|---|---|---|
|Percentage of respondents experiencing attempted distortion who sought help to resolve the most recent incident.|65% (13 of 20)|70%|59%|70%|64%|60%|
|Of those who sought help, the percentage of respondents experiencing attempted distortion who received help that partially or wholly resolved the most recent incident.|77% (10 of 13)|68%|61%|70%|66%|60%|

Note: questions are presented for survey years 2013 - 2015.

In 2013, the AOPS questions above were redesigned such that continuity with prior year results may be compromised. As such; results for the pertinent 20-cv-02020 (DF-2020-00235) 000045
---
|What Product?|ONI 2015|ONI 2011 - 2015 Rank|ONI 2011 - 2015 Average|AII 2015|AII 2011 - 2015 Rank|AII 2011 - 2015 Average|
|---|---|---|---|---|---|---|
|Written analytic product|75%|#1|81%|85%|#1|79%|
|Intelligence briefing|45%|#2|27%|33%|#2|23%|
|President's Daily Brief (PDB)|5%|#6|6%|8%|#3|15%|
|Non-text analytic product|10%|#5|8%|8%|#6|8%|
|NIC product|10%|#4|11%|7%|#5|8%|
|Other|25%|#3|12%| |#4|10%|
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

Analytic Objectivity & Process Survey: Analysts' Perceptions of Attempted Distortion

|Who Attempted?|ONI|AII|
|---|---|---|
|(Select all that Apply)|2015|2011 - 2015|2011 - 2015|2015|2011 - 2015|2011 - 2015|
| |Rank|Average| |Rank|Average|
|Supervisor in your IC element|40%|#2|41%|72%|#1|57%|
|Intelligence professional outside your IC element|45%|#1|42%|38%|#2|36%|
|Non-supervisory professional in your IC element|60%|#3|37%|38%|#3|28%|
|USG official outside of IC|25%|#4|20%|11%|#4|14%|
|Other|10%|#5|8%|3%|#5|5%|

Footnote: 2

20-cv-02020 (DF-2020-00235) 000047
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Objectivity & Process Survey: Analysts' Perceptions of Attempted Distortion

|What was the Motive?|(Select all that Apply)|2015|2014 - 2015|Rank|Average|
|---|---|---|---|---|---|
|To support the viewpoint or keep from offending IC personnel or management outside your IC element|20%|35%|#1|31%| |
|To support the viewpoint or keep from offending a supervisor within your IC element|10%|28%|#5|17%| |
|To support the viewpoint or keep from offending a government official outside the IC|25%|33%|#3|26%| |
|To avoid conflicting with an existing analytic line|45%|New in 2015|#6|43%| |
|Inadvertently unintended (e.g: formatting; bureaucratic processes)|10%|21%|#4|15%| |
|Unsure of motive|25%|29%|#2|22%| |
|Other|15%|17%|#6|22%| |

Footnote: 2

20-cv-02020 (DF-2020-00235) 000048
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Objectivity & Process Survey: Analysts' Perceptions of Management's Protection of Objectivity

| |IMMEDIATE SUPERVISOR|IMMEDIATE SUPERVISOR|ELEMENT SR MGMT|ELEMENT SR MGMT|ODNI|ODNI|
|---|---|---|---|
| |2015|2011 - 2015|2015|2011 - 2015|2015|2011 - 2015|
|Satisfactory|88%|86%|87%|86%|76%|75%|
|Mixed Performance|9%|11%|9%|10%|20%|21%|
|Unsatisfactory|3%|3%|3%|4%|4%|4%|
|Margin of Error (+/-)|8%| |2%| |8%| |

80% 80% 80% 80% 80% 80%

60% 2014 60% 60% 60% 60% 60%

2011 2012 2015 2011 2012 2013 2014 2015 2011 2012 2013 2014 2015 2012 2013 2014 2015 2011 2012 2013 2014 2015

| |ONI|ONI|ALL|ALL|ONI|ONI|ALL|ALL|ONI|ONI|ALL|ALL|
|---|---|---|---|---|---|---|
|Unsatisfactory| | | | | | | | | | | |
|Mixed Performance| | | | | | | | | | | |
|Satisfactory| | | | | | | | | | | |

15

20-cv-02020 (DF-2020-00235) 000049
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

Analytic Objectivity & Process Survey: Analytic Collaboration

| |2015|2011 - 2015 Avg|Syr Trend|2015|2011 - 2015 Avg|Syr Trend|
|---|---|---|---|---|---|---|
|How often do you interact for analytic purposes with intelligence professionals outside of your immediate employment unit but within your IC element?|63% (87 of 139)|66%|8096|67%|8096| |
| |60%| |7096| |7096| |
| |50%| |5096|50%| | |
| |MOE+/-8%| |40%688888|MOE+-2%|4096 8888| |

The figure below represents the percentage of analysts answering daily or weekly to the question above in 18 different IC elements.

IC AII: 71%

Key:

40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

16

20-cv-02020 (DF-2020-00235) OOOOSU
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Objectivity & Process Survey: Analytic Collaboration

| |2015|2011 - 2015|Syr Trend|2015|2011 - 2015|Syr Trend|
|---|---|---|---|---|---|---|
|Percent indicating "Daily" or "Weekly"| | | | |45%|53%|IC All: 53%|
|How often do you interact for analytic purposes with intelligence professionals outside of your IC element but within the IC?|63 of 139|49%|509|49%|40%|3096|
| |MOE+- 8%| | |MOE+- 2%| |8 888|

The figure below represents the percentage of analysts answering daily or weekly to the question above in 18 different IC elements.

Key: 20-cv-02020 (DF-2020-00235) 00O05T
---
| |2015|2011 - 2015 Average|5yr Trend|2015|2011 - 2015 Average|5yr Trend|
|---|---|---|---|---|---|---|
|Percent indicating "Yes"|92%|85%|82%|85%|70%|75%|
|When making judgments is the consideration of alternatives (aka analysis of alternatives) a routine part of your workflow process?|(111 of 121)|83%|75%|(6)(3)|70%|75%|
| |MOE +-9%| | |MOE +-2%| | |

The figure below represents the percentage of analysts answering Yes to the question above in 18 different IC elements. Among the 18 elements the ONI ranks 2nd:

(6)(3)
82%

50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Objectivity & Process Survey: Access to Information

|Percent indicating "Daily" or "Weekly" or "Yes" (Survey Years 2011-2013)|2015|2011 - 2015 Average|5yr Trend|2015|2011 - 2015 Average|5yr Trend|
|---|---|---|---|---|---|---|
|How often do you have difficulty obtaining access to intelligence reports and/or finished analytic products needed to support your analytic work?|27% (33 of 124)|29%|20%|28%|30%|20%|
|MOE +- 8%|8 8 3 8|MOE +- 2%|88888| | | |

The figure below represents the percentage of analysts answering daily or weekly to the question above in 18 different IC elements. Among the 18 elements the ONI ranks 6th. IC All: 28%

OML [(b)(3) 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70%

Note: Prior to 2014, respondents were asked "Do you have difficulty in obtaining access to intelligence reports and/or finished analytic products needed to support your analytic work?" and were given response options of "Yes" or "No." Responses of "Yes" in 2010-2013 and responses of "Daily" and "Weekly" in 2014 were used in 2014 to produce the chart above.

Key: 19 20-cv-02020 (DF-2020-00235) 000053
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Objectivity & Process Survey:

|Keys|Footnotes|
|---|---|
|A:|Margins of error (MOE) were estimated using a 95% confidence level. Organization population size was estimated from input provided by NIAB membership in response to an April 2013 AIS data call. All MOE calculations were rounded up to the next whole number:|
|ONI|1: MOE +/-2% for All, +/-8% for ONI|
|IC Element result with MOE < +/- 10%| |
|IC Element result with MOE 2 +/- 10%| |
|95% Confidence Interval|2: Results include only respondents indicating that someone attempted to distort or suppress analysis on which they were working in defiance of persuasive evidence during the past year; MOE +/-2% for All, +/-20% for ONI|
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Consumer Interview Program (ACIP): Overview of FY15 Program and Results

Background: AIS interviewed consumers from NSC, DOC, DOD, DOE, DHS, DOJ, ONDCP; State, USAID, and Treasury between March and October 2015. Additionally, AIS staff met with NIMs related to each focus area to create tailored questions in order to collect feedback and provide insight on select targets and goals of interest to the NIMs timeliness; accuracy; usefulness, and sourcing of finished analytic products.

Key Findings: Interviewees gave the IC high marks for the objectivity. Results in 2015 were comparable to the aggregate ratings from 2011-2015 for objectivity; timeliness, accuracy; usefulness, and sourcing. Consumers were most satisfied with usefulness and least satisfied with sourcing.

|2015 focus areas|(b)(1)|(b)(3)|
|---|---|---|
|15% indicated routinely receiving ONI products (consumers primarily from DOD, Office of Naval Research, and DOE)|21|21|
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Consumer Interview Program: Quality of IC-Wide Analytic Products & Support

2015 interviewees provided median ratings comparable to aggregate 2011-2015 scores for the topics of Objectivity, Timeliness, Accuracy; Usefulness, and Sourcing:

| |Objectivity|Timeliness|Accuracy|Usefulness|Sourcing|
|---|---|---|---|---|---|
|Strongly Agree|8|1|8|1|8|
|Moderately Agree|Soetuiness|Accuracy|Moderately Agree|Moderately Agree|22|
|Continued...|Moderately Agree_2011-2015 Median2015 Median| | | | |

Ratings for 2015 continue to show the trend of overall satisfaction.
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

Analytic Consumer Interview Program: Consumer Ratings on the Objectivity of IC Products

2015 interviewees ratings on the Objectivity of IC products varied the most across focus areas and organizational groups.

|100%|(b)(3), (b)(T)(E)|Strongly Agree|(b)(3), (b)(Z)E|Consumers gave|
|---|---|---|---|---|
|80%| |Moderately Agree| |higher ratings for Objectivity than other focus areas, while State consumers|
|60%| |Neither Agree nor Disagree| |were the most critical.|
|40%| |Moderately Disagree| |Analytic judgements are not biased by outside actors/politics Perceived|
|20%| |Strongly Disagree| |biases are attributed to differences in agency perspectives/agendas (63%).|
|0%|Median| | |IC is considered to be objective; but utilized/tasked by policymakers. IC is frequently pulled into policy/political issues (16%).|

2015OrgGroups

20-cv-02020 (DF-2020-00235) 000057
---
|Consumer Ratings on the Timeliness of IC Products|Consumer Ratings on the Timeliness of IC Products|
|---|
|Analytic Consumer Interview Program:|2015 interviewees gave high and similar ratings across all focus areas and organizational groups on the Timeliness of the IC products they have received.|
|100%|[6)(3), (DJ(T7)(E) - Strongly Agree|
|80%|Moderately Agree - highest ratings among this year's focus areas, while State consumers saw IC products to be more timely than other organizational groups|
|60%|Neither Agree nor Disagree|
|40%|Moderately Disagree|
|20%|Strongly Disagree - Information provided in IC analytic products is often OBE, and policymakers cannot wait for finished analysis (36%).|
|0%|Median - products require time; there are limits to reporting (27%). Understandable that finished analytic|
| |products.|
| |2015OrgGroups|
| |20-cv-02020 (DF-2020-00235) 000058|
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Consumer Ratings on the Accuracy of IC Products

Analytic Consumer Interview Program:

2015 interviewees gave mixed ratings on the Accuracy of IC products across focus areas and organizational groups.

| |Strongly Agree|Moderately Agree|Neither Agree nor Disagree|Moderately Disagree|Strongly Disagree|
|---|---|---|---|---|---|
|100%|[6)(3), (b)(Z)(E)|[6J(3) , (BJ(T)(E)| | | |
|80%| |consumers provided the highest ratings on accuracy, while DOD consumers rated IC products' accuracy higher than other organizational groups| | | |
|60%| | | | | |
|40%| | | | | |
|20%| | | | | |
|0%|8|1|3|8|3|

Interviewees were able to distinguish between raw reporting and analytic judgments, but were not always able to understand the rationale behind analytic confidence statements (56%).

IC seen as an honest broker; interviewees confident in the analysis (23%).

20-cv-02020 (DF-2020-00235) 000059
---
# Consumer Ratings on the Usefulness of IC Products

| |2015 Org Groups|
|---|---|
|Strongly Agree|Consumers provided higher usefulness ratings than other focus areas, while DOD consumers saw IC products to be more useful than other organization groups.|
|Moderately Agree| |
|Neither Agree nor Disagree| |
|Moderately Disagree| |
|Strongly Disagree|Intelligence products are considered essential for work; provide mission critical support (66%).|
|Median|IC products are not crucial, but do provide context and/or flesh out topics/issue areas (31%).|
---
# Consumer Ratings on the Sourcing of IC Products

2015 interviewees tended to give the lowest ratings to the Sourcing of IC products among the five topic areas.

| |Strongly Agree|Moderately Agree|Neither Agree nor Disagree|Moderately Disagree|Strongly Disagree|Median|
|---|---|---|---|---|---|---|
|Across all groups; 2015 interviewees|(6)(3), (DJ(Z)(E)|Ratings with consumers from" and other organizations|Avoiding, altogether; to give strongly negative ratings|IC does provide enough source information; trust that the IC is providing good sources (35%).|Analysts often do not explain the rationale behind their source choices and/or the confidence levels they ascribe to sources (35%) 27|2015 Org Groups|
---
OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

INTELLIGENCE INTEGRATION

Backup Slides

20-cv-02020 (DF-2020-00235) 000062 28
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# ONI: Distribution of Ratings

FY 16 Products Evaluated as of 30 June 2016

|Evaluation Ratings by Standard|
|---|
|Sourcing Standard|Uncertainty Standard|Distinguishing Standard|Alternatives Standard|Relevance Standard|Logic Standard|
|Poor|Fair|Good|Excellent|Median|29|

20-cv-02020 (DF-2020-00235) 000063
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Objectivity & Process Survey: Analysts' Perceptions of Attempted Distortion

| |ONI|ALL|
|---|---|---|
|More than three times|29%|32%|
|Three times|12%|9%|
|Twice|22%|28%|
|Once|37%|31%|

Over the last five years, respondents from ONI were more likely to experience only one incident of distortion or suppression as compared to all respondents IC wide. The percentage of respondents IC-wide who experienced only one incident of distortion or suppression has decreased four of the past 5 years.

Footnote: 2

20-cv-02020 (DF-2020-00235) 000064
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Objectivity & Process Survey: Analysts' Perceptions of Attempted Distortion

|Analysts|90%|(18 of 20)|
|---|---|---|
|Managers|10%|(2 of 20)|
|0-5 years|20%|(4 of 20)|
|6-10 years|35%|(7 of 20)|
|10+ years|45%|(9 of 20)|

Key: 20-cv-02020 (DF-2020-00235) 00006S
---
| |Immediate sup:|Sr. Element Mgmt|ODNI|
|---|---|---|---|
|Satisfactory|88%|74%|77%|
|Mixed performance|9%|22%|16%|
|Unsatisfactory|4%|4%|8%|
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# Analytic Objectivity & Process Survey: Analytic Collaboration

| |How often do you interact for analytic purposes with intelligence professionals outside of your immediate employment unit but within your IC element?|How often do you interact for analytic purposes with intelligence professionals outside of your IC element but within the IC?|
|---|---|---|
|Analysts|66% (78 of 119)|47% (55 of 118)|
|Managers|45% (9 of 20)|38% (8 of 20)|
|0-5 years|66% (25 of 37)|47% (18 of 38)|
|6-10 years|61% (17 of 28)|41% (12 of 29)|
|10+ years|62% (45 of 73)|46% (33 of 72)|

20-cv-02020 (DF-2020-00235) 00O067
---
|Analytic Objectivity & Process Survey: Analysis of Alternatives & Access to Information|When making judgments is the consideration of alternatives (aka analysis of alternatives) a routine part of your workflow process?|
|---|---|
|Analysts|Managers|
|Work Role|92% (100 of 109)|92% (11 of 12)|
|0-5 years|88% (29 of 33)|34% (11 of 32)|
|Tenure in the IC|6-10 years 92% (24 of 26)|41% (11 of 27)|
| |10+ years 94% (58 of 62)|17% (19 of 65)|
---
# DIRECTOR OF NATIONAL OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

(U) Analytic Integrity and Standards Evaluation Program:

UNITED STATES OF AMERICA Key Results from 2017 (ONI & IC) " T E G R ^ T

Analytic Integrity and Standards

Office of the Deputy Director of National Intelligence for Intelligence Integration

August 01, 2018

20-cv-02020 (DF-2020-00235) 000023

ICOD: 12 December 2017
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

(UIIFOUO) Rationale for DNI Evaluation Efforts reviews of finished intelligence product or other analytic products" (IRTPA Section 1019)

(U/iOUU) ODNI's Analytic Integrity & Standards (AIS) group "shall perform, on a regular basis, detailed

(UIt=OUO) Evaluation criteria: Intelligence Community Directive (ICD) 203 Analytic Standards

A. Objectivity
B. Independent of Political Considerations
C. Timeliness
D. Based on All Available Sources of Intelligence
E. Exhibits Proper Standards of Analytic Tradecraft
1. Properly describes quality and credibility of underlying sources, data, and mepodologies
2. Properly expresses and explains uncertainties associated wip major analytic judgments
3. Assumptions and judgments
4. Properly distinguishes between underlying intelligence information and analysts' assumptions and judgments
5. Incorporates analysis of alternatives
6. Demonstrates customer relevance and addresses implications
7. Uses clear and logical argumentation
8. Explains change to or consistency of analytic judgments
9. Makes accurate judgments and assessments
10. Incorporates effective visual information where appropriate (FY 2016 - First year rated)

(UitfOUe) The AIS evaluation process and this briefing results from:

- Product evaluations
- Analyst survey
- Consumer interviews

20-cv-02020 (DF-2020-00235) 000024 2
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

(U) Distribution of ONI Product Evaluations

(U) FY 2017 Product Lines

(U) FY 2017 product titles most often contained the terms:

Production Dates: 06OCT16 - 30DEC16

DONI Today

| |23%|
|---|---|
|Baseline Assessment|67%|
|Maritime Intelligence Report|10%|

(U) FY 2015-17 Product Lines

ONI Today

| |30%|
|---|---|
|Baseline Assessment|51%|
|Maritime Intelligence Report|19%|

Legend

5+ Mentions

3-4 Mentions

2 Mentions

Charts are classified UllFeye

20-cv-02020 (DF-2020-00235) 000025

UNCLASSIFIED//FOR OFFICIAL USE ONLY
---
| |Sourcing|Uncertainty|Distinguishing Alternatives|Relevance|Logic|Visuals|
|---|---|---|---|---|---|---|
| | |Poor| | |Fair|Good|Excellent|NA|Median|

23 "Excellent" and 106 "Good" ratings (69% of all ratings).

Poor" ratings on four Standards (2% of all ratings)

23 "NA determinations on Alternatives and Visuals.

ONI median scores:

"Good" on six Standards.

"Fair" on one Standard.
---
|Most Recent Fiscal Year and 3-Year Cumulative Ratings|
|---|
|ONI FY 2017|ONI FY 2015-17|IC FY 2015-17|
|ONI FY 2017|ONI FY 2015-17|IC FY 2015-17|
|ONI FY 2017|ONI FY 2015-17|IC FY 2015-17|
|ONI FY 2017|ONI FY 2015-17|IC FY 2015-17|
|ONI FY 2017|ONI FY 2015-17|IC FY 2015-17|
|ONI FY 2017|ONI FY 2015-17|IC FY 2015-17|
|ONI FY 2017|ONI FY 2015-17|IC FY 2015-17|
|ONI FY 2017|ONI FY 2015-17|IC FY 2015-17|
|Poor|Fair|Good|Excellent|NA|
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# (UIFFOUO) Product Ratings Analysis

(UIFFEHE) FY 2015_17 Percentage of Products with All "Good" or "Excellent" Ratings

|IC|ONI|
|---|---|
|19%|IC Element|

With 19% being the best and 18 being the worst ONI is 9th.

| |(UIFAHA) ONI FY 2017|(UiioUC) ONI FY 2015-17|(Uiifehe) IC FY 2015-17|
|---|---|---|---|
|10%|18%|19%| |
|(3)|19%|21%| |
|37%|(17)|(16)|(b)(3)|
|(11)|53%|60%|(b)(3)|

At least 1 Poor

All Fair; Good, or Excellent

All Good or Excellent

All Excellent (0)

20-cv-02020 (DF-2020-00235) 000028
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# (UIFOUO) Tradecraft Summary

ONI FY 2015-17 IC FY 2015-17

|Standard|Percent Good or Excellent|Top Performers|
|---|---|---|
|Sourcing (Std 1)|81% ONI 49% IC|FBI, ONI, TREAS|
|Distinguishing (Std 3)|81% ONI 81% IC|FBI, NGA, TREAS|
|Uncertainty (Std 2)|68% ONI 65% IC|DHS, NGA, TREAS|
|Visuals (Std 9)|63% ONI 66% IC|DOENASIC, NGA|
|#FY 2016-17| | |
|Relevance (Std 5)|58% ONI 70% IC|DHSFBI, NCTC, NIC , TREAS|
|Logic (Std 6)|57% ONI 57% IC|NGA, NIC, TREAS|
|Alternatives (Std 4)|40% ONI 35% IC|DOE, FBI, NIC , TREAS|

Standards highlighted in yellow will be discussed further in the following slides.

(U) more than three entries is due to aTopPerfommers cells which contain tie between some of the agencies

Table is classified UlFee

20-cv-02020 (DF-2020-00235) 000029 7
---
# Area for Improvement: Alternatives (Std 4)

Performance on Standard Products which performed well provided an alternative assessment; discussed its likelihood, and provided the evidence or reasoning supporting it.

One product did not include an alternative assessment when one was deemed necessary due to the uncertainty expressed in its assessment.

92 percent of ONI survey respondents indicated they routinely conduct Analysis of Alternatives (AOA); 77 percent indicated AOA is included in final products.

Recommend/b)(3), (6)()

ONI FY 2017

ONI FY 2015-17

IC FY 2015-17

|Performance on Standard|FY 2017 Statistics|
|---|---|
|Products which performed well provided an alternative assessment; discussed its likelihood, and provided the evidence or reasoning supporting it|30 products evaluated for Alternatives|
|16 products were determined NA for standard|#of Excellent Ratings: 1|
| |#of Good Ratings: 5|
| |#of Fair Ratings:|
| |#of Poor Ratings:|

Alternatives: Incorporates analysis of alternatives

Chart is classified UIIFOUO

20-cv-02020 (DF-2020-00235) 00O030 8

UNCLASSIFIEDIFER BFFOlI
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

E 4 0 1 N T [ G R A 1 [0 N

# (U) Analytic Objectivity and Process Survey (AOPS)

(U7 Summary of AQPS Objectivity Results

| |ONI|IC|
|---|---|---|
|Percent of IC Respondents Reporting Distortion Incident|10%|13%|
|Distortion Rate| | |
|Organizational Hindrance Rate|15%|19%|
|Organizational Mitigation Effort|62%|73%|
|Senior Mgmt Support|85%|76%|
|First-line Mgmt Support|90%|90%|

2009 2010 2011 2012 2013 2014 2015 2016 2017

Percent reporting distortion incident 138 respondents

(UIFZY~) Distortion rate declined in FY 2017 following upward trend observed between FY 2014 16.

(UIIFOLA) FY 2017 response rate increased significantly from FY 2016.

(UIIOUG) ONIs distortion rate has decreased since FY 2015.

Notes: Overall margin Of error (MOE) +l- 2% for the IC and +/- 8% for ONI 95% confidence level. ONI population size was estimated at [D) MOEs were estimated using from input in response to ODNIs FY17 Human Capital Data Call. All MOE calculations were rounded up to the next whole number:

20-cv-02020 (DF-2020-00235) 000031 SECRETINOFORN
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

(U) Description of Objectivity Concerns

|Politicization/Self-politicization|Politicization/Self-politicization|
|---|---|
|Assessments modified/removed to avoid upsetting policy makers|External pressures from political discourse|
|Particular regions for fear of affecting state-to-state relations|Pressure to downplay drug trafficking in narrow policy-focused taskings|
|Risk-averse management| |

|Bureaucratic Interference|Bureaucratic Interference|
|---|---|
|Management dictating analytic line|Top-down direction for production|
|Analysis misrepresented to tell positive narrative|Analytic/Institutional mindsets|
| |Edits without dialogue|

Coordination

External battles wip oper agencies over narcotics topics
Holding products "hostage" to control disseminated analytic lines
"Lanes in pe Road:"
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

# IC Analytic Objectivity and Process Survey

|(U) ONL reports ability to access information comparable to the IC:|64 percent reported at least occasional access challenges. IC reports 60 percent:|
|---|---|
|Sharing Problems|Complicates coordination.|
|Technical Difficulties|Impedes analysis|
|Compartmentation|Introduces bias towards sensitive reporting.|

UFFOUeLON reports higher rate of same or improved coordination than the IC

- 88 percent - Increased technological platforms.
- Same or Improved - Organizational encouragement.
- Coordination - IC positions on PDB Staff.
- IC - 83 percent - Imperative of data overload.
- 12 percent - Pressure to modify analysis: Too many products to coordinate.
- Harder - POC challenges.
- Coordination - Guidelines unclear and ignored: IC = 17 percent

(U) Notes: Overall margin of error (MOE) +/- 2% for the IC and +/- 8% for ONI
---
# OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

(U) Analytic Consumer Interview Program (ACIP)

(U/FOUO) Background: AIS interviewed [(8J senior consumers from NSC, DOD, State, ATF, CBF, FBI, Commerce, Treasury, Labor, and USTR to collect consumer perspectives on analytic integrity issues.

| |FY 2017 AII|FY 2012-16 All|
|---|---|---|
|Objectivity| | |
|Timeliness| | |
|Accuracy| | |
|Usefulness| | |
|Sourcing| | |
|Sharing| | |

60% 40% 20% 0% 20% 40% 60% 80% 100%

Strongly Disagree Moderately Disagree Neither Agree nor Disagree Moderately Agree Strongly Agree

24 FY 2017 customers indicated routinely receiving ONI products; 22 consumers were DOD while 1 was DHS and 1 was DOS.
---
# ODNI

OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

(U) Analytic Integrity and Standards Evaluation Program: Key Results for ONI from FY 2018

Analytic Integrity and Standards

Office of the Deputy Director of National Intelligence for Mission Integration

April 23, 2019

((b)(3) 20-cv-02020 (DF-2020-00235) 000007
---
# ODNI

UNCLASSIFIED

(U/e8e) Rationale for ODNI Evaluation Efforts

finished intelligence product or other analytic products" (IRTPA Section 1019)ODNI's Analytic Integrity and Standards (AIS) group "shall perform, on regular basis, detailed reviews of

(UmFeUe} Evaluation criteria: Intelligence Community Directive (ICD) 203 Analytic Standards

A. Objectivity
B. Independent of Political Considerations
C. Timeliness
D. Based on All Available Sources of Intelligence
E. Exhibits Proper Standards of Analytic TradecraftProperly describes quality and credibility of underlying sources; data, and mepodologies
2. Properly expresses and explains uncertainties associated wip major analytic judgments
3. Properly distinguishes between underlying intelligence information and analysts' assumptions and judgments
4. Incorporates analysis of alternatives
5. Demonstrates customer relevance and addresses implications
6. Uses clear and logical argumentation
7. Explains change to or consistency of analytic judgments
8. Makes accurate judgments and assessments
9. Incorporates effective visual information where appropriate (FY 2016 - First year rated)

The AIS evaluation process and this briefing result from:

- Analyst survey
- Consumer interviews
- Product evaluations and studies

20-cv-02020 (DF-2020-00235) 000008 UNCLASSIFIED Analytic Integrity and Standards 2
---
# ODNI

(U) Analytic Objectivity and Process Survey (AOPS)

(BHNF) In FY 2018, AIS surveyed nearly 63 analysts, with 111 coming from ONI.

(UIF;ZC) Perceived Distortion or Suppression of Analysis in the Face of Persuasive Evidence

|(U) Percent Reporting Distortion Incidents|(UkFeve) FY 2018 ONI|
|---|---|
|19%|16% Rarely|
|17%|0% Occasionally|
|15%|23%|
|13%|14%|
|11%|Constantly or Frequently|
|9%| |
|7%|77% Chart is classified UIIFOUO|
|5%| |

2010 2011 2012 2013 2014 2015 2016 2017 2018 (U/FFEUE) ONI respondents who say they experience frequent distortion (77% - far exceeds the IC-wide average (33%)

(U) Organizational Hindrances to Objectivity Analytic Ombudsman

|Percent Reporting|(UIFF3UE) 44% of ONI respondents know they have an analytic ombudsman. This is on par with the IC-wide average (45%)|
|---|---|
|ONI 28%|IC 29%|

Table is classified UlFBUB

(U) Notes: Overall margin of error (MOE) +/- 2% for the IC and +/- 9% for ONI

20-cv-02020 (DF-2020-00235) 000009 Analytic Integrity and Standards
---
# Sources and Motives of Distortion/Suppression

|Top Sources|ONI|IC|
|---|---|---|
|Internal Mid/Senior Supervisor|(UIF)|ONI Respondent Comments|
|First-Level Manager|Pressure to maintain analytic line|Management concerned with policy over objectivity|
|External Colleague|DOD senior intervened to suppress assessment|Disagreement on threshold|
|Internal Colleague|Inconsistent application of tradecraft standards|Peer suppressing analysis|
|# of Respondents|%|20%|40%|
|ONI: 13|IC [6)(3)|Chart is classified|UltFeue Information compartmentation challenges|

|Top Motives|ONI|IC|
|---|---|---|
|Maintain Existing Analytic Line|Other|Decrease layers of review|
|Support Operational Equities|Increase tolerance for new assessments differing from IC line| |
|Accede to Internal Colleague|Reconsider mandate that all products adhere to AIS| |
|Prevent Policymaker Action/Response|# of Respondents|%|20%|40%|60%|
|ONI: 13|IC: b)(3)|Chart is classified|UHFeue|
---
# ODNI

(U) Editing and Feedback

|Levels of Review|ONI Respondents|IC Respondents|
|---|---|---|
|80%| | |
|60%| | |
|40%| | |
|20%| | |
|0%| | |
| |2-3 Levels|4-5 Levels|>5 Levels|

# Reviewer Feedback

(UMF 3U3) Most ONI respondents received feedback bop in person and virtually (73%), consistent wip IC results
(U/F3US) 9% "frequently" disagreed wip or didn't understand feedback, alpough 27% "sometimes" did (consistent wip IC results)
Reasons Major Changes Not Discussed (Top Choice)
Press of Business 9%
Based on Additional Intelligence 8%
No Mgmt Not Open Problem wip Edits 6%
Oper 6%
Didn't Want To Be a Problem 68%
Chart is classified UIFFBue 3%

20-cv-02020 (DF-2020-00235) 000011

UNCLASSIFIED Analytic Integrity and Standards
---
# ODNI

(U) Coordination; Information Access; and Data Science

# Coordination

(UMFOUE) 79% of ONI respondents "always" or "very often" coordinate peir products Externally
Location of Coordination
(UIIF3UO) 51% are aware of pe IC Standard on Coordination (ICS 203-01)
(UIFFOUO) The most frequent coordination challenges:
Overlapping Lanes in pe Road
Difficulty Getting Attention Paid to Piece

# Information Access

Intelligence (12% on a daily basis) - consistent wip IC responses

# Data Science

(UIFe8e) 67% reported difficulties accessing data analytics
Have you used data science or data analytics in your intelligence analysis?

| |Yes|No|Don't Know|
|---|---|---|---|
|ONI|24%|58%|18%|
|IC|26%|58%|16%|

Chart is classified UIiFBue

Table is classified UlFFete

20-cv-02020 (DF-2020-00235) 000012 Analytic Integrity and Standards
---
# ODNI

(UlFOUO) Tradecraft and Analysis of Alternatives

# (U) Application of Tradecraft Standards

(U/e1ey79% of ONI respondents reported receiving constructive internal feedback on ICD 203 tradecraft standards, which is higher than the IC average (62%)

(U/F38O) 39% had no difficulty applying tradecraft standards, and those that did cited Analysis of Alternatives (Std 4), Visuals (Std 9), and Sourcing (Std 1) as the most difficult to apply:

Primary concerns were that applying the standards would make products too convoluted or not timely enough.

Analysis of Alternatives (AoA)

(ulFeUE) Percent Who Routinely Conduct AoA

(UItFOUO) FY 2018 ONI

| |Frequency of AoA Publication|
|---|---|
|100%|Rarely|Never|
|90%|4%|3%|
|80%| | |
|70%|75%| |
|60%|Sometimes|Always or Usually|
|50%|23%| |
|40%| | |
| |2010|2011|2012|2013|2014|2015|2016|2017|2018|
|IC Respondents| | | | | | | | | |
| |ONI Respondents| | | | | | | | | |

Chart is classified UMieue

Chart is classified Ul

# 20-cv-02020 (DF-2020-00235) 000013

UNCLASSIFIED Analytic Integrity and Standards
---
# ODNI

UNCLASSIFIED

(U) Analytic Consumer Interview Program (ACIP)

(U) Conducted interviews in EY 2018 with senior State, DOD, and other consumers of intelligence:

|97% agreed intel products were objective|
|---|
|Objectivity|Personal and organizational biases are to be expected|
| |Need less resistance to changing hardened analytic lines|
|88% agreed intel products were timely|
|Timeliness|Frequently tied to downgrade/declassification timeliness|
| |Physical location and technology access also play a role|
| |Value of uncertainty and caveat language "stands out" more|
|92% agreed intel products were accurate|
|Accuracy|
| |Value of products "make a call" (i.e., be explicit with judgments)|
|Usefulness|Sourcing|
| |Some want highlighting of credibility factors & reporting reliability|
|82% agreed intel products provide adequate sourcing information|
|82% agreed intel products were useful|
| |Most consumers "trust" the IC to evaluate its sources|
|Relevance|
| |Desire for improved (and more) opportunity analysis|
| |Value in longer time horizons, and info outside open source|
|Sharing|
| |62% agreed they could share intel products with those who need them|
| |Need "Write to Release" to avoid automatic SCI|
| |Need for improved downgrade/release timelines|

Chart is classified UlFSUe20-cv-02020 (DF-2020-00235) 000014UNCLASSIFIED Analytic Integrity and Standards
---
# ODNI

UNCLASSIFIED

|(UIFFEUO) Distribution of ONI Product Evaluations|(UiiFBU) FY 2018 product titles most often contained the terms:|(U/FOUO) FY 2018 Product Lines|
|---|---|---|
| |Production Dates: October 03, 2017 - December 22, 2017|3%|
|(b)(3)| |27% DONI Today|
| | |70% Baseline Assessment Maritime Intelligence Report|

# Legend

|8+ Mentions|30% ONI Today|51% Baseline Assessment|
|---|---|---|
|5-7 Mentions|19% Maritime Intelligence Report| |
|2-4 Mentions| | |

Charts are classified Ulrete

20-cv-02020 (DF-2020-00235) 000015 UNCLASSIFIEDI Analytic Integrity and Standards
---
# ODNI

UNCLASSIFIED

# (UIFO8O) ONI: FY 2018 Distribution of Ratings

(UlFOUO) Key Findings

- (UFEUE) Evaluation Ratings by Standard

(UHFeUC) Steady rise in the number of "Excellent" ratings since 2016.

| |35 (2018)|23 (2017)|9 (2016)|
|---|---|---|---|
|(UMF3UE) Median ratings have improved since FY 2016.| | | |
| | |FY 2018|FY 2017|FY 2016|
| | |"Good" on seven standards|"Good" on six standards|"Good" on five standards|
| | |"Fair" on one standard| |Between "Fair" and "Good" on Std 6|
| | |"Fair" on one standard| |

(UlFeUe) AIS tends to rate ONI higher compared to ONI's self review.

ONI's self review indicates all standards are in the "Good" range with the exception of Std 4.

18% of all ratings "Excellent" as per AIS's evaluations. This compared to 8% per ONI's self review

20-cv-02020 (DF-2020-00235) 000016 UNCLASSIFIEDI Analytic Integrity and Standards 10
---
# ODNI

UNCLASSIFIED

|ONI: Rating Comparisons|Most Recent Fiscal Year and Previous 3-Year Cumulative Ratings|
|---|---|
|ONI FY 2018|ONIFY 2015-17|
|ONI FY 2018|ONI FY 2015-17|
|ONI FY 2018|ONI FY 2015-17|
|ONI FY 2018|ONI FY 2015-17|
|ONI FY 2018|ONI FY 2015-17|
|ONI FY 2018|ONI FY 2015-17|
|ONI FY 2018|ONI FY 2015-17|
|ONI FY 2018|ONI FY 2015-17|
|ONI FY 2018|ONI FY 2015-17|
|ONI FY 2018|ONI FY 2015-17|
|ONI FY 2018|ONI FY 2016-17*|

#(Uf Visuals (Std 9) was not rated until FY 2016.

20-cv-02020 (DF-2020-00235) 000017 UNCLASSIFIEDI Analytic Integrity and Standards 11
---
| |Standard|Percent Good or Excellent|
|---|---|---|
|Sourcing (Std 1)|87%|53%|
|Relevance (Std 5)|80%|89%|
|Distinguishing (Std 3)|77%|71%|
|Visuals* (Std 9)|68%|69%|
|Logic (Std 6)|67%|72%|
|Uncertainty (Std 2)|57%|48%|
|Alternatives* (Std 4)|53%|56%|

#Numbers do not include NA determinations

20-cv-02020 (DF-2020-00235) 000018 UNCLASSIFIED Analytic Integrity and Standards 12
---
# ODNI

UNCLASSIFIED

|Evaluation Ratings By Product|
|---|
|(UIIFBUBONI FY 2018|(UIFBUBTONI FY 2015-17|
|20%|19%|
|(6)|(17)|
|13%|18%|
|(4)|(16)|
|67%|63%|
|(20)|(57)|
|(UFe1B) IC FY 2018|(UltFa8a) IC FY 2015-17|
|2%|19%|
|17%|21%|
|I(b)(3)|(b)3|
|19%|60%|
|(b)(3)|[b)(3|
|At least 1 Poor|AII Fair, Good, or Excellent|
| |AIl Good or Excellent|
| |AIl Excellent|

* (U) NA determinations do not impact product ratings

20-cv-02020 (DF-2020-00235) 000019 UNCLASSIFIED Analytic Integrity and Standards 13
---
# ODNI

Area for Improvement: Relevance (Std 5)

Performance on Standard

ONI's self review stated about Relevance (Std 5), FY 2018 Statistics

- 30 products evaluated for Relevance
- # of Excellent Ratings:
- # of Good Ratings: 17
- # of Fair Ratings: 6
- # of Poor Ratings: 0
- Top Performers: FBI, NIC, NSA

|ONI FY 2018|IC FY 2018|At least Poor|Fair|Good|Excellent|
|---|---|---|---|---|---|
|20-cv-02020 (DF-2020-00235)|000020|UNCLASSIFIED|Analytic Integrity and Standards|14| |
---
# ODNI

Area for Improvement: Alternatives (Std 4)

Performance on Standard (UlmFeye) FY 2018 Statistics

(U) ONIs self review stated about Alternatives (Std 4), "ONI highly encourages analysts to incorporate analysis of alternatives into products, but does not mandate its inclusion. ONIs FRA checklist applicable. The Tradecraft SIO provides training on Standard #4 in as part of formal training programs, but no ONI-published guidance document exists. ~products had a strong judgment, but failed to present alternatives or dissent

ONI FY 2018 IC FY 2018 At least Poor Fair Good Excellent

|20-cv-02020 (DF-2020-00235) 000021|UNCLASSIFIEDI|Analytic Integrity and Standards|
|---|---|---|
|15| | |

15 products evaluated for Alternatives

#of Excellent Ratings: 2

#of Good Ratings: 6

#of Fair Ratings: 7

#of Poor Ratings: 0

Top Performers: DIA, FBI; NSA
---
# Preliminary Findings from AIS Sourcing Study

In FY 2018, AIS conducted a deep dive on sourcing; including a review of source material and source summary statements. AIS examined the underlying sourcing in 15 of ONIs set of AIS-evaluated products (up to 50 sources per product; 369 sources total): AIS examined 10 of the 15 products for source summary statements (SSS):

- Of the sources reviewed, 195 were accessible and consistent with the text they were intended to support; 6 showed minor inconsistencies that did not affect the key judgments of the reviewed products; 168 sources were not able to be accessed and could not be checked.
- No Standard ratings were changed as a result of the deep dive.
- AIS observed that all 10 products in the SSS review contained source summary statements and the level of detail provided in these statements largely determined the variation in Standard 1 ratings (which ranged from "fair" to "excellent"). Those products rated "Good" or "Excellent" clearly identified their primary reporting streams, assessed their relative merits with respect to qualitative factors, and/or noted their respective contributions to the analysis.

20-cv-02020 (DF-2020-00235) 000022 UNCLASSIFIED Analytic Integrity and Standards 16
---
# ODNI

UNCLASSIFIED

DIRECTOR NATIONAL UNITED STATES AFRICA

OFFICE OF THE DIRECTOR OF NATIONAL INTELLIGENCE

(Unclassified) Analytic Objectivity and Process Survey (AOPS):

FY 2019 ONI Key Results

Analytic Integrity and Standards

Office of the Deputy Director of National Intelligence for Mission Integration

20-cv-02020 (DF-2020-00235) OOO001 UNCLASSIFIED Analytic Integrity and Standards
---
# ODNI (Office of the Director of National Intelligence)

UNCLASSIFIED

# (U//FOUO) Rationale for ODNI Evaluation Efforts

(U//FOUO) ODNI's Analytic Integrity and Standards (AIS) "shall perform, on a regular basis, detailed reviews of finished intelligence product or other analytic products" (IRTPA Section 1019)

(U//FOUO) Evaluation criteria: Intelligence Community Directive (ICD) 203 - Analytic Standards

A. Objective
B. Independent of Political Consideration
C. Timely
D. Based on All Available Intelligence Information
E. Implements and Exhibits Analytic Tradecraft Standards
- Properly describes quality and credibility of underlying sources, data, and mepodologies
- Properly expresses and explains uncertainties associated wip major analytic judgments, assumptions, and judgments
- Properly distinguishes between underlying intelligence information and analysts'
- Incorporates analysis of alternatives
- Demonstrates customer relevance and addresses implications
- Uses clear and logical argumentation
- Explains change to or consistency of analytic judgments
- Makes accurate judgments and assessments
- Incorporates effective visual information where appropriate

20-cv-02020 (DF-2020-00235) 000002 UNCLASSIFIED Analytic Integrity and Standards
---
# Perceived Distortion/Suppression of Analysis

|Respondents Reporting Distortion/Suppression|ONI Frequency of Distortion/Suppression|
|---|---|
|20%|Monthly|
|17%|Once|
|14%|Daily or Weekly|
|13%|Several Times|
|5%|2 - 3 Times|

Graph is classified U/ Aobo

Graph is classified Ul Hoto

Sought Assistance

|ONI|IC|
|---|---|
|71%|56%|

Analytic objectivity is violated with the willful distortion of an analytic assessment due to pressure from an internal/external customer or by purposefully suppressing an alternate or dissenting view:
---
Sources and Motives for Distortion/Suppression
(U/FFOUO)
(UIFCEC; FY 19 Source)
IC Senior Leadership
Interna Mid/Senior Supervisor
First-Level Manager
Non-Supervisory Reviewer
Oper
Internal Colleague
Non-Analytic Senior
Senior IC Reviewer
DNI Element (NIC, NIM, etc:)
External Colleague
External Government Official

Motive
(U/FseS; FY 19 Motive)
Oper
Unsure
Accede to Internal Colleague
Suppress Embarrassing Information
Support Operational Equities
Maintain Existing Analytic Line
Accede to Gov/Policy Official
Accede to Internal Mgmt.
Accede to External Colleague
Prevent Policymaker Action/Response
Accede to External Mgmt.

|ONI|10%|20%|30%|40%|
|---|---|---|---|---|
|20-cv-02020 (DF-2020-00235) 000004| | | | |

|ONI|IIC|
|---|---|
| |0%|10%|20%|30%|40%|
---
# ODNI

UNCLASSIFIED

(U/fFeu8) Analytic Tradecraft

(U/FEEC; Standard 4 (Alternatives)

(U/ Feue} Respondents who Routinely Considered Alternatives

(U/fFeue) Frequency of Alternatives Published

| |2010|2011|2012|2013|2014|2015|2016|2017|2018|2019|
|---|---|---|---|---|---|---|---|---|---|---|
|Always|27%| | | | | | | | | |
|Frequently| | | | | | | | | |92%|
|Sometimes| | | | | | | | | |28%|
|Rarely| | | | | | | | | |49%|
|Never| | | | | | | | | |59%|

Graph is classified U/ Hotv

#FY19 QuestionChange: When thinking through analytic issues and/or preparing analytic products, is consideration of alternative explanations outcomes routine part of the process?

(U/fse:i Standard 9 (Visuals)

(U/FSES; Joint Authorship

(UIFCUC} Data Science

Eighty-five percent of respondents reported incorporating visuals into analytic products in the past year: (IC: 88%)

Thirty-nine percent of respondents jointly authored an analytic product with an analyst from another agency/element: (IC: 42%)

Thirty-two percent of respondents have used data science or data analytics in their intelligence analysis. (IC: 43%)

20-cv-02020 (DF-2020-00235) 000005

UNCLASSIFIED

Analytic Integrity and Standards
---
UNCLASSIFIEDI

20-cv-02020 (DF-2020-00235) 000006 UNCLASSIFIED Analytic Integrity and Standards

Output:
{
"Who": "Office of the Director of National Intelligence",
"What": "Analytic Integrity and Standards",
"When": "20-cv-02020 (DF-2020-00235) from various years including FY2015, FY2016, FY2017, FY2018, FY2019",
"Where": [],
"Security Clearance": "SECRET"
}


Input:
"
Here is some messy data parsed from a pdf. These files may contain redacted information.
Using a json format, ouput the who, what, when, and where of the document, and required security clearence
Use the following json format:
{"Who": "key names or groups", "What": "keywords or ideas from document", "When": "day/month/year", "Where": ["referenced locations"], "Security Clearance": "CONFIDENTIAL, SECRET, or TOP SECRET"
} Text: # THESIS APPROVAL SHEET

|1. NAME:|(3), (6)|CLASS NO:|NJ A970(7|
|---|---|---|---|
|2. THESIS CHAIR:|M|A|Rcc akec Gv|
|3. COMMITTEE MEMBER:|(6), (3), () (6)| | |
|THESIS TITLE:|Clafs Kt&|446|Aked, ftef|
|THIS THESIS RESPONDS TO:|Nat Acce;RESEARCH REQUEST FROM|COMMAND OR AGENCY:|YesIi hrnehaaJJ Fq i|
|IF Yes, Specify:|[understand that this thesis was prepared under the direction of a US Department of Defense institution; and that dissemination outside official U.S Government channels either in whole part; prohibited without|Signature of Student:|[(b) (3); ()(6= 347040|
|COMMITTEE MEMBER COMPLETES ITEM 8:|(6)(3); ()(6)| | |
|THESIS APPROVED:| | | |
| |THESIS CHAIR COMPLETES ITEMS 9 & 10| | |
|THESIS APPROVED:| |Yes| |
|THESIS RELEASE:|THESIS MEETS ALL REQUIREMENTS FOR PUBLIC|Yes|Date: L#u62L|
|ASSOCIATE DEAN COMPLETES ITEMS 11 & 12| | | |
|DATE PGIP STARTED:|TO AWARD THE MSSI DEGREE:|APPROVAL AUTHORIZATION [(6) (3), () (6)|Date|
|this thesis are those of the author not reflect the official policy position of the Department of Defense the U.S. Government| | | |
|Distribution (When filled Original: Filed with|Copy: Student = Filingistra| | |
---
# Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006

# ABSTRACT

TITLE OF THESIS: Classification Policy: Why the United States Can Not Account for the Cost of Classifying Information

STUDENT: [6] (3), () (6)

CLASS NO. NSA Cohort Class 9701 Date: August 2001

THESIS COMMITTEE CHAIR: MaAnre Brooker-Grogen

THESIS COMMITTEE MEMBER: (D) (3), () (6)

Mankind has pursued intelligence collection since ancient times. The Bible records the Israelites sending spies into Canaan to determine if the land was worth fighting to obtain. Intelligence collection and reporting played an important role in the Battle of Marathon, when the Greeks defeated the Persians because of advanced warning. Most civilizations used spies as well to obtain and maintain the advantage over their enemies. The Romans and the Chinese knew the value of learning all they could about their adversaries before attacking them, looking for the weakness that could be exploited.

Safeguarding this information is part of the equation that has largely been ignored. The ancient Greeks were known to shave a man's head, tattoo the information on the man's scalp and then wait for the hair to grow back before sending the messenger back with the information. The hair acted as a classification marking. Similarly, wax was also used on tablets that had information inscribed on it. The wax was used to cover over the inscriptions and thus safeguard.

The United States has evolved a classification management system that has grown over time in response to the situations of the day. In other words, the current classification system can be viewed as an ad hoc system that grew out of perceived needs. Adapting a system used by British and French allies during World War and modifying it only slightly during that time, the United States has a system that is in need of an overhaul.

Several studies have been conducted of the classification management system, starting with Congressional and Defense Department reviews in the mid-1950s through the late 1990s. All the commissions and committees have said the same thing; that classification is not well.
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006

# CLASSIFICATION POLICY:

WHY THE UNITED STATES CAN NOT ACCOUNT FOR THE COST OF CLASSIFYING INFORMATION

National Security Agency Cohort 7, Class 9701

Unclassified thesis submitted to the Faculty of the Joint Military Intelligence College partial fulfillment of the requirements for the degree of Master of Science of Strategic Intelligence

August 2001

The views expressed in this paper are those of the author and do not reflect the official policy or position of the Department of Defense or the U.S. Government
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006

# DEDICATION

A thesis is more that just one person choosing to write about a given topic. It takes family, friends, and advisors to make it happen. I acknowledge the countless hours that my readers spent going over the subject matter with and correcting my work. By extension, she too is now a "subject matter expert".

Likewise, my chairperson, Ms. Anne Brooker-Grogan, kept me on the straight and narrow during this project. She never complained about broken promises on my part to get the manuscript to her on time. Another chairperson might well have given up on me as a hopeless cause.

Finally, my wife and daughters have to be acknowledged for their patience and support. Imagine taking a laptop on vacation to get this thesis done! All the same, they tolerated my imposition on their time to get this work done.

To all the women in my life, my chair, my readers, and my family, I dedicate this work. They are free to send me for a psychiatric evaluation if I ever say I want to do something like this again.

Laurel, Maryland

24 July 2001
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
                                                                    n
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006
---
Approved for release by ODNI on 11/30/2023 FOIA Case DF-2024-00006

Output:
{
"Who": "Various students, committee members, and faculty members",
"What": "Thesis approval process, classification policy, the United States' classification system, intelligence collection throughout history, dedication for a thesis",
"When": "August 2001",
"Where": ["Joint Military Intelligence College", "Laurel, Maryland"],
"Security Clearance": "CONFIDENTIAL"
}


Input:
"
Here is some messy data parsed from a pdf. These files may contain redacted information.
Using a json format, ouput the who, what, when, and where of the document, and required security clearence
Use the following json format:
{"Who": "key names or groups", "What": "keywords or ideas from document", "When": "day/month/year", "Where": ["referenced locations"], "Security Clearance": "CONFIDENTIAL, SECRET, or TOP SECRET"
} Text:

---

---

---

---

Output:
{
    "Who": "N/A",
    "What": "No information available",
    "When": "N/A",
    "Where": [],
    "Security Clearance": "N/A"
}


Input:
"
Here is some messy data parsed from a pdf. These files may contain redacted information.
Using a json format, ouput the who, what, when, and where of the document, and required security clearence
Use the following json format:
{"Who": "key names or groups", "What": "keywords or ideas from document", "When": "day/month/year", "Where": ["referenced locations"], "Security Clearance": "CONFIDENTIAL, SECRET, or TOP SECRET"
} Text: # INTELLIGENCE COMMUNITY STANDARD

# NUMBER 500-27

(U) COLLECTION AND SHARING OF AUDIT DATA

(EFFECTIVE: 2 JUNE 2011)

A. (U) AUTHORITY:

The National Security Act of 1947, as amended; Executive Order 12333, as amended; Intelligence Community Directive (ICD) 101, Intelligence Community Policy System; ICD 500, Director of National Intelligence Chief Information Officer; ICD 501, Discovery and Dissemination or Retrieval of Information within the Intelligence Community; ICD 502, Integrated Defense of the Intelligence Community Information Environment; ICD 503, Intelligence Community Information Technology Systems Security Risk Management, Certification and Accreditation; and other applicable provisions of law.

B. (U) PURPOSE

1. (U//FOUO) Intelligence Community (IC) elements shall audit information resources within the IC information environment (hereafter referred to as IC information resources) to protect national intelligence, identify threats (including insider threats), detect and deter penetration of IC information resources, reveal misuse, identify usage trends and for other lawful purposes. This Standard provides guidance for implementing uniform information security requirements and procedures, as established by ICD 500.
2. (U//:FOUO) Audit data shall be collected on IC information resources for the purposes outlined above and shall be shared with each respective user's gaining and employing IC element, or department or agency as appropriate, to include both contractors and government personnel.
3. (U/IFOUO) This Standard provides for the collection and sharing of audit data to support counterintelligence (CD, information assurance (IA}, business analytics (BA), personnel security (PS), and other community audit needs related to IC information resources.
4. (U//FOUO) The collection and sharing of audit data will:
- (U//F9UO) Enable IC elements to identify and evaluate anomalous activity involving IC information resources.

UNCLASSIFIED//Peee
---
# Approved for release by ODNI on 06-10-2016, FOIA Case #DF-2016-00213

UNCLASSIFIED//~

b. (U//FOUO) Enable IC element-; to identify and assess misuse (intentional or inadvertent), and/or exploitation of IC information resources, wheper pe source is external or internal.
c. (U/lffiOO) Support auporized investigations, oversight, and inquiries.
d. (U//FOUO) Deter unauporized use of IC information resources.
e. (U//FOUO~ Enable IC elements to assess pe effectiveness of intelligence information sharing.

# APPLICABIIJITY

1. This Standard applies to the IC, as defined by the National Security Act of 1947, a-; amended, and such other elements of any other department or agency as may be designated by the President. or designated jointly by the Director of National Intelligence (DNI) and the head of the department or agency concerned, a-; an element of the IC.

# IMPLEMENTATION

1. (U//FOUO) The requirements of this Standard shall be implemented on all IC information resources, consistent with the risk management approach prescribed in ICD 503. Requests for waivers to this requirement shall be submitted in accordance with IC Standard (ICS) 500-20, Intelligence Community Enterprise Standards Compliance. IC elements are to provide to the IC Chief Information Officer (IC CIO) an implementation plan for the requirements set forth in this Standard within 60 days of signature.
2. (U//li'OUO) The events and activities identified in Appendix B of this Standard shall be collected on IC information resources. The IC CIO shall promulgate IC enterprise standards with details for collecting these auditable events consistent with ICS 500-20. IC information resources shall have the ability to collect such audit data through automated means and store the information securely. The information will be marked and handled at the appropriate classification and sensitivity levels.
3. (U//FOUO) IC elements shall share, where lawful and appropriate, audit data identified in Appendix B of this Standard to support Cl, IA, BA, PS, and other community audit needs related to IC information resources. This sharing shall be consistent with access restrictions developed pursuant to Section D.6 of this Standard. IC elements are to use IC enterprise standard, IC Enterprise Audit Exchange Technical Specification (AUDIT.XML) to guide near-term sharing. To standardize the sharing of audit data, the IC CIO shall promulgate an enterprise audit framework that will drive the development of additional IC enterprise standards and identify how the IC shall share audit information consistent with Section D.6. Within 60 days of promulgating the enterprise audit framework, IC elements are to provide to the IC CIO an updated implementation plan that includes requirements outlined in the enterprise audit framework.

UNCLASSIFIED//~
---
# Approved for release by ODNI on 06-10-2016, FOIA Case #DF-2016-00213

UNCLASSIFIED//-f!9Y9.

ICS 500-27
4. (U/troUO) IC information resources shall have pe capability to collect key strokes and full application content (email. chat, imports, exports, etc.), obtain screen captures, and perform file shadowing for all lawful purposes, to include detecting unauporized use or disclosure.
a. (U) This capability shall be used only in accordance wip applicable law, policy, and regulations.
b. (U//FOUO) IC elements shall develop internal processes and procedures for using pese specific capabilities and pe information collected, in consultation wip peir respective legal counsel and civil liberties and privacy officials.
5. (U//FeOO) IC information resources shall display a standard banner to be promulgated by pe IC CIO, pat provides notice of, and obtains user consent to, pe collection and monitoring of all user activities. This standard banner shall be implemented in coordination wip legal counsel, as well as wip civil liberties and privacy officials, to ensure legal, civil rights, civil liberties, and privacy issues are appropriately addressed.
6. (U//POUO) Audit data shall be safeguarded, in accordance wip applicable law, policy, and department or agency regulations, at rest, in transit, and during presentation, to include appropriate limitations on access and use. Audit data shall be protected from unauporized access, modification, or destruction and shall be reviewed at least weekly for action by pe IC element.
7. (U//FeOO) IC elements shall develop procedures for accessing audit data. Each IC element shall ensure pat access to audit data is restricted to personnel who require pe information to perform peir auporized functions. Personnel auporized to access data shall be trained regarding all applicable laws and policies and pe consequences of misuse of audit data.
8. (U//FeOO) IC elements shall implement pis Standard consistent wip ICS 500-20.
9. (U) Audit data shall be retained in accordance wip pe applicable records control schedule.
10. (U) Guidance supporting insider preat detection is provided in ICS 700-2, Use of Audit Data for Insider Threat Detection.

# E. (U) RESPONSIBILITIES

1. (U) IC elements shall:

- a. (U//FOUO) Collect audit data pertaining to IC information resources pursuant to the requirements of this Standard and in accordance with ICS 500-20.
- b. (U//POUO) Ensure audit data is attributable to a unique user and/or IC information resource. To the extent that audit data attributable to a unique user may be shared with others, such sharing shall be limited to the least amount required to assess the threat or to address the concern for which the sharing is requested.
---
# Approved for release by ODNI on 06-10-2016, FOIA Case #DF-2016-00213

UNCLASSIFIED//~

|c.|(U) Ensure that adequate security and privacy controls are implemented to protect the data, including oversight of compliance by audit personnel and monitoring of audit personnel activities. Ensure that personnel authorized to access audit data shall be trained regarding applicable laws and policies and the consequences of misuse of audit data.|
|---|---|
|d.|(U//FeYe) Submit to the IC CJO any request for a waiver to the requirements herein in accordance with ICS 500-20. Waiver requests with regard to the requirements of this Standard shall be approved by the IC CIO in consultation with the National Counterintelligence Executive (NCIX).|
|e.|(U/IF006) Share, as appropriate and consistent with applicable law, information regarding audit events pertaining to users and processes acting on behalf of a user accessing an IC information resource with each respective user's gaining or employing IC element, or department or agency. To enable this sharing, each IC element shall have the capability to receive and store audit data securely and in accordance with the requirements of this Standard. Additionally, each IC element shall determine if the collection and use of such data requires a Privacy Act System of Records Notice (SORN).|
|f.|(U//FeYe) Share audit data regarding detected anomalies on IC information resources that potentially stem from an insider threat in a timely manner with all appropriate organizations responsible for insider threat detection (which may include the user's gaining or employing IC element, or department or agency).|
|g.|(U//FOUO) Provide an implementation plan consistent with Section D.|
|h.|(U) Maintain the record copy of the audit data collected pursuant to the Federal Records Act and in accordance with the IC element's applicable records control schedules.|
|i.|(U) Provide quarterly reports to the IC CIO on the extent to which the requirements of this Standard are implemented on IC information resources. Such reporting shall be presented to the IC CIO and begin October 1, 2011.|
|j.|(U) Ensure that notice of any unauthorized access, use or sharing of audit data containing personally identifiable information is handled consistent with applicable data breach notification policies.|

2. (U) The IC CIO shall:

|a.|(U) Develop specific audit-related guidance necessary for addressing IC audit needs associated with IC information resources, to include guidance related to Section D.3. Such guidance shall be developed in consultation with the Office of General Counsel and the Civil Liberties Protection Officer to ensure privacy and civil liberties considerations are addressed.|
|---|---|
|b.|(U) Promulgate IC enterprise standards consistent with requirements pursuant to the governance process in ICS 500-20. Requirements for the development of IC enterprise standards for audit events to support insider threat detection shall be developed and promulgated in consultation with the NCIX.|

I 50 U.S.C. 403-Jd

UNCLASSIFIED//:fi'OU6
---
Approved for release by ODNI on 06-10-2016, FOIA Case #DF-2016-00213

UNCLASSIFIED//i!etJO

ICS 500-27

- Evaluate and monitor the implementation of this Standard at least annually.
- In support of insider threat detection, notify the NCIX of:
- Non-compliance by the IC elements with this Standard or associated IC enterprise standards, and share with NCIX the IC elements' plans of action and milestones to address non-compliance.
- Waiver requests submitted in accordance with Section E.1.d of this Standard.

EFFECTIVE DATE: This Standard becomes effective on the date of signature.

Al Tarasiuk

Assistant Director of National Intelligence and Intelligence Community Chief Information Officer

UNCLASSIFIED//PGYG.
---
# Appendix A - Terms and Definitions

|Audit:|Provides authorized personnel with the ability to review and examine any action that can potentially cause access to, generation of, or affect the release of classified or sensitive information.|
|---|---|
|Employing element:|The IC element from which an employee on a joint IC duty rotational assignment is detailed. The detailed employee's permanent position of record remains with the employing element, and the detailed employee remains on the permanent rolls of that employing element during the joint IC duty rotational assignment, unless other administrative arrangements are agreed to by the employing and gaining element (ICD 601, Joint IC Duty Assignments, 4 September 2009).|
|File shadowing:|The replication of data to another location in a systematic way that produces an identical copy of that data. The location can be as nearby as another disk in the same server, or in another server or workstation in a completely different geographic location.|
|Gaining element:|The IC element to which an employee is detailed while on a joint IC duty rotational assignment (ICD 601, Joint IC Duty Assignments, 4 September 2009).|
|Information resources:|Information and related resources, such as personnel, equipment, funds, and information technology (IC Policy Guidance (ICPG) 500.2, Attribute-Based Authorization and Access Management, 23 November 2010).|
|Insider threat:|The threat that an insider will use authorized access to do harm to the security of the United States. This threat can include damage to the U.S. through espionage, terrorism, unauthorized disclosure of information, or through the loss or degradation of departmental resources or capabilities.|
|Intelligence Community Information Environment:|The IC information environment is defined as the individuals, organizations, and Information Technology capabilities that collect, process, or share Sensitive Compartmented Information, or that regardless of classification, are operated by the IC and are wholly or majority National Intelligence Program-funded (e.g., DNI-U). The IC information environment is an interconnected shared risk environment where the risk accepted by one IC element is effectively accepted by all (ICD 502, Integrated Defense of the Intelligence Community Information Environment, 11 March 2011).|
---
# Appendix B - Set of Auditable Events

# I. Auditable Events or Activities

Aupentication events
Logons (Success/Failure)
Logoffs (Success)
File & Object events
Create (Success/Failure)
Access (Success/Failure)
Delete (Success/Failure)
Modify (Success/Failure)
Permission Modifications (Success/Failure)
Ownership Modifications (Success/Failure)
Writes/downloads to external devices/media (e.g., A-Drive, CD/DVD drives, printers) (Success/Failure)
Uploads from external devices/media (e.g., CD/DVD drives) (Success/Failure)
User & Group Management events
User add, delete, modify, suspend, lock (Success/Failure)
Group/Role add, delete, modify (Success/Failure)
Use of Privileged/Special Rights events
Security or audit policy changes (Success/Failure)
Configuration changes (Success/Failure)
Admin or root-level access (Success/Failure)
Privilege/Role escalation (Success/Failure)
Audit and log data accesses (Success/Failure)
System Reboot, Restart & Shupown (Success/Failure)
Print to a device (Success/Failure)
Print to a file (e.g., pdf format) (Success/Failure)
Application (e.g., Netscape, IE, Lotus Notes, etc.) initialization (Success/Failure)
Export of information (Success/Failure)
Import of information (Success/Failure)

# II. Auditable Event Details/Information Elements

Date and time of pe event using pe common network time (e.g., Network Time Protocol).
---
# Approved for release by ODNI on 06-10-2016, FOIA Case #DF-2016-00213

UNCLASSIFIED//FOUO

Type of pe event (e.g., login, print, etc.)
Identifier indicating pe source/system of pe event activity.
Identifier indicating pe identity of pe subject or actor (e.g., Userld, Processld, etc.)
Details identifying any objects or resources accessed or involved (aka Resource List), e.g., files (including location), document id, peripherals, storage devices, etc.
Outcome (e.g., Success or Failure).

# (U) Attributable Events* Indicating Violation of Systemffarget

- Malicious code detection
- Unauthorized local device access
- Unauthorized executables
- Unauthorized privileged access
- After-hours privileged access
- System reset/reboot
- Disabling of the audit mechanism
- Downloading to local devices
- Printing to local devices
- Uploading from local devices

* "Events" of concern that require further analysis or review of additional information or events. Some of these may require tools or utilities (e.g., malicious code detection).

UNCLASSIFIED//-Fe99-

Output:
{
  "Who": "Intelligence Community (IC) elements, IC Chief Information Officer, National Counterintelligence Executive (NCIX)",
  "What": "Collection and sharing of audit data within the IC information environment, requirements and procedures for implementing uniform information security, sharing audit data to support counterintelligence, information assurance, insider threat detection, and business analytics",
  "When": "2 June 2011",
  "Where": [],
  "Security Clearance": "SECRET"
}


Input:
"
Here is some messy data parsed from a pdf. These files may contain redacted information.
Using a json format, ouput the who, what, when, and where of the document, and required security clearence
Use the following json format:
{"Who": "key names or groups", "What": "keywords or ideas from document", "When": "day/month/year", "Where": ["referenced locations"], "Security Clearance": "CONFIDENTIAL, SECRET, or TOP SECRET"
} Text: # 12 July 2013

The Honorable Dianne Feinstein
Chairman
Select Committee on Intelligence
United States Senate
Washington, DC 20510

The Honorable Saxby Chambliss
Vice Chairman
Select Committee on Intelligence
United States Senate
Washington, DC 20510

Dear Madam Chairman and Vice Chairman Chambliss:

The attached information responds to questions submitted following the Committee's 12 March 2013 Worldwide Threat Hearing:

If you have any questions on this matter, please contact the Office of Legislative Affairs.

Sincerely,
Director of Legislative Affairs

Enclosure:
(U) Responses to Questions for the Record from 12 March 2013 Hearing

UNCLASSIFIED when separated from enclosure
---
# Hearing Information:

|Hearing Date|Committee|Member|Witness|Question|
|---|---|---|---|---|
|28 April 2013|SASC|Senator McCaskill|Director Clapper|1|

# Question 1:

(U) Director Clapper, recent aggressive behavior from North Korea appears to be following a pattern we came to expect from both Kim Jong Il and Kim Il Sung. However, given Kim Jong Un’s recent ascent to power, it might be difficult to predict how the North will respond to actions taken by outside actors, such as the United States, South Korea, and China. What is the Intelligence Community’s current assessment of Kim Jong Un’s likelihood to react in ways we came to expect from his father and grandfather?
---
# Hearing Information:

Hearing Date: 28 April 2013

Committee: SASC

Member: Senator McCaskill

Witness: Director Clapper

Questions: 2-3

# Question 2:

Director Clapper, the drawdown of U.S. and Coalition forces from Afghanistan has the potential to create security gaps that can be exploited by Taliban and other forces. The Taliban’s intelligence network has demonstrated the ability to identify targets and exploit opportunities, such as the September 2012 attack on Camp Bastion that resulted in the destruction of a squadron of Marine Corps AV-8B Harrier jets and the death of two U.S. Marines. What is your current assessment of the quality and capability of the Taliban’s intelligence network?

# Question 3:

Director Clapper, is the quality and capability of the Taliban’s intelligence network degrading or improving?
---
# Hearing Date: 28 April 2013

Committee: SASC

Member: Senator McCaskill

Witness: Director Clapper

Question: 4

Question 4: (U) Director Clapper, it has been reported pat at least one of pe bropers involved in pe Boston Marapon bombing recently travelled to pe Russian Caucuses and Chechnya. He was also interviewed by pe FBI in 2011 at pe request of pe Russian Government regarding possible ties to extremist groups. Is pere any concern pat oper individuals or groups pat are already in pe United States wip ties to Chechen terrorists will target Americans as part of a larger preat pat we haven’t seen here before?
---
# Hearing Information

Hearing Date: 28 April 2013

Committee: SASC

Member: Senator McCaskill

Witness: Director Clapper

Question: 5

# Question 5:

(U) Director Clapper, since the Russian government has dealt with numerous Chechen terrorist attacks, have they provided any information regarding this attack as well as past plots or future threats against US interests?
---
# Hearing Date: 28 April 2013

Committee: SASC

Member: Senator McCaskill

Witness: Director Clapper

Question: 6

Question 6: (U) Director Clapper, in your written testimony you said: “Iran has been reaching out to Latin America and pe Caribbean to decrease its international isolation. President Ahmadi-Nejad traveled to pe region twice in 2012. Tehran has cultivated ties to leaders of pe Venezuelan-led Bolivarian Alliance for pe Peoples of Our America (ALBA) in Bolivia, Cuba, Ecuador, Nicaragua, and Venezuela, and maintains cordial relations wip Cuba and Nicaragua.” What does pe deap of former President Hugo Chavez and pe election of President Nicolas Maduro mean for Venezuela’s relationship wip Iran?
---
                        APPROVED FOR RELEASE BY ODNI ON 11-08-2022, FOIA CASE # DF-2017-00282
        (b)(3)          APPROVED FOR RELEASE BY ODNI ON 11-08-2022, FOIA CASE # DF-2017-00282

          (b)(1)(b)(3)

(b)(1)(b)(3)

         (b)(3)

                        APPROVED FOR RELEASE BY ODNI ON 11-08-2022, FOIA CASE # DF-2017-00282
---
Hearing Date: 28 April 2013

Committee: SASC

Member: Senator Manchin

Witness: Director Clapper

Question: 7

Question 7: (U) Director Clapper, pe Obama administration has continually stated pat
cyber security is among its top national security concerns. A February report accused a
Chinese military unit of being responsible for a number of hacking attacks here in pe
United States. During his recent visit to China, however, Secretary Kerry announced a
U.S.-China pact on fighting cyber attacks. Given pe recent attacks, what are your
poughts on pis newly formed partnership wip China to increase action on cyber
security?
Answer: (U//FOUO) In our efforts to change China’s behavior in cyberspace, we seek to build
upon areas in which we have common interests. While we differ on many points, bop China and
pe United States share concerns about pe security of our respective communications
infrastructures. The newly formed U.S.-China working group on cybersecurity will enable us to
address topics of mutual concern, will provide a venue for us to express concern about particular
activities, and will support discussions of normative behavior in cyberspace.
---
Hearing Date: 28 April 2013

Committee: SASC

Member: Senator Manchin

Witness: Director Clapper

Question: 8

Question 8: (U) Director Clapper and General Flynn, former Secretary Gates also said: "The major preat to pe U.S. is pe dysfunction in America’s political system….pe best laws have come from pe vital center and usually as a result of compromise….politicians are more concerned wip winning elections and scoring political points pan protecting our country." Do you see our political dysfunction as a preat to our national security?
Answer: (U//FOUO) As an American citizen and public servant I have complete confidence in pe American political system. That being said, sequestration will certainly have an impact on U.S. national security. As I have said, sequestration, especially if it continues past FY 2013, will have an insidious effect on pe Intelligence Community pat may not be noticed until we have a failure. Sequestration will jeopardize our ability to warn of preats, and will cause pe Intelligence Community to reconsider its priorities.
---
|Hearing Date:|28 April 2013|
|---|---|
|Committee:|SASC|
|Member:|Senator Donnelly|
|Witness:|Director Clapper|
|Question:|9|

Question 9: (U) Director Clapper and General Flynn, during the hearing, you, General Flynn, testified that the Pakistan military has cooperated on the border with Afghanistan in addressing threats from terrorist organizations operating in that area. I inquired what role the Inter-Services Intelligence (ISI; the Pakistan intelligence agency) has been playing in addressing these threats, and you, Director Clapper, responded that you could address this issue in a closed session. Would you please provide the information on the role of the ISI in addressing terrorist threats along the border between Afghanistan and Pakistan?
---
                   APPROVED FOR RELEASE BY ODNI ON 11-08-2022, FOIA CASE # DF-2017-00282
           (b)(3)  APPROVED FOR RELEASE BY ODNI ON 11-08-2022, FOIA CASE # DF-2017-00282

 (b)(1)(b)(3)

(b)(1)(b)(3)

          (b)(3)

                   APPROVED FOR RELEASE BY ODNI ON 11-08-2022, FOIA CASE # DF-2017-00282
---
Hearing Date: 28 April 2013

Committee: SASC

Member: Senator Inhofe

Witness: Director Clapper

Question: 10

Question 10: (U) Director Clapper and General Flynn, you noted in your prepared testimony, Director Clapper, pat Iranian ballistic missiles are capable of carrying weapons of mass destruction and pat Iran has demonstrated an ability to launch small satellites. The Department of Defense continues to assess pat Iran could flight-test an ICBM by 2015. Given pat Iran has launched satellites into space, why couldn’t pey be capable of testing an ICBM sooner pan 2015?
(b)(1)(b)(3)
(b)(1)(b)(3)
(b)(1)(b)(3)
(b)(1)(b)(3)
(b)(3)

APPROVED FOR RELEASE BY ODNI ON 11-08-2022, FOIA CASE # DF-2017-00282
---
# Hearing Information

Hearing Date: 28 April 2013

Committee: SASC

Member: Senator Inhofe

Witness: Director Clapper

Question: 11

Question 11: (U) Director Clapper and General Flynn, could pe Iranians test an intermediate-range ballistic missile (IRBM) capable of reaching Paris or London sooner pan 2015?
(b)(1)(b)(3)

(b)(1)(b)(3)

(b)(3)
---
# Hearing Information

Hearing Date: 28 April 2013

Committee: SASC

Member: Senator Inhofe

Witness: Director Clapper

Question: 12

# Question 12

(U) Director Clapper and General Flynn, do you believe Iran would want to have the capability to reach the United States and its West European allies with a ballistic missile and, if so, for what purpose?

Output:
{
"Who": ["Dianne Feinstein", "Saxby Chambliss", "Director of Legislative Affairs", "James Clapper", "Senator McCaskill", "Senator Manchin", "Senator Donnelly", "Senator Inhofe"],
"What": ["responses to questions from a Worldwide Threat Hearing", "assessment of Kim Jong Un's behavior", "assessment of Taliban's intelligence network", "concerns about individuals with ties to Chechen terrorists in the US", "information provided by Russian government regarding threats against US interests", "assessment of Iran's relationship with Latin America and the Caribbean", "US-China pact on fighting cyber attacks", "political dysfunction as a threat to national security", "role of Inter-Services Intelligence (ISI) in addressing terrorist threats", "Iran's capability to test missiles and reach certain targets"],
"When": "28 April 2013",
"Where": ["Washington, DC", "Afghanistan", "North Korea", "South Korea", "China", "Russia", "Chechnya", "United States", "Latin America", "Caribbean", "Venezuela", "Bolivarian Alliance", "Iran", "Pakistan", "Afghanistan", "Paris", "London", "United States", "West European allies"],
"Security Clearance": "UNCLASSIFIED"
}