Background DOD’s organizational structure includes the Office of the Secretary of  Defense, the Joint Chiefs of Staff, the military departments, numerous  defense agencies and field activities, and various unified combatant  commands that contribute to the oversight of DOD’s acquisition  programs. Prior to February 2018, the former Under Secretary of Defense  for Acquisition, Technology, and Logistics also served as the principal  acquisition official of the department and was the acquisition advisor to  the Secretary of Defense. The former Under Secretary also served as the Defense Acquisition  Executive and was the official responsible for supervising the acquisition  of MAIS programs. The former Under Secretary’s authority included  directing the military services and defense agencies on acquisition  matters and making milestone decisions for MAIS and other programs.  This official also had policy and procedural authority for the defense acquisition system, which establishes the steps that DOD programs  generally take to plan, design, acquire, deploy, operate, and maintain the  department’s information systems. However, as of February 2018, the department changed the way it  conducts business and operations with the statutory elimination of the  Office of the Under Secretary of Defense for Acquisition, Technology, and  Logistics. The statute contains a provision that required DOD to establish  a new Office of the Under Secretary of Defense for Research and  Engineering to be responsible for driving innovation and acceleration of  the advancement of warfighting capability. In addition, a new Office of the  Under Secretary of Defense for Acquisition and Sustainment was created  to focus on delivering proven technology more quickly. The creation of  these offices within the department is intended to shift the principal focus  of the Office of the Secretary of Defense from a role of program oversight  to that of directing major department investments. Further, the statutory  creation of a Chief Management Officer to replace the former Deputy  Chief Management Officer is intended to improve the quality and  productivity of the department’s business operations. DOD’s Acquisition  Guidance and Framework  for Managing MAIS  Acquisitions In January 2015, DOD updated its guidelines that outline the framework  for MAIS programs. This framework consists of six models for acquiring  and deploying a program, including two hybrid models that each describe  how a program may be structured based on the type of product being  acquired (e.g., software-intensive programs and hardware-intensive  programs). A generic acquisition model that shows all of the program life- cycle phases and key decision points is depicted in figure 1 and  described below. Materiel solution analysis: Refine the initial system solution (concept)  and create a strategy for acquiring the solution. A decision—referred to as  Milestone A—is made at the end of this phase to authorize entry into the  technology maturation and risk reduction phase. Technology maturation and risk: Determine the preferred technology  solution and validate that it is affordable, satisfies program requirements,  and has acceptable technical risk. A decision—referred to as Milestone  B—is made at the end of this phase to authorize entry of the program into  the engineering and manufacturing development phase and award  development contracts. An acquisition program baseline is first  established at the Milestone B decision point. A program’s first acquisition  program baseline contains the original life-cycle cost estimate (which  includes acquisition and operations and maintenance costs), the schedule  estimate (which consists of major milestones and decision points), and  performance parameters that were approved for that program by the  milestone decision authority. The first baseline is established after the  program has refined user requirements and identified the most  appropriate technology solution that demonstrates that it can meet users’  needs. Engineering and manufacturing development: Develop a system and  demonstrate through testing that the system meets all program  requirements. A decision—referred to as Milestone C—is made during  this phase to authorize entry of the system into the production and  deployment phase or into limited deployment in support of operational  testing. Production and deployment: Achieve an operational capability that  meets program requirements, as verified through independent operational  tests and evaluation, and implement the system at all applicable  locations. Operations and support: Operationally sustain the system in the most  cost-effective manner over its life cycle. Leading Practices for  Managing IT Investments  and Acquisition Programs We have developed and identified leading practices for governing IT  investments to help guide organizations to better manage and oversee  their projects. GAO’s Information Technology Investment Management  guide states that good performance data and stakeholder oversight are  elements that can lead to positive outcomes, such as helping to ensure a  project is keeping to its initial cost, schedule, and performance goals. The  guide also states that projects should be reviewed at regular intervals to  monitor performance so that stakeholders can be aware of and review  any differences between actual outcomes and goals. In addition, we and other entities, such as the Software Engineering  Institute at Carnegie Mellon University, have identified leading practices  to help guide organizations to effectively plan and manage their  acquisitions of major IT systems. Our prior reviews have shown that  proper implementation of such practices can significantly increase the  likelihood of delivering promised system capabilities on time and within  budget. These practices include, but are not limited to:  Requirements management: Requirements establish what the system is  to do, how well it is to do it, and how it is to interact with other systems.  Appropriate requirements management involves eliciting and developing  customer and stakeholder requirements, and analyzing them to ensure  that they will meet users’ needs and expectations. It also consists of  validating requirements as the system is being developed to ensure that  the final systems to be deployed will perform as intended in an  operational environment. Risk management: Risk management is a process for anticipating  problems and developing plans to take appropriate steps to mitigate risks  and minimize their impact on program commitments. It involves  identifying and documenting risks, categorizing them based on their  estimated impact, prioritizing them, developing risk mitigation strategies,  and tracking progress in executing the strategies. DOD’s Policies for  Managing MAIS  Programs Do Not  Always Adhere to  Leading IT  Management  Practices According to GAO’s Information Technology Investment Management  guide, leading practices for managing IT projects include:  instituting the investment board, which is the process for creating and  defining the membership, guiding policies, operations, roles,  responsibilities, and authorities within the organization;  identifying decision authorities for making important acquisition  decisions; providing oversight whereby the organization monitors each project on  its performance progress (e.g., establishing and tracking baseline  estimates on cost and schedule goals, and thresholds to identify high  risk on cost and schedule); and  capturing and providing performance information about a particular  investment (project) to decision makers at regular intervals (e.g.,  quarterly and annually). To align MAIS programs with the functions they perform, DOD recently  made changes in how it characterizes its MAIS programs and, as a result,  different programs must follow different management policies.  Specifically, in April 2017, DOD identified 10 of 34 total MAIS programs  as business programs and the Director, Acquisition Resources and  Analysis announced that these programs would adhere to DOD’s  Instruction 5000.75 policy for management and oversight. Further, in  November 2017, the former Under Secretary of Defense for Acquisition,  Technology, and Logistics announced that non-business MAIS  programs would adhere to DOD’s Instruction 5000.02 policy for  management and oversight. However, the policies used for MAIS business programs are not  consistent in their adherence to leading IT management practices. For  example, while the policy for non-business MAIS programs is consistent  in its adherence to all four of the leading IT management practices, the  policy for MAIS business programs is consistent in its adherence to only  two of the four practices. Table 1 shows our analysis of DOD’s policies for  non-business MAIS programs and MAIS business program and their  adherence to the leading IT management practices. As shown in the table, DOD’s policy for non-business MAIS programs  adheres to all four leading IT management practices. For example, the  policy requires non-business MAIS programs to report the status of each  program’s cost, schedule, and technical performance information  quarterly and annually. The policy also designates specific decision  makers who are responsible for monitoring and overseeing the progress  of non-business system MAIS programs. Further, the policy requires each  program to establish and report their initial baseline estimates and current  estimates on cost and schedule so their performance can be tracked and  monitored. In addition, to identify when programs may be at risk of  significant cost or schedule increases, the policy requires programs to  predetermine cost and schedule threshold estimates as an early warning  indicator on when programs reach the point where they are at increased  risk. In contrast, DOD’s policy for MAIS business programs policy only  adheres to two of the four practices. Specifically, the policy adheres to the  practice of instituting an investment board with processes for creating and  defining the membership, policies, operations, roles, responsibilities, and  authorities within the organization. In addition, the policy identifies  decision authorities for making important executive-level acquisition  decisions. However, the policy does not specify the establishment of  initial and current baseline estimates on cost and schedule, and does not  specify the reporting of threshold cost and schedule estimates to identify  the point when programs may be at high risk. In addition, the policy does  not adhere to leading practices requiring the periodic (quarterly and  annual) reporting of performance information to stakeholders. To help address the need for improved guidance, the former Under  Secretary of Defense for Acquisition, Technology and Logistics  established a cross-functional team that is to examine the future of non- business MAIS programs and MAIS business programs from a policy,  organization, management, and reporting perspective. The team was  expected to provide its recommendations to the Under Secretary of  Defense for Acquisition and Sustainment by March 15, 2018. However,  because no final decisions had been made by the Under Secretary as of  that date, it is unclear what specific actions the department will take  regarding its policy recommendations, among other recommendations, to  improve the management of non-business MAIS programs and MAIS  business programs. Until DOD updates its policy for MAIS business programs to adhere to  leading practices on the establishment of baseline estimates on cost and  schedule to include threshold estimates on cost and schedule to identify  when programs may be at high risk, stakeholders may not have the  information they need to manage and oversee MAIS business programs.  Further, unless the department updates its policy for MAIS business  programs to adhere to the leading practice for periodically (quarterly and  annually) reporting essential performance information, stakeholders may  not have the information they need to make informed decisions for  managing and overseeing MAIS business programs. All Selected MAIS  Programs Had  Changes in Cost and  Schedule Estimates,  and Most Programs  Had Met Performance  Targets All of the 15 selected MAIS programs had either increased or decreased  their planned cost estimates, and 10 of them had delays in their planned  schedule estimates when comparing the first acquisition program  baseline to the most recent acquisition program baseline estimates. The  changes in the cost estimates ranged from a decrease of $1.6 billion (-41  percent) to an increase of $1.5 billion (163 percent), and slippages in the  schedule estimates ranged from a delay of 5 years to a delay of 5  months. Further, 9 of the 15 selected programs had conducted testing in which we  could report on the number of performance targets met for each  program. Of those 9, 6 programs reported that they had met all of their  performance targets. The remaining 3 programs reported that they met  several but not all performance targets. The following table shows the  extent of changes in planned cost and schedule estimates for the  selected MAIS programs since the first baseline estimate, as well as the  number of performance targets met. All Selected MAIS  Programs Had Either  Increases or Decreases in  Their Planned Cost All 15 selected MAIS programs had experienced increases or decreases  in their planned cost estimates when comparing the initial, or first,  baseline estimate to the current estimate. Specifically, 10 programs had  decreases in their cost estimates that ranged from $1.2 million (less than  -1 percent) for the Defense Agencies Initiative, Increment 2 program to  $1.6 billion (-41 percent) for the Air Force’s Base Information Transport  Infrastructure Wired program. Program officials reported that reductions in planned cost estimates were  due to changes in program scope. Specifically, the reasons for reduction  in cost include: Program scope changes. Officials for the Air Force’s Joint Space  Operations Center Mission System Increment 2 program reported that  its 12 percent cost decrease was due to a reduction in its estimate for  operations and support that was changed from 20 years to 10 years.  Officials for the Defense Information Systems Agency’s Global  Combat Support System–Joint Increment 8 program reported that its  20 percent cost decrease was due to a reduction in the program’s  scope for the number of development hours required to meet the  logistics and operational needs. In addition, officials for the Defense  Information Systems Agency’s Teleport Generation 3 program  reported that its 22 percent cost decrease was due to a revised scope  in terms of what is needed at the Milestone C decision point for low  rate production. Design reconfiguration. Officials for the Air Force’s Base Information  Transport Infrastructure Wired program reported that its 41 percent  cost decrease was due to a reduction in the program’s scope when  they changed from a base network system to a critical core  configuration. In addition, 5 of the programs had experienced cost increases. These  cost increases ranged from $2.9 million (less than 1 percent) for the  Army’s Logistics Modernization Program Increment 2 to $1.5 billion (163  percent) for the Army’s Tactical Mission Command program. Program  officials reported a variety of reasons for the increases in planned cost  estimates. These reasons included the following: Underestimating schedule. Officials for the Air Force’s Defense  Enterprise Accounting and Management System Increment 1 program  attributed its 60 percent cost increase to underestimating the level of  effort that was needed to develop the system within the estimated  schedule. For example, the program did not account for software  upgrades and, when this effort was added to the schedule to account  for the work, the cost increased. Contractor issues. Officials from the National Security Agency’s Key  Management Infrastructure Increment 2 program attributed its cost  increase of 14 percent to schedule delays caused by the contractor  and, as a result, increased funding at the Milestone C decision point. Underestimating development and test efforts. Officials from the  Army’s Tactical Mission Command program attributed the cost  increase of 163 percent to higher than expected costs to conduct  research and developmental tests. Ten Selected MAIS  Programs Had Delays in  Their Planned Schedule  Estimates Ten of the 15 selected MAIS programs had experienced changes in their  planned schedule estimates, and 5 programs had no changes to their  schedule estimates. The changes consisted of schedule slippages that  ranged from 5 months for both the Army’s Logistics Modernization  Program Increment 2 and the Defense Health Agency’s Department of  Defense Healthcare Management System program, to 5 years for the  Defense Enterprise Accounting and Management System Increment 1  program. Program officials reported that delays in the planned schedule estimates  were due to unplanned budget reductions or unrealistic expectations  regarding project milestones. Specifically, the reasons for these schedule  slippages included: Aggressive schedule, funding reduction, and contract issues.  Officials for the Air Force’s Joint Space Operations Center Mission  System Increment 2 program attributed its schedule slippage of 2  years and 11 months to funding reductions of $18.9 million in fiscal  years 2013 and 2014. In addition, the officials noted that an  aggressive schedule for a Milestone B decision, contracting issues in  the earlier acquisition phase, and longer than expected time to obtain  personnel had contributed to the slippage. Longer than expected time to reach deployment. Officials for the  Air Force’s Defense Enterprise Accounting and Management System  Increment 1 program reported that its schedule slippage of 5 years  occurred because of a change in the approach to deliver the system  in multiple increments, thereby increasing the amount of time it would  take to reach the deployment decision milestone. Also, officials for the  Defense Information Systems Agency’s Teleport Generation 3  program reported a 3-year and 2-month slip. This schedule delay was  due to the program’s inability to develop the mobile user and system  interface capability by the estimated deployment milestone. Further,  program officials for the Navy’s Consolidated Afloat Networks and  Enterprise Services program attributed its schedule slip of 2 years and  6 months to a longer than expected maintenance period for the test  platform and to a lengthy budget approval process, resulting in a  slippage in the deployment date. Unplanned procurement fund reduction. Officials for the Army’s  Global Combat Support System-Army program reported that its  schedule delay of 11 months was due, in part, to a $16 million dollar  decrease to the fiscal year 2016 budget. This unplanned reduction in  procurement fund affected their ability to field the system as originally  planned. Contractor staffing issues. Officials for the National Security  Agency’s Key Management Infrastructure Increment 2 program  reported significant schedule delays due to the contractor’s inability to  staff the program with software developers that had the required  security clearances. As a result, a critical change was reported in  January 2012 that led to a new independent cost estimate, which  extended program development by 10 months. The new estimate  included additional time to improve the governance structure, such as  increasing discipline across the oversight process, adding more  stakeholder interaction, and improving the use of metrics. Six of Nine MAIS  Programs Had Met All  Performance Targets Among other information, DOD uses key performance parameters as a  metric to report on programs’ progress toward meeting system  performance targets. This information includes a description of the  performance characteristics, the objective and threshold value for each  target and, importantly, whether the target has been met in demonstrating  performance. Of the nine programs we evaluated, six programs reported that they met  all of their performance targets. For example, the Navy’s Common  Aviation Command and Control System, Increment 1 program reported in  May 2017 that both of its technical performance targets had been met.  According to the program, these targets were related to the readiness of  the system to fully support all operational activities and satisfy all  technical requirements for military operations and the fusion of all kinds of  data onto any workstation. In another example, the Army’s Logistics Modernization Program  Increment 2 program reported in June 2017 that all seven of its  performance targets had been met. According to the program, these  targets were related to the system’s ability to support military operations,  exchange information in the network, provide system and information  assurance in a disaster recovery scenario, and be operationally available. Further, three programs reported that they met several, but not all, of their  performance targets. For example, the Navy’s Consolidated Afloat  Networks and Enterprise Services program reported that it met eight of  nine performance targets. According to program officials, the remaining  target (i.e., network shall fully support joint critical operational activities)  had not been met because the program lacked an operational platform  that was required to demonstrate its performance. The Defense Information Systems Agency’s Teleport Generation 3  program reported that it met 8 of 12 performance targets. According to  programs officials, the remaining 4 targets (i.e., coverage to allow  warfighter communications, capacity to provide 100 percent of the  required services, and interoperability with military and commercial  frequencies and wave forms) had not been met because the program  needed to field multiple systems and perform solution testing, which they  expect to be completed in fiscal year 2018. Further, the Air Force’s  Defense Enterprise Accounting and Management System Increment 1  program officials reported that it met 3 of 4 targets (i.e., compliance with  requirements, network ready, and sustainment to ensure materiel  availability). The officials reported that the program did not meet the  remaining target because it was waiting for an evaluation of cyber test  results before proceeding. Selected Programs  Fully Implemented  Most, but Not All, of  the Leading Practices  for Managing  Requirements and  Risk According to the Software Engineering Institute’s Capability Maturity  Model Integration® for Acquisition (CMMI®-ACQ), an appropriate  requirements management process involves establishing an agreed-upon  set of requirements, ensuring traceability between requirements and work  products, and managing any changes to the requirements in collaboration  with stakeholders. Likewise, an effective risk management process  identifies potential problems before they occur, so that risk-handling  activities may be planned and invoked, as needed, across the life of the  project in order to mitigate the potential for adverse impacts. Leading requirements management practices help organizations to better  manage the design, development, and delivery of systems within  established cost and schedule time frames. These practices include developing an understanding with the requirements providers of the  meaning of the requirements, obtaining commitment to requirements from project participants, managing changes to requirements as they evolve during the project, maintaining bidirectional traceability among requirements and work, ensuring that project plans and work products remain aligned with  requirements. An effective risk management process includes the following leading  practices determining risk sources and categories; defining parameters used to analyze and categorize risks and to  control the risk management effort; establishing and maintaining the strategy to be used for risk  identifying and documenting risks; evaluating and categorizing each identified risk using defined risk  categories and parameters, and determining its relative priority; developing a risk mitigation plan in accordance with the risk monitoring the status of each risk periodically and implementing the  risk mitigation plan as appropriate. The three selected MAIS programs that we evaluated had fully  implemented most, but not all, of the five leading practices for managing  requirements and the seven leading practices for managing risks.  Specifically, two of three programs implemented all of the requirements  management practices, while one program implemented most, but not all,  of the practices. Further, one of three programs implemented all of the risk management  practices, while two programs implemented most, but not all of the  practices. Table 3 shows the extent to which practices were implemented  by the three selected programs. Two Programs Fully and  One Program Partially  Implemented Leading  Practices for Managing  Requirements Two of the three programs had fully implemented the requirements  management practices. The other program had partially implemented two  practices and fully implemented three practices. Navy — Navy Consolidated Afloat Networks and Enterprise Services  The Navy had fully implemented the five requirements management  practices for the Consolidated Afloat Networks and Enterprise Services  program. For example, the program developed an understanding with  requirements providers of the meaning of the requirements. Specifically,  there was a plan for documenting, managing, and controlling changes to  requirements throughout the system lifecycle. This plan served as the  primary guidance for integrating the management of all specified and  derived requirements for the Consolidated Afloat Networks and Enterprise  Services system program. In addition, the program had established criteria for determining  requirements providers. Specifically, roles and responsibilities for  requirements management had been identified. Further, the program  managed changes to requirements as they evolved during the project.  For example, the program provided evidence that it maintains a  requirements change history, including the rationale for changes. Defense Logistics Agency — Defense Agencies Initiative, Increment  2  The Defense Logistics Agency had fully implemented the five  requirements management practices for the Defense Agencies Initiative,  Increment 2. For example, the program had established objective criteria  for the evaluation and acceptance of requirements. Specifically, there was  a process in place to develop and finalize deliverables in support of the  business requirements identified by the stakeholders, ensure that  requirements management activities were performed in a timely manner  throughout the life of the project, and review and approve requirements  deliverables. Further, throughout the process, the requirements manager  tracked requirements changes and maintained traceability of end user  needs to the system performance specification. The Defense Health Agency had fully implemented three and partially  implemented two of the five requirements management practices for the  Defense Healthcare Management System Modernization program. For  example, the program had established objective criteria for the evaluation  and acceptance of requirements. Specifically, any new or updated  requirements were presented to a Configuration Steering Board for  review and approval prior to any changes being made. Further,  throughout the process, the requirements manager tracked requirements  changes and maintained traceability to ensure they were documented. The program has not developed an understanding with the requirements  providers on the specific meaning of the requirements. For example,  although the program had developed a requirements management plan  which provided guidance in this area, according to program officials, the  plan was not signed and approved based on the recent shift of the  program from a non-business MAIS program to a MAIS business program  operating under DOD Instruction 5000.75. Program officials stated that  the requirements management plan is not expected to be complete until  final guidance is provided by the Office of the Secretary. Regardless of this recent shift, the program should have already had an  approved requirements management plan in place since program  initiation. In the absence of an approved plan, the program lacks  assurance that it can effectively communicate and manage requirements  practices. Further, the program had not demonstrated that it identified any changes  that should be made to plans and work products resulting from changes  to the requirements baseline. Programs officials stated that efforts to  review modifications to the plan due to requirements changes had not  been conducted, but they expected the review and approval to be done at  some future date. However, they could not provide a specific time frame.  According to CMMI®-ACQ, until project plans and work products are  updated to coincide with changes in requirements, the program will not be  able to effectively identify inconsistencies between requirement changes  and project plans and work products, and initiate corrective actions to  resolve them. One Program Fully and  Two Programs Partially  Implemented Applicable  Risk Management Leading  Practices One program had fully implemented the risk management practices, while  two had fully implemented all but one practice. Navy — Navy Consolidated Afloat Networks and Enterprise Services  The Navy had fully implemented six and partially implemented one risk  management practice for the Consolidated Afloat Networks and  Enterprise Services system program. For example, the program’s risks  defined consistent criteria for evaluating and quantifying risk likelihood  and severity levels. Specifically, the program provided a risk exposure  (e.g., a risk source used to examine and oversee changes that impact the  project), which is the value that is given to a risk event, a product, or the  overall program based on the analysis of the probability and  consequences of the event. Further, the program’s Risk Management  Guide outlined risk performance, cost, and schedule criteria. In addition,  the program demonstrated that it included the cost and benefits of  implementing risk mitigation plans. Specifically, a risk’s description  provided the cost impacts associated with the risk, which in turn provided  evidence that cost and benefits were considered during risk evaluation. However, the Navy partially implemented one practice. Specifically,  although the program provided its failover/recovery plan that is intended  to return the program to a state of readiness after a failure, the plan did  not explicitly identify environmental elements. A program official stated  that environmental factors, such as risks that could negatively affect their  work, is understood, but these factors had not been documented in the  plan. Further, the official stated that the program should update the plan  accordingly, but did not provide a time frame to complete this effort. Until  all potential issues, hazards, threats, and vulnerabilities that could  negatively affect work efforts have been identified in the plan, successful  risk management cannot be ensured. Defense Logistics Agency—Defense Agencies Initiative, Increment 2  The Defense Logistics Agency had fully implemented all seven risk  management practices for the Defense Agencies Initiative, Increment 2.  For example, the program identified program risks, including risk sources,  categories, and stakeholders. In addition, Defense Agencies Initiative,  Increment 2 risks followed consistent criteria for evaluating and  quantifying risk likelihood and severity levels. Specifically, risk level was  based on a combination of factors to include both likelihood and  consequence. In all instances, consensus on the risk levels was required  between the risk owner and the customer counterpart. Further, the  program’s contingency plan provided guidance when outages fell into one  of three disaster categories including natural disasters, man-made  disasters, and technological disasters. Defense Health Agency—Defense Healthcare Management System  Modernization  The Defense Health Agency had fully implemented six and partially  implemented one of the seven risk management practices for the  Defense Healthcare Management System Modernization program. For  example, the program’s risks followed consistent criteria for evaluating  and quantifying risk likelihood and severity levels. Specifically, the  program’s Risk and Issue Management Plan described how to assess the  impact level in each risk area (performance, project and program  schedules, and cost). Further, the program prioritized risks for mitigation.  For example, risks were categorized and charted as low, medium, or  high, and grouped accordingly in the program’s risk register. Further, the  program’s Disaster Recovery Plan provides processes that allowed rapid  support recovery for critical operations during a disaster, including  environmental disasters such as tornadoes. Regarding the partially implemented practice, the program provided an  example of a risk mitigation plan. However, the program indicated that  costs and benefits were not quantified within the program-level risk  mitigation plans. According to CMMI®-ACQ, risk mitigation activities  should be examined for benefits they provide versus resources they will  expend. Just like any other design activity, alternative plans may need to  be developed and costs and benefits of each alternative assessed. However, the program does not require that costs and benefits be  included as part of its risk mitigation planning efforts. As a result, the  information for making an informed decision on cost and benefits of risk  mitigation solutions is limited. Program officials did not indicate whether  they have plans to implement this practice, and did not provide an  explanation as to why they are unable to provide this information. Until  the program quantifies costs and benefits, it will not be able to effectively  select the most appropriate risk mitigation plan to address each risk. Conclusions While DOD’s policy for non-business MAIS programs adheres to all four  leading IT management practices, the department’s policy for MAIS  business programs does not adhere to two leading practices on  establishing initial and current baseline estimates on cost and schedule  and predetermining threshold estimates, as well as reporting periodically  on performance information to stakeholders. Until DOD adheres to these  practices in its policies that govern MAIS business programs, it cannot  ensure that stakeholders will have the information they need to manage  and oversee their investments. Following leading IT acquisition practices on requirements and risk  management is essential to help programs effectively plan and direct their  development and acquisition efforts. All of the leading IT acquisition  practices for requirements and risk management had been fully or  partially implemented by three programs that we reviewed. However, the  Defense Health Agency’s Defense Healthcare Management System  Modernization has not finalized its requirements management plan nor  has it identified changes that should be made to plans and work products  resulting from changes to the requirements baseline. Until the program  addresses these practices, it will lack a comprehensive plan for managing  its requirements and it may not be able to effectively identify  inconsistencies and initiate corrective actions. Further, the Navy Consolidated Afloat Networks and Enterprise Services  program did not fully identify and document risks that could negatively  affect work efforts. In addition, the Defense Health Agency’s Defense  Healthcare Management System Modernization did not quantify costs  and benefits of risk mitigation within its program-level risk mitigation  plans. As a result, successful risk management for avoiding, reducing,  and controlling the probability of risk occurrence cannot be ensured. Recommendations for  Executive Action We are making the following three recommendations to the Secretary of  Defense to direct: The Under Secretary of Defense for Acquisition and Sustainment to  update the policy or guidance for MAIS business programs.  Specifically, the update should include the following elements: establishment of initial and current baseline cost and schedule predetermined threshold cost and schedule estimates to identify  the point when programs may be at high risk, and quarterly and annual reports on the performance of programs to  stakeholders. (Recommendation 1) The Director of the Defense Health Agency to direct the program  manager for the Defense Healthcare Management System  Modernization program to:  finalize and approve its requirements management plan,  identify and document changes that should be made to plans and  work products resulting from changes to the requirements  baseline, and quantify costs and benefits of risk mitigation within its program- level risk mitigation plans. (Recommendation 2) The Secretary of the Navy to direct the program manager for the Navy  Consolidated Afloat Networks and Enterprise Services program to:  identify and document, in the failover/recovery plan, all potential  external environmental issues, such as hazards, threats, and  vulnerabilities that could negatively affect work efforts.  (Recommendation 3) Agency Comments  and Our Evaluation DOD provided written comments on a draft of this report, which are  reproduced in appendix II. In its comments, the department partially  concurred with our first recommendation and concurred with the two other  recommendations. DOD partially concurred with the first recommendation on updating the  policy or guidance for MAIS business programs. Specifically, the Under  Secretary of Defense for Acquisition and Sustainment stated that  regarding establishing baselines, the DOD Instruction 5000.75 requires  establishment of cost, schedule, and performance parameters for each  release before development or delivery. The 5000.75 also requires  consideration of program progress against baselined cost, schedule, and  performance as a criterion at the limited deployment and full deployment  decision points. A baseline requirement thus exists in DOD Instruction  5000.75 but it is not described as an acquisition program baseline, which  may be familiar to readers of DOD Instruction 5000.02. However, the  Under Secretary added that the Army’s implementation guidance includes  guidance that states each increment must have an acquisition program  baseline with its own set of threshold and objective values set by the  user. While we agree that the existing policy requires such parameters to be  captured and included in the department’s decision making process, we  found the policy to be vague in its discussion of these parameters and to  not clearly define what a baseline is, or which baselines are to be used or  reported for comparison purposes. For example, the policy does not  make a distinction between the initial acquisition program baseline,  current baseline, and baseline deviations. Yet, such information is  important because it provides a basis for decision makers to identify the  extent to which a program may have deviated from its initial cost,  schedule, or technical performance baseline. By making these  distinctions in the policy, the department’s policy for its MAIS business  programs will be more consistent with its other policy for non-business  MAIS programs with regard to the way an acquisition program baseline is  defined and the elements that should be captured and reported to its  decision makers. In turn, the program managers who prepare these  reports and the decision makers who rely on them will have information  that is consistently and succinctly prepared for making credible decisions. Regarding adding provisions in its policy for the establishment of  predetermined thresholds, the Under Secretary stated that the 5000.75  states that the milestone decision authority is responsible for delivery  within cost, schedule, and performance parameters, and the milestone  decision authority is to do this by establishing oversight controls for  programs, including procedures to report and address variances. The  Under Secretary added that the 5000.75 does not suggest the practice of  establishing a predetermined threshold for the variance, and DOD will  consider the addition of this feature to the 5000.75 update. Finally, regarding providing periodic annual and quarterly reports to the  department’s leadership, the Under Secretary stated that such a periodic  report would add value only if there had been no recent communication of  program status from the program office to the leadership or stakeholders  communities. While such communication is expected to occur frequently,  its regularity is not specified in current policy or guidance. The Under  Secretary stated that DOD will consider adding a provision for a report to  leadership and functional stakeholder if such communication has not  occurred within the past 3 or 4 months. DOD concurred with the second and third recommendations related to the  department’s implementation of selected IT management practices.  Regarding the second recommendation, the Under Secretary of Defense  for Acquisition and Sustainment agreed to direct the Defense Healthcare  Management System Modernization program manager to update and  approve the requirements management plan, identify and document  changes to the requirements baseline, and quantify the costs and benefits  in the risk mitigation plans. Further, regarding the third recommendation,  the Secretary of the Navy agreed to direct the program manager to  identify and document all potential external environmental issues that  could negatively affect work efforts for the Navy’s Consolidated Afloat  Networks and Enterprise Services program. By taking these steps, these  programs should be better positioned to effectively identify  inconsistencies in managing changes to their requirements, and be more  responsive to the potential for environmental issues. We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense; the Secretaries of the Army, Navy,  and Air Force; the Under Secretary of Defense for Acquisition and  Sustainment; the Director of the Defense Health Agency; and other  interested parties. This report also is available at no charge on the GAO  website at http://www.gao.gov. Should you or your staffs have any questions on information discussed in  this report, please contact me at (202) 512-4456 or harriscc@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III. Appendix I: Objectives, Scope, and  Methodology The National Defense Authorization Act for Fiscal Year 2012 mandated  that we select, assess, and report on selected major automated  information systems (MAIS) programs annually through March 2018.  GAO satisfied the statutory mandate by submitting a draft of this report to  the congressional committees on March 29, 2018. This final version of the  report is the sixth and last report in the series of annual mandated  assessments. Our objectives were to: (1) assess the Department of Defense’s (DOD)  policy for the management and oversight of MAIS programs; (2) describe  the extent to which selected MAIS programs have changed their planned  cost and schedule estimates and met performance targets; and (3)  assess the extent to which selected MAIS programs have used leading  information technology (IT) acquisition practices, including requirements  and risk management. To address the first objective, we identified four leading IT management  practices in GAO’s Information Technology Investment Management  guide and compared DOD’s policy adherence to those practices. These  leading practices are:  instituting the investment board, which is the process for creating and  defining the membership, guiding policies, operations, roles,  responsibilities, and authorities within the organization;  identifying decision authorities for making important acquisition  decisions; providing oversight whereby the organization monitors each project on  its performance progress (e.g., establishing and tracking baseline  estimates on cost and schedule goals, and thresholds to identify high  risk on cost and schedule); and  capturing and providing performance information about a particular  investment (project) to decision makers at regular intervals (e.g.,  quarterly and annually). We then compared DOD’s policies used to manage and oversee the  department’s non-business MAIS programs and MAIS business programs against these leading IT management practices. The department’s policy  documents for managing and overseeing non-business MAIS programs  and MAIS business programs include the: Memorandum by the Under Secretary of Defense for Acquisition,  Technology, and Logistics, dated November 17, 2017, regarding the  regulatory response to the repeal of title 10, United States Code,  Chapter 144A, Major Automated Information System Programs. Memorandum by the Under Secretary of Defense for Acquisition,  Technology, and Logistics, dated April 24, 2017, regarding the  transition of programs to business system categories. DOD Instruction 5000.75, Business Systems Requirements and  Acquisition, effective February 2, 2017. DOD Instruction 5000.02, Operation of the Defense Acquisition  System, effective February 2, 2017. We also interviewed an official from the former Office of the Under  Secretary of Defense for Acquisition, Technology, and Logistics, who was  responsible for the development of plans and policy for the administration  regarding the management and monitoring of non-business MAIS  programs and MAIS business programs  To address the second objective, we used DOD’s official list of 34  business and non-business MAIS programs, as of April 18, 2017, to  establish a basis for selecting programs. Of the 34 programs, we selected  the 15 business and non-business MAIS programs that met our criteria:  programs must be unclassified and have an initial acquisition program  baseline that could be used as a reference point for evaluating cost,  schedule, and technical performance characteristics. We then collected and analyzed key documents, reports, and artifacts for  each program and summarized the information on estimated cost,  schedule, and technical performance goals, including their latest program  status in meeting those estimated goals. Next, we analyzed and  compared each selected program’s first acquisition program baseline cost  estimate to the latest estimate to determine the extent to which planned  program costs had changed. Specifically, we used the total life-cycle cost  estimate and analyzed and compared them to the latest estimate to  determine the extent to which planned program costs had changed.  Similarly, to determine the extent to which these programs changed their  planned schedule estimates, we compared each program’s first  acquisition program baseline schedule to the latest schedule. To determine whether the selected programs met their performance  targets, we analyzed each program’s self-identified system performance  targets and compared them against actual system performance metrics  and latest test reports. We also reviewed additional information on each  program’s cost, schedule, and performance, including program  documentation, such as DOD’s MAIS annual and quarterly reports,  acquisition program baselines, system test reports, and our prior reports.  We then aggregated and summarized the results of these analyses  across the programs. To address the third objective, we started with the list of the 15 programs  from the second objective as a basis for selecting three MAIS programs  as case studies. We used a combination of the following criteria to select  the MAIS programs to review. Programs used in a most recent MAIS review were eliminated from  consideration. The program was not designated as classified. The program had a baseline. Based on these criteria, we chose the following systems: Navy Consolidated Afloat Networks and Enterprise Services; Defense Logistics Agency’s Defense Agencies Initiative, Increment 2; Defense Health Agency’s Defense Healthcare Management System  Modernization. We then analyzed each selected program’s IT acquisition documentation  and compared it to key requirements and risk management and leading practices—including Software Engineering Institute’s Capability Maturity  Model® Integration for Acquisition (CMMI- ACQ) practices—to determine  the extent to which the programs were implementing these practices. In  particular, the requirements management practices we reviewed were: develop an understanding with the requirements providers on the  meaning of the requirements, obtain commitment to requirements from project participants, manage changes to requirements as they evolve during the project, maintain bidirectional traceability among requirements and work, and ensure that project plans and work products remain aligned with  requirements. Specifically, we analyzed program requirements documentation, including  requirements management plans, requirements traceability matrices,  requirements change forms, technical performance assessments, and  requirements board meeting minutes. Additionally, we interviewed  program officials to obtain additional information about their requirements  management practices. The conclusions reached for this objective are not  generalizable to the larger population of 34 business and non-business  MAIS programs. We also reviewed the following risk management practices: determine risk sources and categories; define parameters used to analyze and categorize risks and to control  the risk management effort; establish and maintain the strategy to be used for risk management;  identify and document risks; evaluate and categorize each identified risk using defined risk  categories and parameters, and determine its relative priority; develop a risk mitigation plan in accordance with the risk management  strategy; and monitor the status of each risk periodically and implement the risk  mitigation plan as appropriate. Specifically, we analyzed program risk documentation, including risk  reports, risk-level assignments, risk management plans, risk mitigation  plans, and risk board meeting minutes. Additionally, we interviewed  program officials to obtain additional information about their risks and risk  management practices. To assess the reliability of the data of these programs we used to support  the findings in this report, we corroborated program office responses with  relevant program documentation and interviews with agency officials. We  found no data reliability issues and determined that the data used in this  report were sufficiently reliable for our reporting purposes. We have also  made appropriate attribution indicating the sources of the data. We conducted this performance audit from April 2017 to May 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix II: Comments from the Department  of Defense Appendix III: GAO Contact and Staff  Acknowledgments GAO Contact Staff  Acknowledgments In addition to the contact name above, the following staff also made key  contributions to this report: Eric Winter (Assistant Director), John Ortiz  (Analyst in Charge), Alex Bennett, Neha Bhatt, Chris Businsky, and  Rebecca Eyler.
DOD's MAIS programs are intended to help the agency sustain its key operations. In April 2017, recognizing that MAIS programs met different mission needs, DOD categorized its MAIS programs into business and non-business systems. The National Defense Authorization Act for Fiscal Year 2012 includes a provision for GAO to select, assess, and report on DOD's MAIS programs annually through March 2018. GAO's objectives, among others, were to (1) assess DOD's policies for managing and overseeing MAIS programs and (2) describe the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met technical performance goals. To address these objectives, GAO compared DOD's policies for managing and overseeing all 34 MAIS programs (24 non-business programs and 10 business programs) to leading IT management practices. GAO also compared 15 selected programs' initial cost, schedule, and performance baselines to their current acquisition program estimates. The strength of Department of Defense's (DOD) policies for managing and overseeing major automated information system (MAIS) programs varies. Specifically, the policy for managing 24 non-business MAIS programs adheres to leading information technology (IT) management practices, but the policy for managing 10 MAIS business programs does not always do so (see table). When DOD categorized 10 of the 34 MAIS programs as MAIS business programs, it also directed these programs to adhere to DOD's business systems policy (DOD Instruction 5000.75). However, the department directed those programs to use a policy for the management and oversight of MAIS business programs that was not fully comprehensive. Until DOD updates its business systems policy to address gaps in establishing performance information such as baseline estimates on program cost and schedule goals, identifying thresholds to identify high risk, and requiring periodic reports to be provided to stakeholders at regular intervals, stakeholders will likely not have all the information they need to manage and oversee MAIS business programs. While all 15 business and non-business MAIS programs had either increased or decreased their planned cost estimates and the majority had delays in their planned schedule estimates, the majority of the 9 programs that had performance targets met those performance goals. Specifically, the decreases and increases in cost estimates ranged from a decrease of $1.6 billion (-41 percent) to an increase of $1.5 billion (163 percent). The decreases in planned cost were largely due to scope reduction, while cost increases were due to underestimating levels of effort and contracting issues. The slippages in schedule estimates ranged from a delay of 5 years to 5 months; these delays were caused by unrealistic expectations or unplanned changes. Six of the 9 programs that had performance targets met all of them, while the other 3 met several but not all of their performance targets. The other 6 programs were in the early stages of system development and had not begun performance testing.