Background VBA Disability Benefits  Process VA pays monthly disability compensation to veterans with service- connected disabilities according to the severity of the disability. VA’s  disability compensation claims process starts when a veteran submits a  claim to VA (see fig. 1). A claims processor then reviews the claim and  helps the veteran gather the relevant evidence needed to evaluate the  claim. Such evidence includes the veteran’s military service records,  medical exams, and treatment records from VHA medical facilities and  private medical service providers. If necessary to provide support to  substantiate a claim, VA will also provide a medical exam for the veteran,  either through a provider at a VHA medical facility or through a VBA  contractor. According to VBA officials, VBA monitors a VHA facility’s  capacity to conduct exams and in instances when the facility may not  have capacity to conduct a timely exam, VBA will send an exam request  to one of its contractors instead. For exams assigned to a VBA contractor, VBA sends an exam request to  the contractor, who then rejects or accepts the exam request. Once the  contractor accepts the exam, it assigns a contracted examiner to conduct  the exam and complete an exam report designed to capture essential  medical information for purposes of determining entitlement to disability  benefits. The contractors send the completed report to VBA, which uses  the information as part of the evidence to evaluate the claim and  determine whether the veteran is eligible for benefits. According to  contractor officials, if they need clarification on an exam request, they might reject the request and send it back to VBA who, in turn, will revise  the request before sending it back to the contractor. Use of Contracts to  Complete Disability  Compensation Exams VA has used contracted examiners—through VBA and VHA contracts—to  supplement VHA-provided exams for at least two decades. VBA began  using contractors to conduct disability compensation exams at 10 VBA  regional offices in the late 1990s through a pilot program authorized  under federal law. In 2014, federal law authorized VBA to expand the  pilot to all its regional offices starting in fiscal year 2017. Before fiscal year 2017, VHA and VBA both administered disability exam  contracts. However, since fiscal year 2017, all such contracts have been  administered by VBA and none have been administered by VHA. VBA  awarded 12 contracts to five contractors to begin providing exams in  2016. According to VA officials, performance under 10 of these contracts  was delayed until late September 2017 due, in part, to multiple contract  bid protests. During this delay, VA officials told us that the agency  awarded short-term contracts to allow existing contractors to perform  exams until the bid protests were resolved. VBA’s current contracts cover  exams for veterans in five U.S. geographic districts, one district for  overseas exams, and one district for servicemembers participating in  special programs, such as the Benefits Delivery at Discharge and  Integrated Disability Evaluation System programs (see fig. 2). VBA  awarded two contracts in each of its five U.S. geographic districts and  one contract each in districts 6 and 7, which include special programs and  overseas exams, respectively. VBA also awarded two additional short- term contracts in December 2017 to help address workload issues in  districts 1-5. With the addition of these two contracts, VBA has a total of  14 contracts currently in place. According to agency officials, because VBA wanted to update  performance measures for its contractors, VA issued a Request for  Proposals in May 2018 with plans to award new contracts in fall 2018 for  its U.S. geographic districts. Until it awards the new contracts, VBA will  continue to use the current contracts. According to VBA officials, VA  plans to continue using VBA contractors in the long term to conduct  exams that exceed VHA’s capacity. In recent years, VBA contractors  have completed an increasing number of exams, from roughly 178,000 in  fiscal year 2012 to almost 600,000 in fiscal year 2017, according to VBA- provided data. VA estimates that in fiscal year 2019, contractors will  complete over 1.8 million exam reports for almost 800,000 veterans.  However, VBA officials noted that future projections for contracted exams  might change based on the need to supplement VHA capacity to ensure  timely exams. VBA Contract Exam Office  and Requirements for  Contractors In 2016, VBA established an exam program office to manage and  oversee contractors, monitor their performance, and ensure that they  meet contract requirements. For example, the contracts require that  contractors develop plans outlining how they will ensure examiners are  adequately trained. Contractors are also required to provide VBA with  monthly exam status reports, which include the number of canceled,  rescheduled, and completed exams, among other things. VBA also has  an office dedicated to completing quality reviews of contractors’ exam  reports, which are used to assess contractor performance. The contracts  require that VBA conduct quality reviews of a sample of contractors’ exam  reports. According to VA documents and officials, the results of these  quality reviews, and contractor timeliness scores in completing exams,  are included in quarterly performance reports. The contracts require that  VBA provide these performance reports to the contractors. VBA holds  quarterly meetings with the contractors to discuss their quarterly  performance based on these reports. VBA Licensing and Training  Requirements for Contracted  Examiners VBA contracts require that contracted examiners have full, current, valid,  and unrestricted licenses, and current and valid State Medical Board  certifications, before conducting any exams—the same requirements that  apply to VHA medical providers. According to agency officials, VBA also  requires that contracted examiners complete the same training that VHA  providers must take before they can conduct any disability medical  exams. The required training consists of a set of online courses  developed by VHA’s Disability Medical Assessment Office, such as  courses on VA’s disability claims process and one on completing exam  reports. In addition, examiners who provide some specialized exams,  such as posttraumatic stress disorder exams and traumatic brain injury  exams, are required to take additional courses. In addition to VHA- developed training, VBA contracts require that contractors provide  examiners with a basic overview of VA programs. VBA Quarterly Contractor  Performance Targets for  Quality and Timeliness The contracts also outline quality and timeliness performance targets that  VBA uses to assess contractor performance (see table 1). VBA can use  contractors’ performance in meeting these targets to determine financial  incentives. VBA’s performance measures are as follows: Contractor quality: VBA calculates quality scores for each contractor  based on a sample of exam reports that VBA’s quality office selects  for review on a quarterly basis for each contract. According to VBA  documents, the quality score represents the percentage of exam  reports reviewed that had no errors as measured against specific  criteria. Errors identified in quality reviews could range from  incomplete information (e.g., an examiner’s medical specialty  information is not listed on exam report) to completing the wrong  exam report for a given condition. Contractor timeliness: VBA calculates timeliness scores for each  contractor based on the average timeliness of all exams completed in  a given quarter for each contract. VBA measures timeliness as the  number of calendar days between the date the contractor accepts an  exam request and the date the contractor initially sends the completed  exam report to VBA. VBA Reported  Contractors Missed  Exam Quality Targets,  and VBA Could Not  Accurately Measure  Performance On  Timeliness Targets Contractors Missed  Quality Targets in First  Half of 2017; More Recent  Data Are Not Yet Available  for Most Districts VBA reported that almost all contractors missed VBA’s quality target of 92  percent in the first half of calendar year 2017, and more recent data are  not yet available for most districts. More specifically, VBA-determined  quarterly quality scores—the percentage of disability compensation exam  reports with no errors as measured against VBA criteria—for the seven  contracts used by VBA in calendar year 2017 showed that contractors  were frequently well below the quality target. Quarterly quality scores  ranged from 62 percent to 92 percent (see fig. 3). According to VBA  data, only one contractor’s quality score in one quarter met VBA’s target  of 92 percent while the vast majority of contractors’ scores were classified  by VBA as “unsatisfactory” performance. VBA has not yet completed all of the quality reviews used to calculate  contractor quality scores, particularly for exams that were completed in  the second half of 2017. VBA is hiring and training additional quality  review staff to complete these reviews and help manage the workload  moving forward. According to VBA officials, staff will complete the  remaining quality reviews and finalize the quality scores for 2017 by  December 2018. VBA Could Not Accurately  Measure Contractor  Timeliness Against  Targets, but Our  Aggregate Analysis Shows  About Half of Exams Were  Completed Within 20 Days According to agency officials, VBA has not calculated contractor  timeliness as it is outlined in the contracts. VBA measures timeliness as  the number of days between the date the contractor accepts an exam  request and the date the contractor initially sends the completed exam  report to VBA. According to officials, this measure does not include any  time contractors may spend correcting an exam report returned to them  by VBA. Returned exam reports are few in number, VBA officials said.  However, once a contractor submitted a corrected or clarified exam  report, VBA officials said the exam management system did not preserve  the date the exam was initially completed. At that point, the system only  tracked the date VBA received the corrected or clarified report. As a  result, the number of days in VBA’s system could include time contractors  took to correct any issues identified by VBA after submitting the initial  report. While VBA’s data does not allow it to reliably assess contractor  performance against the targets in the contracts, VBA’s data can be used  to measure timeliness in other ways. For example, we were able to use  the data to calculate the entire amount of time it took to complete exams,  which includes time contractors took to correct any issues identified by  VBA. As such, the results of our analysis should not be interpreted as  reflecting contractor compliance with timeliness targets under the  contracts. However, to provide timeframes that are similar to VBA’s  targets, we chose 20 days for districts 1-5 and 30 days for districts 6-7 as  timeframes for our analysis. Moreover, we analyzed timeliness across  all contractors rather than for individual contractors. In particular, we  analyzed VBA data on 646,005 contracted exams completed from  February 2017 to January 2018, which included 575,739 exams in  districts 1-5 and 70,266 exams in districts 6-7. Our analysis of VBA data shows that 53 percent of exams were  completed within 20 days for districts 1-5, and 56 percent were completed  within 30 days for districts 6-7. However, some exams took at least twice  as long to complete. For example, 12 percent of exams in districts 1-5  took more than 40 days to complete (see fig. 4). Contractor officials  described a number of reasons why exams might take longer in some  cases. For example, they said that scheduling delays might occur due to  a veteran’s availability or severe weather, and that it can be challenging  to find specialists for certain exam types in rural locations. Our analysis of timeliness focused on exams that were completed, and it  did not include exams that have been requested and not yet completed  by a contractor. For example, a contractor may have accepted an exam  request from VBA, but not yet scheduled an appointment with the  veteran. Alternatively, a contractor may have conducted an exam with the  veteran, but not yet sent the exam report to VBA. As of late June 2018,  VBA-calculated data showed that 87,768 requested exams had not yet  been completed, including 37,077 exams that had already exceeded  VBA’s timeliness targets. Tracking these exams is important because a  large volume of such exams could ultimately increase the amount of time  veterans have to wait for their claims to be processed. VBA officials  stated that the agency closely monitors contractors’ workloads and helps  expedite requested exams that have exceeded VBA’s targets for  completing exams. In addition, VBA included a performance measure in  its May 2018 Request for Proposals to track the percentage of requested  exams that have been with a contractor for more than seven days. Such a  measure could help VBA identify whether contractors have a backlog of  exams and better assess whether veterans are receiving timely exams. Delayed Quality  Reviews and  Performance Reports,  and Data Limitations,  Hinder VBA’s  Monitoring of  Contractors VBA Identified Some  Contractor Performance  Problems but Was  Delayed in Completing  Quality Reviews and  Performance Reports VBA Identification of  Performance Problems VBA’s contract exam program office, primarily through its Contracting  Officer’s Representatives (COR), has identified some contractor  performance problems, such as delays in completing specific exams,  through its oversight of contractor performance. This oversight includes  day-to-day monitoring of contractor workloads and frequent contact with  contractor officials. Through such contact and reviews of contractors’  daily and weekly exam status updates, the CORs work with contractor  officials to identify ways to expedite disability compensation exams for  veterans who have been waiting longer than VBA’s 20-day or 30-day  targets. In addition, VBA contract quality staff who review samples of  contractor exam reports hold teleconferences with the CORs and  contractor officials to provide feedback and discuss issues arising from  their reviews, such as specific types of errors. The VBA contract exam program office also oversees and manages  contractors through supplemental guidance memos, contractor site visits,  and reviews of veteran customer satisfaction surveys. For example, in  November 2017, VBA sent a supplemental guidance memo to all  contractors to clarify guidance on conducting and documenting hearing  loss exams. Further, VBA has conducted site visits to all five contractors’  headquarters or clinic sites since September 2017. Headquarters visits  include reviews of contractors’ procedures, such as those for assigning  exam requests, and contractors’ information systems, such as those for  tracking the status of exams. VBA visits to contractor clinics focus on  facility issues, such as accessibility and safety. According to VBA  officials, the CORs also review reports on satisfaction surveys completed  by veterans after their exam appointments to identify veterans’ concerns  regarding contractors and to follow up with contractors, when needed.  For example, in response to one veteran’s survey comment regarding a  contracted examiner who did not show up to conduct a scheduled exam,  VBA officials told us they followed up with the contractor and learned that  the examiner’s car broke down. According to VBA, it reimbursed the  veteran for round-trip transportation costs to the clinic. Additionally, VBA’s contract quality review staff have conducted special  focused reviews to investigate concerns raised by veterans and by staff in  VBA regional offices and VHA medical facilities. For example, VBA  conducted a review of one contracted examiner who had high rates of  diagnosing severe posttraumatic stress disorder. After reviewing this  examiner’s reports, VBA found their overall quality to be poor. As a result,  VBA requested that the contractor no longer use this examiner. In addition to identifying and addressing problems with individual exams  and examiners, VBA has identified broader challenges faced by  contractors in meeting VBA’s demand for exams and providing timely  reports. For example, VBA identified two contractors who were not  prepared to perform all of their assigned exams because they did not  have enough examiners, particularly in rural locations, which led to delays  and a backlog of exam requests, according to VBA officials. VBA officials  described how they worked with these contractors over several months to  adjust and closely monitor the volume of exams sent to the contractors to  address the backlog. However, according to VBA officials, by December  2017, VBA determined that one of the contractors was not able to meet  the demand for exams, and the agency stopped sending new exam  requests to this contractor. According to VBA, by late June 2018, it had  discontinued all work with this contractor. VA officials said that to obtain  additional exam capacity to make up for the two contractors’ shortages,  they awarded short-term contracts in December 2017 to two other  contractors who were providing exams in other VBA districts. VBA Delays in Assessing  Quality and Completing  Reports VBA has not completed all required quarterly quality reviews and  accompanying quarterly performance reports on contractors, according to  VBA officials. These reviews and reports are key components to  effectively assessing contractor performance in a timely manner.  Specifically, in late June 2018, VBA officials said that they had conducted  almost all their quality reviews for contracted exams completed in districts  1-5 during the second half of 2017, but that they needed to finalize the  quality scores. They also said that they were beginning their quality  reviews for contracted exams completed in 2018. At the time of our  review, VBA had released one quarterly performance report for the fourth  quarter of calendar year 2017, and officials said they were drafting  others. VBA officials attributed delays in completing quality reviews and  quarterly performance reports primarily to a lack of VBA quality review  staff. The quarterly performance reports provide contractors with  information on their performance against VBA quality and timeliness  targets. For example, prior reports included detailed breakouts of quality  errors by type and suggestions for performance improvements. As  officials of one contractor said, delays in receiving quarterly performance  reports limit VBA’s ability to provide contractors with timely and valuable  feedback they can use to improve the quality of their exams. The delay in completing the quarterly reviews and reports also has  implications for VBA’s ability to allocate exam requests across contractors  and administer potential financial incentives across contractors. More  specifically, VBA can use performance data to help determine how to  allocate exams in each district that has two contractors, as outlined in the  contracts. For example, VBA can decide to allocate more exams to the  contractor with higher performance results. Further, the contracts outline  how VBA can use performance data to administer financial incentives  linked to performance targets. For example, VA is to provide a bonus to a  contractor who meets or exceeds the 92 percent quality standard for a  quarter, and meets or exceeds the 20- or 30-day timeliness standard.  However, because of its delays in completing quality reviews and the lack  of reliable data on contractor timeliness, VA has not yet administered  these incentives. VA officials told us that the agency will determine if it will  administer the 2017 incentives after it completes its performance  assessments of contractors. VBA officials said they are currently hiring more staff to address the lag in  quality reviews and subsequent reports to contractors, as well as to  provide more oversight of contractors. At the time of our review, VBA did  not have its authorized level of 15 quality analysts and 2 senior quality  reviewers, but VBA officials said that they expected to complete hiring to  bring the quality reviewer staff up to 17 full-time positions by the end of  fiscal year 2018. In addition, VBA officials acknowledged that they did not  have enough CORs in VBA’s exam program office to oversee the 14  exam contracts (including the two short-term contracts). As of April  2018, VBA officials said the office had 3 CORs, but hiring was expected  to bring the number up to 14 by the end of fiscal year 2018. VBA officials  said that they determined staffing levels for VBA’s contract exam program  office—including CORs and exam quality reviewers—based on an  assessment of the resources needed to expand the program, among  other factors. Although VBA did not provide documentation outlining how it determined its workforce needs, the agency provided us with updated  organizational charts in June 2018 demonstrating increased staff levels  for the exam program office. VBA’s Data Limitations  Hinder Its Ability to  Oversee Certain Contract  Provisions, and VBA Has  Not Conducted  Comprehensive  Performance Analysis VBA’s lack of reliable data on the status of exams, including insufficient  exams—exam reports that VBA returns to contractors to be corrected or  clarified—limits its ability to effectively oversee certain contract  provisions. VBA officials acknowledged that they could not calculate the  number of completed exams that were once marked as insufficient or how  long they had remained in that status due to the data limitations of the  exam management system the agency used until spring 2018. The  contracts require that contractors correct insufficient exams within a  certain number of days and bill VBA for these exams at half price.  However, VBA’s lack of complete and reliable information on insufficient  exams hinders its ability to ensure that either of these requirements is  met. VBA officials also indicated that they were unable to fully assess  individual contractor timeliness against VBA’s performance targets  because the exam management system did not include the date the initial  exam report was submitted to VBA, which is needed to calculate  timeliness as outlined in the contracts. In March 2018, VBA began implementing a new exam management  system designed to collect more comprehensive and accurate information  on the status of exams. VA documentation on the new system shows that  it will include detailed data on insufficient exams, which, according to VBA  officials, should allow VBA to track whether contractors are properly  discounting their invoices for those exams. However, in June 2018, VBA  stated that three of its five contractors did not have complete functionality with VBA’s new exam management system. As a result, VBA officials  said the agency still did not have complete data in the new system that  would allow it to track insufficient exams. Officials said they were working  to address these issues. More broadly, as described in VA system documents, the new system is  designed to allow VBA to track more detailed data on exam completion  dates and on other points throughout the exam process, such as dates for  initial requests for clarification from contractors, and dates when  appointments are scheduled. However, VBA is in the early stage of this  transition, and agency officials stated that unexpected technical issues  have affected communication between the new exam management  system and other VBA systems. While they work to resolve the issues,  VBA officials said that they have been manually moving some exam  requests through the system each day. Further, VBA has not documented  how it plans to ensure the additional data is accurate and use it to  oversee contractor performance as outlined in the contracts, particularly  for insufficient exams. Federal internal control standards state that  management should use quality information to achieve key objectives.  In addition, management should formulate plans to achieve those  objectives. For example, agencies should assess collected data and  ensure it is accurate so that it can be used to provide quality information  to evaluate performance. In the absence of a plan for how it will capture  and use data in its new exam management system to assess  performance, VBA risks overpaying contractors for insufficient exams and  continuing to inaccurately measure contractor timeliness. Further, according to agency officials, VBA has not conducted  comprehensive analyses of performance data that would allow it to  identify and address higher-level trends and program-wide challenges  across contractors, geographic districts, exam types, or other relevant  factors. Agency officials told us they have no plans to conduct such  analyses. Federal internal control standards state that management  should establish and operate monitoring activities and evaluate the  results of those activities. In addition, management should evaluate  deficiencies both at the individual and aggregate level. While VBA officials  acknowledged that higher-level analyses could improve program  oversight, they explained that analyzing performance data has been  challenging due to the limitations of the exam management system. Thus,  VBA has prioritized addressing contractor-specific problems and resolving  long-standing pending exams over in-depth analysis of the performance  data. However, with the expected improvements provided by VBA’s new exam  management system and increased staff to manage the program and  conduct quality reviews, VBA should be better positioned to conduct  analyses of performance data in the future. By conducting higher-level  analyses across contractors, geographic districts, exam types, or other  relevant factors, VBA could make a more informed assessment of the  challenges contractors and examiners face and where additional  workload capacity and training may be needed. In addition, better  analyses would allow VBA to determine if the contract exam program is  achieving its quality and timeliness goals in a cost effective manner. Auditor Verifies  Contracted Examiner  Licenses, but VBA  Does Not Verify  Training Completion  or Collect Information  on Training  Effectiveness VBA Uses a Third-Party  Auditor to Verify  Contracted Examiner  Licenses VBA has a third-party auditor who verifies that all active contracted  examiners have a current, valid, and unrestricted medical license in the  state where they examined a veteran. The auditor provides regular  reports of its audits to VBA. Specifically, the auditor verifies the license  numbers of all active contracted examiners in the states where they  perform VA disability compensation exams; National Provider Identifiers;  and any prior or current sanctions or restrictions resulting in a revoked or  suspended license at the time of a VA exam. In addition, contractors  send VBA monthly reports of examiners’ medical license, specialty, and  accreditation based on the contractors’ verification of this information.  Every 2 months, VBA sends the auditor a consolidated report of this  information covering all five contractors. The auditor verifies examiners’  information in that report before sending a final audit report to VBA, noting  if the auditor was or was not able to verify examiners’ licenses. After  reviewing the report, VBA contacts the contractors to gather additional  information to resolve any issues, and in cases in which licensing  requirements are not met, VBA stops using the examiner and offers new  exams to veterans who have been seen by the examiner. VBA and auditing firm officials noted that audit results show that almost  all examiners have current and valid licenses, and contractors are  required to stop using those who do not meet licensing requirements.  VBA and auditing firm officials said that issues identified in the audits are  usually due to typos or differences in how information is captured across  different licensing databases. However, based on an audit, VBA provided  an example of an examiner with a restricted medical license who had  completed exams for one contractor. In this case, VBA notified the  contractor, who then stopped using the examiner and said it was taking  action to prevent errors in its license verification process from occurring  again. In addition, the contractor reimbursed VBA for the cost of exams  conducted by the examiner and also offered new exams to veterans who  had been seen by the examiner. VBA Relies on Exam  Contractors to Verify  Training is Completed and  Does Not Review Training  Records for Accuracy VBA relies on contractors to verify that their examiners complete required  training, and agency and contractor officials told us that VBA does not  review contractors’ self-reported training reports for accuracy or request  supporting documentation, such as training certificates, from  contractors. As required by the contracts, contractors must track and  maintain records demonstrating each examiner has completed required  training. Each of VBA’s five contractors has its own process for ensuring  that required training is provided to and completed by their examiners, but  generally, contractors export the courses from VA’s online training system  into their own online training systems for their examiners to access. The  contractors, rather than VBA, access the contractor training systems to  verify that examiners have completed the required training before they  are approved to conduct exams. When requested by VBA, contractors  are required to send VBA reports demonstrating that their examiners  have met training requirements. As stated in the latest version of the  contracts, contractors must immediately stop using any examiner found to  have not completed required training, notify VBA, and re-examine the  involved veterans at no cost to VBA, if requested by the agency. Although VBA currently does not verify the accuracy of training self- reported by contractors to the agency, VBA officials said that they plan to  enhance monitoring through spot checks of training records and a new  training system. Specifically, in fiscal year 2019, VBA officials said they  plan to start conducting spot checks of some examiners’ training records  for accuracy and compliance during site visits to contractor headquarters  and clinics. However, VBA has not provided details or documentation on  these planned checks, such as how it will determine which records to  review or the steps it will take to verify the accuracy of training records.  VBA officials also said they are planning to develop an online system that  would allow VBA to certify that examiners have completed required  training, rather than relying on contractors for this information. However,  as of July 2018, VBA had yet to determine when this system would be  developed and had not documented plans to do so in order to use such  information for monitoring training. VBA also said it would hire staff to  manage contractor training, but has yet to do so. GAO’s prior work has emphasized tracking and other control mechanisms  to ensure that all employees receive appropriate training. While VBA  said it would enhance its monitoring of training records, documenting and  implementing a plan and processes to verify training could help ensure  examiners have met training requirements. Without such a plan, VBA  risks using contracted examiners who are unaware of the agency’s  process for conducting exams and reporting the results, which could lead  to delays for veterans as a result of poor-quality exams that need to be  redone and insufficient exam reports that need to be corrected. VBA Does Not Collect  Information to Determine if  Training Effectively  Prepares Examiners VBA does not collect information from contractors or examiners to help  determine if required training effectively prepares examiners to conduct  high quality exams and complete exam reports. VBA has provided  additional guidance to contractors for some specialty exams. However,  VBA identified these issues after some contractors requested guidance in  monthly meetings, rather than through VBA efforts to proactively or  regularly collect information from contractors or examiners to inform  potential changes to training. VBA is considering including a component  in the online training system that would collect information on the  effectiveness of required training. However, VBA has not outlined  additional details on collecting such information. VBA officials said that  VBA did not collect such information in the past, in part, because staff  were focused on program oversight. To assess progress toward achieving results and to make changes to  training if needed, GAO has found that evaluation is a key component of  any training program. Given that VBA officials told us that the agency  plans to issue new contracts in fall 2018, the number of contracted  examiners who are new to VA processes may increase. Thus, collecting  and assessing regular feedback on training from contractors and  examiners, such as through surveys, discussion groups, or interviews,  could help VBA determine if training effectively prepares examiners to  conduct exams and complete exam reports. Further, information on the  effectiveness of training could supplement data on contractor  performance and results from VBA’s quality reviews to help assess if  additional training courses are needed across contractors or for specific  exam types. Conclusions As VBA increasingly relies on contractors to perform veterans’ disability  compensation exams, it is important that the agency ensures proper  oversight of these contractors. VBA’s lack of accurate and up-to-date data  and reports on contractor performance hampers its ability to oversee the  quality and timeliness of exams provided through contractors. VBA’s new  exam management system provides opportunities to improve oversight  through more comprehensive and accurate data. These improvements  might be limited, however, without a plan to use the data to produce the  quality information needed by VBA to monitor insufficient exams, ensure it  pays contractors the correct amount for those exams, and help it  accurately calculate contractor timeliness. Further, the new system provides an opportunity for VBA to conduct  analyses that could identify high-level trends and challenges facing the  program across contractors and districts, such as delays in completing  exams in specific parts of the country or contractor performance issues  related to specific exam types. Despite these capabilities, VBA has not  outlined plans for using improved information in this manner. Without  doing so, the agency may miss opportunities to improve the program and,  ultimately, its service to veterans. VBA could better prepare contracted examiners for their role by taking  actions to ensure required training has been completed and by collecting  information to assess and improve training. Such actions could help  improve the quality of exams and exam reports, which could mitigate the  need for exam rework and, ultimately, delays in determining veterans’  benefits. With VBA planning to award new contracts and potentially more  new contracted examiners coming on board, verifying that required  training is completed and collecting information on the effectiveness of  training are critical. As VA continues to rely on contracted examiners, it is  important that the agency is well positioned to carry out effective  oversight of contractors to help ensure that veterans receive high-quality  and timely exams. Recommendations for  Executive Action We are making the following four recommendations regarding contracted  disability compensation exams to VA. The Under Secretary for Benefits should develop and implement a plan  for how VBA will use data from the new exam management system to  oversee contractors, including how it will capture accurate data on the  status of exams and use it to (1) assess contractor timeliness, (2) monitor  time spent correcting inadequate and insufficient exams, and (3) verify  proper exam invoicing. (Recommendation 1) The Under Secretary for Benefits should regularly monitor and assess  aggregate performance data and trends over time to identify higher-level  trends and program-wide challenges. (Recommendation 2) The Under Secretary for Benefits should document and implement a plan  and processes to verify that contracted examiners have completed  required training. (Recommendation 3) The Under Secretary for Benefits should collect information from  contractors or examiners on training and use this information to assess  training and make improvements as needed. (Recommendation 4) Agency Comments  and Our Evaluation We provided a draft of our report to the Department of Veterans Affairs  (VA) for its review and comment. VA provided written comments, which  are reproduced in appendix II. VA concurred with all our  recommendations and described the Veterans Benefits Administration’s  (VBA) plans for taking action to address them. Regarding our first  recommendation, VA outlined improvements in the information collected  through VBA’s new exam management system, and said that VBA is  currently testing a mechanism to validate exam invoices submitted by  contractors.  We noted these improvements to the system in our draft  report sent to the agency for comment. We maintain that it will be  important for VBA to take the next step of developing and implementing a  plan for how it will use information from the new system to ensure both  accurate timeliness data and proper exam invoicing. Regarding our  second recommendation, VA stated that VBA will use improved data in  the new exam management system to regularly monitor and assess  aggregate performance data, identify error trends, and monitor contractor  performance and program-wide challenges. Regarding our third and fourth recommendations, VA stated that VBA  plans to develop and implement a training plan for contractors that will  include a mechanism to validate that required training has been  completed and to assess the effectiveness of this training through  feedback from trainees, contractors, and quality review staff in VBA’s  contract exam program office. VA stated that VBA will use this data to  improve the implementation and content of training. VA requested that  GAO combine these two recommendations into one. However, we believe  they are two distinct recommendations and have kept them as such. VBA  could meet the intent of each recommendation with the development and  implementation of one plan that covers both training verification and  assessment, as outlined in its comments. As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Secretary of Veterans Affairs, and other  interested parties. In addition, the report will be available at no charge on  the GAO website at http://www.gao.gov. If you or your staff have any questions, please contact me at (202) 512- 7215 or curdae@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made significant contributions to this report are listed in  Appendix III. Appendix I: Additional Information on  Selected Methodologies Review of VBA Contracts To evaluate VBA monitoring of contractor performance and VBA  oversight of contracted examiners’ qualifications and training, we  reviewed relevant federal laws, regulations, and VA guidance on the use  of contracted examiners for disability compensation exams. To identify  relevant contract provisions and requirements related to contractor  performance, monitoring of such performance, licensing, and training,  among other areas, we reviewed selected provisions of selected versions  of the 12 current VA Medical Disability Examination contracts originally  awarded in 2016, of 5 short-term contracts VA awarded in early 2017,  and of 2 short-term contracts VA awarded in December 2017. With  regard to the 12 current contracts, we reviewed the selected provisions in  the originally awarded contract from 2016 and in the most recently  amended version of the contract (as provided to us by VBA officials).  Based on our review of these two versions of the contract, the selected  provisions appeared to remain in place, unless noted otherwise in this  report. However, we did not review the various contract modifications  that, according to VBA, occurred in the interim period to confirm whether  the selected provisions we focused on in our review actually remained in  place during the period between the original contract and the most recent  amendment. With regard to the 2 short-term contracts awarded in  December 2017, we reviewed the selected provisions in the original  December contract. According to VBA officials, there have been no  subsequent modifications to these short-term contracts. With regard to the 5 short-term contracts awarded in early 2017, we only  reviewed selected provisions relating to contractor quality and timeliness  performance. Thus, any statements in this report relating to other aspects  of the contracts are not based on these short-term contracts. Further, we  only reviewed such provisions in the originally awarded short-term  contract, and we did not review the various contract modifications that,  according to VBA, occurred subsequently, to confirm that those  provisions remained in place over time. However, we found that those  selected provisions were generally in place in all of the various contracts  we reviewed. Analysis of VBA Data on  the Timeliness of  Contracted Exams To answer what is known about the timeliness of VBA contracted exams,  we analyzed VBA data on disability compensation exams completed by  five contractors between February 2017 and January 2018. VBA’s Office  of Performance Analysis and Integrity provided exam-level data that it  maintains in the agency’s Enterprise Data Warehouse, including data on  the exam request date, the date the contractor accepted the request, the  date the contractor completed the exam, and the VBA district where the  exam was conducted, among other information. These data were  created from data originally collected in VBA’s Centralized Administrative  Accounting Transaction System (CAATS), which is the system that VBA  used to request exams from contractors until spring 2018. According to VBA officials, the status of exam requests (e.g., pending,  completed, cancelled) was not always accurate in CAATS. To create  more reliable data and identify the most current information on the status  of exams, the Office of Performance Analysis and Integrity identified and  replaced missing or incorrect data in CAATS by running checks against  other VBA systems, including the Veterans Benefits Management  System, which maintains veterans’ benefits claims records. We assessed  the reliability of the data we received from VBA by conducting electronic  testing for missing data and errors, and by interviewing VBA officials  about their data collection and quality control procedures. We determined  that the data were sufficiently reliable for our purposes of reporting the  time it took to complete exams within districts. Our analysis included 646,005 contracted exams completed between  February 2017 and January 2018. We selected February 2017 as our  starting point because it was the first full month of data available that  covered most of VBA’s current contractors. To allow for 12 full months of  data, we selected January 2018 as our ending point. In addition, we  limited our population to include exams that were requested on or after  January 13, 2017 in districts 1-5 or on or after April 1, 2016 in districts 6- 7, based on the periods of performance in the contracts for those  districts. We calculated timeliness at the level of the exam request. We calculated  the number of days between the date an exam request was accepted by  the contractor and the date the exam report was completed by the  contractor. The timeliness values we calculated may include additional  time needed to request and receive contractors’ corrections or  clarifications on previously submitted exam reports. In our report, we refer  to these exams as “insufficient exams.” VBA officials acknowledged that  due to data limitations the new exam management system is intended to  resolve, VBA’s CAATS system did not retain data on the number of  exams that were once marked as insufficient or how long they remained  in that status. While VBA officials acknowledged that this data limitation  affects the agency’s ability to assess individual contractor timeliness on  VBA’s performance targets outlined in the contracts, the limitation did not  prevent us from analyzing the timeliness of contracted exams overall. The  overall timeliness values we calculated represent the total time taken to  complete exams regardless of whether additional time was needed for  corrections. To put the timeliness values we calculated in context, we calculated the  percentage of exams that were completed within VBA’s timeliness targets  of 20 days for districts 1-5 and 30 days for districts 6-7 for the entire 12- month period of our analysis. We also calculated the percentage of  exams that were completed within other timeframes (e.g., 21-40 days,  more than 40 days). According to the contracts, contractors are not  expected to complete all exams within the timeliness target, but rather  should meet the timeliness target on average in a given quarter, so our  analysis was different from one that VBA might conduct in order to  determine contract compliance. Because VBA does not retain detailed  data on exam completion dates necessary to assess contractor  performance against VBA’s timeliness targets, and because we  calculated timeliness across contractors, the percentages we calculated  do not represent an assessment of whether contractors met VBA’s  timeliness targets. GAO did not conduct a legal analysis of the various  contractors’ compliance with the contract requirements. Alternate Timeliness Values Given that the start of VBA’s timeliness measure is the date the  contractor accepts the exam request (rather than the date VBA requests  the exam), we calculated alternate timeliness values to account for  potential delays in accepting exam requests. VBA officials stated that  VBA requests contractors accept or reject exam requests within 3 days.  For all exam requests that contractors took more than 3 days to accept,  we calculated alternate totals that included the additional days. For  example, if a contractor took 5 days to accept the exam request and  completed the exam 20 days later, we calculated an alternate total of 22 days to complete the exam. We used these alternate values to calculate  adjusted percentages for each category presented in Figure 4 of our  report. For example, using the alternate timeliness values, about 50  percent of exams in districts 1-5 would have been completed in 20 days  and 53 percent in districts 6-7 would have been completed within 30  days, rather than the respective 53 percent and 56 percent shown in  Figure 4. Moreover, we found that about 82 percent of exam requests  during our period of analysis were accepted within 3 days. Pending Exams To report more recent data on exams that were accepted but not yet  completed by contractors—pending contracted exams—VBA provided  aggregate data on the number of pending exams as of June 25, 2018.  For example, for districts 1-5, it provided data on the number of exams  that had been pending for 20 days or fewer, 21-40 days, 41-60 days, 61- 100 days, and more than 100 days. We calculated percentages based on  the VBA-provided totals. Appendix II: Comments from the Department  of Veterans Affairs Appendix III: GAO Contact and Staff  Acknowledgments GAO Contact Elizabeth Curda, (202) 512-7215 or curdae@gao.gov. Staff  Acknowledgments In addition to the contact named above, Nyree Ryder Tee (Assistant  Director); Teresa Heger (Analyst-in-Charge); Alex Galuten; Justin  Gordinas; and Greg Whitney made key contributions to this report. Also  contributing to this report were James Bennett, Matthew T. Crosby,  Teague Lyons, Sheila R. McCoy, Jessica Orr, Claudine Pauselli, Samuel  Portnow, Monica Savoy, Almeta Spencer, and April Van Cleef.
In 2016, VBA awarded 12 contracts to five private firms for up to $6.8 billion lasting up to 5 years to conduct veterans' disability medical exams. Both VBA contracted medical examiners and medical providers from the Veterans Health Administration perform these exams, with a growing number of exams being completed by contractors. Starting in 2017, VBA contracted examiners conducted about half of all exams. GAO was asked to review the performance and oversight of VBA's disability medical exam contractors. This report examines (1) what is known about the quality and timeliness of VBA contracted exams; (2) the extent to which VBA monitors contractors' performance; and (3) how VBA ensures that its contractors provide qualified and well-trained examiners. GAO analyzed the most recent reliable data available on the quality and timeliness of exams (January 2017 to February 2018), reviewed VBA and selected contract documents and relevant federal laws and regulations, and interviewed agency officials, exam contractors, an audit firm that checks examiners' licenses, and selected veterans service organizations. The Veterans Benefits Administration (VBA) has limited information on whether contractors who conduct disability compensation medical exams are meeting the agency's quality and timeliness targets. VBA contracted examiners have completed a growing number of exams in recent years (see figure). VBA uses completed exam reports to help determine if a veteran should receive disability benefits. VBA reported that the vast majority of contractors' quality scores fell well below VBA's target—92 percent of exam reports with no errors—for the first half of 2017. Since then, VBA has not completed all its quality reviews, but has hired more staff to do them. VBA officials acknowledged that VBA also does not have accurate information on contractor timeliness. VBA officials said the exam management system used until spring 2018 did not always retain the initial exam report completion date, which is used to calculate timeliness. In spring 2018, VBA implemented a new system designed to capture this information. VBA monitoring has addressed some problems with contractors, such as reassigning exams from contractors that did not have enough examiners to those that did. However, the issues GAO identified with VBA's quality and timeliness information limit VBA's ability to effectively oversee contractors. For example, VBA officials said they were unable to track the timeliness of exam reports sent back to contractors for corrections, which is needed to determine if VBA should reduce payment to a contractor. The new system implemented in spring 2018 tracks more detailed data on exam timeliness. However, VBA has not documented how it will ensure the data are accurate or how it will use the data to track the timeliness and billing of corrected exam reports. VBA also has no plans to use the new system to analyze performance data to identify trends or other program-wide issues. Without such plans, VBA may miss opportunities to improve contractor oversight and the program overall. A third-party auditor verifies that contracted examiners have valid medical licenses, but VBA does not verify if examiners have completed training nor does it collect information to assess training effectiveness in preparing examiners. While VBA plans to improve monitoring of training, it has not documented plans for tracking or collecting information to assess training. These actions could help ensure that VBA contractors provide veterans with high-quality exams and help VBA determine if additional training is needed.