Background The Defense Laboratories The National Defense Authorization Act (NDAA) for Fiscal Year 1995  authorized the Secretary of Defense to conduct personnel demonstration  projects at the department’s laboratories designated as Science and  Technology Reinvention Laboratories. The demonstration projects were  established to give laboratory managers more authority and flexibility in  managing their civilian personnel. These projects function as the vehicles  through which the department can determine whether changes in  personnel management concepts, policies, or procedures, such as  flexible pay or hiring authorities, would result in improved performance  and would contribute to improved DOD or federal personnel  management. Table 1 presents a list of the 15 defense laboratories  included in the scope of our review. The Defense Laboratories Office—within the Office of the Undersecretary  of Defense for Research and Engineering (Research and Engineering)— carries out a range of core functions related to the defense labs, including  the aggregation of data, analysis of capabilities, and alignment of  activities, as well as advocacy for the defense labs. The National Defense  Authorization Act for Fiscal Year 2017 gave authority to conduct and  evaluate defense laboratory personnel demonstration projects to the  Under Secretary of Defense for Research and Engineering and,  accordingly, the Defense Laboratories Office. The Defense Laboratories  Office supports the Research and Engineering mission by helping to  ensure comprehensive department-level insight into the activities and  capabilities of the defense laboratories. The LQEP was chartered on April 15, 1994 to improve productivity and  effectiveness of the defense laboratories through changes in, among  other things, personnel management and contracting processes. The  NDAA for Fiscal Year 2017 established a new organizational structure for  the program, adding two new panels while also specifying that two  previously existing subpanels on personnel and infrastructure would  continue to meet. The NDAA for Fiscal Year 2017 requires the  department to maintain a LQEP Panel on Personnel, Workforce  Development, and Talent Management—one of the four panels  established by a February 14, 2018 charter signed by the Under  Secretary of Defense for Research and Engineering. The purpose of the  panel is to help the LQEP achieve the following goals: (1) review and  make recommendations to the Secretary of Defense on current policies  and new initiatives affecting the defense laboratories; (2) support  implementation of quality enhancement initiatives; and (3) conduct  assessments and data analysis. The LQEP Panel on Personnel,  Workforce, Development, and Talent Management includes  representatives from each of the defense laboratories, as well as from the  Army, Navy, Air Force, appropriate defense agencies, and Office of the  Under Secretary of Defense for Research and Engineering. Hiring Authorities A hiring authority is the law, executive order, or regulation that allows an  agency to hire a person into the federal civil service. Among other roles,  hiring authorities determine the rules (or a subset of rules within a broader  set) that agencies must follow throughout the hiring process. These rules  may include whether a vacancy must be announced, who is eligible to  apply, how the applicant will be assessed, whether veterans preference  applies, and how long the employee may stay in federal service. Hiring  authorities may be government-wide or granted to specific agencies. Government-wide  (Title 5)  Hiring Authorities Competitive (Delegated) Examining. This is the traditional method  for making appointments to competitive service positions, and it  requires adherence to Title 5 competitive examining requirements.  The competitive examining process requires agencies to notify the  public that the government will accept applications for a job, screen  applications against minimum qualification standards, apply selection  priorities such as veterans preference, and assess applicants’ relative  competencies or knowledge, skills, and abilities against job-related  criteria to identify the most qualified applicants. Federal agencies  typically assess applicants by rating and ranking them based on their  experience, training, and education. Figure 1 depicts the Office of  Personnel Management’s (OPM) 80-day standard roadmap for hiring  under the competitive process. Governmentwide (Title 5) Direct Hire Authority. This authority  allows agencies to appoint candidates to positions without regard to  certain requirements in Title 5 of the United States Code, with OPM  approval. A direct hire authority expedites hiring by eliminating  specific hiring rules. In order for an agency to use direct hire, OPM  must determine that there is either a severe shortage of candidates or  a critical hiring need for a position or group of positions. When using  the direct hire authority, agencies must adhere to certain public notice  requirements. The Pathways Programs. These programs were created to ensure  that the federal government continues to compete effectively for  students and recent graduates. The current Pathways Programs  consist of the Internship Program, the Recent Graduates Program,  and the Presidential Management Fellows Program. Initial  hiring is  made in the excepted service, but it may lead to conversion to  permanent positions in the competitive service. Veterans-Related Hiring Authorities. These include both the  Veterans Recruitment Appointment Authority and the Veterans  Employment Opportunities Act authority. The Veterans Recruitment  Appointment authority allows for certain exceptions from the  competitive examining process. Specifically, agencies may appoint  eligible veterans without competition under limited circumstances or  otherwise through excepted service hiring procedures. The Veterans  Employment Opportunities Act authority is a competitive service  appointment authority that allows eligible veterans to apply for  positions announced under merit promotion procedures when an  agency accepts applications from outside of its own workforce. DOD-specific Hiring Authorities The Defense Laboratory Direct Hire Authorities. These include the  following four types of direct hire authorities granted to the defense  laboratories by Congress for hiring STEM personnel: (1) direct hire  authority for candidates with advanced degrees; (2) direct hire  authority for candidates with bachelor’s degrees; (3) direct hire  authority for veterans; and (4) direct hire authority for students  currently enrolled in a graduate or undergraduate STEM program.  The purpose of these direct hire authorities is to provide a streamlined  and accelerated hiring process to allow the labs to successfully  compete with private industry and academia for high-quality scientific,  engineering, and technician talent. The Expedited Hiring Authority for Acquisition Personnel. This  authority permits the Secretary of Defense to designate any category  of positions in the acquisition workforce as positions for which there  exists a shortage of candidates or there is a critical hiring need; and to utilize specific authorities to recruit and appoint qualified persons  directly to positions so designated. The Science, Mathematics, and Research for Transformation  (SMART) Scholarship-for-Service Program. This program was  established pursuant to 10 USC §2192a, as amended, and is funded  through the National Defense Education Program. The SMART  scholarship for civilian service program provides academic funding in  exchange for completing a period of full-time employment with DOD  upon graduation. The Defense  Laboratories Have  Used Direct Hire  Authorities and Other  Incentives to Help  Hiring Efforts, but  Officials Reported  Challenges in Hiring  Highly Qualified  Candidates The labs have used the defense laboratory-specific direct hire authorities  more than any other category of agency-specific or government-wide  hiring authority. Defense laboratory officials we surveyed reported that  these direct hire authorities had been the most helpful to the labs’ efforts  to hire highly qualified candidates for STEM positions, and also reported  that the use of certain incentives had been helpful in this effort. However,  even with access to the authorities, these defense laboratory officials  identified challenges associated with the hiring process that affected their  ability to hire highly qualified candidates. Defense Laboratories  Used the Direct Hire  Authorities Most  Frequently for Hiring  STEM Candidates, and  the Use of These  Authorities Has Increased  since 2015 For fiscal years 2015 through 2017, the defense laboratories used  laboratory-specific direct hire authorities more often than any other  category of hiring authorities when hiring STEM personnel. Moreover, the  defense laboratories’ use of these direct hire authorities increased each  year from fiscal year 2015 through fiscal year 2017. Of the 11,562 STEM  hiring actions in fiscal years 2015 through 2017, approximately 46 percent  were completed using one of the defense laboratory direct hire  authorities. The second and third most used hiring authorities were  internal hiring actions and the expedited hiring authority for acquisition  personnel, each of which comprised approximately 12 percent of the  hiring actions during the time period. Table 2 provides information on the  overall number of hiring actions by hiring authority for fiscal years 2015  through 2017. The laboratory-specific direct hire authorities include the direct hire  authorities for candidates with advanced degrees, candidates with  bachelor’s degrees, and candidates who are veterans—authorities were  granted by Congress in prior legislation. Among the defense laboratory  direct hire authorities, the direct hire authority for candidates with  bachelor’s degrees was used for 55 percent of all direct hires, for a total  of 2,920 hiring actions for fiscal years 2015 through 2017. During the  same time frame, the labs used the direct hire authority for candidates  with advanced degrees for approximately 36 percent (1,919 hiring  actions) of all direct hires, and the direct hire authority for veteran  candidates for approximately 9 percent (455 hiring actions). In addition,  for less than one percent of the direct hires, either the labs used another  category of laboratory-specific direct hire authority or we were unable to  determine which type of direct hire authority was used during those same  three fiscal years. See table 3 for information on the defense labs’ use  of the defense laboratory-specific direct hire authorities for fiscal years  2015 through 2017. In fiscal year 2017 the defense labs used the defense laboratory direct  hire authorities for 54 percent of STEM hiring actions completed,  representing an increase of approximately 16 percentage points relative  to fiscal year 2015, when 38 percent were hired under defense lab direct  hire authorities. For additional information on the labs’ use of hiring  authorities in fiscal years 2015 through 2017, as well as hiring authority  data by laboratory, see appendix IV. One laboratory official explained that  the increased use of the direct hire authorities could be a result of the  NDAA for Fiscal Year 2016, which increased the laboratories’ allowable  use of the direct hire authority for candidates with bachelor’s degrees from 3 percent to 6 percent, and use of the direct hire authority for  veterans from 1 percent to 3 percent, of the total number of scientific and  engineering positions at each laboratory at the end of the preceding fiscal  year. The direct hire authority for candidates with bachelor’s degrees was  used most often—for 1,151 out of 1,835 hiring actions—as compared with  the other direct hire authorities in fiscal year 2017. See table 4 for more  information on the laboratories’ use of all hiring authorities in fiscal year  2017. In addition, table 5 provides more information on the labs’ use of  the direct hire authorities in fiscal year 2017. Laboratory Officials  Reported That Certain  Hiring Authorities and  Incentives Have Helped  Defense Laboratories Hire  Highly Qualified  Candidates Defense laboratory officials we surveyed most frequently identified the  three defense laboratory-specific direct hire authorities as having helped  to hire highly qualified candidates (see figure 2) and to hire quickly (see  figure 3). Specifically, 15 of 16 respondents to our survey stated that each  of the three direct hire authorities had been helpful in hiring highly  qualified candidates, and that the direct hire authorities for veterans and  for candidates with an advanced degree had helped them to hire quickly.  Moreover, all 16 survey respondents stated that the direct hire authorities  for candidates with a bachelor’s degree had helped them to hire quickly.  Among the three direct hire authorities, the one for candidates with  bachelor’s degrees was reported to be the most helpful to the  laboratories’ hiring efforts, according to our survey results. A majority of the laboratory officials we surveyed also stated that the  Expedited Hiring Authority and the Science, Mathematics, and Research  for Transformation (SMART) Program had both helped facilitate their  efforts to hire highly qualified candidates and to hire them quickly.  According to our survey, the least helpful hiring authority that lab officials  reported using was the delegated examining unit authority. Six of 16  survey respondents stated that the delegated examining unit authority  had helped them to hire highly qualified candidates, while 9 of 16 stated  that the authority had hindered this effort. Three of 16 survey respondents  stated that the delegated examining unit authority had helped them to hire  quickly, while 12 of 16 stated that the use of this authority had hindered  their ability to hire quickly. During our interviews with laboratory officials, hiring officials and  supervisors described the defense laboratory direct hire authorities as  being helpful in their hiring efforts. For example, hiring officials from one  lab stated that the direct hire authorities were the easiest authorities to  use, and that since their lab had started using them, job offer acceptance  rates had increased and their workload related to hiring had decreased. A  hiring official from another laboratory stated that the use of direct hire  authorities had allowed their lab to be more competitive with the private  sector in hiring, which is useful due to the high demand for employees in  research fields. A supervisor from one lab stated that the use of direct  hire authorities was not only faster than the competitive hiring process,  but it also allowed supervisors a greater ability to get to know candidates  early in the process to determine whether they met the needs of a  position. In comparison, hiring managers we interviewed at one laboratory  stated that the Pathways Program is not an effective means of hiring  students because the program requires a competitive announcement.  Supervisors also stated that the application process for Pathways can be  cumbersome and confusing for applicants and may cause quality  applicants to be screened out early. Defense laboratory officials who  responded to our survey also stated that the process takes too long and  that quality applicants may drop out of the process due to the length of  the process. Defense laboratory hiring data also indicated that use of the defense  laboratory direct hire authorities resulted in faster than median hiring  times. As shown in table 6, the median time to hire for STEM positions at  the defense laboratories in fiscal year 2017 was 88 days. The median  time to hire when using the defense laboratories’ direct hire authorities,  Pathways, or the SMART program authority was faster than that of the  median for all categories combined. The median time to hire when using  the competitive hiring process was approximately twice as long as when  using the labs’ direct hire authorities. Our full analysis of defense  laboratory hiring data, including the time to hire by hiring authority  category, for fiscal years 2015 through 2017 can be found in appendix V. Defense laboratory officials also cited the use of incentives as helpful in  hiring highly qualified candidates, as shown in figure 4. According to our  survey results, the defense laboratories’ flexibility  in pay setting under  their demonstration project authority was generally considered to be the  most helpful incentive, with 13 of 16 survey respondents stating that this  incentive had very much helped them to hire highly qualified candidates. During interviews, laboratory officials described the use of these  incentives as being particularly helpful if a candidate is considering  multiple job offers because the incentives can help make the lab’s offer  more competitive with offers from other employers. Multiple hiring officials  stated that they would generally not include such incentives in an initial  offer, but that if the candidate did not accept that offer, they would  consider increasing the salary or offering a bonus. A hiring official from  one lab stated that his lab has not offered many recruitment bonuses in  recent years, because their acceptance rate has been sufficiently high  without the use of that incentive. Many of the recently hired lab employees whom we interviewed also cited  incentives, including bonuses and student loan repayment, as factoring  into their decisions to accept the employment offers for their current  positions. For example, one recently hired employee stated that the lab’s  student loan repayment program was a significant factor in his decision to  accept employment at the lab rather than with private industry. Recently  hired employees also cited less tangible benefits of working at the labs,  including the  work environment, job stability, and type of work  performed, as key factors in their decisions to accept their current  positions. One newly hired employee stated that, while she could earn  more money in a private-sector job, the defense laboratory position would  afford her the freedom to pursue the type of work she is currently doing,  and that this was a major consideration in her decision to accept it.  Another newly hired employee similarly stated that he was interested in  the type of research conducted at the lab where he now works, and that  he was attracted to the opportunity to contribute to the national defense,  while also taking advantage of benefits that support the pursuit of higher  education. Defense Laboratory  Officials We Surveyed  Identified Challenges That  Affect Their Ability to Hire  Highly Qualified  Candidates Defense laboratory officials we surveyed reported that, although the  available hiring authorities and incentives are helpful, they experience a  range of challenges to their ability to hire highly qualified candidates, as  shown in figure 5, ranging in order from the most to the least frequently  cited. In addition, figure 6 shows the extent to which officials reported  selected top challenges that hindered their respective labs’ abilities to hire  highly qualified candidates. Defense laboratory officials described how hiring challenges identified in  our survey affect their ability to hire high quality candidates. Specifically,  these challenges are as follows: Losing quality candidates to the private sector: Fifteen of 16  survey respondents stated that this was a challenge, and 12 of the 15  stated that this challenge had somewhat or very much hindered their  lab’s ability to hire highly qualified candidates for STEM positions  since October 2015. Hiring officials and supervisors we interviewed  stated that private-sector employers can make on-the-spot job offers  to candidates at college career fairs or other recruiting events,  whereas the labs are unable to make a firm job offer until later in the  hiring process. Government-wide hiring freeze: Fifteen of 16 survey respondents  identified this as a challenge, with 13 of those reporting that it had  either somewhat or very much hindered their lab’s ability to hire highly  qualified candidates for STEM positions since October 2015. Multiple  hiring officials and supervisors we interviewed stated that they had  lost candidates whom they were in the process of hiring because the  candidates had accepted other offers due to the delays created by the  hiring freeze. In addition, some officials stated that, although the  freeze had been lifted, their labs’ hiring efforts were still affected by  backlogs created by the freeze, or were adapting to new processes  that were implemented as a result of the freeze. Delays with the processing of security clearances: Fifteen of 16  survey respondents cited this as a challenge; 12 of the 15 stated that  this challenge had somewhat or very much hindered their lab’s ability  to hire highly qualified candidates for STEM positions since October  2015. A supervisor from one lab stated that he was in the process of  trying to hire two employees whose hiring actions had been delayed  due to the security clearance process. The supervisor stated that he  had been told it could potentially take an additional 6 months to 1 year  to complete the process, and that he believed this may cause the  candidates to seek other employment opportunities. In other cases,  hiring officials stated that employees may be able to begin work prior  to obtaining a clearance, but that they may be limited in the job duties  they can perform while waiting for their clearance to be granted. The  government-wide personnel security clearance process was added to  GAO’s High Risk List in 2018, based on our prior work that identified,  among other issues, a significant backlog of background  investigations and delays in the timely processing of security  clearances. Inability to extend a firm job offer until a final transcript is  received: Fourteen of 16 survey respondents stated that this was a  challenge, with 10 of the officials responding that it had somewhat or  very much hindered their lab’s ability to hire highly qualified  candidates. One hiring official stated that top candidates will often  receive 5 to 10 job offers prior to graduation, and that his lab’s may be  the only one of those offers that is characterized as tentative. Multiple  officials noted that career fairs can often occur several months prior to  graduation, so the lab would have to wait for the duration of this time  before extending a firm offer to a candidate who has been identified. Delays with processing personnel actions by the external human  resources office: Thirteen of 16 survey respondents stated that this  presented a challenge, and 9 of the 13 stated that this challenge had  somewhat or very much hindered their lab’s ability to hire highly  qualified candidates for STEM positions since October 2015. Multiple  hiring officials stated that employees at their human resource offices  may not have an understanding of either the technical nature of the  positions being filled at the lab or the lab’s unique hiring authorities,  and that this lack of knowledge could create delays. Other officials  noted that their servicing human resource offices seemed to be  inflexible regarding certain paperwork requirements. For example,  officials at one lab stated that their human resource office requires  candidates’ resumes to be formatted in a particular way, and that they  have been required to ask candidates to make formatting changes to  their resumes. An official at another lab stated that the lab has faced  similar challenges with regard to the formatting of transcripts and has  had to request clarifying documentation from the university. In both  cases, the officials described these requirements as embarrassing,  and as a delay to the hiring process. Further, both a supervisor and a  newly hired employee we interviewed noted that it is difficult to learn  the status of an application when it is being processed by the human  resource office. Overall length of the hiring process: Twelve of 16 survey  respondents cited this as a challenge; 11 of the 12 stated that this  challenge had somewhat or very much hindered their lab’s ability to  hire highly qualified candidates for STEM positions since October  2015. Hiring officials and supervisors we interviewed stated that their  lab had lost candidates due to the length of the hiring process. One  supervisor we interviewed stated that he has encountered candidates  who really wanted to work at his lab but had had to pursue other  opportunities because they could not afford to wait to be hired by the  lab. Multiple newly hired employees we interviewed described the  process as slow or lengthy, but described reasons why they were  willing to wait. For example, some employees were already working at  their lab in a contractor or post-doctoral fellowship position, and  accordingly they were able to continue in these positions while  completing the hiring process for the permanent positions they now  hold. One employee stated that if the process had gone on any  longer, he likely would have accepted another offer he had received,  while another employee stated that he knew of at least two post- doctoral fellows at his lab who chose not to continue in the hiring  process for a permanent position at the lab due to the length of the  hiring process. DOD and the  Defense Labs Track  Hiring Data, but the  Defense Laboratories  Office Has Not  Obtained and  Monitored These  Data or Evaluated the  Effectiveness of  Hiring at the  Laboratories The department and the defense laboratories track hiring data that can be  used to evaluate some aspects of the individual labs’ hiring efforts, but  the Defense Laboratories Office has not routinely obtained or monitored  these data or evaluated the effectiveness of hiring, including the use of  hiring authorities, across the defense laboratories as a whole.  Laboratory hiring data are captured at the department level in the  Defense Civilian Personnel Data System (DCPDS)—the department’s  system of record for personnel data. In addition, the individual  defense  laboratories track hiring data, including the type of hiring authority used  and certain milestone dates that can be used to measure the length of the  hiring process, known as time to hire. According to OPM guidance and our prior work, time to hire is a  measure that may inform about the effectiveness of the hiring process,  and federal agencies are required to report time to hire for certain types of  hiring actions to OPM. Defense laboratory officials stated that, from their  perspectives, the time- to-hire metric does not sufficiently inform about  the effectiveness of the use of specific authorities, particularly when using  the most commonly tracked milestones—from the initiation of a request  for personnel action to an employee’s entrance-on-duty date. For  example, officials stated that when a direct hire authority is used to hire a  candidate who is completing the final year of his or her educational  program, the lab may identify and provide a tentative offer to this  candidate several months prior to graduation, consistent with private- sector recruitment methods. In this case, officials stated that the length of  time between the initiation of the request for personnel action and the  candidate’s entrance-on-duty date, following his or her graduation, could  span a period of several months. According to defense laboratory  officials, the total number of days for this hiring action gives the  appearance that the use of the hiring authority was not efficient in this  case; however, officials stated that it would have been effective from the  supervisor’s perspective, because the use of the hiring authority resulted  in the ability to recruit a highly qualified candidate in a manner that was  more competitive with the private sector. Further, time-to-hire data, as reflected by the milestone dates that are  currently tracked across the defense laboratories, may not reflect a  candidate’s perception of the length of the hiring process. More  specifically, a candidate may consider the hiring process to be completed  upon receiving a job offer (either tentative or final), which could occur  weeks or months before the candidate’s entrance-on-duty date, the  commonly used end-point for measuring time to hire. According to  officials, the length of time from when the offer is extended to entrance on  duty can be affected by a candidate’s individual  situation and  preferences, such as the need to complete an educational program or  fulfill family or professional responsibilities prior to beginning work in the  new position. In other cases, certain steps of the hiring process, such as  completing the initial paperwork or obtaining management approval, may  occur after a candidate has been engaged but prior to the initiation of a  request for personnel action—the commonly used start-point for  measuring time to hire. In this situation, the candidate’s perception of the  length of the hiring process may be longer than what is reflected by the  time-to-hire data. For the reasons described above, some defense laboratories measure  time to hire using milestones that they have determined more  appropriately reflect the effectiveness of their hiring efforts. For example,  officials from one lab stated that they have sought to measure the length  of the hiring process that occurs prior to the request for personnel action,  while officials from some labs stated that they measure time to hire using  the tentative offer date as an end-point. In addition, some laboratories  informally collect other types of data that they use in an effort to evaluate  their hiring efforts, such as the reasons why candidates decline a job offer  or feedback on the hiring process from newly hired employees. However, officials from the Defense Laboratories Office stated that their  office has not conducted any review of the effectiveness of defense  laboratory hiring, including the use of hiring authorities, across the labs.  The National Defense Authorization Action for Fiscal Year 2017 gave  authority to conduct and evaluate defense laboratory personnel  demonstration projects to the Office of the Under Secretary of Defense for  Research and Engineering, under which the Defense Laboratories Office  resides. Defense Laboratories Office officials stated that the office has not  evaluated the effectiveness of defense laboratory hiring because it does  not have access to defense laboratory hiring data, has not routinely  requested these data from the labs or at the department level to monitor  the data, and has not developed performance measures to evaluate the  labs’ hiring. As noted, laboratory hiring data are captured at the  department level in DCPDS and in a variety of service- and laboratory- specific systems and tools. However, the Defense Laboratories Office  does not have access to these data and, according to one official, the  office would not have access to defense laboratory hiring data unless  officials specifically requested them from the labs or from the Defense  Manpower Data Center, which maintains DCPDS. According to the  official, the Defense Laboratories Office has not routinely requested such  data in the past, in part because their role did not require evaluation of  such data. In addition, the Defense Laboratories Office has not developed  performance measures to evaluate the effectiveness of hiring across the  defense laboratories or the labs’ use of hiring authorities. An official from  the Defense Laboratories Office stated that the office may begin to  oversee the effectiveness of the defense laboratories’ hiring efforts and,  in doing so, may consider establishing performance measures to be used  consistently across the labs, which could include time-to-hire or other  measures. However, as of March 2018, the office had not established  such measures for use across the defense laboratories nor provided any  documentation on any planned efforts. Standards for Internal Control in the Federal Government states that  management should design appropriate types of control activities to  achieve the entity’s objectives, including top-level reviews of actual  performance and the comparison of actual performance with planned or  expected results. Further, consistent with the principles embodied in the  GPRA Modernization Act of 2010, establishing a cohesive strategy that  includes measurable outcomes can provide agencies with a clear  direction for implementation of activities in multi-agency cross-cutting  efforts. We have previously reported that agencies are better equipped  to address management and performance challenges when managers  effectively use performance information for decision making. Without routinely obtaining and monitoring defense laboratory hiring data  and developing performance measures, the Defense Laboratories Office  cannot effectively oversee the effectiveness of hiring, including the use of  hiring authorities, at the defense laboratories. Specifically, without  performance measures for evaluating the effectiveness of the defense  laboratories’ hiring, and more specifically the use of hiring authorities, the  department lacks reasonable assurance that these authorities—in  particular, those granted by Congress to the defense laboratories—are  resulting in improved hiring outcomes. In addition, without evaluating the  effectiveness of the defense laboratories’ hiring efforts, the department  cannot understand any challenges experienced by the labs or determine  appropriate strategies for mitigating these challenges. As a result, the  department and defense laboratories may be unable to demonstrate that  they are using their authorities and flexibilities effectively, or that such  authorities and flexibilities should be maintained or expanded for future  use. DOD Does Not Have  Clear Time Frames  for Approving and  Implementing  New  Hiring Authorities for  the Defense  Laboratories DOD does not have clear time frames for its process for approving and  implementing new hiring authorities for the defense laboratories. Section  1105 of the Carl Levin and Howard P “Buck” McKeon National Defense  Authorization Act for Fiscal Year 2015 established a direct hire authority  for students enrolled in a scientific, technical, engineering, or  mathematics course of study at institutions of higher education on a  temporary or term basis. Officials from the Defense Laboratories Office  stated that the labs were unable to use the authority because the  department’s current process—the publication of a federal register  notice—for allowing the laboratories to use the hiring authority took longer  than anticipated. On June 28, 2017—2 ½ years after the authority was  granted in the NDAA for Fiscal Year 2015—the department published a  federal register notice allowing the defense laboratories the authority to  use the direct hire for students. DOD officials stated that the department has typically published a federal  register notice whenever the defense laboratories are granted a new  hiring authority in legislation—for example, when an NDAA is issued, or  when certain modifications to the demonstration projects are made. The  Defense Civilian Personnel Advisory Service—through its personnel  policymaking role for the department—at the time required that the  federal register notice process be used to implement any hiring  authorities granted to the defense labs by Congress in legislation.  These procedures were published in DOD Instruction 1400.37. DOD  officials identified coordination issues that occurred during the approval  process of the federal register notice across the relevant offices as the  cause of the delay associated with this federal register notice. Changes to DOD organizational structures further complicated the  process of implementing new hiring authorities for defense laboratories.  Specifically, in late 2016 a provision in the NDAA for Fiscal Year 2017  shifted the authority to conduct and evaluate defense laboratory  personnel demonstration projects from the Office of the Under Secretary  of Defense for Personnel and Readiness to the Office of the Under  Secretary of Defense for Research and Engineering. Within the Office of  the Under Secretary of Defense for Research and Engineering, the  Defense Laboratories Office has been tasked with the responsibility for  matters related to the defense laboratories. According to the Director of  the Defense Laboratories Office, informal discussions about the transition  began shortly after the NDAA for Fiscal Year 2017 was passed in late  2016. According to that official, despite the shift in oversight responsibility,  coordination between the offices of the Under Secretaries for Research  and Engineering and for Personnel and Readiness is required on issues  related to civilian personnel, including defense laboratory federal register  notices. Although a formal process for coordination did not exist at the start of our  review, officials from the Defense Laboratories Office stated that  representatives from the offices have met approximately five times since  December 2016 and were taking steps to establish a coordination  process for implementing new authorities. According to officials from the  Defense Laboratories Office, during those meetings as well as during  other, less formal interactions, officials have taken steps to formalize the  roles and responsibilities of the relevant offices. According to officials  from the Defense Laboratories Office, as of May 2018 the office was  drafting a memorandum to formalize the roles and responsibilities of the  Defense Laboratories Office and the Office of the Under Secretary of  Defense for Personnel and Readiness to correspond to the federal  register notice approval process; however, officials did not provide a  completion date. The Defense Laboratories Office established and documented its own  federal register approval process in spring 2017 and updated it in early  2018. The aforementioned memorandum would further describe the roles  and responsibilities for the Offices of the Under Secretary for Research  and Engineering and the Deputy Assistant Secretary of Defense for  Civilian Personnel Policy in carrying out the updated process. According  to officials, this is the process the office will use moving forward for  coordination and approval of any future federal register notices. On March  6, 2018, the Office published a federal register notice that rescinds the  earlier instruction published by the Defense Civilian Personnel Advisory  Service of the Office of the Under Secretary of Personnel and Readiness.  By rescinding that instruction—including the earlier process for approving  requests from the labs and federal register notices—the Defense  Laboratories Office can, according to officials, publish its own process  and guidance. In a 2016 presentation to the Joint Acquisition/Human Resources Summit  on the defense laboratories, the Chair of the Laboratory Quality  Enhancement Program Personnel Subpanel stated that a renewed and  streamlined approval process would be beneficial to the creation of new  authorities, among other things. Although Defense Laboratories Office  officials provided a flowchart of the office’s updated federal register  approval process for coordination, this process did not include time  frames for specific stages of the coordination. Officials stated that they  cannot arbitrarily assign time frames or deadlines for a review process  because any time frames will be contingent on the other competing  priorities of each office, and other tasks may take priority and thus push  review of a federal register notice down in order of priority. Our prior work  has found that other federal agencies identify milestones, significant  events, or stages in the agency-specific rulemaking process, and track  data associated with these milestones. That work also found that,  despite variability  across federal agencies in the length of time taken by  the federal rulemaking process, scheduling and budgeting for rulemaking  are useful tools for officials to manage regulation development and  control the resources needed to complete a rule. Standards for Internal Control in the Federal Government further  establishes that management should design control activities to achieve  objectives and respond to risks. Further, management should also  establish an organizational structure, assign responsibility, and delegate  authority to achieve the entity’s objectives. Moreover, documentation is a  necessary part of an effective internal control system. The level and  nature of documentation may vary based on the size and complexity of  the organization and its processes. The standards also underscore that  specific terms should be fully and clearly set forth such that they can be  easily understood. Our prior work on interagency collaboration has also  found that overarching plans can help agencies overcome differences in  missions, cultures, and ways of doing business, and can help agencies  better align their activities, processes, and resources to collaborate  effectively to accomplish a commonly defined outcome. Without establishing and documenting clear time frames for its process  for departmental coordination efforts related to the approval and  implementation of new hiring authorities, the department cannot be  certain that it is acting in the most efficient or effective manner possible.  Moreover, the defense laboratories may not promptly benefit from the use  of congressionally granted hiring authorities, relying instead on other  existing authorities. Doing so could, according to officials, have the  unintended consequence of complicating the hiring process, increasing  hiring times, or resulting in the loss of highly qualified candidates. Conclusions The future of the department’s technological capabilities depends, in large  part, on its investment in its people—the scientists and engineers who  perform research, development, and engineering. To that end, Congress  has granted the defense laboratories specific hiring authorities meant to  encourage experimentation and innovation in their approaches to building  and strengthening their workforces. The defense laboratories have used  most of these authorities as a part of their overall hiring efforts. However,  without obtaining and monitoring hiring data and developing performance  measures, the Defense Laboratories Office may not be in a position to  provide effective oversight of the defense laboratories’ hiring, including  the use of hiring authorities, or to evaluate the effectiveness of specific  hiring authorities. Moreover, the absence of clear time frames to facilitate  timely decision-making and implementation of any new hiring authorities  may impede the laboratories’ ability to make use of future authorities  when authorized by Congress. Until the department addresses these  issues, it lacks reasonable assurance that the defense laboratories are  taking the most effective approach toward hiring a workforce that is  critical to the military’s technological superiority and ability to address  existing and emerging threats. Recommendations for  Executive Action We are making three recommendations to DOD. The Secretary of Defense should ensure that the Defense Laboratories  Office routinely obtain and monitor defense laboratory hiring data to  improve the oversight of the defense laboratories’ use of hiring  authorities. (Recommendation 1) The Secretary of Defense should ensure that the Defense Laboratories  Office develop performance measures to evaluate the effectiveness of the  defense laboratories’ use of hiring authorities as part of the labs’ overall  hiring to better inform future decision making about hiring efforts and  policies. (Recommendation 2) The Secretary of Defense should ensure that the Defense Laboratories  Office, in collaboration with the Under Secretary of Defense for Personnel  and Readiness and the Laboratory Quality Enhancement Panel’s  Personnel Subpanel, establish and document time frames for its  coordination process to direct efforts across the relevant offices and help  ensure the timely approval and implementation of hiring authorities.  (Recommendation 3) Agency Comments We provided a draft of this report to DOD for review and comment. In its  written comments, reproduced in appendix VI, DOD concurred with our  recommendations, citing steps the department has begun and plans to  take to improve oversight and coordination of the defense laboratories’  hiring efforts. DOD also provided technical comments, which we  incorporated as appropriate. We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, and other interested parties,  including the Defense Laboratories Office and defense laboratories. In  addition, this report is available at no charge on the GAO website at  http://www.gao.gov. If you or your staff have any questions regarding this report, please  contact Brenda Farrell at (202) 512-3604 or farrellb@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix VII. Appendix  I: Department of Defense Science,  Technology,  Engineering,  and Mathematics  (STEM) Occupations The term “STEM” refers to the fields of science, technology, engineering,  and mathematics. The following figure identifies the Department of  Defense’s broad categories of STEM occupations, as well as the specific  occupational series within each category. Appendix  II: Copy of GAO Questionnaire  Administered to the Defense Laboratory  Officials Appendix  III: Objectives, Scope, and  Methodology This report examines (1) the defense laboratories use of existing hiring  authorities and what officials view as the benefits of authorities and  incentives and the challenges in hiring; (2) the extent to which the  Department of Defense (DOD) evaluates the effectiveness of hiring,  including hiring authorities, at the defense laboratories; and (3) the extent  to which DOD has time frames for approving and implementing new hiring  authorities. To address these objectives, we included in the scope of our review  science, technology, engineering, and mathematics (STEM) hiring at the  15 defense laboratories designated as Science and Technology  Reinvention Laboratories (STRL) that were implemented at the time of  our review within the Army, Navy, and Air Force. We included 9 Army  laboratories: Armament Research, Development, and Engineering  Center; Army Research Laboratory; Aviation and Missile Research,  Development, and Engineering Center; Communications-Electronics  Research, Development, and Engineering Center; Edgewood Chemical  and Biological Center; Engineer Research and Development Center;  Medical Research and Materiel Command; Natick Soldier Research,  Development, and Engineering Center; and Tank Automotive Research,  Development, and Engineering Center. We included 5 Navy laboratories:  Naval Air Systems Command Warfare Centers, Weapons Division and  Aircraft Division; Naval Research Laboratory; Naval Sea Systems  Command Warfare Centers, Naval Surface and Undersea Warfare  Centers; Office of Naval Research; and Space and Naval Warfare  Systems Command, Space and Naval Warfare Systems Center, Atlantic  and Pacific. We included 1 Air Force laboratory: Air Force Research  Laboratory. We excluded 2 additional defense laboratories within the  Army—the Army Research Institute and the Space and Missile Defense  Command—because these defense laboratories were in the process of  being implemented at the time of our review. For our first objective, we obtained and analyzed documentation,  including past National Defense Authorization Acts (fiscal years 1995  through 2017), guidance related to government-wide hiring authorities,  and federal register notices on existing hiring authorities used by the  defense laboratories to hire STEM personnel. We obtained data that were  coordinated by the Defense Manpower Data Center and prepared by the  Defense Civilian Personnel Advisory Service’s Planning and  Accountability Directorate. These data included, among other things,  hiring process milestone dates and type of hiring authority used for each  civilian hire at the defense laboratories for fiscal years 2015 through  2017. We selected these years because they were the three most recent  years for which hiring data were available, and because doing so would  allow us to identify any trends in the use of hiring authorities or the length  of time taken to hire. The data we obtained were extracted from DCPDS  using the Corporate Management Information System. The team refined the data to include only those hiring actions that were  made by the 15 defense laboratories included within the scope of our  review. In addition, we excluded hiring actions that used a 700-series  nature of action code, which denotes actions that relate to position  changes, extensions, and other changes, which we determined should  not be included in our analysis. We included actions that used nature of  action codes in the 100-series (appointments) and 500-series  (conversions to appointments). For the purpose of calculating time to hire,  we also excluded records with missing dates and those for which the  time-to-hire calculation resulted in negative number (that is, the record’s  request for personnel action initiation date occurred after the enter-on- duty date). Specifically, we excluded 92 actions for which no request for  personnel action initiation date was recorded and 205 actions for which  the date occurred after the enter-on-duty date, for a total of 2.57 percent  of all hiring actions. We included in our calculation 7 actions for which the  request for personnel action initiation date was the same date as the  enter-on-duty date, resulting in a time to hire of zero days. To determine the extent to which the defense laboratories use existing  hiring authorities, based on the department’s data, we analyzed the  current appointment authority codes identified for individual hiring actions.  Current appointment authority codes are designated by the Office of  Personnel Management and are used to identify the law, executive order,  rule, regulation, or other basis that authorizes an employee’s most recent  conversion or accession action. Based on our initial review of the data,  we determined that, in some cases, more than one distinct current  appointment authority code could be used to indicate the use of a certain  hiring authority. Alternately,  a single current appointment authority code  could in some cases be used for indicating more than one type of  authority. In these cases, the details of the specific type of hiring authority  that was used for the hiring action can be recorded in the description field  associated with the current appointment authority code field. For this  reason, in order to determine the type of hiring authority used, it was  necessary to analyze the description fields for the current appointment  authority code when certain codes were used. Two analysts  independently reviewed each description and identified the appropriate  hiring authority. Following this process, the two analysts compared their  work and resolved any instances in which the results of their analyses  differed. A data analyst used the results to produce counts of the number  of times various categories of hiring authorities were used, as well as the  average time to hire for each hiring authority category. For those instances where the analysts could not identify a hiring  authority on the basis of the three digit codes or the description fields, the  hiring actions were assigned to an “unknown” category. We note that the  “unknown” category included 591 hiring actions, or approximately 5  percent of the total data for fiscal years 2015 through 2017. In addition,  within the laboratory-specific direct hire authority category, if a  determination could not be made about the specific type of laboratory- specific direct hire authority used, the hiring action was captured in the  “direct hire authority, unspecified” category because the action was  clearly marked as one of the laboratory-specific direct hire authorities but  the type of authority (for example, direct hire for veterans) was unclear. Of  the 5,303 hiring actions identified as a laboratory-specific direct hire  authority, 0.1 percent of the hiring actions fell into the unspecified  category. Based on the aforementioned steps and discussions with  officials from the Defense Civilian Personnel Advisory Service and the  Defense Manpower Data Center and reviews of additional documentation  provided to support the data file, as well as interviews with officials from  13 of the laboratories about their data entry and tracking, we determined  that these data were sufficiently reliable for the purposes of reporting the  frequency with which the labs used specific hiring authorities and  calculating the time it takes the labs to hire, or time to hire, for fiscal years  2015 through 2017. To describe officials’ views of hiring authorities and other incentives, we  conducted a survey of officials at each of the defense laboratories on (1)  their perceptions of the various hiring authorities and incentives, (2)  whether those authorities and incentives have helped or hindered hiring  efforts, (3) the extent to which they experienced barriers to using hiring  authorities, and (4) any challenges during the hiring process, among other  things. We administered the survey to the official at each defense  laboratory who was identified as the Laboratory Quality Enhancement  Program Personnel, Workforce Development, and Talent Management  Panel point of contact, because we determined that this individual would  be the most knowledgeable about his or her lab’s hiring process and use  of hiring authorities. One laboratory—the Space and Naval Warfare  Systems Command Centers—had two designated Laboratory Quality  Enhancement Program Personnel, Workforce Development, and Talent  Management Panel points of contact, one for each of its command  centers (Atlantic and Pacific). Because the contacts would each be  knowledgeable about his or her lab’s hiring processes for their respective  command centers, we chose to include both command centers in our  survey. As a result, we included a total of 16 laboratory officials in our  survey. We drafted our questionnaire based on the information obtained from our  initial interviews with department, service, and laboratory personnel. We  conducted pretests to check that (1) the questions were clear and  unambiguous, (2) terminology was used correctly, (3) the questionnaire  did not place an undue burden on agency officials, (4) the information  could feasibly be obtained, and (5) the survey was comprehensive and  unbiased. We conducted five pretests to include representatives from  each of the three services, as well as from corporate research  laboratories and from research, development, and engineering centers.  We conducted the pretests—with the assistance of a GAO survey  specialist—by telephone and made changes to the content and format of  the questionnaire after each pretest, based on the feedback we received.  Key questions from the questionnaire used for this study are presented in  appendix II. We sent a survey notification email to each laboratory’s identified point of  contact on July 6, 2017. On July 10, 2017, we sent the questionnaire by  email as a Microsoft Word attachment that respondents could return  electronically after marking checkboxes or entering responses into open  answer boxes. One week later, we sent a reminder email, attaching an  additional copy of the questionnaire, to everyone who had not responded.  We sent a second reminder email and copy of the questionnaire to those  who had not responded 2 weeks following the initial distribution of the  questionnaire. We received questionnaires from all 16 participants by  August 4, 2017, for a 100 percent response rate. Between July 26 and  October 5, 2017, we conducted additional follow-up with 11 of the  respondents via email to resolve missing or problematic responses. Because we collected data from every lab, there was no sampling error.  However, the practical difficulties of conducting any survey may introduce  errors, commonly referred to as non-sampling errors. For example,  differences in how a particular question is interpreted, the sources of  information available to respondents, how the responses were processed  and analyzed, or the types of people who do not respond can influence  the accuracy of the survey results. We took steps in the development of  the survey, the data collection, and the data analysis to minimize these  non-sampling errors and help ensure the accuracy of the answers that  were obtained. For example, a survey specialist designed the  questionnaire, in collaboration with analysts having subject matter  expertise. Then, as noted earlier, the draft questionnaire was pretested to  ensure that questions were relevant, clearly stated, and easy to  comprehend. The questionnaire was also reviewed by internal subject  matter experts and an additional survey specialist. Data were electronically extracted from the Microsoft Word  questionnaires into a comma-delimited file that was then imported into a  statistical program for quantitative analyses and Excel for qualitative  analyses. We examined the survey results and performed computer  analyses to identify inconsistencies and other indications of error, and we  addressed such issues as necessary. Quantitative data analyses were  conducted by a survey specialist using statistical software. An  independent data analyst checked the statistical computer programs for  accuracy. To obtain information on department- and service-level involvement in  and perspectives of defense laboratory hiring, we interviewed officials at  the Defense Personnel Advisory Service, Defense Laboratories Office,  Army Office of the Assistant G-1 for Civilian Personnel, and Navy Office  of Civilian Human Resources. In addition, we interviewed hiring officials,  first-line supervisors, and newly hired employees from a non- generalizable sample of six defense laboratories or subordinate level  entities within a laboratory (for example, division or directorate) to obtain  their perspectives on the hiring process. We selected the six laboratories  based on the following two criteria: (1) two laboratories from each of the  three services, and (2) a mix of both corporate research laboratories and  research and engineering centers. In addition, because some hiring  activities can occur at subordinate levels within a laboratory—such as a  division or directorate—we included at least one subordinate level entity  for each service. In total, we selected: Army Research Laboratory  Sensors and Electron Devices directorate; Aviation and Missile Research,  Development, and Engineering Center (Army); Naval Research  Laboratory; Naval Air Warfare Center Weapons Division; Air Force  Research Laboratory Information directorate; and Air Force Research  Laboratory Space Vehicles directorate. For each lab, we requested to  interview the official(s) most knowledgeable about the lab’s hiring  process, supervisors who had recently hired, and newly hired employees. We initially requested to interview one group each of supervisors and  newly hired employees. Following our first round of interviews at one  laboratory, we requested to interview two groups each of supervisors and  newly hired employees. Subsequent to this request, at one lab we were  able to conduct one supervisor interview and at a second lab we were  able to conduct one newly hired employee interview, due to scheduling  constraints. The views obtained from these officials, supervisors, and  recent hires are not generalizable and are presented solely for illustrative  purposes. For our second and third objectives, we reviewed guidance and policies  for collecting and analyzing laboratory personnel data related to the  implementation and use of hiring authorities by these labs. We  interviewed DOD, military service, and defense laboratory officials to  discuss and review their hiring processes and procedures for STEM  personnel, the use of existing hiring authorities, and efforts to document  and evaluate time-to-hire metrics. We also met with DOD officials from  the Office of the Under Secretary of Defense for Personnel and  Readiness and the Office of the Under Secretary of Defense for Research  and Engineering to discuss processes and procedures for implementing  new hiring authorities granted by Congress. We evaluated their efforts to  determine whether they met federal internal control standards, including  that management should design appropriate types of control activities to  achieve the entity’s objectives, including top-level reviews of actual  performance, and should establish an organizational structure, assigning  responsibilities and delegating authority to achieve an organization’s  objectives. We conducted this performance audit from November 2016 to May 2018  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix  IV: The Department of Defense  Laboratories’ Use of Hiring Authorities  for  Fiscal Years 2015, 2016, and 2017 We analyzed three years of Department of Defense hiring data obtained  from the Defense Civilian Personnel Data System to identify the defense  laboratories’ use of hiring authorities. We found that the defense  laboratories completed a total of 11,562 STEM hiring actions in fiscal  years 2015 through 2017 and used the defense laboratory direct hire  authorities the most often when hiring STEM personnel. Table 7 provides  information on the laboratories’ use of hiring actions by hiring authority for  fiscal years 2015, 2016, and 2017. Table 8 provides a breakdown of the individual  labs’ use of hiring  authorities in fiscal years 2015 through 2017. Appendix  V: Defense Laboratory Time to  Hire Data by Hiring Authority  Category for  Fiscal Years 2015, 2016, and 2017 We analyzed three years of the DOD hiring data to identify time to hire  using various types of hiring authorities when hiring for Science,  Technology, Engineering, and Math (STEM) occupations at the defense  laboratories. Tables 9, 10, 11, and 12 below show the frequency of  actions for each hiring authority category and the average, minimum,  maximum, median, 25th percentile, and 75th percentile of the number of  days to hire for each category in fiscal years 2015 through 2017 and for  all three years combined. Appendix  VI: Comments from the  Department of Defense Appendix  VII: Contact and Staff  Acknowledgments GAO Contact: Staff  Acknowledgments: In addition to the contact named above, Vincent Balloon (Assistant  Director), Isabel Band, Vincent Buquicchio, Joseph Cook, Charles  Culverwell, Serena Epstein, Christopher Falcone, Robert Goldenkoff,  Cynthia Grant, Chelsa Gurkin, Amie Lesser, Oliver Richard, Michael  Silver, John Van Schaik, Jennifer Weber, and Cheryl Weissman made  key contributions to this report.
DOD's defense labs help sustain, among other things, U.S. technological superiority and the delivery of technical capabilities to the warfighter. Over time Congress has granted unique flexibilities—such as the ability to hire qualified candidates who meet certain criteria using direct hire authorities—to the defense labs to expedite the hiring process and facilitate efforts to compete with the private sector. Senate Report 114-255 included a provision for GAO to examine the labs' hiring structures and effective use of hiring authorities. This report examines (1) the defense labs use of existing hiring authorities and officials' views on the benefits of authorities and challenges of hiring; (2) the extent to which DOD evaluates the effectiveness of hiring, including hiring authorities at the defense labs; and (3) the extent to which DOD has time frames for approving and implementing new hiring authorities. GAO analyzed DOD hiring policies and data; conducted a survey of 16 defense lab officials involved in policy-making; interviewed DOD and service officials; and conducted nongeneralizable interviews with groups of officials, supervisors, and new hires from 6 labs—2 from each of the 3 military services, selected based on the labs' mission. The Department of Defense's (DOD) laboratories (defense labs) have used the laboratory-specific direct hire authorities more than any other category of agency-specific or government-wide hiring authority for science, technology, engineering, and mathematics personnel. As shown below, in fiscal years 2015—2017 the labs hired 5,303 personnel out of 11,562 total hires, or 46 percent using these direct hire authorities. Lab officials, however, identified challenges to hiring highly qualified candidates, such as delays in processing security clearances, despite the use of hiring authorities such as direct hire. Source: GAO analysis of Department of Defense data. | GAO-18-417 . a Other includes all other defense laboratory-specific direct hiring authorities used. b All other includes remaining five categories of hiring authorities. c Percentages may not sum to total due to rounding. DOD and the defense labs track hiring data, but the Defense Laboratories Office (DLO) has not obtained or monitored these data or evaluated the effectiveness of the labs' hiring, including the use of hiring authorities. While existing lab data can be used to show the length of time of the hiring process, effectiveness is not currently evaluated. According to lab officials, timeliness data do not sufficiently inform about the effectiveness of the authorities and may not reflect a candidate's perception of the length of the hiring process. Further, the DLO has not developed performance measures to evaluate the effectiveness of hiring across the defense laboratories. Without routinely obtaining and monitoring hiring data and developing performance measures, DOD lacks reasonable assurance that the labs' hiring and use of hiring authorities—in particular, those granted by Congress to the labs—result in improved hiring outcomes. DOD does not have clear time frames for approving and implementing new hiring authorities. The defense labs were unable to use a direct hire authority granted by Congress in fiscal year 2015 because it took DOD 2½ years to publish a federal register notice—the process used to implement new hiring authorities for the labs—for that authority. DOD officials identified coordination issues associated with the process as the cause of the delay and stated that DOD is taking steps to improve coordination—including meeting to formalize roles and responsibilities for the offices and developing a new approval process—between offices responsible for oversight of the labs and personnel policy. However, DLO's new federal register approval process does not include time frames for specific stages of coordination. Without clear time frames for its departmental coordination efforts related to the approval and implementation of new hiring authorities, officials cannot be certain they are taking action in a timely manner.