Background NRFU is a field-based operation that the Bureau administers following the  self-response period so that it can (1) determine the occupancy status of  individual non-responsive housing units and (2) enumerate them. In most  instances, the Bureau typically allows up to six enumeration attempts for  each nonresponsive housing unit, or case. If the Bureau is unable to  enumerate the housing unit in the field, it may have to impute attributes of  the household based on the demographic characteristics of surrounding  housing units as well as administrative records. Within the test site in Providence County, Rhode Island, the Bureau set  up an area census office to administer field operations. Figure 1 provides  an overview of the managerial hierarchy of the area census office. The  area census office manager oversees day-to-day operations within the  office and acts as a liaison with the Bureau’s New York Regional Census  Center, which is a Bureau regional office with jurisdiction over the  Providence area census office. Census field managers are to monitor  operational progress and performance indicators to understand any areas  of concern and shift resources as needed within the test site. Census  field supervisors are to act as front-line supervisors for individual  performance and payroll processes and receive procedural questions  from enumerators, who conduct the count. The Bureau has another operation—Group Quarters—to enumerate  those living or staying in a group facility that provides housing or services.  Such facilities can include skilled nursing facilities, college and university  student housing, and correctional facilities. Within the Group Quarters  enumeration, the Bureau also enumerates places such as soup kitchens,  homeless shelters, and other service-based enumeration facilities. Prior to Group Quarters enumeration in the field, the Bureau attempts to  establish the facilities’ approximate population count and preferred  enumeration method through the Advance Contact operation. These  facilities can choose among methods including paper listing, where the  facility provides a roster of residents as of census day to the Bureau, and  in-person enumeration, where a team of enumerators count residents.  For the 2020 cycle, the Bureau is also adding an “eResponse” option,  tested on a small scale in 2016, whereby facility administrators can  electronically submit enumeration data at a date of their choosing within  operational time frames. The Bureau’s testing of the peak operations that we observed during the  2018 Census Test was intended to test collection of census data from  those either not responding themselves via paper, telephone, or over the  Internet or those living in group quarters. Prior to the start of NRFU during  the 2018 test, roughly 45 percent of anticipated housing units in the test  area of Providence County, Rhode Island, self-responded, leaving more  than 140,000 remaining housing units to be attempted by NRFU itself,  which took place between May 9 and July 31, 2018. The Bureau  conducted a test of its Group Quarters enumeration from July 25 through  August 24, 2018, including service-based enumeration. Both portions of  Group Quarters fieldwork were preceded by the Advance Contact activity.  Dates for the peak operations we observed during the 2018 test are listed  in table 1. The Bureau’s operational plans for this phase of the fieldwork for the  2018 test incorporated two innovation areas that the Bureau hopes will  produce savings for 2020. Reengineered field operations. For most of NRFU during the 2018  test, the Bureau relied on automated data collection methods,  including a system-based, automated process for assigning work to  enumerators, a smartphone-based application for collecting  enumeration data in the field, and system-generated supervisory  alerts. Use of administrative records. To help reduce costly NRFU visits  during the 2018 test, the Bureau reviewed and, where appropriate,  applied administrative records—information already provided to the  government as it administers other programs, such as Social Security,  the Selective Service, or the Special Supplemental Nutrition Program  for Women, Infants, and Children—to determine the occupancy status  of housing units and thus remove vacant housing units from the  NRFU workload, as well as to provide population counts of  households not responding. The Bureau also tested multiple operational features for the first time this  decennial cycle under full census-like production conditions in 2018: NRFU Closeout. During the 2018 test, the Bureau tested how best to  relax certain business rules and enumeration procedures late in the  NRFU operation so that it can enumerate persistently non-responsive  housing units. Examples of procedural modifications include  increasing the maximum allowable number of enumeration attempts  for each housing unit and manually assigning cases to the highest- performing enumerators. Office-Based Group Quarters Advance Contact. In the 2010  Census, the Bureau sent enumerator crews in person to each facility  in advance of enumeration to establish the facility’s preferred method  of enumeration and to obtain an approximate population count. In the  2018 Census Test, the Bureau implemented a new method for 2020  that instead involved clerical staff contacting facilities by telephone  and updating the group quarters address list and enumeration  information remotely to reduce expenses associated with field visits  for its enumerator crews. The Bureau  Experienced  Operational Planning,  Workforce  Management, and  Other Issues during  Its 2018 Test of Peak  Operations The Bureau Did Not  Determine Procedures for  Late-NRFU Data  Collection until after It  Started the Work The Bureau began the last phase of NRFU data collection in the 2018  test without having yet determined the procedures it would use for that  critical phase. Bureau planning documentation from February 2018  described a late-operation “closeout” phase of NRFU that would attempt  to resolve cases that had not yet responded. However, we found that the  Bureau had not determined the procedural modifications this phase would  involve, either in terms of rules enumerators followed or business rules for  how cases were to be assigned. By late May, nearly 3 weeks into the  operation, the Bureau issued a set of closeout procedures to census  areas where most cases had either been completed or where at least four  of the six allowable enumeration attempt day assignments had been  made. The Bureau also placed a priority on having high-performing  enumerators—in terms of their ability to complete cases—available to  work these cases during this phase of the NRFU testing. Table 2 summarizes the chronology for when the Bureau implemented  and documented procedural changes governing the transition from early  to late-NRFU data collection, as well as the nature of those changes. In late June 2018, the Bureau began testing the third phase of NRFU data  collection, what it referred to as the “final attempt” phase, with officials  citing a high incidence of non-interviews during prior phases as the  reason. However, the Bureau had also begun this phase’s data collection  before it had established the procedural modifications it would be using.  The modifications were intended to further increase the chances of  enumerators completing cases in the field, such as by removing the limit  on the number of attempts enumerators could make at each remaining  case before NRFU ended. Standards for Internal Control in the Federal Government states that  agencies should implement control activities by, for example,  documenting policies. However, the Bureau did not determine  procedures for the final attempt phase until after testing for this phase of  NRFU had begun. Enumerators and census field supervisors thus began  working closeout and final attempt cases without a standardized set of  test procedures. Without determining the procedural changes the Bureau would be  testing—or the business rules guiding when to make those changes—the  Bureau was not well positioned to collect data to assess the alternatives it  used during the test to inform planning for 2020. Bureau officials shared  with us that they believed their automated case assignment approach is  most effective during initial data collection but that it is less effective at  targeting the toughest cases to resolve late in NRFU data collection. Yet,  in part because the Bureau had not established when the transition from  automated to manual case management would occur—or the business  rules for determining when—some of the highest-performing enumerators  were unavailable to receive assignments when the Bureau needed to  begin the final attempt phase, according to the area census office  manager. By not establishing the scope and timing of procedural changes  for late-NRFU data collection in 2020, the Bureau may not be in a position  to efficiently shift from its automated assignment approach to a manual  one at the right time and position its most effective enumerators to  receive assignments when needed. In November 2018, the Bureau provided a draft contact strategy for  NRFU in 2020 that included an outline of a multi-phase strategy for late- NRFU data collection. By including multiple phases of (1) shifting away  from a fully-automated case assignment process and (2) relaxing  management controls to complete as much casework as possible in  areas with continued high non-response rates, this strategy appears to  follow what the Bureau ultimately implemented during the 2018 test. It will  be important, however, for the Bureau to determine the business rules for  procedural changes and their timing in advance so that it can maximize  the value of NRFU in reducing the number of housing units that have to  be imputed for the 2020 Census. The Bureau Did Not Fully  Ready Its Field Workforce  for Enumeration  Challenges Census field supervisors were not integrated into casework  management. As described in the Bureau’s training and operational  planning documents, census field supervisors were to be the primary  points of contact in fielding and addressing enumerator questions.  Census field managers—the next step above census field supervisors— were to focus their efforts on monitoring progress in completing the  caseload, reviewing cases flagged by enumerators as problematic in one  of a small number of pre-defined ways (e.g., dangerous addresses), and  resolving significant performance issues. Among field supervisors’ key  responsibilities, according to the Bureau’s plan for NRFU, were providing  guidance to help enumerators understand procedural matters and to offer  coaching and problem-solving support to enumerators who may need it.  They also led enumerator training prior to the beginning of NRFU and  generally were to train their specific team of enumerators. However, census field managers and enumerators indicated that census  field supervisors were often not the primary actors involved in fielding and  addressing enumerator questions. Instead, enumerators and census field  managers reported having direct contact with each other over procedural  questions. Moreover, seven of the nine enumerators participating in the  Bureau’s operational debrief focus group who responded said they  thought finding someone who could answer their questions was either  difficult or very difficult. We found that census field supervisors went underutilized in part because  the Bureau did not recruit and position them to assume front-line  supervising and coaching responsibilities. As outlined in training  documentation, the Bureau vested supervisory review authority (for  special cases, such as resident refusals and language barrier issues)  within census field managers, the area census office manager, clerks,  and office operations supervisors instead of census field supervisors.  Additionally, as part of the Bureau’s reengineered field operations for  2020, census field supervisors are given automated tools to monitor  enumerators, and enumerators we observed told us that they generally  did not interact in person with their supervisors apart from training. We  believe that the combination of these factors resulted in census field  supervisors having limited exposure to NRFU casework and any  problematic situations enumerators might encounter. Officials also told us  that the Bureau did not screen census field supervisors for their  supervisory or coaching skillsets, though officials noted that this has been  the practice in prior censuses, too. Rather, they hired census field  supervisors based on their scores on the online enumerator training and  because they reported an interest in supervising. Additionally, census field supervisors lacked access to certain data  streams from the test that could have helped them answer or troubleshoot  enumerator questions. According to two census field managers, the  Bureau did not regularly share consolidated records of procedural  changes with census field supervisors. Information technology (IT) and  census field managers also noted that the Bureau did not share or  compare observations between the census field supervisor hotline and  the decennial IT hotline, even though enumerators could potentially call  either or both with technical or procedural questions. As a result, without  sharing how best to respond to similar questions across support lines,  enumerators could receive different answers for related questions  depending on which hotline they contacted. Standards for Internal Control in the Federal Government states that  agency management should demonstrate a commitment to recruit,  develop, and retain competent individuals. Management should establish  expectations for competence in key roles and should consider the level of  assigned responsibility and delegated authority when establishing  expectations. Yet, the role the Bureau envisioned census field  supervisors having was not aligned with the authority supervisors were  given, the skills for which the Bureau hired them, or the access to  information that they had for the 2018 test. When we raised this issue related to using census field supervisors,  Bureau officials agreed and cited feedback they had received that census  field managers felt inundated with the combination of the volume of  supervisory review cases that flowed to them and with troubleshooting  day-to-day enumerator questions. In October 2018, the Bureau provided  documentation to us proposing a set of questions that they could use in  screening applicants for the census field supervisor position to identify  supervisory skills. Officials also said they were still evaluating options for  granting census field supervisors more supervisory review authority. As  the Bureau continues to learn from the 2018 test as part of its planning for  2020, it will be important to align census field supervisor roles with their  authorities, skills, and information flows so that the Bureau does not  underutilize a key portion of its field management chain. Doing so could  also lessen the operational burden on higher-level census field managers. Enumerators did not receive training to address mid-operation  issues. Prior to the start of 2018 NRFU testing, the Bureau trained  enumerators with a series of online training modules and assessments  and one full day of in-person training facilitated by census field  supervisors. The training included modules on data stewardship  requirements, payroll responsibilities, and procedural directions for  conducting respondent interviews. However, officials acknowledged that  when the Bureau implemented its closeout and final attempt phases of  NRFU, it did not provide standardized training to enumerators on the  rollout of procedural changes. Five enumerators we observed during  these stages said they relied on informal communications from their  census field supervisors or census field managers for guidance. The  initial practice had been for enumerators to receive daily assignments and  follow pre-specified case sequencing and routing based on the Bureau’s  automated system. During the final attempt phase, enumerators were  given discretion over the sequencing, routing, and number of attempts to  make for cases that could be manually assigned, yet they were not given  standardized training on how to handle this shift. During our field observations, some enumerators we spoke with said they  were uncertain about core procedures. For example, enumerators were  not consistently aware that they had some discretion in large multi-unit  settings to deviate from the assigned sequence of their cases provided by  the automated system. Enumerators we observed and spoke to were also  not always clear on how to flag within their field enumeration application  the commonly occurring cases with confusing address markings and  numberings. For example, enumerators had the option of selecting a case  outcome of “missing unit designation,” but they were not always sure  whether this selection would capture the nuances of what they were  seeing on the ground or how it differed from other selection options. Standards for Control in the Federal Government states that agencies  should demonstrate a commitment to competence by, for example,  tailoring training based on employee needs and helping personnel adapt  to an evolving environment. Targeted informational training would help  the Bureau ensure that staff understand mid-operation procedural  changes, and the training could be an opportunity for the Bureau to  address commonly-observed and persistent implementation issues that  may be arising. By developing brief, targeted mid-operation training,  either as formal modules, guidance, or other standardized job aids, such  as “frequently asked questions” worksheets, the Bureau could better  position itself to react nimbly to enumerator feedback. We have previously reported challenges the Bureau faces with its field  work in other locations, such as connecting to the Internet during testing  of address canvassing in rural West Virginia in 2017 and dealing with  language barriers and other circumstances in unincorporated  communities in southern Texas or with migrant and seasonal farmworkers  in southern California during the 2000 Census. All challenges are not  universal to all locations. Given that some of the enumeration challenges  enumerators encountered in 2018 NRFU testing might not occur  everywhere, and that some other areas of the country will have their own  types of challenges, locally- or regionally-specific training or guidance  may better address some needs. By relying solely on pre-NRFU training,  the Bureau risks having little opportunity to course-correct with  enumerators who may not have absorbed all of the training and are  experiencing difficulty completing interviews or not collecting quality data. The Bureau Is Assessing  Other Implementation  Issues That Arose during  the 2018 Test We observed and discussed with Bureau officials in real time several  other implementation issues that occurred during the 2018 test. Bureau  officials acknowledged these issues and, as of September 2018, were  assessing them and developing mitigation strategies as part of their test  evaluation process. These issues include:  Training certification. Census field managers estimated that roughly  100 enumerators were unable to transmit their final test scores because  the Bureau’s online learning management system had an erroneous  setting. According to Bureau officials, this problem delayed the start of  unsupervised work for these otherwise-qualified enumerators by an  average of 2 days per enumerator and resulted in the attrition of some  who were able to quickly find other work. Bureau officials told us they  have fixed the system setting and are considering an alternative means to  certify training, such as by having the option of trainees taking and  verifying their final assessment as part of their final capstone day of  classroom training. According to Bureau officials, development of this  backup strategy will begin in December 2018. Assigning cases manually in batches. During the 2018 test, the  Bureau’s automated case management system was not configured for  non-Headquarters staff to manually assign multiple cases to an  enumerator at once. Rather, according to officials, census field managers  were faced with having to manually assign thousands of cases  individually during latter stages of NRFU. According to field management,  this problem presented an unexpected burden on them, delayed  assignments of the hardest-to-count cases, and contributed to high- performing enumerators not receiving work timely and in some cases for  days in a row. Officials told us that, as a work-around, the Bureau shifted  responsibility for assigning cases to a headquarters official with access  rights in the system to assign large numbers of cases at once. Bureau  officials acknowledge the unsustainability of this work-around if needed at  a national level and the importance of resolving this before the 2020  Census. As of October 2018, Bureau officials showed us system  screenshots of how census field managers would be able to manually  assign batches of cases and indicated that this functionality would be  ready for the 2020 Census. Monitoring operational progress. The Bureau’s reporting on its  progress in completing the NRFU casework for the 2018 test emphasized  a process-oriented measure that overstated the extent to which the NRFU  efforts were resulting in completed workload. In planning documentation,  the Bureau listed the outcomes of interview attempts that it considered  complete and thus not in need of further enumeration assignments.  These outcomes—such as a full interview of the household or  confirmation of a housing unit being vacant or nonexistent—would also  result directly in reduction in the number of incomplete cases needing to  have some of their missing data imputed by the Bureau later. Yet the daily Bureau progress report and “dashboard” the Bureau  provided us for the 2018 test, which decennial leadership also identified  as their primary monitoring report, did not reflect these pre-planned  definitions of completed workload. Rather, as officials acknowledged, it  included cases that the Bureau had unsuccessfully attempted to  enumerate the maximum number of allowable times for the initial phase  of NRFU being tested, even though those cases could still—and did— receive additional attempts during later phases of NRFU. Officials noted  that the measure reported could be helpful during early stages of the  operation in determining whether enough employees had been hired, or  whether case assignments were being worked quickly enough. Figure 2 demonstrates the gap that arose during 2018 NRFU test  implementation between the reported progress measure and the number  of cases actually being completed. The totals reflected in the Bureau’s  reported measure include those that either have to be re-worked in the  field during the final attempt phase as discussed or have their data  imputed after fieldwork had ended. By contrast, an outcome-based  measure of operational progress, like the one the Bureau designed,  would capture only those cases where the Bureau had completed  enumeration of the nonresponding housing units and thus be a more  accurate representation of the operation’s status. Bureau officials acknowledged the need to maintain measures that focus  on process as well as outcomes—such as avoiding having to impute data  for cases after field work—when measuring progress completing NRFU.  They said that managers in the field and in Bureau headquarters had  access to alternative measures and reports that more closely identified  outcomes. The officials noted that the reporting mechanism expected to  be used in 2020 was not fully available in time for the beginning of NRFU  testing in 2018, so the reporting format and measures will likely differ. In  addition, in October 2018, the Bureau provided a draft dashboard for  2020 that included greater detail on the number of cases that could still  require work to enumerate. Such detail could help assist with determining  when to transition to the final-attempt phase of the operation to address  cases without sufficient information yet collected. Integrating key systems settings. At the beginning of NRFU test  implementation, the Bureau’s case assignment and case sequencing  systems were operating as if they were on time zones 4 hours apart.  Bureau officials said that this resulted in enumerators receiving  mismatched case assignment times, which hampered early NRFU  production, and census field supervisors having to process erroneous  “work not started” supervisory alerts. Bureau officials said they addressed  the problem within the first week of the operation and that they would  ensure that future updates and key settings would be coordinated across  systems. Tracking employees and equipment. The Bureau used two different  sets of employee identification numbers to track their payroll status and  use of Bureau-issued equipment (e.g., smart phones), respectively,  without cross-walking them. According to census field managers, this  resulted in extra work when trying to monitor changes in enumerators’  employment statuses and whether enumerators had returned their  equipment to the Bureau. The managers noted that office staff had to  spend extra time comparing different lists of staff, while one manager  developed a spreadsheet listing all staff by their two different identification  numbers. Bureau officials said they considered this a priority issue to be  resolved during final systems development for 2020 and had already  developed a fix within their case management system so that the cross- walk between the two systems would be integrated within their  management system. This would eliminate the need for manually  reconciling the differing identification numbers. Having more enumerators work weekends. Until the latter stages of  2018 NRFU testing, the Bureau assigned cases to enumerators based on  the alignment of the Bureau’s estimated probability of finding respondents  at home at certain times and enumerator-reported work availability.  Bureau officials told us that Saturdays are generally one of the best days  to find a household member home to respond to the census. However,  during the test, our analysis showed that Saturdays had the second- fewest number of enumerators assigned to cases of any day of the week.  Bureau officials said that they would review whether the incentive  structure for working on Saturdays should be altered and that they would  examine ways to ensure that more enumerators are working on those  days. This includes exploring the feasibility of hiring and assigning work to  applicants who may only want to work weekends and being clearer with  enumerators about what the expected peak enumeration hours are. Increasing electronic completion rates for Group Quarters. The  Bureau hopes to reduce field costs for Group Quarters (such as skilled  nursing homes, college and university student housing, and correctional  facilities) by, for the first time, encouraging facilities to self-enumerate  electronically, when possible. As previously discussed, clerical staff first  establish facility enumeration preferences during Group Quarters  Advance Contact, and enumeration (either in person or otherwise) takes  place afterward. During the 2018 test, the Bureau reported that only 25 of  the 75 facilities that selected the “eResponse” enumeration option during  Advance Contact submitted responses by the enumeration deadline. As  of September 2018, Bureau officials said they were still evaluating  potential causes of the low response rate by Group Quarters facilities but  noted that issues with the required format for the submission of response  files may have prevented some submissions. Bureau officials also  acknowledged the need to conduct more active follow-up with these  facilities during the eResponse period to ensure a full and accurate count  of Group Quarters facilities. The Bureau Has  Made Progress but  Continues to Face  Challenges  Addressing  Implementation  Issues Identified in  Prior Tests The Bureau Has Improved  Its Collection of Case  Notes, but Enumerators  Remain Unclear on How  to Report Fieldwork Issues We reported to the Bureau during the 2015 Census Test that information  enumerators were typing into their case notes did not appear to be  systematically used by their managers. We also reported that, during  the 2016 Census Test, the Bureau was not reviewing case notes written  by enumerators providing respondent information on better times-of-day  for future NRFU visits to their housing unit, and enumerators did not  always have this note-taking feature available. During the 2018 test,  enumerators were trained to take notes and, when appropriate, identify  special cases that would later require supervisory review. We also  observed that enumerators appeared to be consulting case notes from  prior enumerator visits when planning NRFU visits to the same housing  unit. Enumerators can use markings within their automated interview  instrument to describe certain types of cases (e.g., hearing barriers and  dangerous situations), which the automated system would then route to  receive supervisory review. Enumerators could select other case  outcomes that the automated system would apply predetermined  business rules to either reassign the cases (e.g., refusal, no one home) or  treat them as completed (verified vacant or not a housing unit). However, we identified multiple scenarios in which enumerators had  described cases in their case notes but for which the enumerators had  not selected the corresponding case flag for the situation that would have  resulted automatically in a supervisory review. One census field manager  described discovering several dozen cases that had been inactive where  enumerators had written case notes describing language barriers  encountered but had not specifically marked the flag within the device for  “language barrier.” Because these cases thus were not triggered for  supervisory review, they were eligible to be reassigned by the Bureau’s  standard automated system. As a result, the Bureau was not controlling  for the requisite language skills in assigning the cases for subsequent  enumeration attempts. The Bureau’s use of automated systems to apply business rules to  efficiently manage field casework for 2020—including identifying which  cases receive supervisory review—relies critically on field staff  understanding how to reflect what they are seeing on the ground within  the choices provided to them with which to flag cases in their interview  device. The Bureau’s use of remote management as part of reengineered  field operations also relies on enumerators knowing when and how to  report issues to their supervisors. We observed multiple field scenarios that called these conditions into  question, however. For example, enumerators we observed during NRFU  told us that they indicated address listing issues in their case notes, such  as if the unit designation was missing or incorrectly marked. Yet, these  enumerators did not know how to flag such cases in their interview  instrument to trigger supervisory review. According to Bureau officials,  this type of address listing issue turned out to be a broadly experienced  challenge within the test area. Additionally, during Group Quarters, an  enumerator we observed received supplemental information about the  number of residents at a neighboring facility after that facility had been  enumerated. The enumerator made note of this discrepancy and included  the original facility’s identifying information but was uncertain about how,  if at all, to alert the supervisor about the discrepancy. Moreover, we saw little evidence that census field supervisors or  managers were systematically reviewing case notes for the purpose of  identifying either cases not being marked properly or for which the  selected flags may not have been fully describing the case  characteristics. For example, a census field manager confirmed that case  notes recorded at Group Quarters facilities that were enumerated would  not be reviewed during clerical processing, leaving the possibility—such  as we observed—that if enumerators relied on case notes to  communicate information about the accuracy of data collected, it would  not be acted on. Bureau officials told us that reviewing all case notes  could require more staff time than budgeted for, and changing the  automation process to selectively present unflagged cases for  supervisory review could necessitate requirements changes to systems  whose development is already pressed for time. Standards for Internal Control in the Federal Government states that  agencies should use quality information by, for example, processing  reliable information to help it make informed decisions. Bureau  enumerators can record useful local knowledge about their cases with  their choice of case type flags and within their descriptive case notes.  While the Bureau has anticipated a broad range of types of cases for  enumerators to select from when documenting their casework,  enumerators we observed did not uniformly understand those options,  and the descriptors did not fully anticipate what enumerators were  encountering. Improving training and guidance to field staff on the  intended use of case notes and on alternative ways to communicate their  concerns about cases, such as flags for different types of cases, can help  ensure the Bureau has reliable data on the cases during its field  operations relying on automated interviewing instruments. Leveraging Information on  Enumerator-Reported  Technical and Procedural  Issues Remains a  Challenge During the 2015 Census Test, we reported that certain technical and  procedural problems that enumerators were encountering in the field  were going unreported and that enumerators did not always know who to  contact for assistance. We further noted that the Bureau was not  systematically assessing or tracking the extent of these issues during  testing, and we recommended that it enable such capture of information  by training enumerators on where to record issues and whom to  contact. The Bureau agreed with our recommendation. During the 2018 test, the Bureau had both an information technology (IT)  hotline and a census-field-supervisor hotline established for technical and  procedural questions, respectively. Yet, enumerators we observed told us  they did not always report to their support lines the technical issues that  they were easily able to resolve by, for example, turning their devices on  and off to reset. This lack of reporting kept the Bureau from getting  information on commonly-occurring challenges that might be useful real- time feedback in the testing environment. Moreover, a Bureau IT  manager noted that the Bureau does not formally review and share  observations and troubleshooting notes from IT hotline and census field  supervisor hotline calls. Because enumerators may call either or both  hotlines when having difficulty operating their Bureau-issued smart  phone, operators of these hotlines could be unaware of the prevalence of  or solutions to a given problem if the Bureau does not monitor  troubleshooting information across the two operational silos. For the  Bureau to be informed on any additional training needs or other  operational decisions for 2020, it will need to continue to expand its  efforts in collecting information on enumerator-reported problems per our  2015 recommendation. The Bureau Continues to  Assess How to Ensure  Efficient Staffing Levels for  Group Quarters Depending on the size of a Group Quarters facility (e.g., skilled nursing  facility, college and university student housing), the Bureau can use  varying sizes of enumerator crews to conduct an onsite count. During the  2010 Census, we observed overstaffing during the service-based  enumeration (e.g., homeless shelters and soup kitchens) portion of Group  Quarters. While determining staffing levels at these facilities can be  challenging, such overstaffing can lead to poor productivity and  unnecessarily high labor costs. We recommended that the Bureau  determine and address the factors that led to this overstaffing prior to  2020. The Bureau agreed with our recommendation. However, the Bureau has faced challenges determining the right staffing  ratios in light of complications with the Advance Contact phase of Group  Quarters. As previously noted, the Bureau used this phase to establish  facilities’ enumeration method preferences. For the 2018 test, most Group  Quarters facilities selected the facility-provided paper listing and the  eResponse enumeration options. Therefore, the Bureau allocated a large  share of its enumerator and census field supervisor workforce in the test  area to the 44 known service-based enumeration facilities, which were  restricted in terms of the enumeration options they could select and  tended to select in-person enumeration. However, only 11 of these  facilities responded to initial inquiries, so the Bureau had less work than  anticipated for its enumerator crews. At multiple sites we observed in the  test area, enumerators appeared either idle or underutilized. Moreover,  several of the Group Quarters facilities we observed had changed their  initial choice of enumeration method on the day of enumeration. Enumerator crews thus ran the risk of either being overstaffed (in the  case of switching to a facility-provided paper listing) or understaffed (in  the case of switching to an in-person enumeration). The Bureau’s Advance Contact activities have a potential benefit—if the  Bureau can get accurate information on the method of enumeration and  approximate population within a facility ahead of time, Bureau managers  and enumerator crews can more proactively allocate resources and  prepare for the count. Bureau officials said they are still assessing  outcomes of Advance Contact to see if these gains were realized and  may have completed the assessment by as early as January 2019. Doing  so will help the Bureau determine appropriate staffing sizes and thus  address our prior recommendation. The Bureau Is Working to  Determine the Causes of  Elevated Non-Interview  Rates According to preliminary data from the 2018 Census Test, the Bureau  experienced similarly high rates of cases coded as non-interviews as it  did during its last major field test of NRFU in 2016. Non-interviews are  cases where enumerators collect no data or insufficient data from  households either because enumerators made the maximum number of  visits without a successful interview, or because of special circumstances  like language barriers or dangerous situations. When this happens, the  Bureau may have to impute the census data for the case, such as  whether the housing unit is vacant or not, the population counts of the  households, or demographic characteristics of their residents. In January  2017, we reported that, during the 2016 Census Test, the Bureau  incurred what it considered high non-interview rates (31 and 22 percent  across the two test sites, respectively, as the Bureau preliminarily  reported at the time), and we recommended that the Bureau determine  the causes of these rates. Using the same method to calculate the rate of non-interviews for the test  as in 2016, the 2018 Census Test had similarly high non-interview rates— 33 percent of all NRFU cases. Bureau officials said they are still  examining causes of these elevated non-interview rates and whether final  attempts helped to mitigate the non-interview rate and will report out on  what they learn as part of their comprehensive assessment of the test,  planned for December 2018. A draft of the Bureau’s revised contact  strategy for NRFU, provided in November 2018, indicates that as part of  enumerator training in 2020 the Bureau will need to incorporate  messaging that emphasizes the importance of obtaining sufficient data  from interview attempts. Officials noted that any interim lessons learned  from this assessment process would inform updates to the field  enumeration contact strategies for 2020. The Bureau Experienced a  Programming Error While  Implementing Procedures  for Proxy Interviews Enumerators are directed to try and complete a NRFU case by  interviewing a proxy for a household respondent, like a neighbor, after  multiple failed attempts have been made to contact someone in the  household for that case. We previously observed in the 2016 Census  Test that enumerators did not seem to understand the procedures for  conducting these interviews and, as a result, underutilized the  interviewing method. In our January 2017 report, we therefore  recommended that, as part of determining the causes of its non-interview  rate, the Bureau revise and test any needed changes to proxy procedures  and associated training. The Bureau agreed with our recommendation  and subsequently developed automated supervisory performance alerts  for census field supervisors and census field managers that would inform  them when an enumerator was not following prompts to conduct proxy  interviews for eligible cases. However, in implementing proxy interview procedures for the 2018  Census Test, the Bureau experienced a technical glitch resulting in some  confusion among some enumerators and their supervisors about related  procedures. Early in NRFU data collection for the test, a programming  error within the field enumeration application was prompting enumerators  to make more than the allowable three attempts to interview a proxy  respondent. The Bureau reported promptly implementing a technical fix  to this issue; yet, enumerators we observed reported receiving varying  guidance from their supervisors on whether to abide by the erroneous  prompts. While some of these enumerators appeared to understand the  importance of attempting proxy interviews, some did not appear to  understand Bureau guidance that enumerators should make no more than three attempts to interview a proxy respondent, and some appeared  conditioned to follow the erroneous prompts. Proxy interviews can be a substantial portion of completed interviews  during the census. In 2018 NRFU testing, interviews of proxy respondents  accounted for 27 percent of all successful interview-based enumerations  of occupied housing units—compared to 24 percent during the 2010  Census and 9 percent during the 2016 Census Test. Given the role that  proxy interviews play in completing census data collection, it will be  important for the Bureau to fully implement our recommendation so that  enumerators are properly pursuing and conducting these interviews. Enumerators We  Observed Were Uncertain  of How to Proceed When  Property Managers Were  Unavailable Initial visits to property managers of multi-unit residences can help the  Bureau identify vacant and occupied housing units before sending  enumerators to individual units within the facilities. We have previously  reported that property managers can also be a helpful source of  information on respondents who are not at home, thereby making  subsequent follow-up visits to individual units more productive. During  the 2016 Census Test, we observed that enumerators were uncertain of  how to handle individual cases within a multi-unit once they were  unsuccessful in contacting a property manager initially. As a result, we  recommended in January 2017 that the Bureau revise and test procedural  and training modifications as needed to aid enumerators and their  supervisors in these cases. The Bureau agreed with this  recommendation and indicated that the evaluations of the 2018 test would  inform its strategies for 2020. However, we observed a similar issue during the 2018 Census Test in  that enumerators were unclear on what, if any, proxies to attempt if they  were unsuccessful in finding the listed property manager. Additionally, we  observed multiple enumerators leave voicemails with their contact  information—not a central number—for the listed property manager, but it  was unclear how these voicemails would produce a successful interview  because, later, the automated system could reassign other enumerators  to visit the manager. When we raised this concern with Bureau officials,  they acknowledged that they need to continue to refine procedures for  handling initial property manager visits for 2020. The Bureau Provided  Enumerators Access to  Incomplete Closed Cases,  but Enumerators Were Not  Consistently Aware They  Had Access Previously, during the 2016 Census Test, we observed that enumerators  were unable to re-open closed non-interview cases even if they happened  upon the respondent in question soon after and nearby. We noted this  inefficiency, since these cases would get re-assigned later, and in  January 2017, we recommended that the Bureau revise and test  procedures that would grant flexibility to enumerators to access cases in  these circumstances. The Bureau agreed with our recommendation. For the 2018 Census Test, the Bureau provided a list in the field  enumeration application of the cases that had been worked by the  enumerator that day but that had not been submitted for processing or  reassignment. Training for enumerators described this enumeration  option, and enumerators were authorized to access these cases when  needed, but not all enumerators we observed were consistently aware of  how to do so. Enumerators we spoke with cited uncertainty over how to  access these cases and whether enumerators were allowed to do so as  considerations. Continuing to review the procedures and guidance to  enumerators on this flexibility for completing interviews, consistent with  our prior recommendation, will help the Bureau make better use of it in  2020. The Bureau May Add an  Extra Check on NRFU  Addresses Identified by  Administrative Records as  Vacant or Nonexistent As we reported in 2017, the Bureau previously modified how it would treat  some of the households that did not respond to the 2020 Census and that  the Bureau’s use of administrative records had determined to be not  occupied. The Bureau’s earlier testing had determined that the Bureau  should require two—instead of just one—notices from the United States  Postal Service that mail could not be delivered to these households  before removing their addresses from the NRFU workload. After we  provided a draft of this report to the Department of Commerce to obtain  agency comments, Bureau officials provided us with findings from an  evaluation of the 2018 Census Test. In the evaluation, Bureau officials  observed that there were households for which they had received multiple  notices from the United States Postal Service that mail was undeliverable  but that Bureau enumerators recorded as occupied. While Bureau  officials believe, based on their follow-up research, that these addresses  may likely be vacant or not housing units, they are concerned about  possible undercounting from not enumerating people who may be at  these addresses. As of November 2018, the Bureau was considering  adding one physical visit for each of these cases. Bureau officials said  they are continuing to analyze these evaluation results and expect to  document and include changes within its final operational plan for the  2020 Census due in January 2019. Conclusions The 2018 Census Test offered the Bureau its last opportunity to test key  procedures, management approaches, and systems under decennial-like  conditions prior to the 2020 Census. As the Bureau studies the results of  its NRFU and Group Quarters testing to inform 2020, it will be important  that it address key program management issues that arose during  implementation of the test. Namely, by not establishing the intended  procedural changes for late-NRFU data collection ahead of time, the  Bureau risked not getting the most out of NRFU to minimize the number  of housing units having to have their information imputed by the Bureau  later. Additionally, by not aligning the skills, responsibilities, and  information flows for census field supervisors, the Bureau limited their role  in support of enumerators within the reengineered field operation. The  Bureau also lacks mid-operation training or guidance, which, if  implemented in a targeted, localized manner, could further help  enumerators navigate procedural modifications and any commonly- encountered problems when enumerating. Finally, without enumerators  understanding how to use case notes and flags for various types of cases  in their enumeration device and to report enumeration challenges to  supervisors and managers, the Bureau may be unaware of field work  issues that could affect the efficiency of its operations and the quality of  its data. We provided near real-time feedback to the Bureau across a range of test  implementation issues. Some, such as those related to staffing ratios for  the Group Quarters operation, build on long-standing implementation  issues that, if addressed, can contribute to the efficiency and  effectiveness of 2020 field operations. Others, like not having NRFU  progress measures that provide true indications of completed workload,  are issues specific to this test that the Bureau is assessing as part of its  2018 Census Test evaluations. It will be important for the Bureau to  prioritize its mitigation strategies for these implementation issues so that it  can maximize readiness for the 2020 Census. Recommendations for  Executive Action We are making four recommendations to the Department of Commerce  and the Census Bureau:  The Secretary of Commerce should ensure that the Director of the  Census Bureau determines in advance of Non-Response Follow-Up what  the procedural changes will be for the last phases of its data collection  and what the business rules will be for determining when to begin those  phases, which cases to assign, and how to assign them.  (Recommendation 1) The Secretary of Commerce should ensure that the Director of the  Census Bureau identifies and implements changes to align census field  supervisor screening, authorities, and information flows to allow greater  use of the census field supervisor position to provide supervisory support  to enumerators. (Recommendation 2) The Secretary of Commerce should ensure that the Director of the  Census Bureau enables area census offices to prepare targeted, mid- operation training or guidance as needed to address procedural changes  or implementation issues encountered locally during Non-Response  Follow-Up. (Recommendation 3) The Secretary of Commerce should ensure that the Director of the  Census Bureau improves training and guidance to field staff on the  intended use of case notes and flags, as well as on alternative ways to  alert supervisors and managers when case characteristics are not readily  captured by those flags. (Recommendation 4) Agency Comments  and Our Evaluation We provided a draft of this report to the Secretary of Commerce. In its  written comments, reproduced in appendix II, the Department of  Commerce agreed with our findings and recommendations and said it  would develop an action plan to address them. The Census Bureau also  provided technical comments and an update on their evaluation of the  test, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Commerce, the  Undersecretary of Economic Affairs, the Acting Director of the U.S.  Census Bureau, and the appropriate congressional committees. In  addition, the report is available at no charge on the GAO website at  http://www.gao.gov. If you or your staff have any questions about this report please contact  me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix III. Appendix I: Objectives Scope &  Methodology This report examines (1) how peak field operations were implemented  during the 2018 Census Test; and (2) the extent to which implementation  issues raised in prior 2020 Census tests have been addressed, and what  actions the Census Bureau (Bureau) is taking to address them. To address these objectives, we reviewed 2018 Census Test and 2020  Census operational planning and training documentation. We also  reviewed our prior reports and documentation on prior census testing  operations. Non-Response Follow-Up (NRFU) operations took place from  May 8 through July 31, 2018, while Group Quarters took place from July  25, 2018 through August 24, 2018, with the service-based enumeration  portion taking place July 25, 2018 through July 27, 2018. To review the Bureau’s test implementation and mitigation strategies for  previously-identified implementation issues for peak operations, we  visited Providence, Rhode Island, multiple times between May and  August 2018 to observe enumerators, census field supervisors, and  management operations. NRFU visits took place between mid-May and  late-July 2018, while we also conducted two iterations of visits of Group  Quarters in late July and early-August 2018. These multiple iterations  both across and within operations enabled us to see how, if at all,  implementation of procedures varied over time. It also enabled us to get  direct feedback from Bureau field managers on how various phases of  test operations were proceeding. These visits consisted of non- generalizable observations of field enumeration and office clerical work,  as well as interviews with local managers. For each of these visits, we  developed data collection instruments to structure our interviews and to  cover topics that were pertinent to the given phase of the operation we  were observing. We also observed debrief sessions with multiple levels of  the Bureau’s field workforce following the field work. To translate our observations into actionable feedback for the Bureau, we  shared high-level observations in near real-time to Bureau headquarters  management overseeing the operations so that the Bureau could mitigate  and adapt to known issues in a timely manner. We also discussed any  mitigation or evaluation strategies developed in response to our  observations with the cognizant Bureau headquarters officials. For  objective two specifically, we reviewed Bureau test planning  documentation and our work from prior tests to examine how, if at all, the  Bureau planned to address known implementation issues. To gain insight into how implementation was proceeding when we were  not directly observing test implementation, we received daily  management progress reports from the Bureau throughout the NRFU  operation testing that included information on the total number of NRFU  cases, the final outcomes of each case, and the number of cases that the  Bureau reported as completed for each day of the NRFU operation. We  also received Periodic Management Reports that summarized high level  outcomes of both the NRFU and Group Quarters workload. To fully understand the source of the Bureau’s daily progress reports, we  requested and received all transactional data collected during NRFU  production. We reconciled case totals and outcomes with the final  numbers in the NRFU progress reports and then used these data to  analyze the Bureau’s progress during NRFU production. We also  received and analyzed Bureau payroll data on enumerator hours worked  during NRFU operations. Specifically, we assessed the number of  enumerators working each day, the number of enumerator’ hours paid  each day, and the days of the week that were worked the most by  enumerators. We found the Bureau’s transactional and payroll data  sources to be sufficiently reliable for our reporting purposes. We conducted this performance audit from April to December of 2018 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix II: Comments from the Department  of Commerce Appendix III: GAO Contact and Staff  Acknowledgments GAO Contact Staff  Acknowledgments In addition to the contact named above, Ty Mitchell (Assistant Director),   Devin Braun, Karen Cassidy, Joseph Fread, Robert Gebhart, Krista  Loose, Kathleen Padulchick, Lisa Pearson, Kayla Robinson, Robert  Robinson, and Cynthia Saunders made significant contributions to this       report.
The cost of the decennial census has steadily increased over the past several decades, with self-response rates declining over the same period. The largest and costliest operation that the Bureau undertakes, NRFU is the Bureau's attempt to enumerate households not initially self-responding to the census. GAO was asked to review NRFU implementation during the 2018 Census Test as well as the Bureau's overall readiness for peak field operations, which cover the actual enumeration of residents. This report examines (1) how peak field operations, including NRFU, were implemented during the test; and (2) the extent to which prior test implementation issues have been addressed. GAO reviewed test planning and training documentation, as well as production and payroll data. At the test site, GAO observed and interviewed enumerators, field supervisors, and managers conducting peak operations. In preparation for the 2020 census, the Census Bureau (Bureau) set out to enumerate over 140,000 housing units during the 2018 Census Test at a site in Providence County, Rhode Island. The 2018 Census Test marked the Bureau's last chance to test enumeration procedures for peak field operations under census-like conditions before 2020. Implementation of this test identified the following concerns: The Bureau experienced operational issues during implementation of the Non-Response Follow-Up (NRFU) as part of the 2018 Census Test. For example, the Bureau had not finalized procedures for data collection during late phases of NRFU (e.g., after multiple attempts to interview had been made) until after the work had already started. As a result, enumerators and their supervisors did not have standardized procedures during the test, which made it difficult to evaluate the effectiveness of the test procedures. GAO also observed a range of other NRFU implementation issues during the test, such as the Bureau's use of progress reporting that overstates the number of NRFU cases not needing any additional fieldwork and the Bureau having fewer of its enumerators work Saturdays, which can be among the most productive interview days. The Bureau is taking steps to assess and mitigate these and other issues that GAO identified. The Bureau's field workforce was not fully prepared to face all of the enumeration challenges that arose during the test. For instance, the Bureau expects census field supervisors to provide front-line coaching to enumerators but did not screen these employees to ensure they had the needed skills. Moreover, it did not provide them with the authorities and information that would have helped them serve that role. As a result, we believe that supervisors did not have the casework expertise, information, or authority to help enumerators with procedural questions, and higher-level census field managers ended up providing direct support to enumerators. While the Bureau provided extensive online and in-person training to enumerators prior to NRFU fieldwork for the 2018 Census Test, the Bureau lacked any standardized form of mid-operation training or guidance as new procedures were implemented. GAO observed that during the test some enumerators continued to have questions and were uncertain about procedures. Developing targeted, location-specific training could help ensure that, in 2020, enumerators receive the guidance they need to collect census data consistently and in accordance with NRFU procedures. The Bureau has made progress addressing prior test implementation issues but still faces challenges. For example, the Bureau improved its collection of enumerator case notes, which reflect real-time knowledge gained during enumeration. However, enumerators did not always report cases using flags built in to their interviewing device that would benefit from supervisory review, such as for language barriers. Moreover, supervisors were not systematically analyzing case notes to identify cases not flagged properly. As a result, critical data on fieldwork challenges were not being communicated effectively to those who could analyze and use them.