Background DHS’s Homeland Security  Grant Program The federal government has provided financial assistance to public and  private stakeholders for preparedness activities through various grant  programs administered by DHS through its component agency, FEMA.  Through these grant programs, DHS has sought to enhance the capacity  of states, localities, and other entities, such as ports or transit agencies,  to prevent, prepare for, protect against, respond to, and recover from, and  mitigate a natural or manmade disaster, including terrorist incidents. Two  of the largest preparedness grant programs are the SHSP and UASI  grant programs. SHSP grants provide federal assistance to support states’  implementation of homeland security strategies to address the  identified planning, organization, equipment, training, and exercise  needs at the state and local levels to prevent, prepare for, protect  against, and respond to acts of terrorism. SHSP grants are annually  awarded to all the nation’s 56 states and territories. SHSP grant  awards are calculated in two parts. All states and territories are to  receive a minimum grant amount required by law, based on a  percentage of the total amount of SHSP and UASI appropriations in a  given fiscal year. The remaining award amounts are based on  FEMA’s risk-based grant assessment model. UASI grants provide federal assistance to address the unique needs  of high-threat, high-density urban areas, and assists the areas in  building an enhanced and sustainable capacity to prevent, prepare  for, protect against, respond to acts of terrorism. Since 2015,  Congress has instructed through the Explanatory Statements  accompanying the annual DHS Appropriations Acts that the UASI  grants should be awarded to urban areas that reflect up to 85 percent  of nationwide risk. For the UASI program, FEMA uses the risk-based  grant assessment model each year to identify those urban areas that  will be eligible to receive funding. Annual funding for the SHSP and UASI programs have generally declined  over the period of fiscal years 2008 through 2018, but have remained  consistent since fiscal year 2016. Figure 1 shows the changes to SHSP  and UASI programs’ annual funding during this period. For example,  annual funding for SHSP decreased from about $861 million in fiscal year  2008, to $402 million in fiscal year 2018. During this same period, annual  funding for UASI also declined, from about $782 million in fiscal year  2008 to $580 million in fiscal year 2018. However, annual funding for the  UASI program has been higher than the SHSP program since fiscal year  2010. FEMA’s Risk-based Grant  Assessment Model for  Distributing Funding  Awards Risk=Threat x Vulnerability x Consequence  Threat–A natural or man-made occurrence,  individual, entity, or action that has or  indicates the potential to harm life,  information, operations, and/ or property  Vulnerability–Physical feature operational  attribute that renders an entity, asset, system,  network, or geographic area open to  exploitation or susceptible to a given hazard.  Consequence–Effect of an event, incident, or  occurrence, commonly measured in four  ways: human, economic, mission, and  psychological, but may also include other  factors such as impact on the environment. FEMA’s risk-based grant assessment model uses three variables: Threat,  Vulnerability, and Consequence. The purpose of this model is to apply a  risk management process to provide a structured means of making  informed trade-offs and choices about how to use finite resources  effectively, and monitoring the effect of those choices. Specifically,  inherent “uncertainty” is associated with any effort to develop a risk model  such as assessing the risk of terrorist attacks, and thus, requires the  application of policy judgments and analytic assumptions. The effect that  uncertainty has on the results of the risk model can be especially  important if the model produces materially different results in response to  even small changes in assumptions, often referred to as the “sensitivity”  or “robustness” of a model’s assumptions and results. As we reported in June 2008, FEMA’s risk-based grant methodology and  its continuous improvement efforts in estimating risk were part of a  reasonable process to assist in determining SHSP and UASI grant  allocations. For example, the risk-based grant assessment model used  from fiscal year 2001 through 2003 largely relied on measures of  population to determine the relative risk of potential grantees, and  evolved to measuring risk as the sum of threat, critical infrastructure and  population density calculations in fiscal years 2004 and 2005. Further, the  fiscal year 2006 process introduced a risk assessment model that  included measures of Threat, Vulnerability and Consequences. In June 2008, we reported that the way the risk-based grant assessment  model measured vulnerability across states and urban areas was  limited. We found that the model considered all states and urban areas  equally vulnerable to a successful attack, and as a result, the final risk  scores were determined exclusively by the Threat and Consequence  scores. Specifically, the risk model did not measure vulnerability for each  state and urban area; rather it assigned a vulnerability score of 1.0 to  every state and urban area. We recommended that DHS and FEMA  formulate a methodology to measure variations in vulnerability across  states and urban areas. DHS components concurred with our  recommendation to measure vulnerability in a way that captures  variations across states and urban areas and apply this measure in future  iterations of FEMA’s model. In August 2011, FEMA reported that the  agency, in coordination with other DHS components, established a  Vulnerability Index for the fiscal year 2011 risk-based grant assessment  model to better capture the risk to states and urban areas, thereby  addressing our recommendation. Other Reviews of FEMA’s  Risk Methodology DHS and the National Research Council (NRC) have also performed  reviews of FEMA’s risk assessment methodologies, providing their own  conclusions and recommendations, since our 2008 review. For example,  in 2010, the NRC reported that FEMA should strengthen its scientific  practices, such as documentation, analyses to determine how changes to  a model could affect its results, and peer review by technical experts  external to DHS, in order to further develop an understanding of the  uncertainties in its terrorism-related risk analyses. Additionally, in 2016, the Homeland Security Advisory Council reported  that processes by which FEMA uses to assess risk should be made more  inclusive, comprehensive and effective. The Homeland Security  Advisory Council recommended the following actions to strengthen this  process: FEMA should continue to send risk profiles to states and urban areas  to promote timely and meaningful feedback, and enable FEMA to  evaluate recommended adjustments. Before each year’s budget submission, FEMA should discuss with  congressional appropriators the current grant allocation mechanism. We discuss FEMA’s progress in implementing these recommendations  later in this report. Various Factors  Affected SHSP and  UASI Grant  Allocations to States  and Urban Areas  From Fiscal Years  2008 Through 2018 SHSP Allocations Reflect  Both a State’s Relative  Risk Score and the  Minimum Allocation by  Law While all states and territories receive minimum SHSP program grant  allocations by law, the risk-based grant assessment model also informs  the grant allocation of the remaining funds to each state. However, for a  majority of states each year, their SHSP grant awards are primarily based  on a legal minimum amount. For example, in fiscal year 2012, 34 states,  like New Mexico, were awarded $2,801,000, which included $2,745,000  based on the minimum amount by law, and $56,000 was based on its risk  level. By contrast, New York was one of the high-risk states based on the  risk model. For that same fiscal year (2012) New York received a total of  $55,610,000, which included $2,745,000 based on the minimum amount  by law, plus $52,865,000 based on its risk level. Over the period from fiscal years 2008 through 2018, the number of low- risk states whose SHSP grant awards were primarily based on the legal  minimum amount had varied from year to year, from 19 states in fiscal  year 2008, to 37 states in fiscal year 2018, as shown in table 1. In addition, from fiscal year 2008 through fiscal year 2018, there was a  decrease in the percent of total SHSP funds awarded to states and  territories based on FEMA’s risk model. The percent of total SHSP  funding awarded to states and territories based on FEMA’s model ranged  from a high of 63 percent in fiscal year 2009 (about $536 million of the  $851 million of total SHSP funds), to 51 percent (about $149 million of  $294 million of total SHSP funds) for fiscal year 2012. For fiscal year  2018, the total SHSP funds awarded to states and territories based on the  risk-based grant assessment model was 55 percent—about $220 million  of $402 million. For specific details on SHSP grant allocations for fiscal years 2008  through 2018 by states and territories, see appendix I, table 4. UASI Grantee Eligibility  and Allocations Reflect  Results from FEMA’s Risk- Based Grant Assessment  Model The UASI program uses FEMA’s risk-based grant assessment model to  identify which of the 100 of the nation’s largest urban areas are eligible for  grant awards in a particular fiscal year. Then, FEMA’s risk model also  helps inform DHS leadership’s decisions on the final funding amounts for  each grantee, according to FEMA officials. Specifically, FEMA annually  assesses the risk of the 100 most populous metropolitan statistical  areas—a geographical region with a relatively high population density at  its core and close economic ties throughout the area—as defined by the  Office of Management and Budget, in determining the eligible urban  areas. From these 100 eligible urban areas, the risk-based grant  assessment model identifies those urban areas that reflect recent  congressional intent that up to eighty-five percent (85%) of nationwide  risk is funded each year. Those urban areas below this 85 percent  threshold are ineligible for UASI grant awards in that fiscal year,  according to FEMA officials. From fiscal years 2008 through 2018, the number of UASI grantees has  remained relatively stable since fiscal year 2011. As figure 2 shows, the  annual number of grantees has fluctuated from fiscal years 2008 through  2018, ranging from 60 to 64 grantees during fiscal years 2008, 2009 and  2010. However, since fiscal year 2011 the number of UASI grantees has  averaged 31 urban areas, with a high of 39 urban areas in fiscal year  2014 and a low of 25 urban areas in fiscal year 2013. For fiscal year  2018, 32 urban areas were UASI grantees. For additional details on UASI grant awards for fiscal years 2008 through  2018 by urban areas, see appendix I, table 5. Because the UASI grant program is required by annual congressional  guidance to fund only those urban areas that comprise up to 85 percent  of risk nationally, this eligibility cut off can result in different urban areas  being eligible from one year to the next. Specifically, as we demonstrated  in June 2008, the variation of risk across urban areas takes on the  distribution curve illustrated in figure 3. The few urban areas with the highest relative risk score are represented  along the steep part of the relative risk curve. For example, those urban  areas receiving the highest awards, informed by their risk scores and  ranks, are generally the same each fiscal year: New York City, Los  Angeles, and Chicago, as seen in table 2. Those urban areas that have less relative risk are represented along the  flat section of the curve. There are urban areas with less risk that may not  fall within the 85 percent of risk nationally during a specific year and thus  would be ineligible to receive UASI funding during that year. Table 3 lists  the lowest-funded urban areas for the last 5 fiscal years, based on our  analysis of the funding amounts each received within each fiscal year. For  example, during the period of fiscal year 2008 through fiscal year 2018  San Antonio, Texas, and Hampton Roads, Virginia only received awards  in fiscal years 2008, 2009, 2014, 2017, and 2018. In addition to changes to urban areas’ risk ranking from one year to the  next, the amount that an urban area received of the total amount of UASI  funds in a given year can change. FEMA has established a process for  developing grant award funding options based on the results of the risk- based grant assessment model. These funding options are provided to  the Secretary of Homeland Security for consideration and final approval.  According to FEMA officials, the options may vary each year based on  DHS leadership’s priorities and concerns at the time; however, all options  represent only those eligible grantees that represent up to 85 percent of  the nation’s risk, as determined by the risk-based grant assessment  model.  In fiscal year 2013, FEMA shifted its UASI grant funding to a  process referred to as “funding bands.” In fiscal year 2018, for example,  UASI grantees such as Orlando, Florida; Hampton Roads, Virginia; and  San Antonio, Texas each received a $1.5 million UASI grant, whereas a  grouping of UASI grantees that included Sacramento, California;  Pittsburgh, Pennsylvania; and Portland, Oregon each received $2.5  million. According to FEMA officials, grouping jurisdictions with similar risk scores  into funding bands is an effort to stabilize and retain grantees’ funding  levels over multiple years, as annual UASI grants will fund projects that  are multiyear investments and carried out over a 24 to 36-month  performance period. For example, if one jurisdiction increased by four  ranks and another jurisdiction in the same group dropped six ranks, the  two jurisdictions would stay in the same funding band if the overall risk  scores remained close together. The purpose of the funding bands is to  ensure that some consistency in funding exists for jurisdictions, given  minor changes in the relative risk ranking. FEMA looks at the natural risk  breaks and historical grant allocation data for each year. For example,  each year FEMA presents for consideration by DHS leadership the  historical funding and the number of urban areas that have been placed in  specific funding bands in prior grant years, if any, and the differences  between the relative risk scores in the current fiscal year. According to  FEMA officials, the last few grant years had produced similar funding  bands, which are subject to change depending on DHS leadership’s final  decisions. FEMA Has Improved  Its Risk-based Grant  Assessment Model,  but Additional Steps  Could Further  Strengthen Its Model FEMA Has Taken a  Number of Steps to  Improve the Risk-based  Grant Assessment Model  for Allocating SHSP and  UASI Grants Since 2008, FEMA has taken a number of steps to assess and improve  its risk-based grant assessment model for allocating grants based on past  reviews, our prior recommendations, and various changes related to  evolving terrorist threats and real-world scenarios. For example, FEMA  added a Vulnerability Index to its risk model in 2011 in response to our  2008 recommendation. Most recently, for fiscal year 2018, FEMA has  included a “soft target index.” According to FEMA officials, this index was  added to account for the current threat for areas where crowds  congregate. Figure 4 illustrates the timeline of FEMA changes to the risk- based assessment model and prior assessments. Figure 5 depicts the risk-based grant assessment model used for fiscal  year 2018 SHSP and UASI grant awards. Figure 6 depicts the changes in the Threat, Vulnerability, and  Consequence indexes used in the risk-based grant assessments model  for fiscal year 2008, compared to 2018. As we noted above, the 2008 risk  model did not measure Vulnerability for each state and urban area, and  risk scores were essentially determined by Threat and Consequences  indexes. Changes to the Consequence Index can have the most impact on the  relative risk scores because of the weight of this index (50 percent),  relative to the weights for the Threat and Vulnerability indexes. Further,  the weight for population within the Consequence Index represented 30  percent of the total fiscal year 2018 risk model value. As a result, the  weight for the population index was greater than the weights of either the  Threat Index or Vulnerability Index, each 25 percent. FEMA has  decreased the weight for the population index over time, from 40 percent  in 2008 to 30 in 2011, where it has remained consistent through 2018. For fiscal year 2018, FEMA modified how the population index was  calculated within the Consequence Index to better account for attacks  staged by individuals, so-called lone wolves. FEMA did so, in part, by  reducing the importance of population density within the population index.  In past risk models, the population index had favored high-density, high- rise urban areas, commensurate with building destruction scenarios —  the 9/11-style attack scenarios that focused on large building destruction  events, according to FEMA officials. The 2018 change to cap population  density in the population index reduces the impact those extremely-dense  population areas have in the methodology, according to FEMA officials. The other measures used to make up the Consequence Index remain  relatively unchanged since our review in 2008, although FEMA has  renamed the indexes. Vulnerability Index As explained earlier, FEMA added a Vulnerability Index to its risk-based  grant assessment model in 2011, in response to our 2008  recommendation. According to FEMA officials, the Vulnerability Index  helps support what DHS is trying to protect, primarily the protection of  citizens and critical infrastructure. For example, the Vulnerability Index  includes a measure designed to assess the extent that certain types of  national critical infrastructure assets may be considered for possible  attack. This Targeted Infrastructure Index measure uses actionable  intelligence on types of critical infrastructure targets, such as aviation,  mass transit and commuter rail. FEMA works with DHS’s National  Protection and Programs Directorate to match its critical infrastructure  dataset to actionable intelligence from DHS’s Office of Intelligence &  Analysis to compile this measure. Vulnerability Index   Designed to measure the likelihood of a  successful attack in a state or urban area,  based on a) intelligence information of those  critical infrastructure assets identified by  foreign or domestic terrorists; b) the extent of  international borders entries (land, sea and  air) located in a state or urban area, and c)  special events where crowds congregate and  are susceptible to homegrown extremism and  lone wolf attacks. For the fiscal year 2018 grant, FEMA has included a “soft target index.”  According to FEMA officials, this index was added to account for the  current threat for areas where crowds congregate. Based on previous  feedback received through this process, FEMA updated the fiscal year  2018 risk methodology to better account for the nation’s current threat  environment. The soft target index is composed of two new data  elements: Visitors—domestic and international—using the same data used in  the calculation of the Population Index; and Special events measure—uses Special Event Assessment Rating  data from DHS Office of Operations Coordination to identify large  events that are state and local events that may require federal  assistance. Examples of such events include the Super Bowl, the  Boston Marathon and New Year’s Eve in Times Square. In fiscal year 2018, FEMA added a new “isolation” measure to account for  the challenges of response for those states, territories, and urban areas  outside the contiguous United States, who rely on prompt mutual aid from  neighboring jurisdictions. According to FEMA officials, the isolation data  element was included as a response to challenges the agency witnessed  as a result of the 2017 Hurricane season, specifically the unique  challenges of distant U.S. territories receiving timely mutual aid from other  states. For example, if Hawaii, Guam or American Samoa were attacked,  there would be little to no outside help for a number of days. As a result,  FEMA modified the fiscal year 2018 Border Crossings data element  weight, which was dropped from 6 percent to 4 percent, in order to  establish a 2 percent weight for the isolation measure. Threat Index The weight of the Threat Index was raised from 20 percent to 30 percent  in fiscal year 2011, and has been modified again for fiscal year 2018.  Specifically, according to FEMA and DHS officials, DHS leadership made  a policy decision to reduce the Threat Index’s weight from 30 percent in  2017, to 25 percent in 2018, due to the change in current threat  environment, since Congress directed FEMA in the Explanatory  Statement accompanying the FY 2017 DHS Appropriations Act to review  the risk model to account for this changing threat environment. FEMA  officials further stated that they assumed, as domestic terrorism and soft  targets are considered to be prevalent nationwide and pose more of a  challenge in identifying the source of actionable threats. FEMA officials  stated that this modification to the Threat Index better reflects real-world  scenarios. Since fiscal year 2012, FEMA has included information on domestic  terrorism as well as international terrorism in its Threat Index. According  to DHS officials, home grown extremism is also a likely threat, often  through lone wolf attacks. DHS officials decided to assign all urban areas  a minimum threat score to reflect the fact that all areas have some level  of threat. According to DHS officials, the addition of a domestic terror  threat measure resulted in a decrease in the variation of threat scores  across states and urban areas. According to DHS officials, lone wolf  attacks are difficult to determine who the actors may be, or when and  where they will attack. Stakeholder Feedback FEMA annually transmits risk profile information to states and urban  areas to promote timely and meaningful feedback. According to FEMA  officials, draft risk profiles are sent to all 56 states and territories and 100  eligible urban areas closely after the enactment of DHS’s annual  appropriations. States and urban areas are given a 2-week period prior to  the release of the Notices of Funding Opportunity to review their draft risk  profiles and provide FEMA any comments or data corrections that should  be considered. According to FEMA officials, it encourages and welcomes  stakeholders to make suggestions for new or different data sets for the  subsequent fiscal year's risk assessment at any time during the year  convenient to the stakeholder. FEMA also conducts webinars during this  period to can explain the risk profiles in detail, as well as discuss any  updates to data sets and/or any enhancements to the risk assessment.  This will often result in feedback on data elements and the methodology  of the risk-based grant assessment model, according to FEMA officials.  According to FEMA officials, this feedback process has been used to help  guide FEMA’s consideration of enhancements to the risk-based grant  assessment model. For example, FEMA officials noted that this process  helped them in their efforts to develop the soft targets index into the 2018  risk model. FEMA Does Not Fully  Make Use of Recognized  Scientific Practices in  Maintaining Its Risk  Assessment Model In 2010, the National Research Council (NRC) recommended that  incorporating scientific practices can provide decision makers a further  understanding of the effects of its policy judgments and assumptions—i.e.  addressing uncertainties—in its terrorism-related risk analyses. The NRC  identified “good scientific practice” for model-based work. Specifically, the  NRC recommended that detailed documentation for all risk models,  including rigorous mathematical formulations, be implemented  department-wide. Additionally, the NRC recommended that all risk  models undergo verification and validation—or a sensitivity analysis at the  least—of its risk-based grant assessment model. Finally, the NRC  recommended that FEMA should undertake an external peer review by  technical experts outside of DHS, and review its risk-informed formulas in  order to identify issues such as logic flaws, evaluate the ramifications of  the choices of weightings and parameters, and improve the risk model’s  transparency. However, FEMA has not fully adopted these scientific  practices for its risk-based grant assessment model. Documentation: FEMA documentation on the sources of data used for  the model’s calculations does not include information that would enable a  reviewer to understand the underlying assumptions that form the basis for  its risk-based grant assessment model—such as the size of the weights  assigned to Threat, Vulnerability, and Consequence, or the justification  for changes to these weights from one year to the next. FEMA officials  stated that they focus their limited time and resources on developing the  executive summary-level materials that DHS leadership will use to  determine final grant eligibility and grant allocation amounts. Also, to a  lesser extent, FEMA officials said they rely on the expertise of the subject  matter experts from DHS’s Office of Intelligence and Analysis, and DHS’s  National Protection and Preparedness Division’s Office of Cyber and  Infrastructure Analysis, parts of DHS that contribute to the annual risk  assessment process. In April 2018, we identified documentation as one of the key  methodological elements to the baseline structure of an economic  analysis. Specifically, the elements include that the analysis is clearly  written with a plain language summary, has clearly labeled tables that  describe the data used and results, and has a conclusion that is  consistent with these results. The analysis cites all sources used and  documents that it is based on the best available economic information.  The analysis documents that it complies with a robust quality assurance  process and, where applicable, the Information Quality Act, and should  disclose the use and contributions of contractors and outside consultants.  FEMA officials agreed with our analysis of FEMA’s supporting  documentation, and officials stated that maintaining additional  documentation could further assist reviewers. Documenting how subject  matter expert assumptions are made would help FEMA increase the  transparency of the model for key internal and external stakeholders. In-Depth Analyses: Similarly, we could not determine whether FEMA  sufficiently performed all the analyses of the model’s sensitivity needed to  determine how changes to its risk-based grant assessment model could  affect the resulting risk scores. FEMA officials stated that they have only  analyzed the effect of a data element when it has been added to the  model (e.g.: the Soft Target Index in 2018). Further, FEMA officials were  unable to provide us with documentation on their sensitivity analyses  processes or their results. DHS’s Risk Lexicon states that sensitivity analysis can be used to  examine how individual variables can affect the outputs of risk  assessment methodologies. In addition, OMB Circular A-94 recommends  that the outcomes from a risk model should be analyzed to determine  how sensitive such outcomes are to changes in the model’s  assumptions. The assumptions that deserve the most attention will  depend on the dominant elements and the areas of greatest uncertainty  of the program being analyzed. In addition, research in the actuarial  sciences also states that sensitivity analysis “is of fundamental  importance to risk analysts, especially in the presence of complex  computational models with uncertain inputs.” As we stated earlier, understanding the extent that uncertainty has on the  results of the model can be especially important if the model produces  materially different results in response to even small changes in  assumptions—often referred to as the “sensitivity” or “robustness” of a  model’s assumptions and results. We have reported on FEMA’s risk- based grant assessment model in June 2008 and March 2013, where we  found grant years when the risk model was sensitive to even small  changes. For example, we noted that a potential increase or decrease  in a measure would have resulted in one urban area displacing the  eligibility of another, thereby potentially shifting funding as well. FEMA  officials stated that they focus their limited time and resources on  developing the executive summary-level materials that DHS leadership  will use to determine final grant eligibility and grant allocation amounts.  FEMA officials agreed that they could better document the steps used in  their analyses across all the model’s measures and weights so that a  complete understanding of potential impacts are documented and can be  made available to leadership when making decisions about changes. FEMA’s implementation of sensitivity analyses could help the agency to  assess changes to the risk-based grant assessment model including the  introduction of new data elements into Threat, Vulnerability, and  Consequence indexes, the modifications to how existing data elements  are calculated, and the changing of the weights assigned to the Threat,  Vulnerability, and Consequence indexes. Further, FEMA’s  implementation of sensitivity analyses has the ability to show decision  makers the impact or predicted impact of adjustments to FEMA’s risk- based grant assessment model, including with potential shifts in funding  towards or away from certain grantees. Use of External Peer Review: FEMA has not subjected its risk-based  grant assessment model to a peer review by independent, external  technical experts, as previously recommended in 2010 by the NRC.  According to FEMA officials, its risk assessment methodology has  undergone comprehensive internal reconsideration over time to better  reflect real-world scenarios, but such reviews have not included external  peer reviews. FEMA officials stated that its risk-based grant assessment  model has gone through past reviews including a review as part of DHS’s  quadrennial review in 2014, and the model is reviewed by internal subject  matter experts from DHS’s Office of Intelligence and Analysis, and DHS’s  National Protection and Preparedness Division’s Office of Cyber and  Infrastructure Analysis as part of the annual risk assessment process.  FEMA officials stated that the agency is exploring the possibility of  participating in a DHS collaborative group to internally review and provide  feedback on the model’s underlying assumptions and methods. Such a  group could review the underlying components of the current risk-based  grant assessment model and suggest improvements, as well as present  and evaluate other risk assessment theories and approaches. FEMA  officials told us they have encountered time and resources constraints on  establishing an external peer review process. As we have previously reported, independent external peer reviews can  increase the probability of success by improving the technical quality of  projects and the credibility of the decision-making process, and provide  reasonable assurance that the agency’s approach is reproducible and  defensible. In addition, in December 2004, OMB issued the  memorandum “Final Information Quality Bulletin for Peer Review” which  established government-wide guidance aimed at enhancing the practice  of peer review of government science documents. OMB noted that peer  review can increase the quality and credibility of the scientific information  generated across the federal government, which was an effort to improve  the quality of the scientific information upon which policy decisions are  based. OMB also noted that, while peer review may take a variety of  forms, agencies will need to consider at least the following issues when  coordinating an external peer review: individual versus panel review;  timing; scope of the review; selection of reviewers; disclosure and  attribution; public participation; disposition of reviewer comments; and  adequacy of prior peer review. These scientific processes are designed to help decision makers better  understand the impact or predicted impact of risk management  alternatives, and provide greater confidence in the reliability of the risk  assessment model’s results. Full implementation of these processes  better position FEMA to provide further assurances that their risk-based  grant assessment model and grant allocation approaches are reasonable,  of high-quality, and credible. Conclusions Given that risk management has been endorsed by the federal  government as a way to direct finite resources to states and those urban  areas that are most at risk of terrorist attack, it is important that FEMA’s  risk-based grant assessment model supports the application of policy  judgments and analytic assumptions in the model’s role of allocating  those limited resources. Decreased funding levels for SHSP and UASI  grant programs have increased the importance of using risk management  techniques to more effectively target finite federal dollars. DHS and FEMA  have strengthened its risk-based grant assessment model for allocating  grants, taking into account analysis and recommendations from a variety  of reviews. These improvements include the addition of a Vulnerability  Index and modifications to the Threat Index. We have identified  opportunities where FEMA could strengthen its scientific practices. First,  documenting the model’s underlying assumptions and the results of  sensitivity analysis can assist decision makers in better understanding the  predicted impact of risk management alternatives. Second, expanding the  use of sensitivity analysis could further enhance the model. Developing a  greater understanding of the how uncertainty affects its risk-based grant  assessment model’s results helps achieve the objectives of risk  management. Third, coordinating an independent external peer review of  the methodology of its risk-based grant assessment model would better  position the agency to provide reasonable assurance that FEMA’s risk  model and grant allocation approach that FEMA uses for its SHSP and  UASI programs are reasonable, of high-quality, and credible. Applying  such scientific practices could assist FEMA in further strengthening its  risk-based grant assessment model. Recommendations for  Executive Action We are making the following three recommendations to FEMA. The FEMA Administrator should fully document the underlying  assumptions and justifications that form the basis of the risk-based  grant assessment model, such as the size of the weights assigned to  Threat, Vulnerability, and Consequence, or the justification for  changes to these weights from one year to the next. The FEMA Administrator should perform sensitivity analyses to verify  how changes to the risk-based grant assessment model could affect  the resulting risk scores, and document the results. The FEMA Administrator should take steps to coordinate an  independent, external peer review of its risk-based grant assessment  model. Agency Comments  and Our Evaluation We provided a draft of this product to the FEMA and DHS for comment.   In its comments, reproduced in appendix II, FEMA generally concurred  with our findings and three recommendations. In FEMA’s concurrence to our first recommendation that the agency fully  document the underlying assumptions and justifications that form the  basis of the risk-based grant assessment model, FEMA requested that  GAO consider this recommendation resolved and closed as implemented.  As part of FEMA’s response, they reiterate their process of providing draft  Risk Profiles to all 100 urban areas and 56 states and territories and their  annual communications to Congress on how FEMA calculated risk and  computed grant awards.  We recognized FEMA’s stakeholder feedback  efforts in this report. However, as we noted, FEMA’s documentation on  the sources of data used for the model’s calculations does not include  information that would enable a reviewer to understand the underlying  assumptions that form the basis for its risk-based grant assessment  model.   Further, as stated earlier, documentation is one of the key  methodological elements to the baseline structure of this type of analysis,  documenting that it complies with a robust quality assurance process and,  where applicable, the Information Quality Act, and should disclose the  use and contributions of contractors and outside consultants. In order to  fully implement this recommendation, documenting how subject matter  expert assumptions are made would help FEMA increase the  transparency of the model for key internal and external stakeholders, and  will further support the efforts of an independent external peer review of  FEMA’s risk-based assessment model. Regarding the second recommendation, FEMA concurred, stating that the  agency will expand the use of sensitivity analysis to review the entire risk  methodology, and will also document these results for leadership review,  as appropriate.  Finally, regarding the third recommendation, FEMA  concurred, stating that they will coordinate an independent external peer  review and develop a detailed written response to leadership for further  appropriate action. FEMA and DHS also provided technical comments, which we  incorporated as appropriate. We are sending copies of this report to the appropriate congressional  committees, the Secretary of Homeland Security, and other interested  parties. This report will also be available at no charge on our Web site at  http://www.gao.gov. Should you or your staff have any questions  concerning this report, please contact me at (202) 512-8777 or  CurrieC@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report. Appendix I: Grant Funding and Awards for State  Homeland Security Grant Program (SHSP), and  the Urban Area Security Initiative (UASI) Grant  Program for Fiscal Years 2008 Through 2018 Fiscal Years  Mississippi  Total Award  Amount above  legal minimum Fiscal Years  West Virginia  Total Award  Amount above  legal minimum Appendix II: Comments from the Department  of Homeland Security Appendix III: GAO Contact and Staff  Acknowledgments GAO Contact Chris P. Currie, at (202) 512-8777 or CurrieC@gao.gov. Staff  Acknowledgments In addition, key contributors to this report were Aditi Archer, Chris  Keisling, Assistant Director, John Vocino, Analyst-in-Charge, Chuck  Bausell, Dominick Dale, Dorian Dunbar, Eric Hauswirth, Serena Lo, Heidi  Nielson, and Hadley Nobles.
FEMA, a component of DHS, provides preparedness grants to state, local, tribal, and territorial governments to help prepare for, prevent, protect against, respond to, recover from and mitigate terrorist attacks or other disasters. SHSP grants fund the nation's 56 states and territories, while UASI grants fund eligible urban areas. Grant allocations have been based, in part, on FEMA's risk-based grant assessment model, with states and urban areas deemed to be at higher risk receiving more grant dollars than those deemed at lower risk. Since 2008, GAO and others have assessed the model and made recommendations to strengthen it. This report 1) describes SHSP and UASI grant awards during fiscal years 2008 through 2018, and factors affecting grant distributions; and 2) examines the steps that FEMA has taken to strengthen its risk assessment model for allocating preparedness grants, and any additional opportunities to improve the model. GAO analyzed the information in FEMA's model, and data on SHSP and UASI grant awards for fiscal years 2008 through 2018. GAO also interviewed FEMA and DHS officials and collected documents. GAO found that various factors affected Federal Emergency Management Agency (FEMA) State Homeland Security Program (SHSP) and Urban Area Security Initiative (UASI) grant awards from fiscal year 2008 through 2018. SHSP grant awards to states were based on two factors—(1) minimum amounts set in law each year, and (2) FEMA's risk model. For example, in fiscal year 2012, each state was to receive a minimum of approximately $2.74 million, with each state receiving additional funds based on its relative risk score. Conversely, UASI grant awards are made based on its FEMA's risk-based grant assessment model, which ranks each urban area relative to others in that year, and Department of Homeland Security (DHS) leadership decisions on how funding should be allocated. From fiscal year 2008 through 2018, the number of USAI grantees varied from year to year (see figure below). Since 2008, FEMA has taken steps to strengthen its risk-based grant assessment model, but has not incorporated additional scientific practices into its model. For example, in 2011 FEMA included more information in its model on potential targets and their vulnerability in each state and urban area, addressing a prior GAO recommendation. More recently in 2018, FEMA added additional factors to better assess vulnerability in each state and urban area, such as the number of special events where large crowds gather and soft targets susceptible to lone wolf attacks, among other things. However, GAO found that FEMA does not fully utilize scientific practices recognized by the National Research Council and the Office of Management and Budget as best practices. Specifically, FEMA did not fully document its model's underlying assumptions, such as the weights in its model or the justification for changes to these weights. FEMA also did not perform the level of analysis needed to determine how changes to its model could affect the resulting risk scores. Finally, FEMA has not coordinated an independent external peer review of its model. Applying such scientific practices could assist FEMA in further strengthening its model.