# Observations of peak electric load growth in ERCOT with the rise of
electrified heating and its implications for future resource planning

Matthew J. Skiles  Joshua D. Rhodes, PhD  Michael E. Webber, PhD  Walker
Department of Mechanical Engineering, The University of Texas at Austin, 204 E
Dean Keeton St, Austin, TX 78712, USA

###### Abstract

This analysis quantitatively compares the evolution in summer and winter peak
demands in the Electric Reliability Council of Texas (ERCOT) service area from
1997 through 2021. Weather data for the days in which peak demand occurred
were also compiled to investigate the relationship between peak heating and
cooling loads and ambient temperature. This relationship was then applied
along with population projections and a climate scenario with medium to high
radiative forcing to create winter and summer peak demand growth scenarios for
2025 through 2050. This analysis informs resource planners about how ERCOT
peak demand might change in the future and provides new insight into how
electric load growth and non-flexible electrified heating demand could have
contributed to the February 2021 ERCOT blackouts. We found that historically,
summer peak demand growth has been generally stable and approximately linear
with time. The stable summer peak load is likely a consequence of fairly
constant temperatures observed on summer peak demand days. Conversely, the
winter peak demand growth has been less consistent, varying much more around
the broader trend. This phenomenon is likely a consequence of high residential
electrical heating load on winter peak demand days, which saw temperatures
that varied widely from the mean value. Future peak winter and summer
electricity demand scenarios indicated that while average temperatures on
winter peak demand days will remain fairly constant, they will be more erratic
than temperatures on summer peak demand days. As a result, winter peak demand
will remain more erratic and will sporadically surpass summer peak demand
between 2025 and 2050. Thus, resource planners in ERCOT should place less
certainty on winter peak demand projections and an increased level of winter
preparedness on both the supply and demand sectors appears warranted.

###### keywords:

Electrification, Distributed Energy Resources, ERCOT, Demand Response, Energy
Security and Risk Assessment, Peak Demand

††journal: Journal of LaTeX Templates

##  1 Introduction

Heating is the largest global energy end-use (about 50% of energy demand)
ahead of transport (29%) and electricity (21%). Forty-six percent of this heat
energy is consumed within buildings for space and water heating and to a
lesser extent, cooking [1]. Natural gas is used to heat 60% of U.S. households
in cold and very cold climates and 47% of U.S. households overall [2]. This
end-use consumption of fossil fuels releases significant greenhouse gases from
leaks and combustion-related emissions. The electrification of heating is
becoming a standard component of decarbonization efforts [3, 4], and bans on
fossil fuel based space heating equipment have already been administered in
numerous locations [5, 6, 7]. Additionally, legislation such as the Inflation
Reduction Act provides tax incentives for electric heat pump installation and
might cause rapid adoption of this technology over the next decade. As a
result, electrical heating equipment as a portion of global heating technology
sales for residential and service buildings has been steadily increasing for
years, a trend that is expected to continue [8]. These factors have led U.S.
regional electric grid operators to begin to anticipate a large expansion of
space heating demand from the residential and commercial sectors and some to
predict a potential switch from a traditional summer peak to a winter peak [9,
10]. Questions remain about how this new source of electricity demand will
affect electric grid operations. In particular, how will grid resiliency be
impacted by the electrification of space heating? How will electric load for
residential space heating, which already drives winter peak power demand [11],
and is sensitive to severe weather events, be affected by climate change?
Figure 1 demonstrates this phenomenon in one regional electric grid, the
Electric Reliability Council of Texas (ERCOT).

Figure 1: The residential sector in ERCOT (the Electric Reliability Council of
Texas) is responsible for approximately half of peak demand in the winter and
summer [11].

The ERCOT regional power grid is an ideal test bed for answering questions
surrounding the electrification of heating. The ERCOT grid is comprised of a
generation mix with high fractions of wind and solar, which much of the U.S.
system may soon replicate [12]. Additionally, a large portion of Texas is in a
semi-arid temperate climate and has population centers in hotter and more
humid parts of the state. Consequently, electrical heating equipment, which
has historically been designed for more moderate temperatures, is prevalent.
The percentage of Texas household heating that is met by electricity is
increasing over time and is the 7th highest among U.S. states [13]. Thus, the
demand-side of ERCOT also resembles an evolving, future decarbonized grid.
Figure 2 demonstrates the high penetration of electric home heating in Texas
relative to the rest of the contiguous United States.

Figure 2: As of 2019, Texas had one of the highest rates of electric heating
in the residential sector in the United States [13]. Generally speaking,
electric heating is more prevalent in warmer climates (e.g. the southern part
of the United States) or where there is abundant hydroelectric power (e.g. the
Pacific Northwest).

As a result of the warm climate, ERCOT resource planners have prioritized
managing the natural gas system and electricity grid such that they can meet
the demand from large amounts of electrically-operated air-conditioners on hot
summer afternoons. However, the events of the winter storm in February 2021,
which caused hundreds of deaths and blackouts and boil water notices for
millions of people, should serve as a reminder that Texas is not immune to
severe winter weather that strains infrastructure reliability [14, 15, 16,
17]. These winter grid resiliency challenges could yield important lessons for
future grid operators in increasingly electrified economies.

The winter event of February 2021 left millions of Texans in ERCOT without
power for multiple days during some of the coldest and most widespread winter
weather seen in the state in decades. In fact, February 2021 was the first
time in recorded history that all 254 counties of Texas were under a winter
storm watch at the same time. [18]. This event presents the opportunity for a
unique case study on a highly electrified region subjected to severe winter
weather.

Having lost roughly half of all power generation capacity due to freezing
equipment, fuel shortages, and other issues, the grid was minutes away from
conditions that could have triggered a total system-wide blackout that then
could have required weeks or months for full recovery [16].

Because Texas is the largest energy producing and consuming state in the
United States [19], it was a surprise to many that the state could run out of
energy. As such, there has been continued public outrage and demands for
change to prevent a similar disaster. Almost every relevant government agency
including the Governor’s office, the State Legislature, Railroad Commission
(which is the oil and gas regulator in Texas), the Public Utility Commission
of Texas, and ERCOT have faced intense scrutiny or had its top leadership
replaced. Because leaders are under pressure to implement new regulations or
market reforms to prevent a future disaster, there is a need to inform the
policymaking process in Texas and other regions that may face similar
challenges with data about key underlying trends.

While many reports have assessed the underlying acute causes of the February
event itself and its meteorological underpinnings [14, 16, 17, 20, 21], to the
authors’ knowledge, none have rigorously examined how seasonal peak demand has
changed over time nor investigated how non-flexible heating electrification
could have contributed to the disaster. This analysis seeks to fill that
knowledge gap as well as present scenarios for future seasonal ERCOT peak
demand growth based on population projections and a future climate scenario.
In addition to providing insight into ERCOT grid performance, this analysis
presents a relevant case study for a highly electrified economy subject to
severe winter weather and thus could have useful insights for grid planners in
other regions.

##  2 Methods

The methodology for this analysis is simple in principle yet allows for some
important observations. Peak demand data for the past 25 years in ERCOT were
compiled for the winter (December through February) and summer (June through
August) seasons [22, 23, 24, 16, 25]1112001 summer peak demand sourced from
[25] as ”July Load at ERCOT Coincident Peak kW”.. A linear regression was used
to assess their growth over the 25-year period and the effect of ambient
temperatures on these events was investigated.

### Weather influence on peak demand

Degree day data were calculated to assess the sensitivity of peak demand to
outdoor temperatures. A degree day compares the mean outdoor temperature for a
location to a base temperature as a measure of cooling or heating load [26].
Cooling degree days (CDD) and heating degree days (HDD) are calculated using
ambient temperature readings collected at a particular location throughout the
day. The time in days between two temperature readings is multiplied by the
number of degrees by which the ambient temperature was above or below the base
temperature over the period to get the degree days [27]. The further the
ambient temperature is above or below the base temperature, the higher the CDD
or HDD respectively, and thus the higher the cooling or heating load.

$CDD=\sum time\;between\;readings\;(days)\times(ambient\;temp\;-\;base\;temp)$
(1)
$HDD=\sum time\;between\;readings\;(days)\times(base\;temp\;-\;ambient\;temp)$
(2)

In this study, we calculated the number of degree days during each peak demand
day. The base temperature for calculating both CDD and the HDD was set to
18.5° C (65.3° F) and an ERCOT-wide degree day (DD) value was calculated by
taking DD values from the largest city in each ERCOT weather zone [28, 29] and
weighting each DD value by the population in the weather zone2222021
population projected based on 2019-2020 percent population growth. [30] as
described by Equations 1-3 and Figure 4.

$ERCOT\;DD=\sum
DD\;from\;largest\;city\;in\;each\;zone\times\frac{zone\;population}{ERCOT\;population}$
(3)

Figure 3: ERCOT uses eight weather zones for planning purposes [31]. Figure 4:
This flow diagram shows the methodology used to calculate population-weighted
CDD and HDD in ERCOT.

### Future CDD and HDD scenarios

Future CDD and HDD were calculated using daily average near-surface
temperature as processed from downscaled global climate models produced by the
NASA Center for Climate Simulation (NCCS) [32]. NCCS describes the dataset as
“comprised of global downscaled climate scenarios derived from the General
Circulation Model (GCM) runs conducted under the Coupled Model Intercomparison
Project Phase 6 (CMIP6) and across two of the four “Tier 1” greenhouse gas
emissions scenarios known as Shared Socioeconomic Pathways (SSPs)” [32]. This
dataset was developed for the Sixth Assessment Report produced by the
Intergovernmental Panel on Climate Change. The dataset chosen for this
analysis was the SSP3-7.0 scenario, which represents the medium to high end of
the range of future radiative forcing pathways [33].

##  3 Results and discussion

Figure 5 shows how the winter333Winter peaks in 2011 and 2021 were estimated
because firm load shedding prevented the full load from being served. The
peaks for those years were taken from ERCOT estimates of what load would have
been absent load shed. Winter peak in 2001 was taken from an ERCOT report as
2001 load data were not available. and summer444Summer peak in 2001 was taken
from ERCOT Coincident Peak Calculations as 2001 load data were not available.
peaks have grown relative to each other along with a linear regression for
each season. General observations of the peak demand linear regression results
yield two main conclusions; 1) ERCOT’s winter peak is growing about 15% faster
than its summer peak, based on the slopes of the linear regressions and 2) the
winter peak is more erratic than the summer peak, based on the lower R-squared
values of the same regression. The winter peak is, on average, about 3.5 GW
off (above or below the mean of the absolute values of the regression model
errors) the linear model estimation, while the summer peak is only about 1 GW
off, on average. Figure 6 shows how each year’s summer and winter peaks differ
from the linear model’s prediction.

Figure 5: Winter and summer peak electrical demand in ERCOT grew from 1997 to
2021. The bars are actual peak demands (or estimated peak demands if load shed
happened) and the dotted lines are the linear fit estimations of peak demand
for each season and each year. Figure 6: The difference between the estimated
seasonal peak and the actual peak for each year shows that winter peak demand
is much more variable (and therefore harder to predict) than summer peak
demand.

The linear fit in Figure 5 shows that the summer peak demand growth is more
“stable” (or consistent) than the winter peak. A visual inspection of the fit
(dotted lines) versus the actual555Estimated peak demands in case of any firm
load shed. peak demands (bars) shows that the summer peak growth is more
linear than the winter model. This conclusion is also indicated by the
R-squared values of the respective models (higher R-squared values correspond
to higher linearity): R-squared value of 0.74 in the winter vs 0.97 in the
summer.

The lower R-squared value of the winter model indicates that the growth in
winter peak is less predictable. The summer peak is presumably a combination
of consistent population and economic growth trends offset somewhat by
efficiency improvements to space conditioning appliances. The winter peak has
similar factors with two confounding weather factors from climate change:
slowly increasing average prevailing temperatures (which reduces average total
seasonal energy for heating) and ongoing risk for intense winter storms.

Since the general formation of ERCOT, the annual winter peak has never
exceeded the summer peak. If the current rate of change shown in Figure 5
continues, it would take over 100 years from the start of the analysis (1997)
for the average winter peak to systematically surpass the average summer peak.
However, if the ERCOT grid had been able to deliver as much power as was
estimated to have been demanded in the winter storm of February 2021 [16] the
winter peak would have surpassed the summer peak for the first time since the
Texas grid operator began recording data for its modern grid footprint.
Furthermore, if electrified heating deployment rates accelerate, then the
summer peak demand will grow less quickly (because of commensurate upgrades to
air conditioning efficiency) and the winter peak demand will grow more
quickly, in which case it is reasonable to anticipate that the winter peak
will regularly exceed the summer peak much sooner than a century from 1997.

Because of the higher variability, peak demand projections for ERCOT, which
are generally based on a “normal weather year” and are generally linear [34],
should include more uncertainty on the forecasted winter peak than the summer
peak.

While the summer cooling season ramps up and down over several months, the
winter heating season is much more erratic. For homes with electric heating,
the increase in demand for home heating when outside temperatures drop to
levels seen in February 2021 is larger than the demand is for home cooling
when a heat wave pushes temperatures to summer peak levels [35].

While the residential sector is generally responsible for most of the large
swings in demand, residential demand response in ERCOT is small relative to
commercial and industrial demand response [36]. As such, residential demand
response programs that seek to reduce peak demand are mostly an untapped
potential solution for grid reliability.

### Peak demand sensitivity to weather

Degree day data indicate that, over the past 25 years, temperatures were more
erratic on winter peak demand days than on summer peak demand days. CDD on
summer peak demand days were fairly constant near the mean CDD value. However,
HDD for winter peak demand days varied widely from the mean value (Figure 7).
The unpredictable nature of the temperatures on winter peak demand days and
more consistent temperatures on summer peak demand days is mirrored by the
heating and cooling loads, respectively.

Weather Zone |  City |  Weather Station ID |  Average Summer Peak Demand Day CDD |  Average Winter Peak Demand Day HDD
---|---|---|---|---
Coast |  Houston |  KIAH |  12.5 |  16.2
East |  Tyler |  KTYR |  13.0 |  19.8
Far West |  Midland |  KMAF |  11.9 |  20.9
North |  Wichita Falls |  KSPS |  13.6 |  22.6
North Central |  DFW |  KDFW |  14.8 |  20.4
South |  Corpus Christi |  KCRP |  11.6 |  13.4
South Central |  Austin |  KAUS |  12.6 |  17.6
West |  Abilene |  KABI |  12.7 |  21.6
Table 1: Summary of weather stations used in calculation of state-wide
population-weighted CDD and HDD values and average CDD and HDD values. Figure
7: HDD on winter peak demand days are more variable than CDD on summer peak
demand days. ERCOT-wide DD values are calculated from DD values for the
largest population center in each ERCOT weather zone weighted by the
population of each weather zone.

These weather trends that drive heating and cooling loads impact power grid
operations. Overall, about 55% of residential heating equipment in the West
South Central census region of the US, which includes Texas, is electric. Of
those electric heaters, about 85% are electric resistance and the remaining
15% are heat pumps [37]. Electrical resistance heating has very high power
draws as compared to other forms of heating, such as heat pumps. Heat pumps
can provide heating using much less electricity most of the time. However,
many of the heat pumps installed in the region are not cold-weather heat
pumps, rather they are usually designed for mild climates, and thus are only
able to operate down to a specified low outdoor temperature before switching
to backup or auxiliary heating modes that rely heavily on electric resistance
heating or natural gas [38, 39]. When this switch happens, it can create a
large jump in the electricity demand of each individual heating system that in
aggregate can create a large upward disruption in grid demand as a cold front
moves through. An analysis of per capita winter peak demand versus HDD
exhibits this effect: per capita winter peak demand appears to have a
polynomial relationship with degree days, becoming increasingly high for
higher HDD (Figure 8). Additionally, data points from more recent years with
higher penetration of electric heating tend to reside above the regression
line, indicating that more electricity is consumed per capita when more
electrical heating equipment is connected to the grid.

Figure 8: The relationship between 1997-2021 per capita winter peak demand and
winter peak demand day HDD appears to be polynomial. Percent heating
electrification data for this figure was sourced from the American Community
Survey [13] and the Residential Energy Consumption Survey [37] and
interpolated or extrapolated for years in which data was not available.

Normalizing the peak demand data by population and weather demonstrates that
the summer per-capita, per cooling degree day peak demand is decreasing,
indicating that the electrical efficiency of meeting the summer peak demand is
increasing (Figure 9).

However, the winter per-capita, per heating degree day peak demand is
increasing, indicating that over time progressively more electricity is being
consumed to meet the same per capita heating load. However, unlike the summer
peak demand which has been driven by electrically-powered air-conditioners for
the entire period of our dataset, the makeup of heating equipment has changed
over time. For example, in 1997, only about 40% of homes in Texas used
electrical heating, but that number has increased over time, meaning that the
percentage of homes heating with other fuels, such as natural gas, has
declined [37, 13]. Analyzing the flows of natural gas is beyond the scope of
this analysis, but the positive slope, in Figure 9, of the winter per-capita,
per heating degree day peak demand linear fit does indicate (although weakly)
that electricity use in the winter is generally increasing even when
population and weather are considered. This is likely capturing the dual
impacts of heat pumps replacing electrically-operated AC units and gas
furnaces. By replacing older AC units with a new heat pump, the electrical
efficiency of cooling improves. By replacing gas furnaces with heat pumps,
heating load is shifted from natural gas infrastructure to the power grid.

Figure 9: Per capita peak demand per DD is decreasing over time for summer
peak demand days and increasing over time for winter peak demand days.

### Future peak demand scenarios

Scenarios of winter and summer peak demand were constructed by extrapolating
the per-capita, per degree day peak demand linear fits out for 2025 through
2050 and utilizing population projections [40] and DD for a future climate
scenario (Figure 10). In this peak demand scenario for 2025 through 2050,
winter peak demand day HDD and winter peak demand continue to be more erratic
than summer peak demand day CDD and summer peak demand. Additionally, while
average winter peak demand day HDD remains about the same, average summer peak
demand day CDD rises by $\sim$15% as temperatures rise through mid-century.
Summer peak demand through 2050 is more erratic than it has been historically,
but is still more stable than winter peak demand through 2050 and is generally
increasing at a slower rate than winter peak demand due to changing heating
and cooling efficiencies that arise due to the installation of electrical
heating equipment.

Figure 10: Winter and summer peak demand scenarios through 2050 based on
extrapolated historical per capita peak demand per DD with population
projections and DD calculated for a future climate scenario demonstrate that
winter peak demand will likely continue to be more erratic than summer peak
demand, and winter peak demand will begin to regularly surpass summer peak
demand before 2050.

### Suggestions for system and resource planning

As Texas and other regions electrify heating, more attention will need to be
given to the impacts of extreme periods of cold weather on winter peak demand.
Increased efficiency and cold-weather standards for heat pumps could allow the
continued electrification of heating to avoid making as big of an impact on
winter peak demand by delaying the individual heater’s switch into auxiliary
(electric resistance) mode, thereby reducing how often demand grows
polynomially.

Increased residential building envelope efficiency could also reduce the
amount of heating needed by the average home which would reduce the amount of
individual coincident heating systems operating and thus lower overall heating
demand from the residential sector. Most of Texas is located in “hot” climate
zones and thus more attention has been paid to constructing homes to withstand
high summer temperatures. Homes in Texas are generally designed to have
adequate insulation levels and heating systems to withstand -4C (25F) [30] and
are thus not prepared to cope with the much colder temperatures from events
such as the winter storm in February 2021.

On the demand side, Texas can avert future load shed by improving its mediocre
energy efficiency performance as a cheaper alternative to building new thermal
power generation [41]. It’s also worth noting that because half of the peak
demand in winter and summer is from space heating and cooling, these end-uses
represent a significant opportunity for demand response (e.g. dispatching an
intentional reduction in peak demand by turning off heating or cooling
devices). These factors have led grid operators to posit that water heating
and HVAC are good candidates for load control, and could allow buildings to
provide grid services as thermal batteries [9]. If equipment with flexible
features is installed, then rotating shut-offs to electric air conditioners
and heaters can help preserve grid reliability and can be cheaper and quicker
to install than to build additional generating supply.

On the grid supply side, it should be noted that by the end of 2022 it is
possible that 4-5 GW of Texas power generation (or $\sim$5% of total Texas
generation capacity) will come from the distribution grid [42]. These
distributed energy resources (DERs) could provide valuable grid services if
properly coordinated. Price signals and/or incentives should be administered
to ensure that DERs are optimally placed to provide grid reliability and cost
benefits [42]. Additionally, increased attention to resource adequacy levels
in the winter should be a focus for future operations. While beyond the scope
of this work, it is possible that the addition of a new winter reliability
product for the ERCOT market might be necessary to meet the additional
challenges that are associated with the increased uncertainty in predicting
the winter peak demand.

This analysis corroborates previous NERC conclusions that grid planners can
have reasonable confidence that each summer in ERCOT will be hot, but should
place less certainty around how cold each winter will be [43]. These weather
trends hold true historically, and are likely to continue for future climate
scenarios through 2050. These findings would indicate the ERCOT grid might
need more reserve capacity (higher reserve margin) to handle higher levels of
uncertainty in peak demand in the wintertime than in the summer, particularly
if the electrification of heating continues.

Currently, generation maintenance outages for generators and transmission
assets are restricted between May 15 and September 15 so that most assets are
ready to meet the summer peak demand. However more attention might need to be
placed on having more capacity online during winter months to cover the higher
levels of uncertainty. But squeezing more generation outages into smaller
periods of time might then create supply shortages in the shoulder months.

##  4 Conclusions

This analysis assessed the relative change in historical summer vs. winter
peaks in ERCOT from 1997 through 2021 and scenarios of peaks from 2025 through
2050. It was shown that while historical summer peak demand growth is
relatively stable, the winter peak demand is growing more quickly and more
erratically. This trend is likely the result of the electrical efficiency
increases for cooling and growth in electricity consumption as a result of the
installation of electrical heating equipment. Additionally, it was shown that
the historical instability of the winter peak demand growth is a consequence
of erratic winter weather and electrical heating that becomes increasingly
inefficient at lower temperatures. When these trends in heating and cooling
efficiency are extrapolated through 2050 and combined with population
projections and a future climate scenario, winter peak demand continues to
grow more quickly and erratically than summer peak demand. This results in
winter peak demand regularly surpassing summer peak demand before 2050. These
results imply that special attention will need to be paid to how ERCOT plans
for winter heating seasons, particularly given the trend of heating
electrification in the residential sector, which was already the swing
consumer of electricity in both the summer and winter seasons. Namely,
planners should use higher uncertainty in their estimates for peak winter
demand. Increased general efficiency standards for buildings and in particular
heat pumps could mitigate some of the demand side winter issues and an
increased level of uncertainty placed on the aggregate electricity grid winter
peak demand estimates could help drive policies to increase winter reserves.
Considering that the electrification of heating is a primary component of
decarbonization efforts in all climates, lessons from this case study should
be considered in colder regions as they seek to replace fossil fuel heating
equipment with electric heating.

##  5 CRediT authorship contribution statement

Matthew J. Skiles: Data curation; Formal Analysis; Investigation; Methodology;
Visualization; Writing — original draft, Writing — review & editing. Joshua D.
Rhodes: Conceptualization; Data curation; Formal analysis; Investigation;
Methodology; Visualization; Writing — original draft, Writing — review &
editing. Michael E. Webber: Supervision; Writing — review & editing; Project
administration.

##  6 Acknowledgements

We would like to thank Gene Preston, whose question about the relative changes
in ERCOT’s seasonal peak demand prompted this analysis. This work was
supported by the Texas State Energy Conservation Office with additional
partial support from the U.S. Army Corps of Engineers’ Engineer R&D Center
(ERDC) through a subcontract with Artesion, the Energy Foundation, and Austin
Energy. This material is based upon work supported by the National Science
Foundation Graduate Research Fellowship Program under Grant No. DGE-1610403.
Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views
of the National Science Foundation. The sponsors had no direct involvement in
this work. The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence the
work reported in this paper. In addition to the research work on topics
generally related to energy systems at the University of Texas at Austin, one
of the authors (Webber) has an affiliation with Energy Impact Partners (a
venture investment firm) and two of the authors (Webber and Rhodes) are
partners in IdeaSmiths LLC (an engineering consulting firm). Any opinions,
findings, conclusions or recommendations expressed in this material are those
of the authors and do not necessarily reflect the views of the sponsors,
Energy Impact Partners, or IdeaSmiths LLC. The terms of this arrangement have
been reviewed and approved by the University of Texas at Austin in accordance
with its policy on objectivity in research.

## References

  * [1] Renewables 2019 – Analysis.
URL https://www.iea.org/reports/renewables-2019

  * [2] C. Lawrence, C. Berry, U.S. households’ heating equipment choices are diverse and vary by climate region (Apr. 2017).
URL https://www.eia.gov/todayinenergy/detail.php?id=30672

  * [3] I. Sheikh, D. Callaway, Decarbonizing Space and Water Heating in Temperate Climates: The Case for Electrification, Atmosphere 10 (8), place: Basel, Switzerland Publisher: MDPI AG (Aug. 2019).  doi:https://doi.org/10.3390/atmos10080435.
URL https://www.proquest.com/docview/2309963397/abstract/8223CA650C334BC3PQ/1

  * [4] A. S. Gaur, D. Z. Fitiwi, J. Curtis, Heat pumps and our low-carbon future: A comprehensive review, Energy Research & Social Science 71 (2021) 101764.  doi:10.1016/j.erss.2020.101764.
URL https://linkinghub.elsevier.com/retrieve/pii/S221462962030339X

  * [5] T. Gruenwald, M. Lee, 2020: Watt a Year for Building Electrification! (Dec. 2020).
URL https://rmi.org/2020-watt-a-year-for-building-electrification/

  * [6] A. Derrick, Mayor Durkan Announces Ban on Fossil Fuels for Heating in New Construction to Further Electrify Buildings Using Clean Energy (Dec. 2020).
URL https://durkan.seattle.gov/2020/12/mayor-durkan-announces-ban-on-fossil-
fuels-for-heating-in-new-construction-to-further-electrify-buildings-using-
clean-energy/

  * [7] T. DiChristopher, Gas Ban Monitor: West Coast pushes new boundaries; pro-gas state bills stall (Apr. 2022).
URL https://www.spglobal.com/marketintelligence/en/news-insights/latest-news-
headlines/gas-ban-monitor-west-coast-pushes-new-boundaries-pro-gas-state-
bills-stall-69969602

  * [8] Heating – Analysis, Tech. rep., International Energy Agency.
URL https://www.iea.org/reports/heating

  * [9] MISO Electrification Insights, Tech. rep., Midcontinent Independent System Operator (Oct. 2021).
URL https://cdn.misoenergy.org/Electrification%20Insights538860.pdf

  * [10] 2020 Load & Capacity Data, Tech. rep., The New York Independent System Operator, Inc.
URL https://www.nyiso.com/documents/20142/2226333/2020-Gold-Book-Final-
Public.pdf/

  * [11] Load Profiling, Tech. rep., Electric Reliability Council of Texas.
URL https://www.ercot.com/mktinfo/loadprofile

  * [12] H. Reeve, The Distribution System Operator with Transactive (DSO+T) Study, Tech. Rep. PNNL-32170-Sum, 1842482 (Jan. 2022).  doi:10.2172/1842482.
URL https://www.osti.gov/servlets/purl/1842482/

  * [13] Public Use Microdata Sample, Tech. rep., American Community Survey (2019).
URL https://www.census.gov/programs-surveys/acs/microdata.html

  * [14] J. W. Busby, K. Baker, M. D. Bazilian, A. Q. Gilbert, E. Grubert, V. Rai, J. D. Rhodes, S. Shidore, C. A. Smith, M. E. Webber, Cascading risks: Understanding the 2021 winter blackout in Texas, Energy Research & Social Science 77 (2021) 102106.  doi:10.1016/j.erss.2021.102106.
URL https://linkinghub.elsevier.com/retrieve/pii/S2214629621001997

  * [15] Y. R. Glazer, D. M. Tremaine, J. L. Banner, M. Cook, R. E. Mace, J. Nielsen-Gammon, E. Grubert, K. Kramer, A. M. K. Stoner, B. M. Wyatt, A. Mayer, T. Beach, R. Correll, M. E. Webber, Winter Storm Uri: A Test of Texas’ Water Infrastructure and Water Resource Resilience to Extreme Winter Weather Events, Journal of Extreme Events (2021) 2150022doi:10.1142/S2345737621500226.
URL https://www.worldscientific.com/doi/abs/10.1142/S2345737621500226

  * [16] C. W. King, J. D. Rhodes, J. Zarnikau, N. Lin, E. Kutanoglu, B. Leibowicz, D. Niyogi, V. Rai, S. Santoso, D. Spence, S. Tompaidis, H. Zhu, E. Funkhouser, B. Austgen, The Timeline and Events of the February 2021 Texas Electric Grid Blackouts 101.
  * [17] FERC, NERC Staff Review 2021 Winter Freeze, Recommend Standards Improvements, Tech. rep., Federal Energy Regulatory Commission (Sep. 2021).
URL https://www.ferc.gov/news-events/news/ferc-nerc-staff-review-2021-winter-
freeze-recommend-standards-improvements

  * [18] M. Villarreal, All 254 Texas counties under winter storm warning as arctic blast heads east (Feb. 2021).
URL https://www.cbsnews.com/news/texas-snow-winter-storm-warning-arctic-
blast-2021-02-14/

  * [19] U.S. Energy Information Administration - EIA - Independent Statistics and Analysis.
URL https://www.eia.gov/state/?sid=TX

  * [20] C. T. M. Clack, A. Choukulkar, B. Cote, S. A. McKee, ERCOT Winter Storm Uri Blackout Analysis (February, 2021), Tech. rep., Vibrant Clean Energy (Mar. 2021).
  * [21] J. Doss-Gollin, D. J. Farnham, U. Lall, V. Modi, How unprecedented was the February 2021 Texas cold snap?, Environmental Research Letters 16 (6) (2021) 064056.  doi:10.1088/1748-9326/ac0278.
URL https://iopscience.iop.org/article/10.1088/1748-9326/ac0278

  * [22] Hourly Load Data Archives, Tech. rep., Electric Reliability Council of Texas.
URL https://www.ercot.com/gridinfo/load/load_hist

  * [23] EILS in the CDR, Tech. rep., Electric Reliability Council of Texas (Dec. 2009).
  * [24] Report on outages and curtailments during the Southwest cold weather event, Tech. rep., The Federal Energy Regulatory Commission and the North American Electric Reliability Corporation.
  * [25] ERCOT Four Coincident Peak Calculations, Tech. rep., Electric Reliability Council of Texas.
URL https://www.ercot.com/mktinfo/data_agg/4cp

  * [26] Degree-days \- U.S. Energy Information Administration (EIA).
URL https://www.eia.gov/energyexplained/units-and-calculators/degree-days.php

  * [27] Calculating Degree Days.
URL https://www.degreedays.net/calculation

  * [28] Heating & Cooling Degree Days – Free Worldwide Data Calculation.
URL https://www.degreedays.net/

  * [29] Weather History & Data Archive | Weather Underground.
URL https://www.wunderground.com/history

  * [30] Population and Housing Unit Estimates Tables, section: Government.
URL https://www.census.gov/programs-surveys/popest/data/tables.html

  * [31] Maps.
URL https://www.ercot.com/news/mediakit/maps

  * [32] NASA Earth Exchange Global Daily Downscaled Projections (NEX-GDDP-CMIP6) | NASA Center for Climate Simulation.
URL https://www.nccs.nasa.gov/services/data-collections/land-based-
products/nex-gddp-cmip6

  * [33] Abram, N., J.-P. Gattuso, A. Prakash, L. Cheng, M.P. Chidichimo, S. Crate, H. Enomoto, M. Garschagen, N. Gruber, S. Harper, E. Holland, R.M. Kudela, J. Rice, K. Steffen, and K. von Schuckmann, 2019: Framing and Context of the Report Supplementary Material. In: IPCC Special Report on the Ocean and Cryosphere in a Changing Climate [H.-O. Pörtner, D.C. Roberts, V. Masson-Delmotte, P. Zhai, M. Tignor, E. Poloczanska, K. Mintenbeck, A. Alegría, M. Nicolai, A. Okem, J. Petzold, B. Rama, N.M. Weyer (eds.)]. In press., Tech. rep.
  * [34] Report on the Capacity, Demand and Reserves (CDR) in the ERCOT Region, 2022-2031, Tech. rep., Electric Reliability Council of Texas (Dec. 2021).
URL
https://www.ercot.com/files/docs/2021/12/29/CapacityDemandandReservesReport_December2021.pdf

  * [35] NRG Energy, Inc.’s Coments on the Commission’s September 2, 2021 Questions for Comment, Tech. rep., NRG Energy.
URL https://www.nrg.com/assets/documents/energy-policy/p-52373_nrg-comments-
to-september-2-2021-questions.pdf

  * [36] P. Du, N. Lu, H. Zhong, Demand Responses in ERCOT, in: Demand Response in Smart Grids, 2019, pp. 85–119.
  * [37] Residential Energy Consumption Survey (RECS), Tech. rep., U.S. Energy Information Administration.
URL https://www.eia.gov/consumption/residential/data/2015/hc/php/hc6.8.php

  * [38] P. R. White, J. D. Rhodes, Electrification of Heating in the Texas Residential Sector, Tech. rep., Ideasmiths, LLC (2019).
  * [39] P. R. White, J. D. Rhodes, E. J. Wilson, M. E. Webber, Quantifying the impact of residential space heating electrification on the Texas electric grid, Applied Energy 298 (2021) 117113.  doi:10.1016/j.apenergy.2021.117113.
URL https://linkinghub.elsevier.com/retrieve/pii/S0306261921005559

  * [40] 2022 State Plan Population Projections Data | Texas Water Development Board.
URL https://www.twdb.texas.gov/waterplanning/data/projections/2022/popproj.asp

  * [41] J. Lee, Texas Has an Obvious, Affordable Fix for Its Electricity Problem, Wall Street Journal (Dec. 2022).
URL https://www.wsj.com/articles/texas-has-an-obvious-affordable-fix-for-its-
electricity-problem-11671067175

  * [42] D. Lewin, Texas grid regulator eyes big changes for DERs (May 2022).
URL https://www.renewableenergyworld.com/solar/texas-grid-regulator-eyes-big-
changes-for-ders/

  * [43] 2022 Long-Term Reliability Assessment, Tech. rep., North American Electric Reliabiliaty Corporation (Dec. 2022).


# Brightness-Restricted Adversarial Attack Patch

Mingzhen Shao
Kahlert School of Computing
University of Utah
Salt Lake City, UT 84108
<EMAIL_ADDRESS>

###### Abstract

Adversarial attack patches have gained increasing attention due to their
practical applicability in physical-world scenarios. However, the bright
colors used in attack patches represent a significant drawback, as they can be
easily identified by human observers. Moreover, even though these attacks have
been highly successful in deceiving target networks, which specific features
of the attack patch contribute to its success are still unknown. Our paper
introduces a brightness-restricted patch (BrPatch) that uses optical
characteristics to effectively reduce conspicuousness while preserving image
independence. We also conducted an analysis of the impact of various image
features (such as color, texture, noise, and size) on the effectiveness of an
attack patch in physical-world deployment. Our experiments show that attack
patches exhibit strong redundancy to brightness and are resistant to color
transfer and noise. Based on our findings, we propose some additional methods
to further reduce the conspicuousness of BrPatch. Our findings also explain
the robustness of attack patches observed in physical-world scenarios.

##  1 Introduction

Deep neural networks (DNNs) have experienced significant success across
various domains in recent years, such as image classsification [11, 5], object
detection [10, 16], speech recognition [19], and natural language processing
[24]. However, the nature of DNNs also makes them vulnerable to adversarial
attacks that are crafted by adding carefully designed perturbations on normal
examples [2, 9, 13, 18, 20, 21]. Since DNNs have become more critical to some
safety-critical applications, such as autonomous driving and biometric
authentication, their susceptibility to adversarial attacks raises serious
concerns about their safety and reliability [17, 7, 4].

Adversarial attacks in computer vision tasks can be categorized into two
domains: digital and physical. In the digital domain, attackers can access the
digital values of inputs and make arbitrary pixel-level changes to inputs.
However, such ideal conditions are challenging to achieve in the real world.
Data security measures in well-designed software are typically difficult to
breach. Once an attacker successfully bypasses these measures, further
manipulation of the DNNs may become unnecessary. Physical domain attacks, on
the other hand, may be more realistic in practice as they assume that only the
physical layer objects, such as the environment or objects that the system
interacts with, can be manipulated.

Image-dependent and image-independent are the two main types of attacks in the
physical domain. Image-dependent attacks require a design of each attack
tailored to the specific target image. In most conditions, these attacks need
to replace the target with a modified object. For example, to make DNNs
misclassify a stop sign, a modified stop sign image needs to be created and
substituted for the original one. However, this approach can be highly
inefficient and even impossible if the target is not an image. In contrast,
image-independent attacks use an additional object (patch) to get rid of this
requirement. The patch is trained to create a physical world attack without
prior knowledge of the other items within the scene. The patch can be put in
any environment to launch an attack without replacing the targets.

In spite of the great success the attack patches have shown in deceiving
target networks, the bright and vivid coloration of these patches can also be
a significant drawback. In many real-world scenarios, attack patches must
remain low visibility to human observers, particularly in security
applications where an attacker may seek to evade visual surveillance. In order
to reduce the visibility of attack patches, a straightforward question to ask
is whether their vivid coloration is strictly necessary for a successful
attack because reducing this feature can significantly decrease the patch’s
visibility to human observers.

In this work, we first introduced a brightness-restricted patch (BrPatch) to
reduce visibility while maintaining a high attack success rate. By using
optical characteristics (brightness) to minimize detectability, the BrPatch
can still maintain image independence, which means the BrPatch does not
require additional training for specific scenarios. We then conducted an
analysis of the impact of various image features (such as color, texture,
noise, and size) on the effectiveness of an attack patch in a physical-world
deployment. Based on our findings, we proposed a hue mapping method to further
reduce the visibility of the BrPatch. Furthermore, we evaluated the
performance of the proposed BrPatch in the physical world to demonstrate that
the BrPatch still achieves an attack success rate comparable to the original
adversarial attack patch in the real world. To the best of our knowledge, this
study is the first attempt to use optical characteristics (brightness) to
reduce the visibility of an adversarial attack patch and analyze the
mechanisms underlying its effectiveness.

The main contributions of this paper can be summarized as follows:

  * •

We propose a brightness-restricted adversarial attack patch (BrPatch) to
reduce visibility while preserving image independence.

  * •

Our experiments show that adversarial attack patches exhibit strong redundancy
to brightness restrictions.

  * •

We demonstrate that color transfer and random noise in the physical world will
not significantly affect the performance of attack patches.

##  2 Related work

Adversarial attacks for deep learning were first introduced by szegedy  _et
al._ [18]. Since the publication of their seminal work, numerous researchers
have proposed more efficient methods for generating adversarial attacks. These
adversarial attacks commonly modify each pixel by only a small amount and can
be found using a number of optimization strategies such as the Fast Gradient
Sign Method (FGSM) [9], Projected Gradient Descent (PGD) [14], and Skip
Gradient (SGD) [22]. Some other attacks seek to modify a small number of
pixels in the image [15], or a small patch at a fixed location of the image
[17]. However, all these approaches assume that the attacker has digital-level
access to the inputs, which limits the range of scenarios in which the attacks
can be used.

Therefore, Kurakin  _et al._ [12] proposed the first physical-domain attack
model by printing digital adversarial examples onto paper. They found that a
significant portion of the printed adversarial examples deceived the image
classifier. Athalye  _et al._ [1] improved on this work by creating
adversarial objects that remain effective even when viewed from different
angles. They achieved this by modeling small-scale transformations
synthetically when generating adversarial perturbations. They demonstrated
that the resulting adversarial objects deceived target classifiers. In their
paper, they claimed that the algorithm is robust to rotations, translations,
and noise as long as the transformation can be modeled synthetically. Eykholt
_et al._ [8] also developed an attack algorithm that can generate physical
adversarial examples. In contrast to Athalye  _et al._ , they modeled image
transformations both synthetically and physically. Their target dataset
included certain image transformations, such as changes in viewing angle and
distance. They synthetically applied other transformations, such as changes in
lighting, when generating adversarial examples. Their work suggested that by
relying solely on synthetic transformations, subtleties in the physical
environment can be overlooked, resulting in a less robust attack. However, all
these works require the design of each attack to be tailored to the specific
target image, which dramatically limits their applicability in various
scenarios.

In order to provide an image-independent adversarial attack model, Brown  _et
al._ [3] proposed a new approach for creating an adversarial attack patch.
This patch can be placed anywhere within the field of view of a classifier and
launch an attack. Because this patch is scene-independent, it allows attackers
to create a physical-world attack without prior knowledge of the lighting
conditions, camera angle, type of classifier being attacked, or even the other
items within the scene. Since then, many studies have focused on improving
physically feasible attacks aimed at deceiving classifiers or object
detectors, such as traffic signs [7], cloaks [23], or vehicles [25]. However,
as Brown  _et al._ mentioned in their paper, these attack patches are not
restricted to imperceptible changes. These patches have striking color, which
is very conspicuous to human observers. One approach to reducing the
visibility of the attack patch has been proposed by Duan  _et al._ [6]. They
modified the patch into some natural styles that appear legitimate to human
observers. Their approach can be an effective way to evade detection, but it
requires generating a new patch for each attack scenario. This regeneration is
contrary to the main advantage of using attack patches, which is the ability
to be trained once and deployed universally, without being dependent on
specific images. Furthermore, in their physical-world experiments, they chose
to replace the target object with the modified image rather than using the
patch with the original target. These experiments simplified the 3D position
relationships between the patch and the target and also suffered the drawback
that some targets cannot be replaced with printed images.

##  3 Method

In this secession, we first provide a method to generate the brightness-
restricted attack patch (BrPatch). Then we provide an easy hue mapping method
in the RGB color model to further reduce the visibility of attack patches.

###  3.1 Generating brightness-restricted patch

Given an image $x\in{[0,1]}^{w\times h\times c}$ with class $y$, and an attack
patch $p$, let $T$ be a transformation function that can involve location,
rotation, and scale. We define $T(p,x)$ as the input image obtained by
applying $T$ to patch $p$ to get the transformed patch, and then overlaid onto
$x$.

The adversarial loss for a targeted attack can be formed as follows:

$L_{adv}=log(softmax(Pr(\hat{y}|T(p,x))))$ (1)

where the $\hat{y}$ is the target class and $\hat{y}\neq y$, $Pr$ is the
prediction of the target model with respect to class $\hat{y}$.

In order to reduce the intensity of vivid colors in an image, it is necessary
to manipulate the brightness component in the HSB (Hue, Saturation,
Brightness) color model. However, to prevent the need for switching between
multiple color models, we introduced a brightness-restricting loss function
within the RGB color space:

$L_{b}=log(1-mse(p,p_{r}))$ (2)

where $mse(p,p_{r})$ calculates the mean square error between the patch $p$
and a reference patch $p_{r}$. In printing systems, white is used to represent
the absence of any ink. Therefore, we use an all-white patch as the reference
patch $p_{r}$.

The final loss is a combination of $L_{adv}$ and $L_{b}$:

$L=L_{adv}+\lambda L_{b}$ (3)

where $\lambda$ is a parameter that adjusts the strength of the brightness-
restricting loss.

###  3.2 Hue mapping in RGB color model

Given a patch $p$ and a target region on original image $x_{t}$, calculate the
hue difference between the patch and target region:

$\Delta H=ave_{c}(p)-ave_{c}(x_{t})$ (4)

where $ave_{c}(\cdot)$ computes the average value of each channel (red, green,
and blue) of the input patch $p$ and the target region $x_{t}$, respectively.

We use a threshold $H_{t}$ to prevent potential overflows of values in the
patch $p$ after the hue mapping process:

$p^{\prime}=\begin{cases}p+\Delta H,&\text{if }\Delta H\leq H_{t}\\\
p+H_{t},&\text{otherwise}\end{cases}$ (5)

where $p^{\prime}$ is the patch after hue mapping.

##  4 Experimental results and analysis

In this section, we first outline the experimental setup. Then we show the
performance of the proposed BrPatch in different settings and demonstrate that
BrPatch can significantly reduce the visibility of attack patches without
compromising their effectiveness in deceiving the target network. The
experiments also show that adversarial attack patches have strong redundancy
to brightness restrictions. After that, we conduct an analysis of the impact
of various image features (such as color, texture, noise, and size) on the
effectiveness of an attack patch in the physical world deployment. Based on
our findings, we use hue mapping to further reduce the visibility. Finally, we
evaluate the robustness of the BrPatch in physical-world attacks.

###  4.1 Experimental setup

In our experiments, we use a gray-box setting for the threat model: the source
and target networks are both ResNet50 networks but were trained separately on
ImageNet1K. We perform 40 epochs of training for each setting and select the
patch that achieves the best performance as the final output patch. The input
images have a size of 224x224, and their values range is $[0,1]$. In order to
test the attack success rate, we randomly select 1,000 images in ImageNet1K as
source images, and apply the attack patch to these images.

For a physical-world test, we use an HP laser printer to print adversarial
attack patches and a Google Pixel6a to take photos. To further reduce the
visibility of the BrPatch during physical deployment, we first capture a photo
of the target and apply hue mapping to the BrPatch. Then we print the hue
mapped BrPatch only on A4 paper. We place the printed BrPatch next to the
target and take another photo that includes both the target and BrPatch. All
our experiments focus on targeted attacks, as untargeted attacks can be viewed
as a special case of targeted attacks where the target can be any valid target
value.

###  4.2 Performance of brightness-restricted patch

Figure 1: Attack success rate with different patch sizes and brightness
ranges.

Figure 1 shows the performance variations for different brightness ranges and
patch sizes on ResNet50, where each line corresponds to a specific patch size.
The impact of patch size on the performance is apparent from the figure, with
larger patches showing better performance. We also find that all patches show
redundancy to brightness, with larger patches being more tolerant to stricter
restrictions than smaller ones.

Figure 2: Patches with different brightness restriction and their brightness
distribution. (Top: Different brightness-restricted patches; Bottom:
Corresponding brightness distributions to each patch.)

Some brightness distributions of different BrPatch are shown in Figure 2. We
choose a patch size of 70x70 due to its high sensitivity to brightness
restriction and ability to achieve high performance without brightness
restriction (original adversarial attack patch). For the following analysis,
we will continue to use attack patches of the same size.

Referring to the performance shown in Figure 1, our results indicate that the
adversarial patches can achieve a high attack success rate in deceiving the
target network with only a small range of brightness values. For instance, the
performance remains consistent for the patch sizes of 70x70 even when up to
65% of the brightness is lost. This finding allows us to decrease the
brightness of an attack patch without sacrificing its performance, effectively
reducing the visibility of the patch.

Figure 3: Images with different patches. (Top: AdvPatch; Bottom: BrPatch)

Figure 3 demonstrates a set of images with an original adversarial attack
patch (AdvPatch) and a BrPatch (brightness range=0.24). The proposed BrPatch
is observed to be less visible to a human observer in all scenarios and does
not require any additional training. The performance of the BrPatch is also
comparable to that of the AdvPatch.

###  4.3 Effectiveness of different features in the physical world

####  4.3.1 Color transfer and texture blurring

Figure 4: Patch with different feature changes. (left: original patch; middle:
color transfer patch; right: texture blurring patch)

In physical-world deployments, attack patches are often affected by color
transfer due to lighting conditions or blurring caused by camera focus or
smudging. To analyze if these changes will affect the performance of the
attack patches, we compare the performance of an original patch, a color-
adjusted patch, and a local texture adjusted patch. The color-adjusted patch
is applied by adding a value ($\delta$) to all values in RGB channels and
making sure $\delta$ will not lead to an overflow. This color transfer will
not change the texture information of the patch. The local texture adjustment
is applied by using a 3x3 Gaussian blur. Figure 4 demonstrates the two types
of feature adjustments applying to a patch with brightness range=0.24.

Table 1: Performance with color transfer and Gaussian blur Brightness range | Original | Color transfer | Gaussian blur
---|---|---|---
$1$ (AdvPatch) | 89.4% | 90.8% | 47.8%
$0.35$ | 89.5% | 87.9% | 22.7%
$0.24$ | 74.2% | 75.3% | 10.1%

The performances of different patches are shown in Table 1. Regardless of the
lightness restriction, the color transfer patch achieves almost the same
success rate as the original patch. This performance shows that the patch does
not need to maintain a specific color to deceive the target network. On the
other hand, the blurred patch exhibits a significant decrease in success rate,
suggesting that local texture is the key feature in deceiving target networks.
Using these findings, we can apply the proposed hue mapping method to adjust
the color of the patch and enhance its integration with the target
environment, resulting in further reduced visibility. This process does not
require any learning and can be quickly applied when deploying the patch in
the physical world.

####  4.3.2 Random color variations

When printing an attack patch, it is important to consider that normal
printers are not able to produce a patch with precisely the same color as the
digital version. Therefore, the patch’s robustness to random color variations
must be evaluated.

To replicate the color drift that commonly occurs during printing, we generate
random noise within a restricted range that corresponds to a percentage of the
original value. This approach allows us to simulate different levels of drift,
and the results are shown in Table 2.

Table 2: Performance with different color drift Brightness range | Original | 10% drift | 15% drift | 20% drift
---|---|---|---|---
$1$ (AdvPatch) | 89.4% | 87.6% | 85.9% | 83.3%
$0.35$ | 89.5% | 84.2% | 77.6% | 67.2%
$0.24$ | 74.2% | 68.6% | 43.2% | 27.3%

We notice that even with 10% random color drift, the attack patches can still
maintain an attack rate as reliable as that for the original clean patches.
This performance suggests that the patch is relatively robust and can
withstand slight color variations during printing. Figure 5 demonstrates a set
of attack patches with different noise levels. As the level of noise
increases, the patch experiences certain color deviations. However, adjusting
the color of the patch is distinct from this process because it uniformly
alters all the pixels to maintain the local texture while achieving the
desired color.

Figure 5: Patches with different color drift.

####  4.3.3 Patch scaling

In practical deployments, it is often the case that the environment can
accommodate larger sized attack patches. As demonstrated in the previous
sections, training larger sized patches can result in improved attack
performance. However, in such scenarios, there may not be enough time to train
a new patch with a custom size. Therefore, we are motivated to investigate
whether we can enhance the performance of a pre-existing attack patch by using
interpolation.

We use a patch with brightness range=0.24, and its initial attack success rate
is 74.2%. Then we use bilinear interpolation to get patches with sizes 84x84,
98x98, and 112x112. The success rates of the different patches are shown in
Table 3.

Table 3: Performance across different sizes Patch size | 70x70 | 84x84 | 98x98 | 112x112
---|---|---|---|---
Success rate | 74.2% | 83.4% | 87.4% | 88.4%

We notice that the attack success rate increases as the patch size grows.
However, we also observe a diminishing marginal utility with each increment in
size. In fact, we find that the success rate of the bilinear interpolation
patch, even when it is the same size as the original patch, is lower. Some
images with different scaled patches are shown in Figure 6.

Figure 6: Patch scaling. (left: original 70x70 patch; middle: scaled 98x98
patch; right: original 98x98 patch)

###  4.4 Physical-world attacks

The physical-world deployability of normal attack patches has been
demonstrated in many works. However, a reasonable concern about the proposed
BrPatch is whether the restricted brightness reduced the robustness of the
patch in physical-world attacks. In order to address this concern, we designed
several physical-world deploy instances to show that the proposed attack patch
can still work robustly in the physical world.

We have selected two typical scenarios for the experiment: outdoor with
natural light and indoor with artificial light. The artificial light has a
color temperature of around 3000K (warm white), whereas the natural light has
a color temperature of around 6500K (daylight). For each scenario, we took
several images from different angles and distances.

Figure 7 shows some figures and the predictions with different patches. It is
first observed that although the conspicuousness of the printed BrPatch is not
as insignificant as in the digital domain, it still greatly reduces the
visibility compared to the normal attack patch (especially with the hue
mapping). We can then find that the probability score of the proposed patch
does not obviously decrease compared to the original adversarial attack patch.
This experiment demonstrates that the proposed BrPatch remains sufficiently
robust for physical-world deployment even when losing over 65% of the
brightness.

Figure 7: Comparison between AdvPatch and BrPatch in the physical world

##  5 Conclusion

This paper presents a novel approach to reduce the visibility of adversarial
attack patches in the physical world: the brightness-restricted patch
(BrPatch). The BrPatch is the first attempt to use optical characteristics
(brightness) to minimize detectability. This approach allows BrPatch to
maintain image independence and significantly enhances its practical
applicability in physical-world attacks. We analyze the impact of various
image features (color, texture, noise, and size) on the effectiveness of an
attack patch in physical-world deployment and show that attack patches exhibit
strong redundancy to brightness and are robust to color transfer and noise.
Our experiments also demonstrate the robustness of the BrPatch to different
physical world scenarios. The BrPatch presents an advanced camouflage
technique, offering a reliable and effective solution for safeguarding objects
or individuals from being detected by DNN-based equipment.

## References

  * [1] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok.  Synthesizing robust adversarial examples.  In International conference on machine learning, pages 284–293, 2018.
  * [2] Yang Bai, Yan Feng, Yisen Wang, Tao Dai, Shu-Tao Xia, and Yong Jiang.  Hilbert-based generative defense for adversarial examples.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4784–4793, 2019.
  * [3] Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer.  Adversarial patch.  arXiv preprint arXiv:1712.09665, 2017.
  * [4] Yinpeng Dong, Hang Su, Baoyuan Wu, Zhifeng Li, Wei Liu, Tong Zhang, and Jun Zhu.  Efficient decision-based black-box adversarial attacks on face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7714–7722, 2019.
  * [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [6] Ranjie Duan, Xingjun Ma, Yisen Wang, James Bailey, A Kai Qin, and Yun Yang.  Adversarial camouflage: Hiding physical-world attacks with natural styles.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1000–1008, 2020.
  * [7] Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song.  Robust physical-world attacks on machine learning models.  arXiv preprint arXiv:1707.08945, 2(3):4, 2017.
  * [8] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song.  Robust physical-world attacks on deep learning visual classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [9] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  * [12] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio.  Adversarial examples in the physical world.  In Artificial intelligence safety and security, pages 99–112. 2018\.
  * [13] Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey.  Characterizing adversarial subspaces using local intrinsic dimensionality.  arXiv preprint arXiv:1801.02613, 2018.
  * [14] Aleksander Madry, Aleksander Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial examples.  arXiv preprint arXiv:1706.06083, 2017.
  * [15] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami.  The limitations of deep learning in adversarial settings.  In 2016 IEEE European symposium on security and privacy (EuroS&P), pages 372–387, 2016.
  * [16] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [17] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter.  Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition.  In Proceedings of the 2016 acm sigsac conference on computer and communications security, pages 1528–1540, 2016.
  * [18] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199, 2013.
  * [19] Yisen Wang, Xuejiao Deng, Songbai Pu, and Zhiheng Huang.  Residual convolutional ctc networks for automatic speech recognition.  arXiv preprint arXiv:1702.07793, 2017.
  * [20] Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu.  On the convergence and robustness of adversarial training.  arXiv preprint arXiv:2112.08304, 2021.
  * [21] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu.  Improving adversarial robustness requires revisiting misclassified examples.  In International Conference on Learning Representations, 2020.
  * [22] Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma.  Skip connections matter: On the transferability of adversarial examples generated with resnets.  arXiv preprint arXiv:2002.05990, 2020.
  * [23] Zuxuan Wu, Ser-Nam Lim, Larry S Davis, and Tom Goldstein.  Making an invisibility cloak: Real world adversarial attacks on object detectors.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 1–17, 2020\.
  * [24] Min Zeng, Yisen Wang, and Yuan Luo.  Dirichlet latent variable hierarchical recurrent encoder-decoder in dialogue generation.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1267–1272, 2019.
  * [25] Yang Zhang, Hassan Foroosh, Philip David, and Boqing Gong.  Camou: Learning physical vehicle camouflages to adversarially attack detectors in the wild.  In International Conference on Learning Representations, 2019.