# Lincoln AI Computing Survey (LAICS) Update
††thanks: This material is based upon work supported by the Assistant
Secretary of Defense for Research and Engineering under Air Force Contract No.
FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily
reflect the views of the Assistant Secretary of Defense for Research and
Engineering.

Albert Reuther, Peter Michaleas, Michael Jones,
Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner
MIT Lincoln Laboratory Supercomputing Center
Lexington, MA, USA
<EMAIL_ADDRESS>
###### Abstract

This paper is an update of the survey of AI accelerators and processors from
past four years, which is now called the Lincoln AI Computing Survey – LAICS
(pronounced “lace”). As in past years, this paper collects and summarizes the
current commercial accelerators that have been publicly announced with peak
performance and peak power consumption numbers. The performance and power
values are plotted on a scatter graph, and a number of dimensions and
observations from the trends on this plot are again discussed and analyzed.
Market segments are highlighted on the scatter plot, and zoomed plots of each
segment are also included. Finally, a brief description of each of the new
accelerators that have been added in the survey this year is included.

###### Index Terms:

Machine learning, GPU, TPU, tensor, dataflow, CGRA, accelerator, embedded
inference, computational performance

##  I Introduction

A number of announcements, releases, and deployments of artificial
intelligence (AI) accelerators from startups and established technology
companies have occurred in the past year. Perhaps most notable is the
emergence of very large foundation models that are able to generate prose,
poetry, images, etc. based on training using vast amounts of data usually
collected via internet data crawls. Much technical press has been focused on
how effective the resulting tools will be for various tasks, but also there is
much discussion about the training of these models. But from an accelerator
perspective, it is the very same accelerators that are aimed towards training
more modestly sized models that are used for training these very large models.
The very large models are just using many more accelerators simultaneously in
a synchronous parallel manner, and they are interconnected with very high
bandwidth networks. But beyond that news, not much has changed in the overall
trends and landscape. Hence, this paper just updates what was discussed in
last year’s survey.

For much of the background of this study, please refer to one of the previous
IEEE-HPEC papers that our team has published [1, 2, 3, 4]. This background
includes an explanation of the AI ecosystem architecture, the history of the
emergence of AI accelerators and accelerators in general, a more detailed
explanation of the survey scatter plots, and a discussion of broader
observations and trends.

Figure 1: Peak performance vs. power scatter plot of publicly announced AI
accelerators and processors.

(a)

(b)

(c)

(d)

(e)

Figure 2: Zoomed regions of peak performance vs. peak power scatter plot: (a)
very low power, (b) embedded, (c) autonomous, (d) data center chips and cards,
(e) data center systems.

##  II Survey of Processors

This paper is an update to IEEE-HPEC papers from the past four years [1, 2, 3,
4]. This survey continues to cast a wide net to include accelerators and
processors for a variety of applications including defense and national
security AI/ML edge applications. The survey collects information on all of
the numerical precision types that an accelerator supports, but for most of
them, their best inference performance is in int8 or fp16/bf16, so that is
what usually is plotted. This survey gathers performance and power information
from publicly available materials including research papers, technical trade
press, company benchmarks, etc. The key metrics of this public data are
plotted in Figure 1, which graphs recent processor capabilities (as of Summer
2023) mapping peak performance vs. power consumption, and Table I summarizes
some of the important metadata of the accelerators, cards, and systems,
including the labels used in Figure 1.

The x-axis indicates peak power, and the y-axis indicate peak giga-operations
per second (GOps/s), both on a logarithmic scale. The computational precision
of the processing capability is depicted by the geometric marker used. The
form factor is depicted by color, which shows the package for which peak power
is reported. Blue corresponds to a single chip; orange corresponds to a card;
and green corresponds to entire systems (single node desktop and server
systems). Finally, the hollow geometric objects are peak performance for
inference-only accelerators, while the solid geometric figures are performance
for accelerators that are designed to perform both training and inference.

A reasonable categorization of accelerators follows their intended
application, and the five categories are shown as ellipses on the graph, which
roughly correspond to performance and power consumption: Very Low Power for
wake word detection, speech processing, very small sensors, etc.; Embedded for
cameras, small UAVs and robots, etc.; Autonomous for driver assist services,
autonomous driving, and autonomous robots; Data Center Chips and Cards; and
Data Center Systems. A zoomed in scatter plot for each of these categories is
shown in the subfigures of Figure 2.

TABLE I: List of accelerator metadata and labels for plots. Company | Product | Label | Technology | Form Factor | References
---|---|---|---|---|---
Achronix | VectorPath S7t-VG6 | Achronix | FPGA | Card | [5]
Aimotive | aiWare3 | Aimotive | dataflow | Chip | [6]
AIStorm | AIStorm | AIStorm | dataflow | Chip | [7]
Alibaba | HanGuang 800 | Alibaba | dataflow | Card | [8]
AlphaIC | RAP-E | AlphaIC | dataflow | Chip | [9]
Amazon | Inferentia | AWSi1 | dataflow | Card | [10, 11]
AMD | MI100 | AMD-MI100 | GPU | Card | [12]
AMD | MI210 | AMD-MI210 | GPU | Card | [12]
AMD | MI250 | AMD-MI250 | GPU | Card | [12]
AMD | MI300 | AMD-MI300 | GPU | Card | [13, 14]
ARM | Ethos N77 | Ethos | dataflow | Chip | [15]
Axelera | Axelera Test Core | Axelera | dataflow | Chip | [16]
Baidu | Baidu Kunlun 200 | Baidu-K1 | dataflow | Chip | [17, 18, 19]
Baidu | Baidu Kunlun II | Baidu-K2 | dataflow | Chip | [20]
Biren Technology | br100 | br100 | GPU | Card | [21, 22]
Biren Technology | br104 | br104 | GPU | Card | [21, 22]
Bitmain | BM1880 | Bitmain | dataflow | Chip | [23]
Blaize | El Cano | Blaize | dataflow | Card | [24]
Canaan | Kendrite K210 | Kendryte | CPU | Chip | [25]
Cerebras | CS-1 | CS-1 | dataflow | System | [26]
Cerebras | CS-2 | CS-2 | dataflow | System | [27]
Cornami | Cornami | Cornami | dataflow | Chip | [28]
Enflame | Cloudblazer T10 | Enflame | CPU | Card | [29]
Esperanto | ET-SoC-1 | ES-1 | CPU | Card | [30, 31]
Google | TPU Edge | TPUedge | tensor | System | [32]
Google | TPU1 | TPU1 | tensor | Chip | [33, 34]
Google | TPU2 | TPU2 | tensor | Chip | [33, 34]
Google | TPU3 | TPU3 | tensor | Chip | [35, 33, 34]
Google | TPU4i | TPU4i | tensor | Chip | [35]
Google | TPU4 | TPU4 | tensor | Chip | [36]
GraphCore | C2 | GraphCore | dataflow | Card | [37, 38]
GraphCore | C2 | GraphCoreNode | dataflow | System | [39]
GraphCore | Colossus Mk2 | GraphCore2 | dataflow | Card | [40]
GraphCore | Bow-2000 | GraphCoreBow | dataflow | Card | [41]
GreenWaves | GAP8 | GAP8 | dataflow | Chip | [42, 43]
GreenWaves | GAP9 | GAP9 | dataflow | Chip | [42, 43]
Groq | Groq Node | GroqNode | dataflow | System | [44]
Groq | Groq Node | GroqNode | dataflow | System | [44]
Groq | Tensor Streaming Processor | Groq | dataflow | Card | [37, 45]
Groq | Tensor Streaming Processor | Groq | dataflow | Card | [37, 45]
Gyrfalcon | Gyrfalcon | Gyrfalcon | PIM | Chip | [46]
Gyrfalcon | Gyrfalcon | GyrfalconServer | PIM | System | [47]
Habana | Gaudi | Gaudi | dataflow | Card | [48, 49]
Habana | Goya HL-1000 | Goya | dataflow | Card | [50, 49, 50]
Hailo | Hailo-8 | Hailo-8 | dataflow | Chip | [51]
Hailo | Hailo-15H | Hailo-15 | dataflow | Chip | [52]
Horizon Robotics | Journey2 | Journey2 | dataflow | Chip | [53]
Huawei HiSilicon | Ascend 310 | Ascend-310 | dataflow | Chip | [54]
Huawei HiSilicon | Ascend 910 | Ascend-910 | dataflow | Chip | [55]
Intel | Arria 10 1150 | Arria | FPGA | Chip | [56, 57]
Intel | Mobileye EyeQ5 | EyeQ5 | dataflow | Chip | [24]
Intel | Xe-HPC | Intel-Xe-HPC | GPU | Card | [58, 59, 60]
Intel | Flex140 | Flex140 | GPU | Card | [61]
Intel | Flex170 | Flex170 | GPU | Card | [61]
Kalray | Coolidge | Kalray | manycore | Chip | [62, 63]
Kneron | KL720 | KL720 | dataflow | Chip | [64]
Maxim | Max 78000 | Maxim | dataflow | Chip | [65, 66, 67]
MemryX | MX3 | MX3 | dataflow | Chip | [68, 69]
Mythic | M1076 | Mythic76 | PIM | Chip | [70, 71, 72]
Mythic | M1108 | Mythic108 | PIM | Chip | [70, 71, 72]
NovuMind | NovuTensor | NovuMind | dataflow | Chip | [73, 74]
NVIDIA | Ampere A10 | A10 | GPU | Card | [75]
NVIDIA | Ampere A100 | A100 | GPU | Card | [76]
NVIDIA | Ampere A800 | A800 | GPU | Card | [77]
NVIDIA | Ampere A30 | A30 | GPU | Card | [75]
NVIDIA | Ampere A40 | A40 | GPU | Card | [75]
NVIDIA | DGX Station | DGX-Station | GPU | System | [78]
NVIDIA | DGX-1 | DGX-1 | GPU | System | [78, 79]
NVIDIA | DGX-2 | DGX-2 | GPU | System | [79]
NVIDIA | DGX-A100 | DGX-A100 | GPU | System | [80]
NVIDIA | DGX-H100 | DGX-H100 | GPU | System | [81]
NVIDIA | H100 | H100 | GPU | Card | [82]
NVIDIA | Jetson AGX Xavier | XavierAGX | GPU | System | [83]
NVIDIA | Jetson NX Orin | OrinNX | GPU | System | [84, 85]
NVIDIA | Jetson AGX Orin | OrinAGX | GPU | System | [84, 85]
NVIDIA | Jetson TX1 | Jetson1 | GPU | System | [86]
NVIDIA | Jetson TX2 | Jetson2 | GPU | System | [86]
NVIDIA | Jetson Xavier NX | XavierNX | GPU | System | [83]
NVIDIA | DRIVE AGX L2 | AGX-L2 | GPU | System | [87]
NVIDIA | DRIVE AGX L5 | AGX-L5 | GPU | System | [87]
NVIDIA | L40 | L40 | GPU | Card | [88]
NVIDIA | Pascal P100 | P100 | GPU | Card | [89, 90]
NVIDIA | T4 | T4 | GPU | Card | [91]
NVIDIA | Volta V100 | V100 | GPU | Card | [92, 90]
Perceive | Ergo | Perceive | dataflow | Chip | [93]
Preferred Networks | MN-3 | PN-3 | multicore | Card | [94, 95]
Quadric | q1-64 | Quadric | dataflow | Chip | [96]
Qualcomm | Cloud AI 100 | Qcomm | dataflow | Card | [97, 98]
Qualcomm | QRB5165 | RB5 | GPU | System | [99]
Qualcomm | QRB5165N | RB6 | GPU | System | [100]
Rockchip | RK3399Pro | RK3399Pro | dataflow | Chip | [101]
SiMa.ai | SiMa.ai | SiMa.ai | dataflow | Chip | [102]
Syntiant | NDP101 | Syntiant | PIM | Chip | [103, 104]
Tachyum | Prodigy | Tachyum | CPU | Chip | [105]
Tenstorrent | Tenstorrent | Tenstorrent | multicore | Card | [106]
Tesla | Tesla FSC | Tesla | dataflow | System | [107, 108]
Texas Instruments | TDA4VM | TexInst | dataflow | Chip | [109, 110, 111]
Toshiba | 2015 | Toshiba | multicore | System | [112]
Untether | TsunAImi | TsunAImi | PIM | Card | [113]

For most of the accelerators, their descriptions and commentaries have not
changed since last year so please refer to the papers of the last four years
for descriptions and commentaries. Several new releases are included in this
update.

  * •

Based on similar technology of its Cloud AI 100 accelerator, Qualcomm has
released two versions of its robotics AI system platform, the RB5 and RB6, in
the past few years. Both are competing in the same low power system-on-a-chip
market as the NVIDIA Jetson product line, and are aimed at integration in
applications including robotics, driver assist, modest UAVs, etc. [99, 100].

  * •

The Memryx MX3 AI accelerator chip is a startup that was spun out of the
University of Michigan. It is designed to be deployed with a host CPU to
greatly speed up AI inference, consuming about 1W of power. It computes
activations with bf16 numerical precision, and store model parameter weight at
4-bit, 8-bit, and 16-bit integer precisions, which can be set on a layer-by-
layer basis [68, 69].

  * •

On the heels of it’s Hailo-8 AI accelerator, Hailo has released a lower power
variant, the Hailo-15. The Hailo-15 targets the Internet Protocol (IP) camera
market, and it is a SoC that includes a CPU, a digital signal processor (DSP)
accelerator, and a neural accelerator, which all draw less than 2W [52].

  * •

Startup Esperanto Technologies has released their first processor accelerator
called the ET-SoC-1. Each chip is comprised of 1,088 64-bit ET-Minion RISC-V
cores, each of which have scalar, vector, and tensor units along with L1
cache/scratchpad memory. Their key application is training and inference for
recommender systems, which have a balanced mix of scalar, vector, and tensor
operations [30, 31].

  * •

Baidu has started deploying its second-generation Kunlun accelerator, Kunlun
II. Baidu touted that the Kunlun II is 2-3 times faster than the original
Kunlun [20].

  * •

The Chinese GPU startup Biren emerged from stealth mode to announce and
release two high performance GPUs: the BR100 and BR104. The BR104 is a single
die GPU, while the BR100 combines two dies/chiplets in the same package [21,
22].

  * •

AMD has announced the followup to their Instinct MI250 GPU called the Instinct
MI300A, which will be a multi-chiplet CPU-GPU Accelerated Processing Unit
(APU) integrated package. The announcement showed package photos of two CPU
dies integrated with six GPU dies. [13, 14]

  * •

While Intel announced their high-end AI GPU a few years ago, details continued
to be scarce until this past year. Enough performance numbers were announced
for the Intel Xe-HPC (codename Ponte Vecchio) to include it in this year’s
survey [58, 59, 60]. Along with the Xe-HPC, Intel also announced and started
shipping two inference-oriented GPU cards, the Flex 140 and Flex 170 [61].

  * •

After announcing and shipping their Hopper H100 GPUs in systems at the end of
2022, NVIDIA has started shipping DGX servers, which integrate eight H100 GPUs
[81]. NVIDIA has also released a high-performance Ampere GPU, the A800, that
is aimed at the Chinese market, which reportedly performs at approximately 70%
peak performance of the A100 [77]. Finally, NVIDIA has released a new Ada
Lovelace GPU family which is aimed at the data center inference and graphics
rendering (gaming) farm markets. The first specifications were released for
the L40 GPU, which are included in this survey [88].

##  III Summary

This paper updates the Lincoln AI Computing Survey (LAICS) of deep neural
network accelerators that span from extremely low power through embedded and
autonomous applications to data center class accelerators for inference and
training. We presented the new full scatter plot along with zoomed in scatter
plots for each of the major deployment/market segments, and we discussed some
new additions for the year. The rate of announcements and releases has
continued to be consistent as companies compete for various embedded, data
center, cloud, and on-premises HPC deployments.

##  IV Data Availability

The data spreadsheets and references that have been collected for this study
and its papers will be posted at https://github.com/areuther/ai-accelerators
after they have cleared the release review process.

## Acknowledgement

We express our gratitude to Masahiro Arakawa, Bill Arcand, Bill Bergeron,
David Bestor, Bob Bond, Chansup Byun, Nathan Frey, Vitaliy Gleyzer, Jeff
Gottschalk, Michael Houle, Matthew Hubbell, Hayden Jananthan, Anna Klein,
David Martinez, Joseph McDonald, Lauren Milechin, Sanjeev Mohindra, Paul
Monticciolo, Julie Mullen, Andrew Prout, Stephan Rejto, Antonio Rosa, Charles
Yee, and Marc Zissman for their support of this work.

## References

  * [1] A. Reuther, P. Michaleas, M. Jones, V. Gadepally, S. Samsi, and J. Kepner, “Ai and ml accelerator survey and trends.” IEEE, 9 2022, pp. 1–10.
  * [2] ——, “Ai accelerator survey and trends,” 9 2021, pp. 1–9.
  * [3] ——, “Survey of machine learning accelerators,” 2020, pp. 1–12.
  * [4] ——, “Survey and benchmarking of machine learning accelerators.” Institute of Electrical and Electronics Engineers Inc., 9 2019. [Online]. Available: https://doi.org/10.1109/HPEC.2019.8916327
  * [5] G. Roos, “Fpga acceleration card delivers on bandwidth, speed, and flexibility,” 11 2019. [Online]. Available: https://www.eetimes.com/fpga-acceleration-card-delivers-on-bandwidth-speed-and-flexibility/
  * [6] “aiware3 hardware ip helps drive autonomous vehicles to production,” 10 2018. [Online]. Available: https://aimotive.com/news/content/1223
  * [7] R. Merritt, “Startup accelerates ai at the sensor,” 2 2019. [Online]. Available: https://www.eetimes.com/startup-accelerates-ai-at-the-sensor/
  * [8] T. Peng, “Alibaba’s new ai chip can process nearly 80k images per second,” 2019\. [Online]. Available: https://medium.com/syncedreview/alibabas-new-ai-chip-can-process-nearly-80k-images-per-second-63412dec22a3
  * [9] P. Clarke, “Indo-us startup preps agent-based ai processor,” 8 2018. [Online]. Available: https://www.eenewsanalog.com/news/indo-us-startup-preps-agent-based-ai-processor/page/0/1
  * [10] J. Hamilton, “Aws inferentia machine learning processor,” 11 2018. [Online]. Available: https://perspectives.mvdirona.com/2018/11/aws-inferentia-machine-learning-processor/
  * [11] C. Evangelist, “Deep dive into amazon inferentia: A custom-built chip to enhance ml and ai,” 1 2020. [Online]. Available: https://www.cloudmanagementinsider.com/amazon-inferentia-for-machine-learning-and-artificial-intelligence/
  * [12] R. Smith, “Amd announces instinct mi200 accelerator family: Taking servers to exascale and beyond,” 11 2021. [Online]. Available: https://www.anandtech.com/show/17054/amd-announces-instinct-mi200-accelerator-family-cdna2-exacale-servers
  * [13] T. P. Morgan, “Amd teases details on future mi300 hybrid compute engines,” 1 2023\.
  * [14] ——, “The third time charm of amd’s instinct gpu,” 6 2023. [Online]. Available: https://www.nextplatform.com/2023/06/14/the-third-time-charm-of-amds-instinct-gpu/
  * [15] D. Schor, “Arm ethos is for ubiquitous ai at the edge,” 2 2020. [Online]. Available: https://fuse.wikichip.org/news/3282/arm-ethos-is-for-ubiquitous-ai-at-the-edge/
  * [16] S. Ward-Foxton, “Axelera demos ai test chip after taping out in four months,” 5 2022. [Online]. Available: https://www.eetimes.com/axelera-demos-ai-test-chip-after-taping-out-in-four-months/
  * [17] J. Ouyang, X. Du, Y. Ma, and J. Liu, “Kunlun: A 14nm high-performance ai processor for diversified workloads,” vol. 64, 2 2021, pp. 50–51.
  * [18] R. Merritt, “Baidu accelerator rises in ai,” 7 2018. [Online]. Available: https://www.eetimes.com/baidu-accelerator-rises-in-ai/
  * [19] C. Duckett, “Baidu creates kunlun silicon for ai,” 7 2018. [Online]. Available: https://www.zdnet.com/article/baidu-creates-kunlun-silicon-for-ai/
  * [20] A. Shilov, “Baidu unveils kunlun ii ai chip: Rival for nvidia a100,” 8 2021. [Online]. Available: https://www.tomshardware.com/news/baidu-unveils-kunlun-ii-processor-for-ai
  * [21] O. Peckham, “Chinese startup biren details br100 gpu,” 8 2022. [Online]. Available: https://www.hpcwire.com/2022/08/22/chinese-startup-biren-details-br100-gpu/
  * [22] A. Shilov, “Chinese gpu firm biren plans ipo to better compete against nvidia,” 7 2023. [Online]. Available: https://www.tomshardware.com/news/biren-mulls-ipo
  * [23] B. Wheeler, “Bitmain soc brings ai to the edge,” 2 2019. [Online]. Available: https://www.linleygroup.com/newsletters/newsletter_detail.php%3Fnum=5975%26year=2019%26tag=3
  * [24] M. Demler, “Blaize ignites edge-ai performance,” pp. 1–5, 9 2020. [Online]. Available: https://www.blaize.com/wp-content/uploads/2020/09/Blaize-Ignites-Edge-AI-Performance.pdf
  * [25] L. Gwennap, “Kendryte embeds ai for surveillance,” 3 2019. [Online]. Available: https://www.linleygroup.com/newsletters/newsletter_detail.php?num=5992
  * [26] A. Hock, “Introducing the cerebras cs-1, the industry’s fastest artificial intelligence computer,” 11 2019. [Online]. Available: https://www.cerebras.net/introducing-the-cerebras-cs-1-the-industrys-fastest-artificial-intelligence-computer/
  * [27] T. Trader, “Cerebras doubles ai performance with second-gen 7nm wafer scale engine,” 4 2021. [Online]. Available: https://www.hpcwire.com/2021/04/20/cerebras-doubles-ai-performance-with-second-gen-7nm-wafer-scale-engine/
  * [28] “Cornami achieves unprecedented performance at lowest power dissipation for deep neural networks,” 10 2019. [Online]. Available: https://cornami.com/1416-2/
  * [29] P. Clarke, “Globalfoundries aids launch of chinese ai startup,” 12 2019. [Online]. Available: https://www.eenewsanalog.com/news/globalfoundries-aids-launch-chinese-ai-startup
  * [30] D. R. Ditzel and the Esperanto team, “Accelerating ml recommendation with over 1,000 risc-v/tensor processors on esperanto’s et-soc-1 chip,” _IEEE Micro_ , vol. 42, pp. 31–38, 5 2022.
  * [31] D. Schor, “A look at the et-soc-1, esperanto’s massively multi-core risc-v approach to ai,” 7 2021. [Online]. Available: https://fuse.wikichip.org/news/4911/a-look-at-the-et-soc-1-esperantos-massively-multi-core-risc-v-approach-to-ai/
  * [32] “Edge tpu,” 2019. [Online]. Available: https://cloud.google.com/edge-tpu/
  * [33] N. P. Jouppi, D. H. Yoon, G. Kurian, S. Li, N. Patil, J. Laudon, C. Young, and D. Patterson, “A domain-specific supercomputer for training deep neural networks,” _Commun. ACM_ , vol. 63, p. 67–78, 6 2020. [Online]. Available: https://doi.org/10.1145/3360307
  * [34] P. Teich, “Tearing apart google’s tpu 3.0 ai coprocessor,” 5 2018. [Online]. Available: https://www.nextplatform.com/2018/05/10/tearing-apart-googles-tpu-3-0-ai-coprocessor/
  * [35] N. P. Jouppi, D. H. Yoon, M. Ashcraft, M. Gottscho, T. B. Jablin, G. Kurian, J. Laudon, S. Li, P. Ma, X. Ma, T. Norrie, N. Patil, S. Prasad, C. Young, Z. Zhou, D. Patterson, and G. Llc, “Ten lessons from three generations shaped google’s tpuv4i.” IEEE Computer Society, 6 2021, pp. 1–14.
  * [36] O. Peckham, “Google cloud’s new tpu v4 ml hub packs 9 exaflops of ai,” 5 2022\. [Online]. Available: https://www.hpcwire.com/2022/05/16/google-clouds-new-tpu-v4-ml-hub-packs-9-exaflops-of-ai/
  * [37] L. Gwennap, “Groq rocks neural networks,” 1 2020. [Online]. Available: http://groq.com/wp-content/uploads/2020/04/Groq-Rocks-NNs-Linley-Group-MPR-2020Jan06.pdf
  * [38] D. Lacey, “Preliminary ipu benchmarks,” 10 2017. [Online]. Available: https://www.graphcore.ai/posts/preliminary-ipu-benchmarks-providing-previously-unseen-performance-for-a-range-of-machine-learning-applications
  * [39] “Dell dss8440 graphcore ipu server,” 2 2020. [Online]. Available: https://www.graphcore.ai/hubfs/Leadgenassets/DSS8440IPUServerWhitePaper_2020.pdf
  * [40] S. Ward-Foxton, “Graphcore takes on nvidia with second-gen ai accelerator,” 7 2020\. [Online]. Available: https://www.eetimes.com/graphcore-takes-on-nvidia-with-second-gen-ai-accelerator/
  * [41] M. Tyson, “Graphcore bow ipu introduces tsmc 3d wafer-on-wafer processor,” 3 2022\. [Online]. Available: https://www.tomshardware.com/news/graphcore-tsmc-bow-ipu-3d-wafer-on-wafer-processor
  * [42] “Gap application processors,” 2020. [Online]. Available: https://greenwaves-technologies.com/gap8_gap9/
  * [43] J. Turley, “Gap9 for ml at the edge,” 6 2020. [Online]. Available: https://www.eejournal.com/article/gap9-for-ml-at-the-edge/
  * [44] N. Hemsoth, “Groq shares recipe for tsp nodes, systems,” 9 2020. [Online]. Available: https://www.nextplatform.com/2020/09/29/groq-shares-recipe-for-tsp-nodes-systems/
  * [45] D. Abts, J. Ross, J. Sparling, M. Wong-VanHaren, M. Baker, T. Hawkins, A. Bell, J. Thompson, T. Kahsai, G. Kimmell, J. Hwang, R. Leslie-Hurd, M. Bye, E. R. Creswick, M. Boyd, M. Venigalla, E. Laforge, J. Purdy, P. Kamath, D. Maheshwari, M. Beidler, G. Rosseel, O. Ahmad, G. Gagarin, R. Czekalski, A. Rane, S. Parmar, J. Werner, J. Sproch, A. Macias, and B. Kurtz, “Think fast: A tensor streaming processor (tsp) for accelerating deep learning workloads,” 5 2020, pp. 145–158. [Online]. Available: https://doi.org/10.1109/ISCA45697.2020.00023
  * [46] S. Ward-Foxton, “Gyrfalcon unveils fourth ai accelerator chip — ee times,” 11 2019\. [Online]. Available: https://www.eetimes.com/gyrfalcon-unveils-fourth-ai-accelerator-chip/
  * [47] “Solidrun, gyrfalcon develop arm-based edge optimized ai inference server,” 2 2020\. [Online]. Available: https://www.hpcwire.com/off-the-wire/solidrun-gyrfalcon-develop-edge-optimized-ai-inference-server/
  * [48] L. Gwennap, “Habana offers gaudi for ai training,” 6 2019. [Online]. Available: https://habana.ai/wp-content/uploads/2019/06/Habana-Offers-Gaudi-for-AI-Training.pdf
  * [49] E. Medina and E. Dagan, “Habana labs purpose-built ai inference and training processor architectures: Scaling ai training systems using standard ethernet with gaudi processor,” _IEEE Micro_ , vol. 40, pp. 17–24, 3 2020. [Online]. Available: https://doi.org/10.1109/MM.2020.2975185
  * [50] L. Gwennap, “Habana wins cigar for ai inference,” 2 2019. [Online]. Available: https://www.linleygroup.com/mpr/article.php?id=12103
  * [51] S. Ward-Foxton, “Details of hailo ai edge accelerator emerge,” 8 2019. [Online]. Available: https://www.eetimes.com/details-of-hailo-ai-edge-accelerator-emerge/
  * [52] ——, “Hailo adds vision processor socs for smart cameras,” 3 2023. [Online]. Available: https://www.eetimes.com/hailo-adds-vision-processor-socs-for-smart-cameras/
  * [53] “Horizon robotics journey2 automotive ai processor series,” 2020. [Online]. Available: https://en.horizon.ai/product/journey
  * [54] Huawei, “Ascend 310 ai processor,” 2020. [Online]. Available: https://e.huawei.com/us/products/cloud-computing-dc/atlas/ascend-310
  * [55] ——, “Ascend 910 ai processor,” 2020. [Online]. Available: https://e.huawei.com/us/products/cloud-computing-dc/atlas/ascend-910
  * [56] M. S. Abdelfattah, D. Han, A. Bitar, R. DiCecco, S. O’Connell, N. Shanker, J. Chu, I. Prins, J. Fender, A. C. Ling, and G. R. Chiu, “Dla: Compiler and fpga overlay for neural network inference acceleration,” 8 2018, pp. 411–4117. [Online]. Available: https://doi.org/10.1109/FPL.2018.00077
  * [57] N. Hemsoth, “Intel fpga architecture focuses on deep learning inference,” 7 2018\. [Online]. Available: https://www.nextplatform.com/2018/07/31/intel-fpga-architecture-focuses-on-deep-learning-inference/
  * [58] A. Shilov, “Intel’s ponte vecchio xe-hpc gpu boasts 100 billion transistors,” 3 2021. [Online]. Available: https://www.tomshardware.com/news/intel-xe-hpc-ponte-vecchio-examined
  * [59] “Intel provides details about sapphire rapids cpu and ponte vecchio gpu,” 8 2021\. [Online]. Available: https://www.hpcwire.com/off-the-wire/intel-unveils-details-about-sapphire-rapids-cpu-ponte-vecchio-gpu-ipu/
  * [60] A. Shilov, “Intel’s ponte vecchio is finally in the wild,” 6 2022. [Online]. Available: https://www.tomshardware.com/news/intels-ponte-vecchio-smiles-for-the-camera
  * [61] T. P. Morgan, “Different gpu horses for different datacenter courses,” 10 2022\. [Online]. Available: https://www.nextplatform.com/2022/10/04/different-gpu-horses-for-different-datacenter-courses/
  * [62] B. D. de Dinechin, “Kalray’s mppa® manycore processor: At the heart of intelligent systems.” IEEE, 6 2019. [Online]. Available: https://www.european-processor-initiative.eu/dissemination-material/1259/
  * [63] P. Clarke, “Nxp, kalray demo coolidge parallel processor in ’bluebox’,” 1 2020\. [Online]. Available: https://www.eenewsanalog.com/news/nxp-kalray-demo-coolidge-parallel-processor-bluebox
  * [64] S. Ward-Foxton, “Kneron attracts strategic investors,” 1 2021. [Online]. Available: https://www.eetimes.com/kneron-attracts-strategic-investors/
  * [65] ——, “Maxim debuts homegrown ai accelerator in latest ulp soc,” 11 2020. [Online]. Available: https://www.eetimes.com/maxim-debuts-homegrown-ai-accelerator-in-latest-ulp-soc/
  * [66] A. Jani, “Maxim showcases efficient custom ai,” 2 2021. [Online]. Available: https://www.linleygroup.com/newsletters/newsletter_detail.php?num=6274&year=2021&tag=3
  * [67] M. Clay, C. Grecos, M. Shirvaikar, and B. Richey, “Benchmarking the max78000 artificial intelligence microcontroller for deep learning applications,” N. Kehtarnavaz and M. F. Carlsohn, Eds., vol. 12102. SPIE, 2022, pp. 47–52. [Online]. Available: https://doi.org/10.1117/12.2622390
  * [68] S. Leibson, “Adding low-power ai/ml inference to edge devices,” 4 2023. [Online]. Available: https://www.eetimes.com/adding-low-power-ai-ml-interference-to-edge-devices/
  * [69] S. Vicinanza, “Start-up memryx makes mx3 an ai accelerator that rivals established offerings,” 12 2022. [Online]. Available: https://circuitcellar.com/newsletter/start-up-memryx-makes-mx3-an-ai-accelerator-that-rivals-established-offerings/
  * [70] S. Ward-Foxton, “Mythic resizes its ai chip,” 6 2021. [Online]. Available: https://www.eetimes.com/mythic-resizes-its-analog-ai-chip/
  * [71] N. Hemsoth, “A mythic approach to deep learning inference,” 8 2018. [Online]. Available: https://www.nextplatform.com/2018/08/23/a-mythic-approach-to-deep-learning-inference/
  * [72] D. Fick, “Mythic @ hot chips 2018,” 8 2018. [Online]. Available: https://medium.com/mythic-ai/mythic-hot-chips-2018-637dfb9e38b7
  * [73] K. Freund, “Novumind: An early entrant in ai silicon,” 5 2019. [Online]. Available: https://moorinsightsstrategy.com/wp-content/uploads/2019/05/NovuMind-An-Early-Entrant-in-AI-Silicon-By-Moor-Insights-And-Strategy.pdf
  * [74] J. Yoshida, “Novumind’s ai chip sparks controversy,” 10 2018. [Online]. Available: https://www.eetimes.com/novuminds-ai-chip-sparks-controversy/
  * [75] T. P. Morgan, “Nvidia rounds out ”ampere” lineup with two new accelerators,” 4 2021. [Online]. Available: https://www.nextplatform.com/2021/04/15/nvidia-rounds-out-ampere-lineup-with-two-new-accelerators/
  * [76] R. Krashinsky, O. Giroux, S. Jones, N. Stam, and S. Ramaswamy, “Nvidia ampere architecture in-depth,” 5 2020. [Online]. Available: https://devblogs.nvidia.com/nvidia-ampere-architecture-in-depth/
  * [77] A. Shilov, “Nvidia’s chinese a800 gpu’s performance revealed,” 5 2023. [Online]. Available: https://www.tomshardware.com/news/nvidia-a800-performance-revealed
  * [78] P. Alcorn, “Nvidia infuses dgx-1 with volta, eight v100s in a single chassis,” 5 2017. [Online]. Available: https://www.tomshardware.com/news/nvidia-volta-v100-dgx-1-hgx-1,34380.html
  * [79] I. Cutress, “Nvidia’s dgx-2: Sixteen tesla v100s, 30tb of nvme, only $400k,” 3 2018. [Online]. Available: https://www.anandtech.com/show/12587/nvidias-dgx2-sixteen-v100-gpus-30-tb-of-nvme-only-400k
  * [80] C. Campa, C. Kawalek, H. Vo, and J. Bessoudo, “Defining ai innovation with nvidia dgx a100,” 5 2020. [Online]. Available: https://devblogs.nvidia.com/defining-ai-innovation-with-dgx-a100/
  * [81] H. Mujtaba, “Nvidia unveils hopper gh100 powered dgx h100, dgx pod h100, h100 pcie accelerators,” 3 2022. [Online]. Available: https://wccftech.com/nvidia-unveils-hopper-gh100-powered-dgx-h100-dgx-pod-h100-h100-pcie-accelerators/
  * [82] R. Smith, “Nvidia hopper gpu architecture and h100 accelerator announced: Working smarter and harder,” 3 2022. [Online]. Available: https://www.anandtech.com/show/17327/nvidia-hopper-gpu-architecture-and-h100-accelerator-announced
  * [83] ——, “Nvidia gives jetson agx xavier a trim, announces nano-sized jetson xavier nx,” 11 2019. [Online]. Available: https://www.anandtech.com/show/15070/nvidia-gives-jetson-xavier-a-trim-announces-nanosized-jetson-xavier-nx
  * [84] B. Funk, “Nvidia jetson agx orin: The next-gen platform that will power our ai robot overlords unveiled,” 3 2022. [Online]. Available: https://hothardware.com/news/nvidia-jetson-agx-orin
  * [85] “Jetson agx orin for next-gen robotics,” 2022. [Online]. Available: https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/
  * [86] D. Franklin, “Nvidia jetson tx2 delivers twice the intelligence to the edge,” 3 2017. [Online]. Available: https://developer.nvidia.com/blog/jetson-tx2-delivers-twice-intelligence-edge/
  * [87] B. Hill, “Nvidia unveils ampere-infused drive agx for autonomous cars, isaac robotics platform with bmw partnership,” 5 2022. [Online]. Available: https://hothardware.com/news/nvidia-drive-agx-pegasus-orin-ampere-next-gen-autonomous-cars
  * [88] “Nvidia l40,” 2023. [Online]. Available: https://www.techpowerup.com/gpu-specs/l40.c3959
  * [89] “Nvidia tesla p100.” [Online]. Available: https://www.nvidia.com/en-us/data-center/tesla-p100/
  * [90] R. Smith, “16gb nvidia tesla v100 gets reprieve; remains in production,” 5 2018\. [Online]. Available: https://www.anandtech.com/show/12809/16gb-nvidia-tesla-v100-gets-reprieve-remains-in-production
  * [91] E. Kilgariff, H. Moreton, N. Stam, and B. Bell, “Nvidia turing architecture in-depth,” 9 2018. [Online]. Available: https://developer.nvidia.com/blog/nvidia-turing-architecture-in-depth/
  * [92] “Nvidia tesla v100 tensor core gpu,” 2019. [Online]. Available: https://www.nvidia.com/en-us/data-center/tesla-v100/
  * [93] J. McGregor, “Perceive exits stealth with super efficient machine learning chip for smarter devices,” 4 2020. [Online]. Available: https://www.forbes.com/sites/tiriasresearch/2020/04/06/perceive-exits-stealth-with-super-efficient-machine-learning-chip-for-smarter-devices/
  * [94] “Mn-core,” 2020. [Online]. Available: https://projects.preferred.jp/mn-core/en/
  * [95] I. Cutress, “Preferred networks: A 500 w custom pcie card using 3000 mm2 silicon,” 12 2019. [Online]. Available: https://www.anandtech.com/show/15177/preferred-networks-a-500-w-custom-pcie-card-using-3000-mm2-silicon
  * [96] D. Firu, “Quadric edge supercomputer,” 4 2019. [Online]. Available: https://quadric.io/supercomputing.pdf
  * [97] S. Ward-Foxton, “Qualcomm cloud ai 100 promises impressive performance per watt for near-edge ai,” 9 2020. [Online]. Available: https://www.eetimes.com/qualcomm-cloud-ai-100-promises-impressive-performance-per-watt-for-near-edge-ai/
  * [98] D. McGrath, “Qualcomm targets ai inferencing in the cloud,” 4 2019. [Online]. Available: https://www.eetimes.com/qualcomm-targets-ai-inferencing-in-the-cloud/#
  * [99] S. Crowe, “Qualcomm robotics rb5 platform puts 5g, ai in developers’ hands,” 6 2020. [Online]. Available: https://www.therobotreport.com/qualcomm-robotics-rb5-platform-puts-5g-ai-in-developers-hands/
  * [100] “Robotics rb6 platform,” 2023. [Online]. Available: https://www.qualcomm.com/products/internet-of-things/industrial/industrial-automation/robotics-rb6-platform
  * [101] “Rockchip released its first ai processor rk3399pro npu performance up to 2.4tops,” 1 2018. [Online]. Available: https://www.rock-chips.com/a/en/News/Press_Releases/2018/0108/869.html
  * [102] L. Gwennap, “Machine learning moves to the edge,” pp. 1–7, 4 2020. [Online]. Available: https://www.linleygroup.com/uploads/sima-machine-learning-moves-to-the-edge-wp.pdf
  * [103] D. McGrath, “Tech heavyweights back ai chip startup,” 10 2018. [Online]. Available: https://www.eetimes.com/tech-heavyweights-back-ai-chip-startup/
  * [104] R. Merritt, “Startup rolls ai chips for audio,” 2 2018. [Online]. Available: https://www.eetimes.com/startup-rolls-ai-chips-for-audio/
  * [105] A. Shilov, “Tachyum teases 128-core cpu: 5.7 ghz, 950w, 16 ddr5 channels,” 6 2022\. [Online]. Available: https://www.tomshardware.com/news/tachyum-teases-128-core-cpu-57-ghz-950w-16-ddr5-channels
  * [106] L. Gwennap, “Tenstorrent scales ai performance: Architecture leads in data-center power efficiency,” pp. 1–4, 4 2020. [Online]. Available: https://www.tenstorrent.com/wp-content/uploads/2020/04/Tenstorrent-Scales-AI-Performance.pdf
  * [107] E. Talpes, D. D. Sarma, G. Venkataramanan, P. Bannon, B. McGee, B. Floering, A. Jalote, C. Hsiong, S. Arora, A. Gorti, and G. S. Sachdev, “Compute solution for tesla’s full self-driving computer,” _IEEE Micro_ , vol. 40, pp. 25–35, 3 2020. [Online]. Available: https://doi.org/10.1109/MM.2020.2975764
  * [108] “Fsd chip - tesla,” 2020. [Online]. Available: https://en.wikichip.org/wiki/tesla_(car_company)/fsd_chip
  * [109] S. Ward-Foxton, “Ti’s first automotive soc with an ai accelerator launches,” 2 2021. [Online]. Available: https://www.eetimes.com/tis-first-automotive-soc-with-an-ai-accelerator-launches/
  * [110] “Tda4vm jacinto processors for adas and autonomous vehicles,” 3 2021. [Online]. Available: https://www.ti.com/lit/gpn/tda4vm
  * [111] M. Demler, “Ti jacinto accelerates level 3 adas,” 3 2020. [Online]. Available: https://www.linleygroup.com/newsletters/newsletter_detail.php?num=6130&year=2020&tag=3
  * [112] R. Merritt, “Samsung, toshiba detail ai chips,” 2 2019. [Online]. Available: https://www.eetimes.com/samsung-toshiba-detail-ai-chips/
  * [113] L. Gwennap, “Untether delivers at-memory ai,” 11 2020. [Online]. Available: https://www.linleygroup.com/newsletters/newsletter_detail.php?num=6230


# TWO HEADED DRAGONS: MULTIMODAL FUSION AND CROSS MODAL TRANSACTIONS

###### Abstract

As the field of remote sensing is evolving, we witness the accumulation of
information from several modalities, such as multispectral (MS), hyperspectral
(HSI), LiDAR etc. Each of these modalities possess its own distinct
characteristics and when combined synergistically, perform very well in the
recognition and classification tasks. However, fusing multiple modalities in
remote sensing is cumbersome due to highly disparate domains. Furthermore, the
existing methods do not facilitate cross-modal interactions. To this end, we
propose a novel transformer based fusion method for HSI and LiDAR modalities.
The model is composed of stacked auto encoders that harness the cross key-
value pairs for HSI and LiDAR, thus establishing a communication between the
two modalities, while simultaneously using the CNNs to extract the spectral
and spatial information from HSI and LiDAR. We test our model on Houston (Data
Fusion Contest – 2013) and MUUFL Gulfport datasets and achieve competitive
results.

Index Terms—  Hyperspectral, LiDAR, domain mapping, multimodal fusion, cross-
modal inferences

##  1 Introduction

The existing single modal deep learning classifiers perform well on
classification tasks. Having said that, they still have a huge scope of
improvements if presented with a complementary dataset such as a corresponding
LiDAR digital surface model (DSM) map of the area. With multiple modalities,
the classification accuracies of the single modal classifiers are boosted
significantly. For instance, the case where the reflectance information is not
sufficient to identify areas with similar reflectances, such as water and
shadow. In those cases, additional information (such as depth from LiDAR
images), can help in classification [1]. Thus, having a fusion based model can
learn the mapping of hyperspectral and LiDAR attributes to a common space,
where it can infer a combined knowledge for better classification. The general
approaches mainly revolve around two stream CNN structures which at some point
exploit data fusion parameters for achieving high accuracies (refer Fig. 1).

Fig. 1: Traditional HSI-LiDAR fusion module. The HSI and LiDAR features are
concatenated and sent for classification.

The use of optimal data encapsulation for feature reconstruction is
demonstrated to improve accuracies beyond the traditional approaches [2]. This
method establishes the elegance of reconstruction using concatenated deep
features that can hugely improve classification performance. Approach [3]
being a simple yet innovative architecture performs proficiently and paved way
towards concatenation based classification. [1] further improved the fusion
performance by introducing visual attention in HSI-LiDAR fusion. The novel
attention method helps in highlighting important data features that would be
otherwise lost. It was to our knowledge, the first attempt to introduce cross-
modal attention for initiating interaction between different modalities. The
above papers have explored the different mixtures of machine learning/deep
learning approaches and have proved that the deep fusion of features works
best in a two stream deployment. Other methods like [4, 5] which have already
been tried and tested in the field of data fusion, prove their merits in image
classification with little cross-modal interaction.

Since, the previous approaches perform fusion either at the beginning or at
the intermediate level, their models are unable to effectively capture the
representations in the fused space. Hence, we go for cross-modal interaction
at different levels to fuse high and low level features, that are important
for classification. To this end, we draw the idea from the transformer
architecture [6] which played a key role in using cross-attention as well as
self-attention for long sequence NLP tasks. Having seen the transformer define
the state of the art for various cross-domain tasks like machine translation
and sequence to sequence modelling, we attempt to solve a classic cross-domain
fusion problem. Our contributions in this paper would be:

  * •

We redefine queries with cross key-value pairs for multimodal data fusion of
HSI and LiDAR images as images of different domains belong in the same
geographical location.

  * •

We utilize CNNs to capture spatial data while using attention modes to capture
the spectral essence and to achieve a true spatio-spectral fusion module.

  * •

We introduce a modular design for an attempt at maximum cross modal
interaction with spectro-spatial feature selection based on attention with
dual modalities.

##  2 METHODOLOGY

We propose a stacked encoder-decoder architecture where attention based cross
modal information is preserved. The filter block generates a representative
spectral map based on spatial features, such as queries, keys and values for
their respective inputs. The spectral attention decided on the basis of
queries and keys of same modality is captured in the self attention blocks,
which scores closely related pixels higher, thereby giving attention in the
spectral zone. The cross-attention decoder takes the attention scores and
decodes the corresponding cross key-value pairs by highlighting the important
information based on the input attention scores. The ‘CrossOut’ (section 2.2)
ensures that the input data is preserved through the decoding. The
architecture being modular in design, aides in easy stacking of the blocks. As
key-value pairs are cross-modal, there we exploit the cross-modal
interactions.

###  2.1 Encoder block

Fig. 2: Encoder block components. (a) Filter block, (b) Self attention module,
(c) Encoder block. The inputs for the encoder block are the originals
modalities (when placed in the beginning) or the outputs of the previous
decoders (for all subsequent encoders).

Encoder block [Fig. 2 (c)] consists of 2 components, namely, Filter Block and
Self Attention Module. This block generates the queries and corresponding key-
value pairs to generate attention score based weights (inspired from
transformers [6]).

Filter Block: It is a Conv2D → Layer Normalization → Activation implementation
[Fig. 2 (a)], where convolution is done with padding and kernel size is set to
3 with number of filters as ‘$d$’. The filter block being a convolutional
block captures the spatial data for both the modalities. The outputs of the
filter blocks are their corresponding queries with key-value pairs. The
convolutional filter block ensures that the generated key-value pairs are in
the same geolocation but have individual channel signatures. The query-key-
value triplets are used for computing self-attention and cross-attention
features.

Self Attention Module: The Self Attention block [Fig. 2 (b)] is the attention
mechanism inspired from the multiplicative attention based on luong attention
[7]. The scaled dot product attention is defined as $Q\odot K^{T}/\sqrt{d}$
(‘$\odot$’ stands for the vector dot product). It takes queries ($Q$) and keys
($K$) as inputs, computes the scaled dot product and performs softmax on it.
The scaled dot product based attention maps similarity in the spectral space
and the softmax layer pushes the row-wise attention scores to (0, 1). The
scaling helps is managing the dot products when they tend to get high values
[6].

###  2.2 Decoder block

Fig. 3: Decoder block components. (a) CrossOut block, (b) Cross-attention
module, (c) Decoder block. Here, the input comes from the attention layers and
filter blocks of the encoder.

The decoder [Fig. 3 (c)] in its entirety, decodes the cross key-value pairs
effectively and select attention based features using the cross decoder
module. The CrossOut module ensures that the input modal data is retained
along with highlighted cross-modal features to optimally capture data.

Cross-Attention Decoder: It [Fig. 3 (b)] implements the cross-decoding based
on prior key-value pairs. It is an $A\odot V$ implementation. $A$ is the
cross-domain self-attention score and $V$ is the value generated from the
respective filter block.

CrossOut: It is a residual block implementation that follows {Input1 + Input2}
followed by layer normalization [8] [Fig. 3 (a)]. This ensures that the
abstractions includes features from both cross and individual domains.

###  2.3 The full stack: Two headed dragon

Fig. 4: Schematic of full stack with both encoder and decoder.

The full stack [Fig. 4] is the stacking of the Encoder Block and the Decoder
Block in a way that cross-attention is achieved. The number of stacks
($N_{X}$) can be varied as required. Also, this being a modular build, the
number of filters for individual filter blocks can also be user defined. The
stacking also aids in multiple cross interaction with self signature
preservation. The two step attention process selectively filters the relevant
and important information to be preserved for the next step. There are two
outputs which are complementary yet have interacted using cross-attention,
hence the name “Two headed dragon”.

###  2.4 Proposed architecture

Fig. 5: Schematic of complete fusion based classification model.

We use a 4 Stacked Encoder - Decoder architecture [Fig. 5], where the output
from all the stacks is concatenated, which represents a global common space
representation for the HSI and LiDAR patches. This concatenated feature passes
through a classification block for classification. The classification block is
a Global Average Pooling → Dropout → Softmax (number of classes)
implementation. The entire model is trained on categorical crossentropy loss,
given as:

$\mathcal{L}_{CE}=-\sum_{i=1}^{N}y^{i}\log{\hat{y}^{i}}$ (1)

where, $N$ is the number of training samples, $y^{i}$ and $\hat{y}^{i}$ are
respectively groundtruth class and prediction for the $i^{th}$ sample.

Table 1:  Accuracy analysis of 2013 Houston dataset (in %). ‘H’ represents only HSI and ‘H+L’ represents fused HSI and LiDAR. Class |  EncDec [2] |  Two Branch CNN [9] |  Deep Fusion [3] |  FusAt Net [1] |  Ours
---|---|---|---|---|---
1 |  82.81 |  82.62 |  82.87 |  82.53 |  82.05
2 |  83.65 |  84.49 |  84.87 |  85.15 |  84.68
3 |  97.62 |  81.58 |  96.44 |  99.21 |  99.80
4 |  92.52 |  99.43 |  92.14 |  92.80 |  97.63
5 |  100 |  99.53 |  99.43 |  100 |  100
6 |  87.41 |  81.12 |  93.01 |  100 |  96.50
7 |  86.01 |  78.26 |  87.31 |  94.87 |  88.71
8 |  82.91 |  87.94 |  78.82 |  83.48 |  89.55
9 |  80.26 |  83.66 |  88.01 |  85.93 |  91.60
10 |  66.89 |  63.22 |  80.98 |  68.05 |  68.92
11 |  89.28 |  88.71 |  92.41 |  90.32 |  96.77
12 |  96.54 |  88.38 |  84.34 |  92.51 |  97.02
13 |  72.63 |  91.93 |  85.96 |  92.63 |  88.77
14 |  96.55 |  78.14 |  89.88 |  99.60 |  100
15 |  97.25 |  78.44 |  97.89 |  96.41 |  95.77
OA |  86.90 |  85.14 |  88.02 |  88.93 |  90.64
Only H |  83.82 |  79.32 |  84.07 |  85.15 |  85.98
Table 2:  Accuracy analysis of MUUFL dataset (in %). ‘H’ represents only HSI and ‘H+L’ represents fused HSI and LiDAR. Class |  EncDec [2] |  Two Branch CNN [9] |  Deep Fusion [3] |  FusAt Net [1] |  Ours
---|---|---|---|---|---
1 |  89.84 |  88.39 |  87.96 |  91.61 |  93.80
2 |  83.74 |  75.95 |  79.88 |  72.01 |  85.56
3 |  69.68 |  75.92 |  77.38 |  81.42 |  82.87
4 |  93.22 |  93.74 |  95.02 |  97.16 |  96.23
5 |  86.50 |  51.06 |  89.45 |  89.72 |  91.76
6 |  100 |  98.91 |  100 |  99.45 |  99.73
7 |  92.45 |  93.25 |  92.17 |  96.20 |  90.62
8 |  95.77 |  94.38 |  92.83 |  94.58 |  97.43
9 |  77.12 |  82.72 |  67.63 |  80.47 |  78.29
10 |  91.57 |  91.57 |  91.57 |  90.36 |  95.18
11 |  96.45 |  97.04 |  99.41 |  99.41 |  99.41
OA |  87.03 |  82.16 |  86.94 |  89.02 |  91.64
Only H |  86.84 |  81.57 |  85.16 |  88.03 |  90.02
Table 3:  Ablation study for change in stack number. Stack ($N_{x}$) |  1 |  2 |  3 |  4
---|---|---|---|---
Houston |  87.63 |  89.49 |  89.73 |  90.64
MUUFL |  85.49 |  88.18 |  90.06 |  91.64
Table 4:  Ablation study for change in embedding size. Embed Size |  64 |  96 |  128 |  160 |  192
---|---|---|---|---|---
Houston |  86.13 |  87.61 |  90.64 |  90.36 |  90.16
MUUFL |  88.80 |  89.19 |  91.64 |  90.62 |  90.25

##  3 PERFORMANCE EVALUATION

Fig. 6: Dataset. (a) RGB FCC, (b) LiDAR, (c) Ground Truth. Classification
maps. (d) Deep Encoder Decoder, (e) Two Branch CNN, (f) Deep Fusion, (g)
FusAtNet, (h) Ours, (i) Color Scheme

###  3.1 Training protocols

The input image patches are of size 11$\times$11$\times$channels. We compare
our model’s performance with other state of the art HSI-LiDAR fusion models
from [2] (encoder decoder based model, shortened as EncDec), two branch CNN
[9], deep fusion [3] and FusAtNet [1]. We use Adam optimizer [10] for training
all the models, where learning rate is set to 5$\times$10-6 for a total of 500
epochs with glorot weight initialization [11].

###  3.2 Houston 2013 dataset

The Houston 2013 dataset, one of the largest available HSI+LiDAR datasets, was
introduced in GRSS Data Fusion Contest 2013. The HSI image consists of a total
of 144 hyperspectral channels (0.38 µm to 1.05 µm) containing
[349$\times$1905] pixels per channel. The corresponding LiDAR image is a
single channel image with [349$\times$1905] pixels. The total number of ground
truth points are 15029 from 15 different classes, of which 2832 samples are in
the training set and the remaining in the test set [1].

As seen from Table 1, DeepFusion does exceptionally well for the Highway class
(10) which is a bit problematic in other architectures including our proposed
method. FusAtNet performs great for the classes 2, 5, 6, 7 and 13, major boost
being in the Residential class. It captures very well the spatial
relationships of residential areas which can be seen through the
classification map [Fig. 6 (c)]. Two Branch CNN performs well in class 4 by a
good margin. However, EncDec even though has a relatively lower accuracy, is
the only one that gives a steady increase in accuracy throughout its epochs
given the learning rate. Our proposed model outperforms the other models in
the classes 3, 5, 8, 9, 11, 12 and 14, with major accuracy boosts for Trees,
Commercial, Road, Railway and Parking classes. The major pitfalls of our
proposed model are in the classes of Highway, Water and Residential. It can be
seen from our classification map that Trees, Railways and Roads have a
definite form, however lack in the Residential and Water class depiction.

Fig. 7: Dataset. (a) RGB FCC, (b) LiDAR, (c) Ground Truth

###  3.3 MUUFL Gulfport dataset

This is an HSI + LiDAR dataset acquired over the campus of University of
Southern Mississippi Gulf Park, Long Beach Mississippi. The HSI data is
composed of a total of 64 bands, while the LiDAR modality has one raster each
corresponding to elevation and intensity. All the images are coregistered to
the size of 325$\times$220\. A total of 53687 ground truth pixels are
distributed amongst 11 classes (Fig. 7). For training purposes, 100 pixels per
class are selected leaving the rest of 52587 pixels for testing [1]. The
results in Table 2 show major improvements in classes 2, 6 and 10 and marginal
improvements in classes 1, 2 and 5.

###  3.4 Ablation study

Ablation studies were performed to analyze the variation on change of the
stack number and as well as the embedding dimension. The embedding dimension
was fixed to 128 while variation of stack number was performed. The stack
number $N_{x}$ was kept 4 while varying the embedding dimension. It can be
seen that increasing embedding dimension does increase the accuracy to a
specific number but beyond the threshold number, its increase does not
contribute significant improvement in the accuracy (Table 4). However, we can
see an increase in accuracy as the stack number increases (Table 3). This
shows that self-attention along with cross-modal interaction helps in
identifying features for better classification.

##  4 CONCLUSIONS AND FUTURE WORKS

In this paper we propose a novel cross-modal interaction based fusion model
which tries to leverage the two complementary modalities for achieving higher
accuracies. The results show that by increasing the interaction, the model has
an increased overall accuracy as it can identify features better due to the
cross-modal attention mechanism and infer more information via self-attention.
The concatenation aides in information retention in an efficient encapsulation
form. We would like to further extend this architecture as a scope to
introduce more than two modalities. Due to modular nature, and cross-modal
attention being inherent, it can open up new avenues in solving other cross-
domain tasks more efficiently.

##  5 ACKNOWLEDGEMENTS

The authors would like to thank ISRO for the grant RD/0120-ISROC00-005 that
contributed to this research.

## References

  * [1] Satyam Mohla, Shivam Pande, Biplab Banerjee, and Subhasis Chaudhuri,  “Fusatnet: Dual attention based spectrospatial multimodal fusion network for hyperspectral and lidar classification,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 92–93.
  * [2] Danfeng Hong, Lianru Gao, Renlong Hang, Bing Zhang, and Jocelyn Chanussot,  “Deep encoder-decoder networks for classification of hyperspectral and lidar data,”  IEEE Geoscience and Remote Sensing Letters, 2020.
  * [3] Yushi Chen, Chunyang Li, Pedram Ghamisi, Xiuping Jia, and Yanfeng Gu,  “Deep fusion of remote sensing data for accurate classification,”  IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 8, pp. 1253–1257, 2017.
  * [4] Hao Li, Pedram Ghamisi, Uwe Soergel, and Xiao Xiang Zhu,  “Hyperspectral and lidar fusion using deep three-stream convolutional neural networks,”  Remote Sensing, vol. 10, no. 10, pp. 1649, 2018.
  * [5] Quanlong Feng, Dehai Zhu, Jianyu Yang, and Baoguo Li,  “Multisource hyperspectral and lidar data fusion for urban land-use mapping based on a modified two-branch convolutional neural network,”  ISPRS International Journal of Geo-Information, vol. 8, no. 1, pp. 28, 2019.
  * [6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Advances in neural information processing systems, 2017, pp. 5998–6008.
  * [7] Minh-Thang Luong, Hieu Pham, and Christopher D Manning,  “Effective approaches to attention-based neural machine translation,”  arXiv preprint arXiv:1508.04025, 2015.
  * [8] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton,  “Layer normalization,”  arXiv preprint arXiv:1607.06450, 2016.
  * [9] Xiaodong Xu, Wei Li, Qiong Ran, Qian Du, Lianru Gao, and Bing Zhang,  “Multisource remote sensing data classification based on convolutional neural network,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 2, pp. 937–949, 2017.
  * [10] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [11] Xavier Glorot and Yoshua Bengio,  “Understanding the difficulty of training deep feedforward neural networks,”  in Proceedings of the thirteenth international conference on artificial intelligence and statistics, 2010, pp. 249–256.