---
abstract: |
    FPGAs have found increasing adoption in data center applications since a new generation of high-level tools have become available which noticeably reduce development time for FPGA accelerators and still provide high-quality results. There is, however, no high-level benchmark suite available, which specifically enables a comparison of FPGA architectures, programming tools, and libraries for HPC applications.

    To fill this gap, we have developed an OpenCL-based open-source implementation of the HPCC benchmark suite for Xilinx and Intel FPGAs. This benchmark can serve to analyze the current capabilities of FPGA devices, cards, and development tool flows, track progress over time, and point out specific difficulties for FPGA acceleration in the HPC domain. Additionally, the benchmark documents proven performance optimization patterns. We will continue optimizing and porting the benchmark for new generations of FPGAs and design tools and encourage active participation to create a valuable tool for the community.
author:
- 
bibliography:
- 'bibliography/meyer20\_sc.bib'
- 'bibliography/IEEEabrv.bib'
title: Evaluating FPGA Accelerator Performance with a Parameterized OpenCL Adaptation of the HPCChallenge Benchmark Suite
---

FPGA, OpenCL, High Level Sythesis, HPC benchmarking

Introduction
============

In , benchmarks are an important tool for performance comparison across systems. They are designed to stress important system properties or generate workloads that are similar to relevant applications for the user. Especially in acquisition planning they can be used to define the desired performance of the acquired system before it is built. Since it is a challenging task to select a set of benchmarks to cover all relevant device properties, benchmark suites can help by providing a pre-defined mix of applications and inputs, for example SPEC CPU [@SPEC-CPU] and  [@HPCCIntroduction].

There is an ongoing trend towards heterogeneity in , complementing by accelerators, as indicated by the Top 500 list [@top500]. From the top 10 systems in the list, seven are equipped with different types of accelerators. Nevertheless, to get the best matching accelerator for a new system, a tool is needed to measure and compare the performance across accelerators. For well-established accelerator architectures like , there are already standardized benchmarks like SPEC ACCEL [@SPECACCEL]. For , that are just emerging as accelerator architecture for data centers and HPC, existing benchmarks do not focus on and miss to measure highly relevant device properties.

Similar to the compiler for applications, the framework consisting of and takes a very important role to achieve performance on an . The framework translates the accelerator code (denoted as *kernel*), most commonly from , to intermediate languages, organizes the communication with the underlying , performs optimizations and synthesizes the code to create executable configurations (bitstreams). Hence, the framework has a big impact on the used resources and the maximum kernel frequency, which might vary depending on the kernel design. An benchmark suite for should capture this impact and, for comparisons, must not be limited to a single framework.

One of the core aspects of is communication. Some cards offer a new approach for scaling with their support for direct communication to other cards without involving the host CPU. Such technology is already used in first applications [@MLNetwork; @Sano-Multi-FPGA-Stencil] and research has started to explore the best abstractions and programming models for inter-FPGA communication [@SMI; @FPGAEthernet]. Thus, communication between FPGAs out of an framework is another essential characteristic that a benchmark suite targeting should consider.

In this paper, we propose *HPCC FPGA*, an benchmark suite for using the applications of the benchmark suite. The motivation for choosing HPCC is that it is well-established for CPUs and covers a small set of applications that evaluate important memory access and computing patterns that are frequently used in HPC applications. Further, the benchmark also characterizes the HPC system’s network bandwidth allowing to extrapolate to the performance of parallel applications.

Specifically, we make the following contributions in this paper:

1.  We provide FPGA-adapted kernel implementations along with corresponding host code for setup and measurements for all benchmark applications.

2.  We provide configuration options for the kernels that allow adjustments to resources and architecture of the target and board without the need to change the code manually.

3.  We evaluate the execution of these benchmarks on different FPGA families and boards with Intel and Xilinx FPGAs and show the benchmarks can capture relevant device properties.

4.  We make all benchmarks and the build system available as open-source on GitHub to encourage community contributions.

The remainder of this paper is organized as follows: In Section \[sec:related-work\], we give an overview of existing benchmark suites. In Section \[sec:hpcc-benchmark\], we introduce the benchmarks in in more detail and briefly discuss the contained benchmarks and the configuration options provided for the base runs. In Section \[sec:evaluation\] we build the benchmarks for different architectures and evaluate the results to show the potential of the proposed configurable base runs. In Section \[sec:discussion\], we evaluate the global memory system of the boards in more detail and give insights into experienced problems and the potential of the benchmarks to describe the performance of boards and the associated frameworks. Finally, in Section \[sec:conclusion\], we draw conclusions and outline future work.

Related Work {#sec:related-work}
============

There already exist several benchmark suites for and their frameworks. Most benchmark suites like Rodinia [@Rodinia], OpenDwarfs [@OpenDwarfs-first] or SHOC [@SHOC] are originally designed with GPUs in mind. Although both GPU and can be programmed using , the design of the compute kernels has to be changed and optimized specifically for to achieve good performance. In the case of Rodinia this was done [@RodiniaFPGA] for a subset of the benchmark suite with a focus on different optimization patterns for the Intel FPGA (then Altera) SDK for OpenCL. In contrast, to port OpenDwarfs to FPGAs, Feng et al. [@OpenDwarfs-first] employed a research OpenCL synthesis tool that instantiates GPU-like architectures on FPGAs. With Rosetta [@Rosetta], there also exists a benchmark suite that was designed targeting using the Xilinx HLS tools from the start. It focuses on typical FPGA streaming applications from the video processing and machine learning domains. The CHO [@CHO] benchmark targets more fundamental FPGA functionality and includes kernels from media processing and cryptography and the low-level generation of floating-point arithmetic through OpenCL, using the Altera SDK for OpenCL.

The mentioned benchmarks often lack possibilities to adjust the benchmarks to the target architecture easily. Modifications have to be done manually in the kernel code, sometimes many different kernel variants are proposed or the kernels are not optimized at all, making it difficult to compare results for different . A benchmark suite that takes a different approach is Spector [@Spector]. It makes use of several optimization parameters for every benchmark, which allows modification and optimization of the kernels for a architecture. The kernel code does not have to be manually changed, and optimization options are restricted by the defined parameters. Nevertheless, the focus is more on the research of the design space than on performance characterization.

To our best knowledge, there exists no benchmark suite for with a focus on characteristics at the point of writing. All of the mentioned benchmark suites lack a way to measure the inter- communication capability of recent high-end . In some of the benchmarks, the investigated input sizes are small enough to fit into local memory resources of a single . Since actual applications are highly parallel and require effective communication, an focused benchmark must also evaluate the characteristics of the communication network.

HPC Challenge Benchmarks for FPGA {#sec:hpcc-benchmark}
=================================

Benchmark Execution and Evaluation {#sec:evaluation}
==================================

Further Findings and Investigations {#sec:discussion}
===================================

Conclusion {#sec:conclusion}
==========

In this paper, we proposed *HPCC FPGA*, a novel benchmark suite for . Therefore, we provide configurable base implementations and host codes for all benchmarks of the well-established benchmark suite. We showed that the configuration options allow the generation of efficient benchmark kernels for Xilinx and Intel using the same source code without manual modification. We executed the benchmarks on up to three with four different memory setups and compared the results with simple performance models. Most benchmarks showed a high-performance efficiency when compared to the models. Nevertheless, the evaluation showed that the base implementations are often unable to utilize the available resources on an board fully. Hence, it is important to discuss the base implementations and configuration options with the community to create a valuable and widely accepted performance characterization tool for . We made the code open-source and publicly available to simplify and encourage contributions to future versions of the benchmark suite.

Acknowledgements {#acknowledgements .unnumbered}
================

The authors gratefully acknowledge the support of this project by computing time provided by the Paderborn Center for Parallel Computing (PC2). We also thank Xilinx for the donation of an Alveo U280 card, Intel for providing a PAC D5005 loaner board and access to the reference design BSP with SVM support, and the Systems Group at ETH Zurich as well as the Xilinx Adaptive Compute Clusters (XACC) program for access to their Xilinx FPGA evaluation system.
