Discrepancies among Pre-trained Deep Neural Networks: A New Threat to
  Model Zoo Reliability
Training deep neural networks (DNNs) takes signifcant time and resources. A
practice for expedited deployment is to use pre-trained deep neural networks
(PTNNs), often from model zoos -- collections of PTNNs; yet, the reliability of
model zoos remains unexamined. In the absence of an industry standard for the
implementation and performance of PTNNs, engineers cannot confidently
incorporate them into production systems. As a first step, discovering
potential discrepancies between PTNNs across model zoos would reveal a threat
to model zoo reliability. Prior works indicated existing variances in deep
learning systems in terms of accuracy. However, broader measures of reliability
for PTNNs from model zoos are unexplored. This work measures notable
discrepancies between accuracy, latency, and architecture of 36 PTNNs across
four model zoos. Among the top 10 discrepancies, we find differences of
1.23%-2.62% in accuracy and 9%-131% in latency. We also fnd mismatches in
architecture for well-known DNN architectures (e.g., ResNet and AlexNet). Our
findings call for future works on empirical validation, automated tools for
measurement, and best practices for implementation.
Diego Montes, Pongpatapee Peerapatanapokin, Jeff Schultz, Chengjun Gun, Wenxin Jiang, James C. Davis
# Discrepancies among Pre-trained Deep Neural Networks:

###### Abstract.

Training deep neural networks (DNNs) takes significant time and resources. A practice for expedited deployment is to use pre-trained deep neural networks (PTNNs), often from model zoos--collections of PTNNs; yet, the reliability of model zoos remains unexamined. In the absence of an industry standard for the implementation and performance of PTNNs, engineers cannot confidently incorporate them into production systems. As a first step, discovering potential discrepancies between PTNNs across model zoos would reveal a threat to model zoo reliability. Prior works indicated existing variances in deep learning systems in terms of accuracy. However, broader measures of reliability for PTNNs from model zoos are unexplored. This work measures notable discrepancies between accuracy, latency, and architecture of 36 PTNNs across four model zoos. Among the top 10 discrepancies, we find differences of 1.23%-2.62% in accuracy and 9%-131% in latency. We also find mismatches in architecture for well-known DNN architectures (_e.g._, ResNet and AlexNet). Our findings call for future works on empirical validation, automated tools for measurement, and best practices for implementation.

Neural networks, Model zoos, Software reuse, Empirical software engineering +
Footnote †: 2022, Singapore, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Singapore.

+
Footnote †: 2022, Copyright held by the owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/owner/

## 1. Introduction

With the growing energy consumption of training DNNs (26), taking advantage of the re-usability of PTNNs can significantly reduce the costs of training (13). In particular, transfer learning can result in shorter training times and higher asymptotic accuracies compared to other weight initialization methods (22; 36). This kind of technique accelerates model reuse and development. The history of PTNNs and their impact on the development of artificial intelligence has been extensively documented (13; 25). As such, collections of PTNNs have been created, referred to as _model zoos_. Notably, maintainers of popular machine learning frameworks, such as TensorFlow (2), maintain corresponding model zoos developed with their framework, such as the TensorFlow Model Garden (38).

There are many model zoos (18; 23; 38) and an expanding use of PTNNs in production systems (13). Past work has emphasized the difficulties in adopting software engineering practices in machine learning, and specifically, the challenges with reproducing machine learning research papers (4; 17). These reproducibility issues may affect PTNNs, leading to variations across model zoos (28). Disparities in the accuracy, latency, or architecture of a PTNN could negatively affect a deep learning system, threatening PTNNs' reuse potential. Consider a model zoo that has an incorrect implementation of a well-known DNN architecture, which has increased its latency significantly. If an engineer were to use the PTNN from this zoo, they would unknowingly be receiving a lower quality PTNN than they might otherwise have from a different model zoo. The engineer's effort to enable a quick turnaround time with a PTNN would have become harmful. Discovering discrepancies would shain a light on the reliability of model zoos.

To explore the reliability of model zoos, we performed a measurement study to identify discrepancies among 36 image classification PTNN architectures across four model zoos: _TensorFlow
dataset (ImageNet) can be as large as 2.62% (Krizhevsky et al., 2014).1 Similarly, over 20% of the PTNNs measured had latency differences (FLOPs) of 10% or more when comparing PTNNs of the same name across the model zoos. Lastly, we discover architectural differences in several PTNNs, including implementations of _AlexNet_ and _ResNet V2_. We conclude with an agenda for future research on further empirical validation, automated tools for measurement, and best practices for implementing model zoo PTNNs.

Footnote 1: The _LSWC-2012-CLS_ image dataset has 50,000 validation images. A 1% accuracy difference is equivalent to 500 images.

## 2. Background and Related Work

PTNNs are applied in a wide variety of domains (Krizhevsky et al., 2014). With the demand for engineers far exceeding supply (Krizhevsky et al., 2014), companies are looking for best practices that can boost the productivity of their engineers. Major companies (_e.g._, Google and Microsoft) have shared best practices on machine learning development and informed future directions on model reuse (Bannis et al., 2016; Bannis et al., 2016). A case study from SAP indicates possible compatibility, portability, and scalability challenges in machine learning model deployment, which may affect their performance (Krizhevsky et al., 2014). There have been many efforts to improve the quality of model zoos. For example, IBM has developed a tool to extract model metadata (Wang et al., 2017) to support better model management. Bannis _et al._ promote best practices for reproducing and publishing high-quality PTNNs (Bannis et al., 2016). However, the reliability of model zoos has not been validated by prior works.

The ability to replicate the accuracy of a DNN in identical training environments is hindered by non-deterministic factors. Accuracy differences of up to 10.8%, stemming purely from non-determinism, have been reported with popular DNN architectures (Krizhevsky et al., 2015). Closely related, research has investigated and benchmarked the performance variances tied to deep learning frameworks (Krizhevsky et al., 2014; Krizhevsky et al., 2014). This variability threatens the reliability of new deep learning techniques. As such, automated variance testing (Krizhevsky et al., 2015) has been proposed to assure the validity of these comparisons. However, PTNNs in model zoos may also suffer from varying architectural implementations, affecting more than just accuracy. Our work measures the disparities in PTNNs across different model zoos as opposed to attempting to improve the standard in just one (Bannis et al., 2016). Our results enlighten future works validating the quality and promoting the standardization of model zoos.

## 3. Methodology

We perform a measurement study to assess our problem statement: whether discrepancies exist between the accuracy, latency, and architecture of PTNNs across different model zoos.

### Subjects

A _model zoo_ is a collection of PTNNs for various tasks. We carry out a selection process for four model zoos. Our selection criteria included the model zoo being maintained alongside a machine learning framework: this increases the likelihood of the model zoo being actively maintained. Furthermore, to ensure the popularity of the model zoo, the zoo must have a public GitHub repository with at least three thousand stars (Bannis et al., 2016). Using GitHub search2 to identify potential model zoo candidates, 11 model zoos were selected that met the criteria.3 The PTNNs within the 11 model zoos were categorized into deep learning tasks, including image classification, object detection, and natural language processing. We focused on image classification models because it is the most common type in 8 of the 11 model zoos.

Footnote 2: [https://github.com/search](https://github.com/search)

A PTNN availability analysis was done on the candidate model zoos to assess how many model zoos offered the same image classification PTNN architectures. Based on the largest shared availability, we chose _TensorFlow Model Garden_, _ONNX Model Zoo_, _Torchvision Models_, and _Keras Applications_. Within these model zoos, we selected all the image classification PTNN architectures that were present in at least two of the four model zoos, yielding 36 PTNN architectures. The selected PTNNs are either directly downloadable from the model zoos' GitHub repositories or can be pulled using the model zoos' APIs.

### Evaluation Metrics

**Accuracy.** Image classification DNNs' effectiveness is measured in accuracy, which is a critical component of a PTNN. We are measuring discrepancies between the claims of model zoos as opposed to verifying them. _Top-1 accuracy_ is the conventional accuracy where model prediction must exactly match the expected label, while _top-5 accuracy_ measures the fraction of images where any of the top five predicted labels matches the target label (Bannis et al., 2016; Krizhevsky et al., 2014). 35 image classification PTNN architectures reported top-1 ImageNet classification accuracies, meanwhile only 32 reported top-5 ImageNet classification accuracies.

**Latency.** The latency of a DNN is a key factor that engineers consider (Krizhevsky et al., 2014). For example, _MobileNet_ is a DNN image classification architecture that prioritizes low latency on mobile and embedded systems (Krizhevsky et al., 2014). We used open-source tools (Bannis et al., 2016; Wang et al., 2017; Wang et al., 2017) to measure the latency by counting the floating point operations (FLOPs) (Bannis et al., 2016).

Figure 1. Overview of the measurement process. We gather PTNNs from the model zoos with the same name, perform measurements on each PTNN, and compare for discrepancies.
FLOPs are framework and hardware-agnostic, allowing for unbiased comparisons.

**Architecture**. PTNNs are trained weights based on research papers that propose DNN architectures. As a result, model zoos advertise PTNNs by their architecture name. The observed accuracy differences and past work on DNN vulnerabilities motivated us to examine architecture (Han et al., 2017). Qualitative observations of discrepancies in the descriptions, source code, and visualizations of PTNN architectures were employed. Specifically, neutron, an open-source neural network visualizer, was used to inspect the architecture of the PTNNs (Wang et al., 2017). However, not all neural network weight formats are supported, so all PTNNs were converted to the ONNX format for architectural analysis using an appropriate tool for each framework (Zhou et al., 2018; Wang et al., 2019). The source code for the implementations of the PTNNs are present in the model zoos' GitHub repositories and was used as an additional form of PTNN inspection.

## 4. Results and Analysis

### Accuracy

We compared the top-1 accuracy of 35 PTNN architectures and the top-5 accuracy of 32 PTNN architectures by using ImageNet. Notably, 12 of the 35 profiled PTNN architectures had top-1 accuracy differences greater than 0.96%. For top-5 accuracies, 6 of the 32 PTNN architectures had differences greater than 0.94%. The large differences present in Figure 2 have significant consequences. For example, _ResNet V1 152_ from Keras is noticeably less accurate than the PTNN by the same name from Torchvision, with top-1 accuracies of 76.6% and 78.31%, respectively. This difference is pronounced enough that _ResNet V1 101_ from Torchvision with top-1 accuracy of \(77.3^{\prime\prime}\) to more accurate than _ResNet V1 152_ from Keras 4

Footnote 4: ResNet V1 101 was originally reported to be 0.32% less accurate than _ResNet V1 152_(Keras, 2016).

Table 1 shows the aggregation of accuracy differences across model zoos, highlighting how often a model zoo had the highest or lowest top-1 or top-5 accuracy for a given PTNN architecture. As seen, 48% of the PTNNs that were available on Torchvision had the highest top-1 accuracy among the model zoos. On the other hand, Keras had the lowest top-1 accuracy 44% of the time for its selection of PTNNs.

### Latency

36 PTNN architectures were measured for their FLOPs. Figure 3 shows that there are 8 PTNN architectures where the PTNN with the highest amount of FLOPs had greater than 10% more FLOPs than the PTNN with the lowest FLOP count. At the extreme, Torchvision's _SqueezeNet 1.0_, sitting at 819.08 million FLOPs, had 2.31\(\times\) the FLOPs of ONNX's _SqueezeNet 1.0_. Likewise, the three PTNN architectures from the _ResNet V2_ family all had greater than 85% more FLOPs than the lowest FLOPs PTNN. All the high FLOP-count _ResNet V2_ come from TFMG.

We discuss the possible explanations for the FLOPs differences seen in Figure 3. The high FLOPs difference measured in _SqueezeNet 1.0_ can be explained by looking at its successor, _SqueezeNet 1.1_. _SqueezeNet 1.1_ is advertised by ONNX to contain 2.4\(\times\) less computation than the former. However, _SqueezeNet 1.1_ from ONNX has the same number of measured FLOPs as the _1.0_ PTNN offered. ONNX has been advertising _SqueezeNet 1.1_ as its _1.0_ counterpart. Similarly, looking at the _ResNet V2_ from TFMG: a primary contributor to the large amount of FLOPs is the input shape. _ResNet V2_ architectures, according to the origin paper, accept 224\(\times\)224 inputs (Keras, 2016); however, TFMG states that the _ResNet V2_ PTNNs it provides use Inception pre-processing and an input image size of 299\(\times\)299. A trade-off between accuracy and throughput, FLOPs, was potentially made here by the model zoo maintainers.

Across all FLOP-counted PTNNs, Torchvision had the highest FLOPs PTNNs for 78% of the PTNNs it offered. Close behind, TFMG had 69%. Pointedly, Keras never had the highest FLOPs PTNN and had the lowest FLOPs implementation 81% of the time.

### Architecture

We frame our results for architecture in terms of the discrepancies we discovered in our analysis. Specifically, we discuss differences among PTNNs for _AlexNet_, _ResNet V1 101_, _ResNet V2 50_, and _ResNet V2 101_ and against the PTNNs' origin papers.

The _AlexNet_ from Torchvision cites a different origin paper than other model zoos (Zhou et al., 2018; Wang et al., 2019). Both papers contain the same first author; however, only the latter contains an explicit description of a DNN architecture. As such, our analysis pegs the PTNN against the latter paper (Wang et al., 2019). We notice two main discrepancies: the PTNN is missing the response normalization layers and the kernel-size and number of kernels for the convolution layers are incorrect. For instance, Torchvision's PTNN has 64 kernels in the first convolution layer as opposed to the 96 that are described in the origin paper.

\begin{table}
\begin{tabular}{l c c c c}  & Highest Top-1 & Lowset Top-1 & Highest Top-5 & Lowset Top-5 \\ \hline Torchvision Models & 48\% & 41\% & 52\% & 36\% \\ TF Model Garden & 40\% & 33\% & 36\% & 43\% \\ Keras Applications & 37\% & 44\% & 36\% & 40\% \\ ONNX Model Zoo & 35\% & 41\% & 31\% & 44\% \\ \hline \end{tabular}
\end{table}
Table 1. Frequency at which each model zoo had the most or least accurate model ordered by highest top-1 accuracy.

Figure 2. Top 10 largest top-1 accuracy differences. For a PTNN architecture, the accuracy of the PTNN with the lowest reported top-1 accuracy is subtracted from that of the PTNN with the largest top-1 accuracy.
The _ResNet V1 101_ from ONNX and Keras contain convolution shortcuts, which were only introduced in the _ResNet V2_ paper, but not in the _ResNet V1_ origin paper (Keras, 2018; Keras, 2018). Turchvision's and TFMG's _ResNet V1 101_ do not include this shortcut. Also in the _ResNet_ family, both the _ResNet V2 50_ and _ResNet V2 101_ have a shared discrepancy. As seen in Figure 4, Keras' _ResNet V2 50_ implementation contains max pool skip connections, which are not present in the paper, and uses convolutions with larger strides in these residual blocks (Keras, 2018).

The observed discrepancies in architecture may affect the accuracy and latency. For example, the larger convolution strides and max pool skip connection in the _ResNet V2 50_ from Keras allows the network to use less compute, FLOPs, compared to the PTNN from ONNX. This can be seen in the FLOP measurements of the _ResNet V2 50_ from Keras and ONNX. ONNX's _ResNet V2 50_ has 4.12 billion FLOPs while Keras' PTNN only has 3.49 billion FLOPs, an 18.1% difference. Moreover, the Keras PTNN did not sacrifice accuracy through this implementation, reporting a 76% top-1 accuracy, which is greater than ONNX's _ResNet V2 50_ top-1 accuracy of 75.81%. While the Keras maintainers did not implement _ResNet V2 50_ faithfully to the origin paper, they produced a more accurate PTNN with lower latency.

## 5. Discussion and Future Work

**Empirical Validation.** The top-1 accuracy differences depicted in Figure 2 suggest that the choice of model zoo matters. Specifically, 34% of the PTNN architectures having top-1 accuracy differences greater than 0.96% is not easily overlooked. An engineer may receive a PTNN that incorrectly classifies greater than 500 validation images on ImageNet more than a PTNN from a different model zoo. Model zoo choice should not result in a noticeable impact on the accuracy of PTNNs that engineers receive. Although model zoo's currently report the accuracy of the PTNNs they offer, our work has shown that this does not guarantee that there is not another model zoo with the same PTNN at a higher accuracy. Publicly available and actively maintained comparisons of model zoo PTNNs would allow engineers to be more informed when choosing a model zoo. Furthermore, we only studied the accuracies of image classification models at face value. We recommend future works focus on empirical validation on the claims of PTNNs in model zoo to check for the existence of false advertising.

**New Metrics and Automated Tools.** The measured FLOP disparities seen in Figure 3 have consequences, especially in edge devices with limited compute. For example, ONNX incorrectly listing _SqueezeNet 1.1_ as _SqueezeNet 1.0_ may lead to confusion when an engineer switches to _SqueezeNet 1.1_ from _SqueezeNet 1.0_ expecting a drop in latency. Similar confusion may arise from instances like the one seen in TFMG's selection of _ResNet V2_. While the increased input size is stated, the impact on latency is not made clear. To effectively inform engineers of the latency of PTNNs, model zoos should report FLOP counts alongside accuracy. Also of interest is the energy usage of these PTNNs, another important property for edge devices. The lack of reporting of these properties may make choosing PTNNs more difficult. We recommend future works create new metrics to measure the reliability and quality of PTNNs from model zooos and develop tools for automatically measuring these properties. Publishing updated results frequently can support easier decision-making of models for deployment.

**Naming Conventions.** The differences in the architectures of PTNNs may indicate an underlying improper documentation standard and a need for improved naming conventions in model zoos. As indicated in SS4.3, Torchvision's _AlexNet_ did not adhere to the origin paper while still claiming to be _AlexNet_. Seemingly, model zoos are advertising PTNNs labeled as well-known DNN architectures, like _ResNet_ and _AlexNet_, but when they do this, they really mean that the PTNNs are based on the DNN architecture and are not strict implementations. This inadequate naming convention leads to a false sense of equality and thus confusion. We recommend the community comprehensively document PTNN naming conventions to increase cohesion among model zoos. Likewise, we suggest future works investigate the expectations of engineers with regards to the PTNNs from model zoos to see whether they prefer exact reproductions or more accurate and lower latency PTNNs.

Figure 4. _ResNet V2 50_ architecture differences between _Keras Applications_ (left) and _ONNX Model Zoo_ (right). The top-right convolution on the left has a stride size of 2, while the top-right convolution on the right has a stride size of 1.

Figure 3. Top 10 largest FLOPs differences. For a PTNN architecture, the FLOP count of the PTNN with the most FLOPs is divided by the FLOPs of the PTNN with the fewest.
The result of such a study would inform model zoo maintainers on how to best implement and train PTNNs.

## 6. Conclusion

We present an investigation of the discrepancies between 36 image classification PTNN architectures from four popular model zoo through accuracy, latency, and architecture analyses. We find several significant discrepancies among these three axes that challenge the well-established use of PTNNs from model zoos, suggesting that an engineer will receive a PTNN with different characteristics based on the model zoo. The PTNN's goal of shortening model deployment time is diminished because of the time investment needed to verify the properties of the PTNN. We discuss the importance of future works to validate the claims of model zoos, develop automated tools for measurement, and explore best practices for implementing model zoo PTNNs.

###### Acknowledgements.

 This work was supported by gifts from Google and Cisco and by NSF-OAC award #2107230. We thank the anonymous reviewers for their careful reading of our manuscript and their many insightful comments and suggestions.