# Framework for Automatic PCB Marking Detection and Recognition for Hardware
Assurance

Olivia P. Dizon-Paradis, Daniel E. Capecci, Nathan T. Jessurun,
Damon L. Woodard, Mark M. Tehranipoor, and Navid Asadizanjani DISTRIBUTION
STATEMENT A. Approved for public release: distribution is unlimited.
Electrical and Computer Engineering Department, University of Florida
Gainesville, FL, USA
Email<EMAIL_ADDRESS>

###### Abstract

A Bill of Materials (BoM) is a list of all components on a printed circuit
board (PCB). Since BoMs are useful for hardware assurance, automatic BoM
extraction (AutoBoM) is of great interest to the government and electronics
industry. To achieve a high-accuracy AutoBoM process, domain knowledge of PCB
text and logos must be utilized. In this study, we discuss the challenges
associated with automatic PCB marking extraction and propose 1) a plan for
collecting salient PCB marking data, and 2) a framework for incorporating this
data for automatic PCB assurance. Given the proposed dataset plan and
framework, subsequent future work, implications, and open research
possibilities are detailed.

###### Keywords:

PCB assurance; automatic visual inspection; automatic optical inspection; bill
of materials; optical character recognition; logo recognition; text
recognition

##  I Introduction

A Bill of Materials (BoM) is a complete inventory of all components present on
a printed circuit board (PCB), e.g. resistors, capacitors, and integrated
circuits (ICs). BoMs can be used in a variety of applications in the domain of
PCB reverse engineering (e.g. analysis of foreign, competitor, and legacy
devices), hardware assurance (e.g. vulnerability identification and hardware
Trojan detection), industrial assessment (e.g. manufacturing defect detection,
revenue estimation), and academia (e.g. technology trend research) [1].

Consequently, frameworks to automatically extract a PCB’s BoM using optical
images were introduced in [2] and [6]. In this study, the role of PCB
markings, i.e. logos and text, is detailed in the context of these frameworks.

PCBs are used in critical government, military, and biomedical infrastructures
and are often mass-manufactured with hundreds or thousands of components.
Therefore, 100% accuracy is highly desirable, as even a small error in AutoBoM
accuracy means the potential misrepresentation of hundreds of PCB components.
Subject matter experts (SMEs) use PCB markings, such as those shown in Fig. 1,
as a vital indicator for classifying and identifying each component to
construct a BoM.

Figure 1: Example of PCB markings, i.e. text and logos.

In this work, we propose PCB markings as a route to achieve higher AutoBoM
accuracy. For example, components may be associated with reference
designators, or on-board text which typically consist of one or two letters
followed by a number, e.g. R69, C76, and U22 in Fig. 1. These reference
designators can be used to help classify each component. As represented in the
figure, components with reference designators beginning with an ”R” are
typically resistors, ”C” are capacitors, ”U” are ICs, etc. In addition to on-
board markings, on-component text and manufacturer logos, such as those on the
ICs in Fig. 1, provide critical information to specifically identify the exact
component. To leverage on-board and on-component information for component
identification, SMEs must be familiar with reference designation standards
(e.g. ASME Y14.44-2008 [7] and IEEE 315-1975 [8]), as well as manufacturer
logos, computer-aided design (CAD) design rule checks, and best practices.
Such PCB design rules may be used for automatic BoM extraction by informing
parameter constraints and priors.

Several algorithms exist for automatic detection and recognition of text and
logos in a general sense. At present, few existing algorithms are specifically
meant to address PCB markings [16, 17]. Parameter-tuning, method fusion, and
transfer learning methodologies could improve accuracy for detection and
recognition of text, but they are not as effective for detection and
recognition of logos because many types of logos are unique to the electronics
domain. Moreover, there is a lack of annotated PCB marking data in the public
domain to train and/or evaluate potential new algorithms [15, 9]. The FICS-PCB
dataset contains annotations for components, on-board text, and on-component
text, but the text is annotated in a format inconsistent with traditional
optical character recognition (OCR) techniques and logos are not identified
[9]. To address these research gaps, we propose 1) a plan for PCB marking data
collection and 2) a framework for incorporating marking data for automatic PCB
assurance.

The rest of this paper is organized as follows. First, Sec. II elaborates on
the real-word challenges to consider for automatic PCB marking detection and
recognition. Then, Sec. III presents a proposed plan for data collection of
PCB marking data. Sec. IV details a framework for incorporating this data for
automated PCB assurance. Finally, Sec. V concludes the study with a summary of
the paper’s key takeaways and future research directions.

##  II Challenges

As in other domains, there are several difficulties for automatic PCB marking
detection and recognition [14]. Examples of difficult cases are shown in Fig.
2.

###  II-A High Intra-class Variation

There are many different form factors each character or logo may exhibit. In
other words, instances of each marking may display a wide variety of colors,
shapes, and textures. For example, the character “4” on the IC in Fig. 2a, the
capacitor in 2b, and the resistor in 2d all appear different. Though methods
to detect text in various fonts and orientations is improving [4, 3], many of
these algorithms are tuned to images with only a few instances of markings per
image. This is not the case for PCBs, which may consist of hundreds or
thousands of densely-packed markings. In addition, there are also other
characteristics of PCB images that need to be tested.

###  II-B Low Inter-class Variation

As each marking may exhibit different form factors, there are also instances
where one marking may be mistaken for another due to low inter-class
variation. For instance, the STMicroelectronics logo in Fig. 2a, letter “O” in
2b, and “IN” in 2c may be mistaken for the characters “ST”, number “0”, and
“NI”, respectively. Note that, though the surrounding text in Fig. 2c provides
insight into the upside-down orientation of “IN”, there are cases where such
context is not available and the ambiguous orientation can cause
misclassification errors.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: Examples of challenging cases for Automatic PCB Marking Detection
and Recognition.

###  II-C Imbalanced Classes

Another factor to consider is the uneven number of instances for each class of
PCB marking. For example, since PCBs generally possess more resistors and
capacitors than ICs, the associated reference designators “R” and “C” are more
common than “U” (Fig. 2e). Furthermore, logos pose additional challenges
because there are often significantly more characters than logos on PCBs. Even
when considering only logos, there may still be class imbalances between logos
on common off-the-shelf components versus custom-made components. Class
imbalances, if unaddressed, can heavily bias recognition algorithms and
performance metrics.

###  II-D Imaging Challenges

Complications may also arise due to the way the PCBs are imaged. Noise due to
camera sensors, lighting, or the internal digital camera image processing
pipeline can introduce artifacts and/or obfuscate the markings. Markings may
not be properly rendered if the camera resolution is too poor, while the image
data may be difficult to efficiently process if the resolution is too high.
Uneven lighting conditions can introduce shine, shadows, and other color
variations, which thereby increase the intra-class variation of the different
markings. For instance, in Fig. 2c, the IC text is relatively out-of-focus as
the camera’s sensors are focused on the board text. For each imaging
parameter, it is important to consider the possible trade-offs. For example,
shine may obscure some markings, but it can also be beneficial for texture
analysis-based detection of marking defects and component localization [5].
Different imaging setups have their own benefits and limitations in terms of
performance and computation time/speed.

###  II-E Physical Challenges

Challenges may also arise due to physical properties of the PCB components and
the board itself. Text and logos can fade due to age, wear-and-tear, and/or
damage (e.g. Fig. 2a). Markings can also be occluded by components, remnants
of PCB packaging, and/or stickers as in Fig. 2c. Fig. 2d shows an example of
false markings due to reflection of a true marking off a smooth component.
Markings can also be broken or interrupted by components, other markings,
traces, and vias as in Fig. 2f. Physical challenges such as these can often
create false positive detections, false negatives, or misclassifications.
Though some physical challenges can be addressed by altering the image
acquisition setup, some are difficult to overcome without physically altering
the board.

###  II-F Complex Component-Marking Associations

After markings have been detected and recognized, the next step for hardware
assurance would be to relate the markings to the different PCB components.
Though it is relatively straightforward to relate board markings to the
devices they are printed on, it is nontrivial to relate board markings to
their respective components. In an ideal situation, each component is
associated with exactly one board marking, which is a reference designator,
and each component-designator pair is located closest to each other (i.e. they
are unambiguously related). However, in practice, there are components
associated with multiple board markings or none at all, and vice versa. For
example, a reference designator such as “R1-3” may correspond to three
components, with R2 and R3 implicitly defined. Moreover, due to the density of
the components and board markings, it is not always the case that the closest
component-designator pairs are associated. For example, the light brown
capacitor in the upper left of Fig. 2e is not associated with “R102” or
“R111”, but rather “C120”. Such complex associations require the use of board
context and domain knowledge to infer the true relationships.

###  II-G Evolving Scope

In addition, not only is the problem of automatic PCB marking detection and
recognition challenging, but the problem is constantly evolving. As technology
advances, markings tend to become smaller and more compact along with similar
trends of smaller components and more compact PCBs. Along with technological
changes, manufacturing contexts are also constantly shifting. New
manufacturers, mergers and acquisitions, and development of new logo
variations over time (e.g. to accommodate smaller components or different
layouts) can all affect the types of logos that appear on a PCB. In other
words, the number of PCB marking classes is changing over time and the
necessary database to properly map these symbols grows. In addition, other
variabilities such as hardware Trojans, design errors, and manufacturing
defects must be accounted for. The “R69” resistor-like reference designator
for the capacitor in Fig. 2f is one such example. Since the problem is
constantly evolving, there is a need from the hardware assurance community for
increased collaboration and a continually updated dataset, as proposed in the
following section.

##  III Proposed Data Collection Plan

As detailed in Sec. II, there are several challenges for automatic PCB marking
detection and recognition, many of which can be addressed by incorporating
more data and prior knowledge. However, the lack of available PCB marking data
in the hardware assurance community makes this solution currently infeasible.
Furthermore, collecting this data is expensive and time-consuming. Hence, such
PCB marking datasets tend to be quite limited in either the number of distinct
classes or instances of each class. Though there are many datasets of text and
logos in other domains (e.g. text in natural scenes [11], logos of commercial
brands [12] and clothing [13], etc.), few are representative of all the
challenges present in the PCB domain. Therefore, a plan for collecting PCB
marking data is proposed.

To ensure PCB assurance algorithms are robust, there should be a method of
color and scale normalization incorporated in the workflow. For example, a
color checker consisting of an array of different color swatches with known
RGB values and a ruler can be used to normalize the lighting conditions and
resolution of images taken with different cameras. Such normalization of data
taken with different sensors facilitates the development of reproducible
algorithms. This, in turn, helps address the high intra-/low inter-class
variations and the imaging challenges.

Consistent with existing text and logo datasets from other domains, PCB
markings should be annotated by word [14]. Though semantic marking annotations
may improve algorithm performance, there is a considerable time/performance
trade-off. Many stock marking recognition algorithms first find words, then
localize characters, and finally classify the characters [11]. Though accuracy
of these stock algorithms may not be very high, they can be used to help
mitigate the time/performance trade-off of semantic annotation.

Since there is a wide and ever-increasing variety of logos, a lookup table of
logos should be used to ensure classes are annotated as consistently as
possible. Consistent class labelling can address the challenges from arbitrary
additions of unnecessary classes such as low inter-class variation between
equivalent logos and potential class imbalances from over-representing the
same logo under different class names.

Other information that should be annotated include whether the marking is
defective, its orientation, and any devices associated with it. Noting the
defects, especially the type (e.g. wearing versus misprints), helps algorithms
better model the effects of such defects on inter-class variation. Such
algorithms are useful for defect detection, a critical aspect of PCB
assurance. Orientation tracking is used to train and test algorithms to
resolve ambiguous cases (e.g. resistor text “221” can look like “122” upside
down, and vice versa). These annotations can be verified (e.g. against a
published list of standard resistor values) to ensure an accurate grammar
model is produced. These algorithms are useful for specifically identifying
the different PCB components. Finally, tracking associations between markings
and devices is useful for training and testing algorithms to determine
implicit, ambiguous, or other complex relations. Such algorithms assist in
classifying and identifying the different PCB components.

Figure 3: Proposed framework for automatic PCB marking detection and
recognition for hardware assurance.

Note that, though additional information such as font size, text color, and
board color could also improve PCB marking detection and recognition
algorithms, they are not specifically mentioned in the data collection plan.
These properties are excluded because many can be automatically extracted
given the information obtained from the above data collection plan combined
with simple image processing and/or computer vision algorithms. Moreover, the
proposed data collection plan is intended as a starting point to facilitate
research and collaboration within the hardware assurance community. Hence,
additional details for improved general recognition will be incorporated as
the problems are more deeply understood. After the marking data has been
collected, it will be used for PCB hardware assurance. Toward this end, we
propose the following PCB marking detection and recognition framework in the
next section.

##  IV Proposed Framework

The proposed framework for automatic PCB marking detection and recognition for
hardware assurance is shown in Fig. 3. Individual processes are detailed in
the following subsections.

###  IV-A Domain Knowledge

Subject matter expertise concerning PCB markings from design rules, company
standards, physical constraints, etc. can provide helpful details about the
nature of PCB components. Examples of such details include standard resistor
values, lists of component manufacturer logos, and rules for reference
designator placement. Domain Knowledge consists of collecting this information
so that more accurate annotations (and subsequently, automated PCB analysis
algorithms) can be produced. For example, insights for reference designator
placement can help algorithms classify and identify the associated components.
In addition, incorporation of domain knowledge as a prior to PCB assurance
algorithms may improve performance while requiring less data.

###  IV-B Trojan Design

An important factor in reliably detecting Trojans is understanding their
various appearances. Hardware Trojans, which are designed to go undetected,
may possess false markings to disguise themselves. By noting these marking
peculiarities (which may appear different from those used by the true
manufacturers) and researching state-of-the-art Trojan designs, the PCB
assurance framework can maintain a regularly updated library of adversarial
attacks, as in [10]. As a result, hardware assurance algorithms can cross-
reference markings and their associated components against this library to
better identify malicious alterations.

###  IV-C Annotation

During Annotation, SMEs leverage domain knowledge and Trojan design principles
to label PCB images. Here, text and component logos are labeled along with
their associated PCB components to produce ground truth information. The
ground truth is then used to train machine learning algorithms. As discussed
in Sec. III, a plan for collecting PCB marking data is proposed which details
information to annotate. When the domain of automatic PCB assurance has
matured, SME involvement may be reduced from fully-manual efforts to semi-
supervised efforts. This transition to semi-supervised efforts may help
address the challenges in Sec. II by dramatically increasing the amount of
data available for PCB assurance research.

###  IV-D Knowledge Database

All image, marking annotations, component annotations, and associated metadata
(e.g. Trojan exploits and domain knowledge) are then stored in the Knowledge
Database. This data will be readily accessible as both reference material and
ground truth for algorithm parameter tuning and performance evaluation.
Information stored in this database will encompass a wide variety of markings
from different component types, manufacturers, generations, and more. A
comprehensive range of represented markings and associated devices ensures PCB
assurance algorithms are not over-tuned to a select few PCBs. In addition,
this Knowledge Database of information accumulated over time can help address
the challenge of an evolving scope of PCB assurance.

###  IV-E Marking Detection

Information from the previous stages are then processed during Marking
Detection and leveraged for algorithm training, cross-referencing, and result
evaluation. By the end of this procedure, a list of regions where markings may
exist in the PCB images is output.

###  IV-F Marking Recognition

Each region of interest (ROI) of potential markings obtained from Marking
Detection is then recognized. This involves determining the specific
characters or logos within each ROI. Text can be recognized by the characters
they consist of, while logos can be recognized by their respective
manufacturer.

###  IV-G Marking-Component Association

Finally, recognized markings obtained from Marking Recognition are then
associated with PCB components such as resistors, capacitors, ICs, inductor,
no component, etc. Both on-board and on-device markings provide valuable
insight to classify and identify the associated components.

####  IV-G1 On-board Markings

Markings such as on-board reference designators provide valuable insight to
classify the associated components. For example, components with reference
designators beginning with an ”R” are typically resistors, ”C” are capacitors,
”U” are ICs, etc. This can be helpful in ambiguous cases where a component’s
appearance straddles the decision boundary between two classes (e.g. a dark
gray inductor may look similar to a resistor). However, it is important to
note that the process of classifying components using reference designators
should be iterative to account for manufacturing defects and hardware Trojans.
For example, the capacitor in Fig. 2f has the resistor-like reference
designator “R69”. Here, a marking recognition algorithm may be certain the
component is a resistor because of the “R69” reference designator, but a
component recognition algorithm may be certain the component is a capacitor
based on its appearance. In this case, the confidences of both recognition
algorithms should be considered and unclear cases should be flagged for review
by an SME. In this way, markings are not intended to be the only source of
information used to classify the components, only a way to verify and refine
predictions with higher confidence.

####  IV-G2 On-device Markings

Markings such as on-device text and logos provide insight for uniquely
identifying specifics about the associated components, such as a parameter
values, serial numbers, and logos. For instance, a component classified as a
“resistor” with a “221” on it may be identified as a “220 Ohm resistor”. Note
that domain knowledge was used here to infer the last digit is an exponent,
i.e. “221” resistor text = $22\times 10^{1}=220$ Ohms. Similarly, an object
classified as an “IC” may be identified by a logo and serial number, which can
be used to search for the respective specifications sheet published by the
manufacturer. As with classifying components using markings, the process of
identifying them should also be iterative to account for manufacturing defects
and hardware Trojans. In this way, markings are also not intended to be the
only source of information used to identify the components, only a way to
verify and refine predictions with higher confidence.

##  V Conclusion

This paper suggests text and logo information can be useful for automated PCB
assurance. However, at present, extracting and using this marking information
is difficult with the limited amount of data that is publicly available. This
paper elaborates on the real-word challenges with PCB marking data and details
why it is important to develop a dataset specially for PCB text and logos. We
therefore propose a plan for collecting PCB marking data and a framework for
incorporating this marking data for automatic PCB assurance. The proposed data
collection plan and framework are intended as a starting point to facilitate
research and collaboration within the hardware assurance community.

Future work includes collecting and annotating PCB data, as outlined in Sec.
III, and incorporating this data for automated PCB assurance, as detailed in
Sec. IV. From there, the results, benefits, and limitations of algorithms
incorporating and not incorporating marking information will be compared.
These resources and tools will enable progress toward accurate, automated
hardware assurance.

## References

##

  * [1] Quadir, S., Chen, J., Forte, D., Asadi, N., Shahbazmohamadi, S., Wang, L., Chandy, J. & Tehranipoor, M. A Survey on Chip to System Reverse Engineering. J. Emerg. Technol. Comput. Syst.. 13 (2016,4)
  * [2] Azhagan, M., Mehta, D., Lu, H., Agrawal, S., Chawla, P., Tehranipoor, M., Woodard, D. & Asadi, N. A Review on Automatic Bill of Material Generation and Visual Inspection on PCBs. ISTFA 2019: Conference Proceedings From The 45th International Symposium For Testing And Failure Analysis. pp. 256-265 (2019)
  * [3] Shi, B., Bai, X. & Belongie, S. Detecting oriented text in natural images by linking segments. 2017 IEEE Conference On Computer Vision And Pattern Recognition (CVPR). (2017,7)
  * [4] Coates, A., Carpenter, B., Case, C., Satheesh, S., Suresh, B., Wang, T., Wu, D. & Ng, A. Text detection and character recognition in scene images with unsupervised feature learning. 2011 International Conference On Document Analysis And Recognition. (2011,9)
  * [5] Ibrahim, A., Tominaga, S. & Horiuchi, T. Spectral imaging method for material classification and inspection of printed circuit boards. Organ. Ethic.. 49, 057201 (2010,5)
  * [6] Paradis, O., Jessurun, N., Tehranipoor, M. & Asadizanjani, N. Color Normalization for Robust Automatic Bill of Materials Generation and Visual Inspection of PCBs. ISTFA 2020: Papers Accepted For The Planned 46th International Symposium For Testing And Failure Analysis. pp. 172-179 (2020)
  * [7] ASME Reference designations for electrical and electonics parts and equipment : engineering drawing and related documentation practices. (American Society of Mechanical Engineers,2008)
  * [8] IEEE IEEE Standard American National Standard Canadian Standard Graphic Symbols for Electrical and Electronics Diagrams (Including Reference Designation Letters). IEEE Std 315-1975 (Reaffirmed 1993). pp. i-244 (1993)
  * [9] Lu, H., Mehta, D., Paradis, O., Asadizanjani, N., Tehranipoor, M. & Woodard, D. FICS-PCB: A Multi-Modal Image Dataset for Automated Printed Circuit Board Visual Inspection. (FICS,2020), http://eprint.iacr.org/2020/366
  * [10] _Trust-Hub.org_. Accessed: May 04, 2020. [Online]. Available: https://trust-hub.org/home
  * [11] Zhang, H., Zhao, K., Song, Y. & Guo, J. Text extraction from natural scene image: A survey. Neurocomputing. 122 pp. 310-323 (2013)
  * [12] Wang, J., Min, W., Hou, S., Ma, S., Zheng, Y. & Jiang, S. LogoDet-3K: A Large-Scale Image Dataset for Logo Detection. ArXiv Preprint ArXiv:2008.05359. (2020)
  * [13] Liu, K., Liu, T. & Wang, F. CBL: A Clothing Brand Logo Dataset and a New Method for Clothing Brand Recognition. 2020 28th European Signal Processing Conference (EUSIPCO). pp. 655-659 (2021)
  * [14] Ye, Q. & Doermann, D. Text Detection and Recognition in Imagery: A Survey. IEEE Transactions On Pattern Analysis And Machine Intelligence. 37, 1480-1500 (2015)
  * [15] Pramerdorfer, C. & Kampel, M. A dataset for computer-vision-based PCB analysis. 2015 14th IAPR International Conference On Machine Vision Applications (MVA). pp. 378-381 (2015,5)
  * [16] Li, W., Neullens, S., Breier, M., Bosling, M., Pretz, T. & Merhof, D. Text recognition for information retrieval in images of printed circuit boards. IECON 2014 - 40th Annual Conference Of The IEEE Industrial Electronics Society. pp. 3487-3493 (2014,10)
  * [17] Iano, Y., Bonello, D. & Neto, U. Text Recognition in PCBs: An Object Character Recognition (OCR) Algorithm. International Journal Of Development Research. 10 pp. 1-7 (2020,7)