# Learning to Explain: A Model-Agnostic Framework for Explaining Black Box
Models

Oren Barkan∗1 Yuval Asher∗2 Amit Eshel∗2 Yehonatan Elisha1 Noam Koenigstein2
1The Open University 2Tel Aviv University * Denotes equal contribution.

###### Abstract

We present Learning to Explain (LTX), a model-agnostic framework designed for
providing post-hoc explanations for vision models. The LTX framework
introduces an “explainer” model that generates explanation maps, highlighting
the crucial regions that justify the predictions made by the model being
explained. To train the explainer, we employ a two-stage process consisting of
initial pretraining followed by per-instance finetuning. During both stages of
training, we utilize a unique configuration where we compare the explained
model’s prediction for a masked input with its original prediction for the
unmasked input. This approach enables the use of a novel counterfactual
objective, which aims to anticipate the model’s output using masked versions
of the input image. Importantly, the LTX framework is not restricted to a
specific model architecture and can provide explanations for both Transformer-
based and convolutional models. Through our evaluations, we demonstrate that
LTX significantly outperforms the current state-of-the-art in explainability
across various metrics. Our code is available at: https://github.com/LTX-
Code/LTX

###### Index Terms:

Explainable AI, computer vision, transformers, CNNs, deep learning, learning
to explain, black box explanations, model-agnostic explanations

##  I Introduction and Related Work

##  II The Learning to Explain Framework

##  III Experimental Setup and Results

##  IV Conclusion

## Acknowledgment

## References

  * [1] S. Abnar and W. Zuidema.  Quantifying attention flow in transformers.  arXiv preprint arXiv:2005.00928, 2020.
  * [2] V. Badrinarayanan, A. Kendall, and R. Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
  * [3] O. Barkan.  Bayesian neural word embedding.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  * [4] O. Barkan, O. Armstrong, A. Hertz, A. Caciularu, O. Katz, I. Malkiel, and N. Koenigstein.  Gam: Explainable visual similarity and classification via gradient activation maps.  In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 68–77, 2021.
  * [5] O. Barkan, A. Caciularu, O. Katz, and N. Koenigstein.  Attentive item2vec: Neural attentive user representations.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3377–3381. IEEE, 2020.
  * [6] O. Barkan, A. Caciularu, I. Rejwan, O. Katz, J. Weill, I. Malkiel, and N. Koenigstein.  Cold item recommendations via hierarchical item2vec.  In 2020 IEEE International Conference on Data Mining (ICDM), pages 912–917. IEEE, 2020.
  * [7] O. Barkan, A. Caciularu, I. Rejwan, O. Katz, J. Weill, I. Malkiel, and N. Koenigstein.  Representation learning via variational bayesian networks.  In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 78–88, 2021.
  * [8] O. Barkan, Y. Elisha, Y. Asher, A. Eshel, and N. Koenigstein.  Visual explanations via iterated integrated attributions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2073–2084, October 2023.
  * [9] O. Barkan, Y. Elisha, J. Weill, Y. Asher, A. Eshel, and N. Koenigstein.  Deep integrated explanations.  In Proceedings of the 32nd ACM International Conference on Information & Knowledge Management (CIKM), 2023.
  * [10] O. Barkan, Y. Elisha, J. Weill, Y. Asher, A. Eshel, and N. Koenigstein.  Stochastic integrated explanations for vision models.  In 2023 IEEE International Conference on Data Mining (ICDM). IEEE, 2023.
  * [11] O. Barkan, Y. Fuchs, A. Caciularu, and N. Koenigstein.  Explainable recommendations via attentive multi-persona collaborative filtering.  In Proceedings of the 14th ACM Conference on Recommender Systems, pages 468–473, 2020.
  * [12] O. Barkan, E. Hauon, A. Caciularu, O. Katz, I. Malkiel, O. Armstrong, and N. Koenigstein.  Grad-sam: Explaining transformers via gradient self-attention maps.  In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2882–2887, 2021.
  * [13] O. Barkan, R. Hirsch, O. Katz, A. Caciularu, and N. Koenigstein.  Anchor-based collaborative filtering.  In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2877–2881, 2021.
  * [14] O. Barkan, R. Hirsch, O. Katz, A. Caciularu, Y. Weill, and N. Koenigstein.  Cold start revisited: A deep hybrid recommender with cold-warm item harmonization.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3260–3264. IEEE, 2021.
  * [15] O. Barkan, O. Katz, and N. Koenigstein.  Neural attentive multiview machines.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3357–3361. IEEE, 2020.
  * [16] O. Barkan and N. Koenigstein.  Item2vec: neural item embedding for collaborative filtering.  In 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2016.
  * [17] O. Barkan, N. Koenigstein, E. Yogev, and O. Katz.  Cb2cf: a neural multiview content-to-collaborative filtering model for completely cold item recommendations.  In Proceedings of the 13th ACM Conference on Recommender Systems, pages 228–236, 2019.
  * [18] O. Barkan, N. Razin, I. Malkiel, O. Katz, A. Caciularu, and N. Koenigstein.  Scalable attentive sentence pair modeling via distilled sentence embedding.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3235–3242, 2020.
  * [19] O. Barkan, T. Reiss, J. Weill, O. Katz, R. Hirsch, I. Malkiel, and N. Koenigstein.  Efficient discovery and effective evaluation of visual perceptual similarity: A benchmark and beyond.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20007–20018, October 2023.
  * [20] O. Barkan, I. Rejwan, A. Caciularu, and N. Koenigstein.  Bayesian hierarchical words representation learning.  arXiv preprint arXiv:2004.07126, 2020.
  * [21] O. Barkan, T. Shaked, Y. Fuchs, and N. Koenigstein.  Modeling users’ heterogeneous taste with diversified attentive user profiles.  User Modeling and User-Adapted Interaction, pages 1–31, 2023.
  * [22] O. Barkan, S. Shvartzman, N. Uzrad, A. Elharar, M. Laufer, and N. Koenigstein.  Inversynth ii: Sound matching via self-supervised synthesizer-proxy and inference-time finetuning.  ISMIR, 2023.
  * [23] O. Barkan and D. Tsiris.  Deep synthesizer parameter estimation.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3887–3891. IEEE, 2019.
  * [24] O. Barkan, D. Tsiris, O. Katz, and N. Koenigstein.  Inversynth: Deep estimation of synthesizer parameter configurations from audio signals.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):2385–2396, 2019.
  * [25] A. Binder, G. Montavon, S. Lapuschkin, K.-R. Muller, and W. Samek.  Layer-wise relevance propagation for neural networks with local renormalization layers.  In International Conference on Artificial Neural Networks, pages 63–71. Springer, 2016.
  * [26] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229, 2020.
  * [27] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In 2018 IEEE winter conference on applications of computer vision (WACV), pages 839–847. IEEE, 2018.
  * [28] H. Chefer, S. Gur, and L. Wolf.  Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 397–406, 2021.
  * [29] H. Chefer, S. Gur, and L. Wolf.  Transformer interpretability beyond attention visualization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 782–791, 2021.
  * [30] J. Chen, L. Song, M. Wainwright, and M. Jordan.  Learning to explain: An information-theoretic perspective on model interpretation.  In International Conference on Machine Learning, pages 883–892. PMLR, 2018.
  * [31] J. Dai, Y. Li, K. He, and J. Sun.  R-fcn: Object detection via region-based fully convolutional networks.  Advances in neural information processing systems, 29, 2016.
  * [32] A. Défossez, N. Zeghidour, N. Usunier, L. Bottou, and F. Bach.  Sing: Symbol-to-instrument neural generator.  Advances in neural information processing systems, 31, 2018.
  * [33] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In Computer Vision and Pattern Recognition (CVPR), 2009.
  * [34] S. S. Desai and H. G. Ramaswamy.  Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization.  2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 972–980, 2020.
  * [35] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [36] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [37] J. Engel, K. K. Agrawal, S. Chen, I. Gulrajani, C. Donahue, and A. Roberts.  Gansynth: Adversarial neural audio synthesis.  arXiv preprint arXiv:1902.08710, 2019.
  * [38] R. C. Fong and A. Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3429–3437, 2017.
  * [39] R. Fu, Q. Hu, X. Dong, Y. Guo, Y. Gao, and B. Li.  Axiom-based grad-cam: Towards accurate visualization and explanation of cnns.  ArXiv, abs/2008.02312, 2020.
  * [40] K. Gaiger, O. Barkan, S. Tsipory-Samuel, and N. Koenigstein.  Not all memories created equal: Dynamic user representations for collaborative filtering.  IEEE Access, 11:34746–34763, 2023.
  * [41] D. Ginzburg, I. Malkiel, O. Barkan, A. Caciularu, and N. Koenigstein.  Self-supervised document similarity ranking via contextualized language models and hierarchical inference.  arXiv preprint arXiv:2106.01186, 2021.
  * [42] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [43] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [45] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [46] X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang.  Lightgcn: Simplifying and powering graph convolution network for recommendation.  In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 639–648, 2020.
  * [47] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua.  Neural collaborative filtering.  In Proceedings of the 26th international conference on world wide web, pages 173–182, 2017.
  * [48] G. Huang, Z. Liu, and K. Q. Weinberger.  Densely connected convolutional networks.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261–2269, 2017.
  * [49] P.-T. Jiang, C.-B. Zhang, Q. Hou, M.-M. Cheng, and Y. Wei.  Layercam: Exploring hierarchical class activation maps for localization.  IEEE Transactions on Image Processing, 30:5875–5888, 2021.
  * [50] H. Jung and Y. Oh.  Towards better explanations of class activation mapping.  2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1316–1324, 2021.
  * [51] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [52] K. Kumar, R. Kumar, T. De Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. De Brebisson, Y. Bengio, and A. C. Courville.  Melgan: Generative adversarial networks for conditional waveform synthesis.  Advances in neural information processing systems, 32, 2019.
  * [53] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov.  RoBERTa: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [54] Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie.  A convnet for the 2020s.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11966–11976, 2022.
  * [55] J. Lu, D. Batra, D. Parikh, and S. Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In Advances in Neural Information Processing Systems, pages 13–23, 2019.
  * [56] I. Malkiel, O. Barkan, A. Caciularu, N. Razin, O. Katz, and N. Koenigstein.  Recobert: A catalog language model for text-based recommendations.  arXiv preprint arXiv:2009.13292, 2020.
  * [57] I. Malkiel, D. Ginzburg, O. Barkan, A. Caciularu, J. Weill, and N. Koenigstein.  Interpreting bert-based text similarity via activation and saliency maps.  In Proceedings of the ACM Web Conference 2022, pages 3259–3268, 2022.
  * [58] T. Mikolov, W. tau Yih, and G. Zweig.  Linguistic regularities in continuous space word representations.  In NAACL-HLT, 2013.
  * [59] G. Montavon, S. Lapuschkin, A. Binder, W. Samek, and K.-R. Muller.  Explaining nonlinear classification decisions with deep taylor decomposition.  Pattern Recognition, 65:211–222, 2017.
  * [60] W.-J. Nam, S. Gur, J. Choi, L. Wolf, and S.-W. Lee.  Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2501–2508, 2020.
  * [61] V. Petsiuk, A. Das, and K. Saenko.  Rise: Randomized input sampling for explanation of black-box models.  arXiv preprint arXiv:1806.07421, 2018.
  * [62] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [63] A. Shrikumar, P. Greenside, and A. Kundaje.  Learning important features through propagating activation differences.  In International conference on machine learning, pages 3145–3153. PMLR, 2017.
  * [64] K. Simonyan, A. Vedaldi, and A. Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [65] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [66] S. Srinivas and F. Fleuret.  Full-gradient representation for neural network visualization.  In NeurIPS, 2019.
  * [67] M. Sundararajan, A. Taly, and Q. Yan.  Axiomatic attribution for deep networks.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3319–3328. JMLR. org, 2017.
  * [68] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [69] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, 2019.
  * [70] X. Wang, X. He, M. Wang, F. Feng, and T.-S. Chua.  Neural graph collaborative filtering.  In Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval, pages 165–174, 2019.
  * [71] M. D. Zeiler and R. Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.


# Rethinking Detection Based Table Structure Recognition for Visually Rich
Documents

Bin Xiao<EMAIL_ADDRESS>Murat Simsek<EMAIL_ADDRESS>Burak
Kantarci<EMAIL_ADDRESS>Ala Abu Alkheir<EMAIL_ADDRESS>
###### Abstract

Table Structure Recognition (TSR) is a widely discussed task aiming at
transforming unstructured table images into structured formats, such as HTML
sequences. One type of popular solution is using detection models to detect
components of a table, such as columns and rows, then applying a rule-based
post-processing method to convert detection results into HTML sequences.
However, existing detection-based studies often have the following
limitations. First, these studies usually pay more attention to improving the
detection performance, which does not necessarily lead to better performance
regarding cell-level metrics, such as TEDS, after the post-processing. Second,
some solutions over-simplify the problem and can miss some critical
information. Lastly, even though some studies defined the problem to detect
six components, including table, row, column, spanning cell, projected row
header, and column header, to provide as much information as other types of
solutions, these studies ignore the fact this problem definition is a multi-
label detection because row, projected row header and column header can share
identical bounding boxes. Besides these limitations, there is often a
performance gap between two-stage and transformer-based detection models
regarding the structure-only TEDS, even though they have similar performance
regarding the COCO metrics. Therefore, in this study, we revisit the
limitations of existing detection-based solutions, compare two-stage and
transformer-based detection models, and identify the key design aspects for
the success of a two-stage detection model for the TSR task, including the
multi-class problem definition, the aspect ratio for anchor box generation,
and the feature generation of the backbone network. Without proposing
sophisticated new components, we applied simple methods to improve these
aspects of the Cascade R-CNN model, achieved state-of-the-art performance, and
improved the baseline Cascade R-CNN model by 19.32%, 11.56% and 14.77%
regarding the structure-only TEDS on SciTSR, FinTabNet, and PubTables1M
datasets.

###### keywords:

Table Structure Recognition , Information Extraction , Document Processing ,
Object Detection , Visually Rich Document Understanding

††journal: Pattern Recognition

[inst1]organization=School of Electrical Engineering and Computer
Science,addressline=University of Ottawa, city=Ottawa, postcode=K1N 6N5,
state=ON, country=Canada

[inst2]organization=Lytica Inc.,addressline=555 Legget Drive, city=Ottawa,
postcode=K2K 2X3, state=ON, country=Canada

##  1 Introduction

Figure 1: Generic workflow of transforming unstructured tabular data from
document images into a structured format. Figure (a) is a sample document
containing a table and other text areas. (b) represents the table detection
step, and the output of this step is the table area. Note that the samples in
(a) and (b) are only parts of the original results due to the page size
limitation. (c) represents the step of table structure recognition. Following
this study’s definition, its outputs are the detected Tables, Columns, Rows,
Spanning Cells, and Headers. (d) represents the post-processing step
reconstructing the complex table structure based on the detected table
components. (e) is the OCR tool to extract text based on the reconstructed
structure to fully transform the table image into a structured format.
Notably, step (d) can also generate the cell boxes as the inputs of step (e).

(a) Four types of defined table components in the study in [1]. This
definition can infer spanning cells but cannot provide header information.
Besides, the projected row headers are treated as rows, which can lead to a
wrong structure.

(b) Four types of defined table components in the study in [2]. This
definition cannot provide header information, and the projected row headers
are treated as rows, which can lead to a wrong structure.

(c) Six types of defined table components in the study in [3]. The defined
projected row headers in this sample share identical bounding boxes with two
corresponding rows.

Figure 2: Different problem formulations for the detection-base TSR. Figure 3:
Four types of defined table components. In this sample, the defined column
header shares an identical bounding box with a defined row.

Datasheets are widely used in business scenarios and on the internet because
of their user-friendliness to human readers. However, these documents are
usually not structured, which means that it is hard to extract useful
information from them and further analyze the semantic meaning automatically.
Besides the unstructured format of the document, tables in these documents,
which are widely used to summarize critical information, often have very
complex structures and layouts, making it challenging to interpret and analyze
these tables. Therefore, there have been many studies of Table Detection (TD),
Table Structure Recognition (TSR), Text Detection, and Optical Character
Recognition (OCR) as steps of interpreting and analyzing tables from visually
rich documents. Figure 1 shows a typical workflow of extracting tables from
documents and transforming the extracted tables into an HTML format,
consisting of a TD, a TSR, and an OCR steps. Considering that this study
formulates the TSR as an Object Detection problem, a post-processing step is
required after the TSR detection model, as shown in Figure 1(d). Since there
have been many studies discussing Table Detection [4, 5, 6] and OCR tools [7,
8], achieving promising results, we focus on the TSR in this study, which aims
at transforming the tables from visually rich documents into structured
formats.

TSR studies can roughly be categorized into three groups based on their
problem formulations: detection-based models, image-to-sequence models, and
graph-based models. Image-to-sequence models usually follow the encoder-
decoder architecture and directly generate structured outputs, such as HTML
sequences. Some image-to-sequence models [9, 10] also integrate the OCR task
into the model to make the model end-to-end without using extra OCR tools to
extract text contents from the images. However, since these models use auto-
regressive decoders, they often suffer from error accumulation [11], and their
OCR capacity usually cannot generalize well because of the limitation of
training data. On the other hand, graph-based models usually use segmentation
or detection methods to extract table cells and treat extract table cells as
nodes of a graph and further build the relation among the graph nodes. This
graph-based definition makes it easier to deal with the scenarios in which
table images are collected from the wild, such as rotated, distorted tables.
However, graph-based models introduce extra complexity because of building
graph models compared with detection-based models. By contrast, detection-
based models provide an alternative way that is more straightforward to detect
the table components directly and post-process the detection results with a
deterministic rule-based method for reconstructing the table structure.
However, the detection-based method can fail to deal with rotated and
distorted samples. To sum up, these different types of approaches have their
benefits and must be selected based on the application scenarios. This study
focuses on the tables from well-formatted documents and proposes a detection-
based solution.

There have been some studies [12, 13, 1, 2, 3] using detection models together
with a post-processing method to solve the TSR task. However, existing studies
either over-simplify the problem or define a multi-label detection problem,
which is challenging for two-stage object detectors. For example, studies [12,
13] formulate the problem as only detecting columns and rows, making them hard
to deal with spanning cells. TableStrRec [1] further extend the formulation by
defining regular column, regular row, irregular column, and irregular row so
that the spanning cell can be inferred from these four types of components, as
shown in Figure 2(a). study in [2] simplifies the formulation of study in [3],
directly detecting table, column, row, and spanning cell, as shown in Figure
2(b). These three formulations treat the projected row header as a regular row
and do not define headers, resulting in over-simplified table structures. By
contrast, study in [3] defines six types of components, including table,
column, row, spanning cell, column header, and project row header, as shown in
Figure 2(c), which can provide as much structure information as other types of
TSR models. However, study in [3] does not consider that some column headers
and projected row headers can share identical bounding boxes with
corresponding rows. For example, as shown in Figure 2(c), the bounding boxes
of the project row header can also be classified as rows. Similarly, as shown
in Figure 3, a column header’s bounding box is also the row’s bounding box.
Therefore, the problem definition of study in [3] is a multi-label detection
problem. This multi-label definition is challenging for typical two-stage
models, such as Faster R-CNN [14] and Cascade R-CNN [15], because the
classifier for the classification task is for single-label classification
problems, and the feature fed into the classifier are extracted from the
backbone network instead of learned queries in Sparse R-CNN [16] and DETR
[17]. Therefore, this study follows the definition of study in [3], analyzes
the impact of multi-class definition to the Cascade R-CNN and Sparse R-CNN,
which are typical two-stage and transformer-based detectors, and further
refactors this multi-label definition by extending the overlapped samples to
pseudo-classes to improve the performance of our proposed Cascade R-CNN based
solution. Besides, the defined table components can have extremely large
aspect ratios, and the ground truth bounding boxes contain a large number of
white areas that cannot generate discriminate features for the classification
and regression task, as shown in Figures 2(c) and 3. Therefore, suitable
aspect ratios for the anchor generation of two-stage detectors and
discriminate, long-range features from the backbone network are two critical
aspects for the success of a two-stage detector for the TSR. Therefore, this
study tunes the aspect ratio in the Region Proposal Network (RPN), applies
deformable convolution, and introduces a Spatial Attention Module that can be
inserted into the backbone network to enrich long-range and context features.
To sum up, the contributions of this study are four-fold:

  1. 1.

This study revisits popular detection-based TSR definitions and formulates the
TSR problem as detecting six components: table, column, row, spanning cell,
projected row header, and column header, following the definition of the study
in [3]. This formulation is a multi-label detection problem, which is
challenging for two-stage detectors. Therefore, this study extends the
formulation to conventional single-label detection problems by generating
pseudo-classes.

  2. 2.

This study identifies and analyzes critical design aspects for the success of
a TSR model, including the aspect ratios in the anchor box generation and the
discriminate feature generation, then further proposes TSRDet, a Cascade R-CNN
based solution, by adjusting the ratio aspects applying the deformable
convolution and introducing a spatial attention module to build the long-range
dependencies and context information in the backbone network.

  3. 3.

We conduct comprehensive experiments to evaluate our proposed solution on
various datasets, including SciTSR [18], FinTabNet [19], PubTabNet [20] and
PubTables1M [3] with both detection metrics and cell-level metrics. The
experimental results show that our proposed TSRDet can outperform state-of-
the-art models in terms of both cell-level evaluation metrics and detection
metrics.

  4. 4.

We compare the two-stage and transformer-based detectors for the TSR problem,
using the TSRDet and Sparse R-CNN as two examples, and further analyze the
impact of the identified critical design aspects on the transformer-based
detectors.

The rest of this paper is organized as follows: Section 2 discusses related
studies, including studies in Object Detection Models and Table Structure
Recognition models. Section 3 describes our proposed solution. Section 4 shows
the experimental results and discusses the design aspects of the proposed
method. At last, we draw our conclusion and possible directions in Section 5.

##  2 Related Work

###  2.1 Object Detection Models

Object Detection is a fundamental task that has been widely discussed in many
studies. Since deep models have become the dominant solutions, we only discuss
popular deep learning based models in this section. Based on different design
perspectives, popular detection models can be categorized in different ways.
One popular categorization of detectors based on the number of regression
steps is to classify them into one- and two-stage detectors. Two-stage models,
such as Faster R-CNN [14] and Cascade R-CNN [15], usually use a Region
Proposal Network (RPN) to generate region proposals first and then feed the
region proposals to the well-designed model to classify and regress the
proposals. In the RPN network, one key parameter is the aspect ratio, which
defines the height/width ratio when generating anchor boxes. Suitable aspect
ratios are often close to the target objects’ height/width ratio, making the
regression task easier and improving the model performance. In contrast,
popular one-stage models, such as FCOS [21], YOLO [22], and YOLO’s variants
[23, 24, 25], integrate the region proposal generation and other regression
and classification components into a single network. For example, YOLO divides
the images into grids first, then classifies the class of grid cells and
directly predicts the bounding boxes and their confidences. The simple design
of one-stage detectors leads to faster training and inference time compared
with two-stage detectors.

On the other hand, some studies [16, 26] categorize the popular detectors from
the perspective of Non-maximum Suppression (NMS), which is widely used to
reduce the redundant predictions from the detectors. From this perspective,
popular detectors can be categorized into end-to-end and none end-to-end
models based on whether NMS is needed. DETR [17] is a typical end-to-end
detector introducing transformer architecture [27], set prediction loss, and
one-to-one label assignment to the object detection problem. Sparse R-CNN [16]
further refactors the DETR model and proposes to use sparse learnable regional
proposals to replace dense regional proposals and utilize a dynamic instance
interactive head to regress and classify the proposals in an iterative manner.
study in [28] analyses the success of end-to-end detectors and argues that the
one-to-one label assignment method in end-to-end detectors contributes to the
success of the end-to-end models but is not sufficient to fully remove the NMS
from the pipeline. This study further points out that the classification cost
in the matching cost when applying one-to-one label assignment plays a key
role in the success of these end-to-end models. study in [26] further analyzes
combinations of the label assignment methods and queries and argues that
sparse queries with one-to-one label assignment can degrade the recall, and
dense queries with one-to-one label assignment are hard to optimize. To
address these issues, study in [26] proposes a dense distinct queries (DDQ)
method to select distinct queries from dense queries using a class-agnostic
NMS, achieving promising precision and recall. It is worth mentioning that
these end-to-end detector can easily be extended to none end-to-end solutions
by adapting many-to-one label assignments [29] and NMS.

###  2.2 Table Structure Recognition

There have been many studies [30, 31, 32, 2, 33, 34] discussing the TSR
problem in recent years. As mentioned earlier in Section 1, we can roughly
categorize these solutions into image-to-sequence, detection-based, and graph-
based models. Image-to-sequence based models usually define the ground truth
as structured sequences, such as HTML sequences, built on the transformer
architecture [27], and follow an encoder-decoder architecture. For instance,
TableMaster [10] is a typical image-to-sequence based model that can generate
HTML sequences. More specifically, TableMaster follows the architecture of
MASTER [35], which is originally designed for the scene text generation
following the transformer architecture [27], and further improved the encoder
part by introducing a Multi-Aspect Global Context Attention. In contrast,
TableMaster has two branches designed for the HTML sequence generation and
bounding box regression. Similarly, MTL-TabNet [9] also follows the encoder-
decoder architecture but contains three decoders for the cell box regression,
cell content recognition, and HTML sequence generation. DRCC [11] argues that
the error accumulation problem degrades the performance of image-to-sequence
TSR models, especially when the input image is large. Therefore, DRCC proposes
a two-step decoder architecture, which first decodes the input image into rows
and then decodes the rows in cell sequences. VAST [36] pays more attention to
the imprecise bounding boxes of table cells and proposes a Coordinate Sequence
Decoder to improve the model’s ability to generate accurate bounding boxes and
introduces a visual-alignment loss to align the visual and structural
information. To sum up, this type of method is usually based on the encoder-
decoder architecture and can be trained end-to-end without using post-
processing methods.

On the other hand, detection-based models usually define the problem as
detecting different table components and applying a post-processing method to
reconstruct table structures. DeepTabStR [12] proposes to detect columns and
rows to obtain the table cells. However, DeepTabStR ignores the row/column
span in the tables, which means that it cannot recover the hierarchical
structures of tables. TableStrRec [37] extends the DeepTabStR, defining four
types of table components: regular columns, irregular columns, regular rows,
and irregular rows. Then, the spanning cells across multiple columns can be
inferred from the difference between the regular and irregular columns when
they are overlapped, and the spanning cells across multiple rows can be
inferred from regular and irregular rows similarly. PubTables-1M [3] is
another typical detection-based approach that defines six table components:
table, column, row, spanning cell, projected row header, and columns header,
in which projected row header and columns header are for the function analysis
and other components can be used to reconstruct the complex table structure.

At last, graph-based methods usually apply either detection or segmentation
methods to obtain the locations of table cells and further build the relation
among table cells. For instance, TGRNet [38] formulates the cell location
detection and cell logical location prediction jointly in a multi-task
architecture, which is modularized by a segmentation based method and graph
convolutional network (GCN), respectively. Similarly, TSRNet [39] proposes a
unified GNN-based approach modeling table detection and table structure
recognition tasks together. More specifically, TSRNet also employs a semantic
segmentation module to extract primitive regions, then applies k-nearest
neighbors and line-of-sight neighbors to construct the graph and further
classify the graph nodes and edges to filter the noise regions, merge, and
build relations. In contrast, LGPMA [40] proposes a Local Pyramid Mask
Alignment Module and Global Pyramid Mask Alignment Module to localize table
cells, which are formulated as detection and segmentation problems and can be
implemented by MaskR-CNN [41]. To construct the structure of the table, LGPMA
further proposes a pipeline of cell matching, empty cell searching, and empty
cell merging using the Maximum Clique Search algorithm and rule-based methods.
Besides building graph explicitly, some studies [42, 43, 44] predict the table
grids or separators first, and then merge grid elements, which are also
treating grid elements as graph nodes. SPLERGE [42] is a typical method
following this strategy consisting of a Split Model and Merge Model, in which
the Split Model consists of a Row Projection Network and a Columns Projection
Network to obtain the table grid, and the Merge Model is used to merge the
grid cells. Similarly, SEM [43] employs a segmentation model to segment
columns and rows and generate the table grid with a post-processing method.
After the table grid is obtained, SEM introduces an Embedder network to
extract and fuse the features from textual and visual modalities. A Merger
network takes the fused features from Embedder as inputs to merge the grid
elements. TSRFormer-DQ-DETR [45] leverages a DETR [17] based separation line
prediction model, termed DQ-DETR, to predict the reference points on
separation lines, followed by a Relation Network based cell Merging module to
merge grid elements.

##  3 Proposed Method

In this section, we first introduce the overall architecture of the proposed
method, then give the details of the proposed methods, including tuning aspect
ratios for the anchor boxes generation, refactoring the multi-label
definition, and the proposed Spatial Attention Module.

###  3.1 Overall Architecture

Our proposed solution is based on Cascade R-CNN [15], and the overall
architecture is shown in Figure 4. It mainly consists of a backbone network, a
region proposal network (RPN), and a series of cascade heads. The backbone
network is used to extract features of the input images, which is often
followed by Feature Pyramid Network (FPN) [46] to extract and fuse features
from different scales. The extracted features, together with the region
proposals generated by the RPN, are fed into the first cascade head for the
classification and regression tasks, and the regression results would be the
inputs of the subsequent cascade head, as shown in Figure 4.

Since there are multiple cascade heads, all the outputs of these cascade heads
are used to calculate the loss. Moreover, the final loss of the model can be
defined as the sum of these Cascade Heads loss and the RPN loss, as defined by
Equation 1 where $N$ is the number of cascade head. It is worth mentioning
that we follow the most popular Cascade R-CNN model to use three cascade heads
in our model.

$\mathcal{L}=\mathcal{L}_{rpn}+\sum_{i=1}^{N}(\mathcal{L}^{i}_{cls}+\mathcal{L}^{i}_{reg})$
(1)

$e_{cls}=PROJ(ROI\\_POOL(BACKBONE(x),b))$ (2)

$e_{reg}=PROJ(ROI\\_POOL(BACKBONE(x),b))$ (3)

Figure 4: Overall architecture of Cascade R-CNN.

As shown in Figure 4, there are a REG Head and a CLS Head in each cascade head
for the regression and classification tasks, respectively. The input features
of these REG Heads and CLS Heads $e_{cls},e_{reg}$ are extracted by applying
ROI Pooling operations to the features from Backbone Network with the proposal
boxes $b$, which can be defined by Equations 2 and 3. $PROJ,ROI\\_POOL$, and
$BACKBONE$ are the projection layer, ROI Pooling operation, and the backbone
network. Therefore, for a trained model, the features of the CLS Heads
$e_{cls}$ are determined by the input image $x$ and the proposal boxes $b$,
meaning that a single proposal box cannot be classified into multiple classes
because CLS Heads are not multi-label classifiers. As aforementioned, the TSR
definition of this study is a multi-label detection problem, we refactor the
problem formulation by generating pseudo-classes which will be discussed in
Section 3.3. Besides, we also improve the proposal generation of the RPN
network by tuning the aspect ratios, and feature generation of the backbone
network by incorporating deformable convolution and Spatial Attention, which
will be discussed in sections 3.2 and 3.4, respectively.

###  3.2 Aspect Ratio Tuning

Aspect Ratios are the parameters used in the RPN to generate anchor boxes for
the two-stage detectors, which control the shape of the generated anchor
boxes. Popular implementations of Cascade R-CNN, such as Detectron2 [47],
usually use 0.5, 1.0, and 2.0 as default values, which can work well for
detecting common objects, such as the objects in COCO [48] dataset. However,
in the context of TSR, the range of aspect ratios is much larger because of
the shape of the table components. For example, as shown in Figure 2(c), the
aspect ratio of the last row at the table bottom is $20:1084=0.01845$, much
smaller than the popular value $0.5$. Therefore, adjusting the aspect ratios
to the TSR problem is important. Without proposing fancy new modules to select
suitable values, we simply select the values based on the statistics of the
training sets. Taking the FinTabNet dataset as an example, the aspect ratios
of the defined components are shown in Figure 5. The maximum value is 140, far
larger than the popular choices in common object detection. Besides, the
majority of aspect ratios in Figure 5 are in the range between 1 and 60.
Therefore, we extend this parameter for our proposed TSRDet as [0.0125, 0.025,
0.0625, 0.125, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 16, 40, 80]. A further detailed
parameter table is provided in Section 4. It is worth mentioning that when an
aspect ratio is less than 1, its multiplicative inverse is applied to count
the number of aspect ratios in Figure 5. We also didn’t further fine-tune this
parameter through validation, which means that they might not be optimal. But
this parameter has improved the model performance by around 2.7% as shown in
Section 4.3.

Figure 5: Statistics of aspect ratio values of FinTabNet training set. When a
aspect ratio is less than 1, then its multiplicative inverse is applied to
count the number of aspect ratios.

###  3.3 Pseudo-class Generation

As mentioned earlier, this study also formulates the TSR problem as detecting
six components, namely Table, Column, Row, Spanning Cell, Project Row Header
and Column Header, as shown in Figure 2(c) and Figure 3. In the study in [3],
all these six components are treated independently. Formally, the ground truth
$\mathbf{Y}$ of this definition for each image is a set of tuples containing
bounding boxes and their corresponding labels, as defined by Equation 5 where
$b_{i},c_{i}$ are the $i$th bounding box and class, and values from $0$ to $5$
are the Table, Column, Row, Spanning Cell, Project Row Header and Column
Header, respectively. This definition is a multi-label problem for the
classification task of a detection model, because some table components, such
as Column Header and Row can share identical bounding boxes, as shown in
Figure 3. This multi-label definition is challenging for typical two-stage
detectors, because the classifiers in these detectors are designed for single-
label classification.

$Y=\\{(b_{i},c_{i})\\}_{i=1}^{N},c_{i}\in\\{0,1,2,3,4,5\\},\forall i\neq
j,(c_{i}\neq c_{j})\land((b_{i}=b_{j})\lor(b_{i}\neq b_{j}))$ (4)

Therefore, in this study, considering the observation that the defined Project
Row Headers are all Rows at the same time, we only keep the Project Row
Headers samples during the training. Since some Column Headers can share
identical bounding boxes with corresponding Rows, we derive a pseudo class for
these overlapped samples and remove the original overlapped samples.
Therefore, during the training stage, we refactor the ground truth for each
image to the regular single-label classification, as defined by Equation 4
where values $0$ to $6$ are the Table, Column, Row, Spanning Cell, Project Row
Header, Column Header and the Pseudo Class, respectively. During the testing
stage, the results of Project Column Header are duplicated once to generate
their corresponding prediction Rows, and the results of the pseudo-class are
duplicated twice to generate the corresponding prediction Rows and Headers, so
that we can still follow the formulation defined by Equation 4 to evaluate the
model performance.

$Y=\\{(b_{i},c_{i})\\}_{i=1}^{N},c_{i}\in\\{0,1,2,3,4,5,6\\},\forall i\neq
j,(c_{i}\neq c_{j})\land(b_{i}\neq b_{j})$ (5)

###  3.4 Spatial Attention and Deformable Convolution

As discussed in Section 1, building long dependencies for detecting the
defined components is important. Inspired by the recent studies using large
convolution kernels [49, 50], a Spatial Attention Module has been proposed, as
shown in Figure 6. For the design of the Spatial Attention Module, we use a
similar architecture with MSCA [51] containing multiple branches and large
kernel convolutions and use spatial and depthwise separable convolution [52,
53] to reduce the number of parameters. More specifically, for the spatial
separable convolution, we use a pair of $7*1$ and $1*7$ kernels to replace a
typical $7*7$, use the pair of $11*1$ and $1*11$, and the pair of $21\times 1$
and $1\times 21$ to replace $11*11$ and $21*21$ kernels, respectively. For the
depthwise separable convolution, we applied the convolution on each channel of
the feature maps independently. Then, the outputs of the three branches are
concatenated together as the input of a convolution layer with $1\times 1$
kernel to make the channel dimension the same as the inputs. The proposed
Spatial Attention Module can be easily inserted into the backbone network
between two blocks because they do not change the feature shapes. For example,
for a typical backbone network implemented by ResNet [54] containing a STEM
block and four Residual Blocks, as shown in Figure 6, the Spatial Attention
module can be inserted after the last three Residual Blocks to generate the
spatial attention, then the spatial attention can be applied to the original
outputs of each Residual Block by Element-wise Multiplication. It is worth
mentioning that the Spatial Attention Modules shown in Figure 6 have
independent trainable parameters, and all the feature maps are padded
correspondingly to keep the size of the feature maps.

Figure 6: Architecture of proposed Spatial Attention Module. A ResNet backbone
consists of a STEM Block and four stages of Residual Block. Our proposed
Spatial Attention Module are inserted between the blocks of the backbone to
build long dependencies.

On the other hand, deformable convolution [55] has been demonstrated to help
improve the detection performance on the document image detection tasks by
many studies [1, 56, 57]. Deformable convolution uses a learnable grid offset
to sample the grid points from the feature map and apply the convolution
operation, as defined by Equation 6,

$\mathbf{z}_{p_{0}}=\sum_{p_{n}\in R}{w(p_{n})x(p_{0}+p_{n}+\Delta p_{n})}$
(6)

where $p_{0}$ is the location on the output feature map $\mathbf{z}$, $p_{n}$
is the $n$th grid point in grid $R$, and $\Delta p_{n}$ is the $n$th learnable
offset. Since the offset $\Delta p_{n}$ applied to the deformable convolution
is usually obtained by a regular convolution with small kernels, such as a 3 *
3 kernel, it can only improve the local feature instead of building long
dependencies. Therefore, deformable convolution is useful for improving the
detection performance but can contribute little to exploring the structures of
the table. In this study, we apply the proposed Spatial Attention module and
deformable convolution to build the long-dependent and local features
together.

##  4 Experiments and Analysis

###  4.1 Datasets and Experimental Settings

We utilize four datasets in this work, including SciTSR [18], FinTabNet [19],
PubTabNet [20] and PubTables1M [3]. As discussed in the study in [58],
FinTabNet and SciTSR datasets contain noise annotations that are harmful for
the model performance. Therefore, we use their cleaned versions proposed in
the study in [58]. All the four datasets consist of only the table regions of
each image. SicTSR dataset is collected from academic publications, which
contains 7453, 1034 and 2134 samples for the training, validation and testing.
PubTables1M dataset is a large scale dataset for the TSR problem collected
from the PMCOA corpus, containing 758849 training samples, 94959 validation
samples, and 93834 testing samples. Since the PubTabNet dataset does not
provide original PDF files, we cannot process it to make detection
annotations. Besides, its testing is not public available. Therefore, we use
its validation set to evaluate the model trained with the PubTable1M dataset.
Following the study in [58], we use the code base in [59] to process the
datasets and align the formats of all the three datasets. FinTabNet is also a
large dataset widely used for the TSR problem, containing 78537, 9289 and 9650
samples for the training, testing and validation, respectively. FinTabNet is
collected from the annual reports of companies, making its data source
different from other two datasets. Table 1 summarizes the datasets used in
this study for the model evaluation.

Table 1: Summary of datasets. Dataset | Train | Validation | Test
---|---|---|---
SciTSR [18] | 7,453 | 1,034 | 2,134
FinTabNet [19] | 78,537 | 9,650 | 9,289
PubTabNet [20] | 500,777 | 9,115 | -
PubTables1M [3] | 758,849 | 94,959 | 93,834

Since the TSR problem in this study is formulated as an object detection
problem, we use both detection metrics and cell-level TSR metric for the model
evaluation. For the detection metrics, we employ the widely accepted COCO
metrics [48], namely mean Average Precision (mAP), $AP50$, $AP75$, $AP_{s}$,
$AP_{m}$, $AP_{l}$, and object specific AP scores. More specifically, $AP50$,
$AP75$ are the APs using 0.50 and 0.75 as IoU thresholds, respectively.
$AP_{s}$, $AP_{m}$ and $AP_{l}$ are the APs of different size of target
objects, which is defined by Equation 7.

$object\\_size=\begin{cases}small&\text{if area \textless$32^{2}$ px}\\\
medium&\text{if $32^{2}$ \textless area \textless$64^{2}$ px}\\\
large&\text{otherwise}\\\ \end{cases}$ (7)

For the TSR metric, we choose structure-only Tree-Edit-Distance-Based
Similarity(TEDS) [20], which is firstly introduced in the study to overcome
the drawbacks of adjacency relation metrics [60], and can be defined as
Equation 8.

$TEDS(T_{a},T_{b})=1-\frac{EditDist(T_{a},T_{b})}{max(\mid T_{a}\mid,\mid
T_{b}\mid)}$ (8)

where EditDist is the tree-edit distance, and $T$ is the number of nodes in
the tree. It is worth mentioning that we use structure-only TEDS in the study,
which only consider the HTML tags without extracting their contents to avoid
the influence of OCR tools. It is worth mentioning that the testing samples
can be categorized into Sim. and Com. groups based on whether they have cells
spanning across multiple columns and rows.

###  4.2 Implementation Details and Experimental Results

To verify the effectiveness of our proposed solution, we include all three
types of state-of-the-art methods as discussed in Section 2. For the
detection-based methods, Cascade R-CNN [15], Deformable-DETR [61] and Sparse
R-CNN [16] as benchmark models, in which Cascade R-CNN [15] is also the based
model of the proposed methods, Deformable-DETR and Sparse R-CNN are two state-
of-the-art transformer based models. For the image-to-sequence models, EDD
[20], GTE [19], TableFormer [62], TableMaster [10], VAST [36] and MTL-TabNet
[9] are included. TSRFormer-DQ-DETR [45] is a state-of-the-art model following
the pipeline of detecting separation lines and then merging cell grids, as
discussed in Section 2.

We implement Cascade R-CNN and our proposed TSRDet based on the Detection2
[47], build the Deformable-DETR based on detrex [63], train the Sparse R-CNN
with their official codebase. For the Deformable-DETR and Sparse R-CNN, we use
the default parameters. For the Cascade R-CNN baseline, we align the number of
regional proposals and the batch normalization method to the TSRDet, as shown
in Table 2. All these detection models are using ResNet50 [54] pre-trained
with ImageNet [64] as the backbone network. We also re-train TableMaster [10]
with the FinTabNet dataset based on its official code base. For the
implementation of the proposed TSRDet, aspect ratios in the anchor box
generation are set as [0.0125, 0.025, 0.0625, 0.125, 0.25, 0.5, 1.0, 2.0, 4.0,
8.0, 16, 40, 80], and other key parameters are summarized in Table 2. Notably,
to calculate the structure-only TEDS, we use the scripts provided by study in
[59] to generate the HTML sequences from the detected components. All the
models are trained with 240, 120, and 60 epochs for the SciTSR, FinTabNet, and
PubTables1M datasets, respectively.

The experimental results regarding the COCO metrics and structure-only TEDS
are shown in Tables 7, 3, 4, 5 and 6, which can demonstrate the superiority of
the proposed solution. For the SciTSR dataset, the proposed TSRDet can improve
the baseline Cascade R-CNN by 19.32% regarding the structure-only TEDS,
outperforming Deformable-DETR and Sparse R-CNN. When it comes to COCO metrics,
the mAP of the proposed TSR is as good as Deformable-DETR, outperforming other
benchmark models. Similarly, the proposed TSR can outperform benchmark models
regarding both COCO metrics and structure-only TEDS on the FinTabNet and
PubTables1M datasets. It is worth mentioning that FinTabNet does not provide
original PDF files, making it hard to generate detection annotations.
Therefore, the model performance reported in Table 6 is calculated using the
model trained with the PubTable1M dataset. Even though the PubTable1M and
PubTabNet datasets have misalignments regarding the ground truth HTML
sequences, the proposed method still shows competitive performance compared
with other state-of-the-art methods, as shown in Table 6. Figure 7 shows a
prediction sample and its generated HTML sequence after post-processing,
demonstrating the capacities of the proposed solution.

Besides demonstrating the effectiveness of the proposed TSRDet, there are some
important observations from the experimental results. First, structure-only
TEDS is not well aligned with COCO metrics. For example, even though the
baseline Cascade R-CNN model can only reach 87.49% regarding the structure-
only TEDS on the FinTabNet dataset, its detection performance is only slightly
worse than other models. Deformmable-DETR has better mAP, but worse structure-
only TEDS than Sparse R-CNN model on the FinTabNet dataset. Second, benchmark
models, especially Sparse R-CNN also have very promising performance even
though it is not as good as our proposed solution. The discussions and
analysis to these observations will be in Section 4.4.

Table 2: Key training parameters of the proposed model. MAX_ITER and STEPS are for the FinTabNet dataset as examples. Parameter | Value | Description
---|---|---
RESNETS.NORM | nnSyncBN | Batch Normalization for the Backbone Network
MAX_ITER | 112,500 | total number of mini-batch
STEPS | 84,375 | the mini-batch to apply the learning rate schedule
SCHEDULER | MultiStepLR | the scheduler to change the learning rate
NMS_THRESH | 0.9 | non-maximum suppression threshold
PRE_NMS_TOPK_TRAIN | 4000 | RPN proposals to keep before applying NMS
PRE_NMS_TOPK_TEST | 2000 | RPN proposals to keep before applying NMS
POST_NMS_TOPK_TRAIN | 4000 | RPN proposals to keep after applying NMS
POST_NMS_TOPK_TEST | 2000 | RPN proposals to keep after applying NMS
DEFORM_ON_PER_STAGE | [True, True, True, True] | whether to use deformable convolution in each backbone stage
Table 3: Experimental results on SciTSR dataset with structure-only TEDS score. Sim. means the tables without spanning cells and Com. represents the tables with spanning cells. Model | TEDS-struc.(%)
---|---
Sim. | Com. | All
Cascade R-CNN | 77.31 | 84.74 | 79.09
Deformable-DETR | 98.17 | 94.59 | 97.30
SparseR-CNN | $\mathbf{99.08}$ | 95.92 | 98.30
TSRDet(Ours) | 98.59 | $\mathbf{97.88}$ | $\mathbf{98.41}$
Table 4: Experimental results on FinTabNet dataset with structure-only TEDS score. Sim. means the tables without spanning cells and Com. represents the tables with spanning cells. Model | TEDS-struc.(%)
---|---
Sim. | Com. | All
EDD [20] | 88.40 | 92.08 | 90.60
GTE [19] | - | - | 87.14
GTE(FT) [19] | - | - | 91.02
TableFormer [62] | 97.50 | 96.00 | 96.80
TableMaster [10] | 98.36 | 98.28 | 98.32
VAST [36] | - | - | 98.63
MTL-TabNet [9] | 99.07 | 98.46 | 98.79
TSRFormer-DQ-DETR [45] | - | - | 98.40
Cascade R-CNN | 82.17 | 92.50 | 87.49
Deformable-DETR | 98.08 | 97.54 | 97.81
Sparse-RCNN | 98.36 | 97.91 | 98.13
TSRDet(Ours) | $\mathbf{99.08}$ | $\mathbf{99.02}$ | $\mathbf{99.05}$
Table 5: Experimental results on PubTables1M dataset with structure-only TEDS score. Sim. means the tables without spanning cells and Com. represents the tables with spanning cells. Model | TEDS-struc.(%)
---|---
Sim. | Com. | All
Cascade R-CNN | 82.73 | 85.21 | 83.78
Deformable-DETR | 97.54 | 93.14 | 95.73
SparseR-CNN | 99.04 | 95.90 | 97.72
TSRDet(Ours) | $\mathbf{99.19}$ | $\mathbf{97.66}$ | $\mathbf{98.55}$
Table 6: Experimental results on PubTabNet validation set with structure-only TEDS score. Sim. means the tables without spanning cells and Com. represents the tables with spanning cells. The proposed model is trained with PubTable1M dataset, while the benchmark models are trained with PubTabNet dataset. Model | TEDS-struc.(%)
---|---
Sim. | Com. | All
EDD [20] | 91.10 | 88.70 | 89.90
GTE [32] | - | - | 93.01
LGPMA [40] | - | - | 96.70
RobustTabNet [65] | - | - | 97.00
TSRNet [39] | - | - | 95.64
VAST [36] | - | - | 97.23
TableFormer [62] | 98.50 | 95.00 | 96.75
MTL-TabNet [9] | 99.05 | 96.66 | 97.88
TSRDet(Ours) | 96.99 | 94.99 | 96.58
Table 7: Experimental results with Average Precision.

Dataset Model $AP$ $AP50$ $AP75$ $AP_{s}$ $AP_{m}$ $AP_{l}$ table column row
spanning cell projected row header column header SciTSR Cascade R-CNN 93.89
95.27 94.80 95.81 93.89 92.96 98.96 98.63 96.33 88.58 83.80 97.01 Deformable-
DETR 96.28 97.39 97.01 96.75 96.55 96.07 98.96 98.63 97.26 93.84 90.86 98.15
Sparse-RCNN 94.78 96.17 95.48 95.49 95.07 90.08 98.98 98.30 97.93 88.06 86.92
98.49 TSRDet(Ours) 96.28 96.79 96.57 99.01 96.42 95.65 98.97 99.25 98.57 95.30
87.06 98.50 FinTabNet Cascade R-CNN 95.23 97.53 96.90 87.32 95.31 93.08 99.00
96.69 96.96 84.43 96.63 97.64 Deformable-DETR 96.68 98.42 97.98 75.17 95.53
95.58 99.00 97.55 96.95 91.91 96.62 98.04 Sparse-RCNN 96.38 98.37 97.69 62.11
96.22 95.86 99.01 97.79 97.84 88.39 97.29 97.97 TSRDet(Ours) 97.50 98.33 98.09
91.60 97.40 97.15 99.01 98.83 97.99 94.62 96.61 97.93 PubTables1M Cascade
R-CNN 93.40 95.38 94.76 85.75 93.32 92.57 99.01 98.76 87.56 82.18 95.81 97.11
Deformable-DETR 94.82 97.43 96.79 78.33 92.55 94.48 98.99 97.89 95.84 85.04
95.43 95.74 Sparse-RCNN 96.46 98.14 97.60 84.25 95.73 96.45 99.00 98.42 98.03
87.85 97.91 97.57 TSRDet(Ours) 97.72 98.26 98.04 94.76 97.43 97.33 99.01 98.99
98.41 94.21 97.88 97.85

(a) A prediction sample after post-processing.

(b) The generated HTML sequence after post-processing.

Figure 7: A sample of prediction result from the FinTabNet testing set.

###  4.3 Ablation Study

In this section, we conduct experiments on the FinTabNet dataset to
demonstrate the effectiveness of our applied methods, including applying
aspect ratio tuning, pseudo-class generation, deformable convolution and
spatial attention. The experimental results are shown in Tables 8 and 9, in
which Asp_Ratio Tuning, Pseudo-Class Gen, DEFORM, and S_Attn are shorts for
applying aspect ratio tuning, pseudo-class generation, deformable convolution,
and spatial attention, respectively. We use the default implementation of
Cascade R-CNN in Detectron2 [47] as the baseline model. Even though this
default implementation can reach 95.06% regarding the mAP, its overall
structure-only TEDS only reaches 82.70%. After tuning the aspect ratios for
the anchor generation, the structure-only TEDS is increased to 90.23%, even
though the mAP is only increased from 95.06% to 95.54%. Other ablation results
show similar trends compared with the baseline model. It is worth mentioning
that simply applying deformable convolution without other methods can increase
the detection performance significantly, but lead to worse structure-only TEDS
score, , which is a similar phenomenon aforementioned in Section 4.2. This is
because deformable convolution can make model fit better to the extra spaces
that are not necessarily useful for recovering the table structure.

Table 8: Ablation study results on FinTabNet dataset with structure-only TEDS score. Asp_Ratio Tuning, Pseudo-Class Gen, DEFORM, and S_Attn are shorts for applying aspect ratio tuning, pseudo-class generation, deformable convolution, and spatial attention. Model | Asp_Ratio Tuning | Pseudo-Class Gen | DEFORM | S_Attn | TEDS-struc.(%)
---|---|---|---|---|---
Sim. | Com. | All
Cascade R-CNN |  |  |  |  | 82.17 | 92.50 | 87.49
Ablation 1 |  |  | ✓ |  | 81.45 | 87.11 | 84.35
Ablation 2 | ✓ |  |  |  | 84.27 | 95.80 | 90.23
Ablation 3 | ✓ | ✓ |  |  | 95.17 | 98.63 | 96.95
Ablation 4 | ✓ | ✓ | ✓ |  | 96.44 | 99.14 | 97.83
Ablation 5 | ✓ | ✓ |  | ✓ | 96.95 | 98.75 | 97.88
TSRDet(Ours) | ✓ | ✓ | ✓ | ✓ | 99.08 | 99.02 | 99.05
Table 9: Ablation study results regarding Mean Average Precision. The model
names are aligned with models in Table 8.

Model $AP$ $AP50$ $AP75$ $AP_{s}$ $AP_{m}$ $AP_{l}$ table column row spanning
cell projected row header column header Cascade R-CNN 95.23 97.53 96.90 87.32
95.31 93.08 99.00 96.69 96.96 84.43 96.63 97.64 Ablation 1 97.22 98.03 97.90
90.11 96.72 96.76 99.00 98.95 96.16 94.98 96.01 98.19 Ablation 2 95.54 97.54
96.91 87.43 95.79 94.04 99.00 97.04 97.64 84.84 96.67 98.02 Ablation 3 95.51
97.56 96.94 88.43 95.52 93.76 99.00 97.31 97.87 84.74 96.82 97.28 Ablation 4
97.83 98.37 98.13 91.91 97.65 97.58 99.00 98.96 98.33 95.78 96.98 97.93
Ablation 5 96.97 97.84 97.58 90.32 96.88 96.21 99.00 98.83 98.03 91.97 96.58
97.37 TSRDet(Ours) 97.50 98.33 98.09 91.60 97.40 97.15 99.01 98.83 97.99 94.62
96.61 97.93

###  4.4 Observations and Analysis

As mentioned earlier in Sections 4.2 and 4.3, there are some important
observations from the experimental results that can demonstrate the key design
aspects included in this study. In this section, we further discuss and
explain these important observations.

####  4.4.1 Metrics misalignment

The experimental results in Sections 4.2 and 4.3 show that a model with better
mAP performance does not necessarily also have better structural TEDS, which
means that the structure-only TEDS and the COCO metrics are not aligned. For
example, in Table 7, both the Deformable-DETR and our proposed TSRDet can
reach 96.28% regarding mAP on the SciTSR dataset, which is better than that of
Sparse R-CNN. However, when it comes to structure-only TEDS, as shown in Table
3, both TSRDet and Sparse R-CNN can perform better than Deformable-DETR. COCO
metrics evaluate the similarity between the prediction and ground truth boxes
with IoU thresholds. In contrast, the structure-only TEDS evaluates the
structure difference between the detected components and the ground truth
components because the post-processing method uses structure information, such
as the overlaps between columns and rows, to generate table cells. In the
context of detection-based TSR, the defined components often contain large
white areas that can influence the COCO metrics, but cannot influence the
structure. Figure 8 shows the columns’ ground truth of a sample from the
FinTabNet dataset. As shown in Figure 8(a), the minimum boxes required to
infer the correct architecture are much smaller than the ground truth boxes.
Figure 8(b) and 8(c) show two prediction samples after post-processing using
Ablation 1 and Ablation3 models. Ablation 1 model with deformable convolution
can better fit the extra white areas to improve the detection performance, but
it is not necessarily helpful for the structure.

(a) Comparison of the ground truth bounding box and the minimum bounding box
for the structure. We only include the columns’ boxes for simplicity.

(b) A sample prediction result of the Ablation 1 model after post-processing.
The mAP and structure-only TEDS are 97.22 and 84.35, respectively. We only
include the columns’ predictions for simplicity.

(c) A sample prediction result of the Ablation 3 model after post-processing.
The mAP and structure-only TEDS are 95.51 and 96.95, respectively. We only
include the columns’ predictions for simplicity.

Figure 8: Comparison of results from Ablation1 and Ablation3 models. Even
though Ablation 1 can achieve better detection performance, its performance
regarding structure-only TEDS is much lower than that of Ablation 3 model.

####  4.4.2 Comparison with Sparse R-CNN

Besides the metrics misalignment, our other important observation is that the
Deformable-DETR and Sparse R-CNN also achieved promising results even though
they are not as good as the proposed method. Therefore, using Sparse R-CNN as
an example, we further analyze the difference between our proposed TSRDet and
Sparse R-CNN and discuss how Sparse R-CNN implicitly deals with the multi-
label definition.

Sparse R-CNN mainly consists of a backbone network, learnable region proposal,
and a series of Dynamic Heads containing Dynamic Instance Interactive Head.
Therefore, for a input image $x$ with initialized region proposal feature
embedding $e_{pro}$ and corresponding proposal boxes $b$, the inputs to the
$head_{cls}$ and $head_{reg}$ in the first Dynamic Heads can be generated by
Equations 9, 10, 11 and 12, in which $BACKBONE,ROI\\_POOL,SELF\\_ATTN$ and
$PROJ$ are the backbone network, ROI_Pooling, self attention and projection
layers. Since Sparse R-CNN also utilizes a cascade architecture, the $N$th
Dynamic Head takes the outputs of $(N-1)$th Head’s regression result and
$e_{dyn}$ as the proposal boxes and proposal feature embedding. Since Sparse
R-CNN has multiple ways to initialize the region proposal boxes, such as
Random boxes, Image Size boxes, and Grid boxes, we follow the setting of
Section 4 to assume the region proposal boxes are initialized with Image Size
boxes, which means that for an input image $x$, its proposal boxes $b$ can be
treated as constants. Therefore, the classification and regression results are
determined by the learned region proposal feature embedding $e_{pro}$, which
can differ even when their corresponding boxes are identical. Therefore,
Sparse R-CNN can naturally deal with the multi-label definition for the TSR
task.

$e_{roi}=ROI\\_POOL(BACKBONE(x),b)$ (9)

$e_{dyn}=DYN\\_CONV(SELF\\_ATTN(e_{pro},e_{pro},,e_{pro}),e_{roi})$ (10)

$e_{cls}=PROJ_{cls}(e_{dyn})$ (11)

$e_{reg}=PROJ_{reg}(e_{dyn})$ (12)

By contrast, for the classification head of Cascade R-CNN, their inputs are
features extracted by applying RoI Pooling operation, as defined by Equation
2. In other words, the input features of the classification head are
determined by the bounding boxes once the model is trained. Therefore, there
can only be one classification result for a proposal box, making Cascade R-CNN
challenging to deal with the multi-label definition for the TSR task.

##  5 Conclusion and Future Work

In this study, we propose TSRDet, a tailored Cascade R-CNN based solution for
the TSR task, directly detecting six types of table components. The
experimental results show that the proposed solution outperforms different
types of state-of-the-art models, including image-to-sequence and graph-based
models. Considering the definition of six components, a bounding box can
simultaneously be a Row, Projected Row Header, and Column Header, which means
it is a multi-label classification. We refactor the multi-label classification
to regular single-label classification by generating pseudo-classes. Besides,
the proposed Spatial Attention module can be inserted into the backbone
network to build long-range dependencies, so deformable convolution is applied
to further improve the model performance. Since the proposed method is
detection-based and focuses on well-formatted, visually rich documents, one
major limitation is that it may fail to deal with irregular tables, such as
rotated and distorted tables. Integrating instance segmentation with detection
models is included in our future research agenda to deal with irregular tables
because instance segmentation can handle the irregular shape and be guided by
the bounding boxes.

## Acknowledgements

This work was supported in part by Mathematics of Information Technology and
Complex Systems (Mitacs) Accelerate Program and Lytica Inc.

## References

  * [1] J. Fernandes, B. Xiao, M. Simsek, B. Kantarci, S. Khan, A. A. Alkheir, Tablestrrec: framework for table structure recognition in data sheet images, International Journal on Document Analysis and Recognition (IJDAR) (2023) 1–19.
  * [2] B. Xiao, Y. Akkaya, M. Simsek, B. Kantarci, A. A. Alkheir, Efficient information sharing in ict supply chain social network via table structure recognition, in: GLOBECOM 2022-2022 IEEE Global Communications Conference, IEEE, 2022, pp. 4661–4666.
  * [3] B. Smock, R. Pesala, R. Abraham, Pubtables-1m: Towards comprehensive table extraction from unstructured documents, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4634–4642.
  * [4] B. Xiao, M. Simsek, B. Kantarci, A. A. Alkheir, Revisiting table detection datasets for visually rich documents, arXiv preprint arXiv:2305.04833 (2023).
  * [5] B. Xiao, M. Simsek, B. Kantarci, A. A. Alkheir, Table detection for visually rich document images, Knowledge-Based Systems 282 (2023) 111080.  doi:https://doi.org/10.1016/j.knosys.2023.111080.
URL https://www.sciencedirect.com/science/article/pii/S0950705123008304

  * [6] J. Fernandes, M. Simsek, B. Kantarci, S. Khan, Tabledet: An end-to-end deep learning approach for table detection and table image classification in data sheet images, Neurocomputing (2021).
  * [7] Z. Kuang, H. Sun, Z. Li, X. Yue, T. H. Lin, J. Chen, H. Wei, Y. Zhu, T. Gao, W. Zhang, K. Chen, W. Zhang, D. Lin, Mmocr: A comprehensive toolbox for text detection, recognition and understanding, arXiv preprint arXiv:2108.06543 (2021).
  * [8] B. Xiao, Y. Akkaya, M. Simsek, B. Kantarci, A. A. Alkheir, Multi-modal ocr system for the ict global supply chain, in: ICC 2023-IEEE International Conference on Communications, IEEE, 2023, pp. 3096–3101.
  * [9] N. T. Ly, A. Takasu, An end-to-end multi-task learning model for image-based table recognition (2023) 626–634doi:10.5220/0011685000003417.
  * [10] J. Ye, X. Qi, Y. He, Y. Chen, D. Gu, P. Gao, R. Xiao, Pingan-vcgroup’s solution for icdar 2021 competition on scientific literature parsing task b: Table recognition to html, arXiv preprint arXiv:2105.01848 (2021).
  * [11] H. Shen, X. Gao, J. Wei, L. Qiao, Y. Zhou, Q. Li, Z. Cheng, Divide rows and conquer cells: Towards structure recognition for large tables, IJCAI, 2023.
  * [12] S. A. Siddiqui, I. A. Fateh, S. T. R. Rizvi, A. Dengel, S. Ahmed, Deeptabstr: Deep learning based table structure recognition, in: 2019 international conference on document analysis and recognition (ICDAR), IEEE, 2019, pp. 1403–1409.
  * [13] K. A. Hashmi, D. Stricker, M. Liwicki, M. N. Afzal, M. Z. Afzal, Guided table structure recognition through anchor optimization, IEEE Access 9 (2021) 113521–113534.
  * [14] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, Advances in neural information processing systems 28 (2015).
  * [15] Z. Cai, N. Vasconcelos, Cascade r-cnn: Delving into high quality object detection, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018) 6154–6162.
  * [16] P. Sun, R. Zhang, Y. Jiang, T. Kong, C. Xu, W. Zhan, M. Tomizuka, L. Li, Z. Yuan, C. Wang, et al., Sparse r-cnn: End-to-end object detection with learnable proposals, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14454–14463.
  * [17] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in: European conference on computer vision, Springer, 2020, pp. 213–229.
  * [18] Z. Chi, H. Huang, H.-D. Xu, H. Yu, W. Yin, X.-L. Mao, Complicated table structure recognition, arXiv preprint arXiv:1908.04729 (2019).
  * [19] X. Zheng, D. Burdick, L. Popa, P. Zhong, N. X. R. Wang, Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context, Winter Conference for Applications in Computer Vision (WACV) (2021).
  * [20] X. Zhong, E. ShafieiBavani, A. Jimeno Yepes, Image-based table recognition: data, model, and evaluation, in: European conference on computer vision, Springer, 2020, pp. 564–580.
  * [21] Z. Tian, C. Shen, H. Chen, T. He, Fcos: Fully convolutional one-stage object detection, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 9627–9636.
  * [22] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: Unified, real-time object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779–788.
  * [23] J. Redmon, A. Farhadi, Yolo9000: better, faster, stronger, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7263–7271.
  * [24] J. Redmon, A. Farhadi, Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767 (2018).
  * [25] A. Bochkovskiy, C.-Y. Wang, H.-Y. M. Liao, Yolov4: Optimal speed and accuracy of object detection, arXiv preprint arXiv:2004.10934 (2020).
  * [26] S. Zhang, X. Wang, J. Wang, J. Pang, C. Lyu, W. Zhang, P. Luo, K. Chen, Dense distinct query for end-to-end object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7329–7338.
  * [27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017).
  * [28] P. Sun, Y. Jiang, E. Xie, W. Shao, Z. Yuan, C. Wang, P. Luo, What makes for end-to-end object detection?, in: International Conference on Machine Learning, PMLR, 2021, pp. 9934–9944.
  * [29] Q. Hong, F. Liu, D. Li, J. Liu, L. Tian, Y. Shan, Dynamic sparse r-cnn, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 4723–4732.
  * [30] Z. Chi, H. Huang, H.-D. Xu, H. Yu, W. Yin, X.-L. Mao, Complicated table structure recognition, ArXiv abs/1908.04729 (2019).
  * [31] D. Adiga, S. A. Bhat, M. B. Shah, V. Vyeth, Table structure recognition based on cell relationship, a bottom-up approach, in: RANLP, 2019.
  * [32] X. Zheng, D. Burdick, L. Popa, X. Zhong, N. X. R. Wang, Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context, in: Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2021, pp. 697–706.
  * [33] S. Schreiber, S. Agne, I. Wolf, A. R. Dengel, S. Ahmed, Deepdesrt: Deep learning for detection and structure recognition of tables in document images, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) 01 (2017) 1162–1167.
  * [34] H. Liu, X. Li, B. Liu, D. Jiang, Y. Liu, B. Ren, Neural collaborative graph machines for table structure recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4533–4542.
  * [35] N. Lu, W. Yu, X. Qi, Y. Chen, P. Gong, R. Xiao, X. Bai, Master: Multi-aspect non-local network for scene text recognition, Pattern Recognition 117 (2021) 107980.
  * [36] Y. Huang, N. Lu, D. Chen, Y. Li, Z. Xie, S. Zhu, L. Gao, W. Peng, Improving table structure recognition with visual-alignment sequential coordinate modeling, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 11134–11143.
  * [37] J. Fernandes, B. Xiao, M. Simsek, B. Kantarci, S. Khan, A. A. Alkheir, Tablestrrec: framework for table structure recognition in data sheet images, International Journal on Document Analysis and Recognition (IJDAR) (2023).
URL https://api.semanticscholar.org/CorpusID:261633420

  * [38] W. Xue, B. Yu, W. Wang, D. Tao, Q. Li, Tgrnet: A table graph reconstruction network for table structure recognition, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1295–1304.
  * [39] X.-H. Li, F. Yin, H.-S. Dai, C.-L. Liu, Table structure recognition and form parsing by end-to-end object detection and relation parsing, Pattern Recognition 132 (2022) 108946.
  * [40] L. Qiao, Z. Li, Z. Cheng, P. Zhang, S. Pu, Y. Niu, W. Ren, W. Tan, F. Wu, Lgpma: Complicated table structure recognition with local and global pyramid mask alignment, in: International conference on document analysis and recognition, Springer, 2021, pp. 99–114.
  * [41] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969.
  * [42] C. Tensmeyer, V. I. Morariu, B. Price, S. Cohen, T. Martinez, Deep splitting and merging for table structure decomposition, in: 2019 International Conference on Document Analysis and Recognition (ICDAR), IEEE, 2019, pp. 114–121.
  * [43] Z. Zhang, J. Zhang, J. Du, F. Wang, Split, embed and merge: An accurate table structure recognizer, Pattern Recognition 126 (2022) 108565.
  * [44] N. Q. Nguyen, A. D. Le, A. K. Lu, X. T. Mai, T. A. Tran, Formerge: Recover spanning cells in complex table structure using transformer network, in: International Conference on Document Analysis and Recognition, Springer, 2023, pp. 522–534.
  * [45] J. Wang, W. Lin, C. Ma, M. Li, Z. Sun, L. Sun, Q. Huo, Robust table structure recognition with dynamic queries enhanced detection transformer, Pattern Recognition 144 (2023) 109817.  doi:https://doi.org/10.1016/j.patcog.2023.109817.
URL https://www.sciencedirect.com/science/article/pii/S0031320323005150

  * [46] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117–2125.
  * [47] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, R. Girshick, Detectron2, https://github.com/facebookresearch/detectron2 (2019).
  * [48] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, Springer, 2014, pp. 740–755.
  * [49] X. Ding, X. Zhang, J. Han, G. Ding, Scaling up your kernels to 31x31: Revisiting large kernel design in cnns, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11963–11975.
  * [50] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, S. Xie, A convnet for the 2020s, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11976–11986.
  * [51] M.-H. Guo, C.-Z. Lu, Q. Hou, Z. Liu, M.-M. Cheng, S.-M. Hu, Segnext: Rethinking convolutional attention design for semantic segmentation, Advances in Neural Information Processing Systems 35 (2022) 1140–1156.
  * [52] F. Chollet, Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1251–1258.
  * [53] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam, Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv:1704.04861 (2017).
  * [54] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [55] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, Y. Wei, Deformable convolutional networks, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 764–773.
  * [56] S. A. Siddiqui, M. I. Malik, S. Agne, A. Dengel, S. Ahmed, Decnt: Deep deformable cnn for table detection, IEEE access 6 (2018) 74151–74161.
  * [57] A. Mondal, M. Agarwal, C. Jawahar, Dataset agnostic document object detection, Pattern Recognition 142 (2023) 109698.
  * [58] B. Smock, R. Pesala, R. Abraham, Aligning benchmark datasets for table structure recognition, arXiv preprint arXiv:2303.00716 (2023).
  * [59] B. Smock, R. Pesala, Table Transformer (06 2021).
URL https://github.com/microsoft/table-transformer

  * [60] M. Hurst, A constraint-based approach to table structure derivation, in: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings., Vol. 3, IEEE Computer Society, 2003, pp. 911–911.
  * [61] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, J. Dai, Deformable detr: Deformable transformers for end-to-end object detection, arXiv preprint arXiv:2010.04159 (2020).
  * [62] A. Nassar, N. Livathinos, M. Lysak, P. Staar, Tableformer: Table structure understanding with transformers, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4614–4623.
  * [63] T. Ren, S. Liu, F. Li, H. Zhang, A. Zeng, J. Yang, X. Liao, D. Jia, H. Li, H. Cao, J. Wang, Z. Zeng, X. Qi, Y. Yuan, J. Yang, L. Zhang, detrex: Benchmarking detection transformers (2023).  arXiv:2306.07265.
  * [64] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
  * [65] C. Ma, W. Lin, L. Sun, Q. Huo, Robust table detection and structure recognition from heterogeneous document images, Pattern Recognition 133 (2023) 109006.


# Measurements of jet fragmentation and jet substructure with ALICE

Oak Ridge National Laboratory, Oak Ridge, Tennessee, United States
E-mail

###### Abstract:

We discuss the latest results from jet fragmentation and jet substructure
measurements performed with the ALICE experiment in proton-proton and heavy-
ion collisions in a wide range of jet transverse momentum. The jet production
cross sections and cross section ratios for different jet resolution
parameters will be shown in a wide range of $p_{\textrm{T}}$. Results will be
compared to next-to-leading order pQCD calculations.

##  1 Introduction

Jet substructure measurements at low transverse momentum $p_{\textrm{T}}$
allow to constrain various perturbative and non-perturbative effects
contributing to jet production, such as hadronization, perturbative radiation
and the underlying event. For example the measurement of the jet cross section
for various jet resolution parameters $R$ and the ratios of cross sections, in
particular for $p_{\textrm{T}}<100~{}\textrm{GeV}/c$ show different
sensitivity to the various effects [1]. In addition, the large datasets
obtained with LHC Run2 allow for more differential studies. This is used in
the measurement of the groomed momentum fraction $z_{\rm{g}}$ using the
SoftDrop algorithm [2, 3], defined as

$z_{\rm{g}}=\frac{min(p_{\rm{T},1},p_{\rm{T},2})}{p_{\rm{T},1}+p_{\rm{T},2}}>z_{cut}\theta^{\beta}$
(1)

with $p_{\rm{T},1}$ and $p_{\rm{T},2}$ being the $p_{\rm{T}}$ of the hardest
and second hardest subjet. The level of grooming is controlled by the
parameters $z_{cut}$ and $\beta$. The observable is selected as it is closely
related to the QCD splitting function at leading order [1]. No
$p_{\textrm{T}}$-dependence is expected. The difference between the parton and
the reconstructed jet momentum is expected to increase as log($R$) for
perturbative radiation, to decrease as 1/$R$ due to hadronization effects and
to increase as $\sim R^{2}$ due to the contribution from the underlying event
[1]. In heavy-ion collisions the groomed momentum fraction is sensitive to
effects from the hot and dense medium.

Jets are reconstructed using the anti-$k_{\textrm{T}}$ algorithm [4] from the
FastJet package [5] employing the E scheme recombination. Clustering is done
for track-based jets using tracks reconstructed in the ALICE tracking
detectors, requiring a minimum constituent $p_{\textrm{T}}$ of 150 MeV/$c$.
For fully reconstructed jets clusters in the electromagnetic calorimeter with
energy larger than 300 MeV are included in the jet reconstruction.

The data used for the measurement in pp collisions at
$\sqrt{s}=13~{}\textrm{TeV}$ was collected in 2016 and 2017, consisting of a
min. bias dataset with $L_{\textrm{int}}=11.5~{}\textrm{nb}^{-1}$ and a jet-
triggered dataset based on energy deposition in the electromagnetic
calorimeter with $L_{\textrm{int}}=4~{}\textrm{pb}^{-1}$. For the measurements
in heavy-ion collisions the data was collected at
$\sqrt{s_{\textrm{NN}}}=2.76~{}\textrm{TeV}$ in 2011 and consists of 16M
central collisions.

In heavy-ion collisions jet reconstruction is affected by the particles from
the underlying event. Therefore observables with low sensitivity to the
background, like jet-hadron correlations [6] are preferential. For
substructure measurements the effect of the uncorrelated background can be
handled in the detector response. For this PYTHIA events [7, 8] are embedded
into heavy-ion events, and the effect of the background is estimated and
subtracted using area based [9] or constituent based [10] subtraction methods.

##  2 Measurement of the jet production in pp collisions at
$\sqrt{s}=13~{}\textrm{TeV}$

Figure 1: Left: $p_{\textrm{T}}$-differential jet production cross section in
pp collisions at $\sqrt{\textrm{s}}$ = 13 TeV for different $R$. Right: Ratios
of jet production cross sections for jets with $R$ = 0.2 to the cross section
for various $R$. Lines indicate calculations using POWHEG+PYTHIA6.

Figure 2: $p_{\textrm{T}}$-differential jet production cross section for jets
with $R$ = 0.2 (left) and $R$ = 0.4 (right) compared to POWHEG+PYTHIA6
($\mu_{R}$ = $\mu_{F}$ = 1).

The $p_{\textrm{T}}$-differential cross section of jet production is measured
for different jet radii between $R$ = 0.2 and $R$ = 0.6 for
$20~{}\textrm{GeV}/c<p_{\textrm{T}}<320~{}\textrm{GeV}/c$ (Fig. 1 (left)). The
comparison to POWHEG+PYTHIA6 [11, 12, Alioli:2010xa] is shown in Fig. 2 for
$R$ = 0.2 (left) and $R$ = 0.4 (right). POWHEG calculations use $\mu_{R}$ =
$\mu_{F}$ = 1. Calculations are in good agreement with the measured production
cross sections. The ratios of the differential jet production cross section
for jets with $R$ = 0.2 to the cross section for various $R$ is shown in Fig.
1 (right) for $R$ = 0.3 to $R$ = 0.6 in the same $p_{\textrm{T}}$ range. As
many of the systematic uncertainties cancel in the ratios they can be measured
with a higher precision. The cross section ratios are well-described by
POWHEG+PYTHIA6. A good agreement with POWHEG+PYTHIA6 has also been found in
cross section measurements of track-based jets and fully reconstructed jets at
lower centre-of-mass energies [13, 14, 15].

##  3 Measurement of the groomed momentum fraction $z_{g}$ in pp and heavy-ion
collisions

In pp collisions at $\sqrt{s}$ = 13 TeV the groomed momentum fraction
$z_{\textrm{g}}$ has been measured for jets with
$30~{}\textrm{GeV}/c<p_{\textrm{T}}<200~{}\textrm{GeV}/c$ for various ranges
in $p_{\textrm{T}}$. Jets were reclustered with the Cambridge/Aachen
algorithm. The SoftDrop parameters $\beta$ = 0 and $z_{\textrm{g}}$ = 0.1 were
used. No underlying event subtraction has been applied as the grooming already
removes soft components from the underlying event. The distributions are
unfolded back to particle level using a 2-dimensional unfolding based on the
Bayesian method [DAGOSTINI1995487]. The response matrices have been obtained
using a full detector simulation based on GEANT3 [16] of pp events at the same
centre-of-mass energy generated with PYTHIA8 [8].

Figure 3: Left: $z_{\rm{g}}$-differential per-jet yield for jets with $R$=0.4
for different bins in $p_{\textrm{T}}$ (left), jets with different $R$ for
$30~{}\textrm{GeV}/c<p_{\textrm{T}}<40~{}\textrm{GeV}/c$ (middle), and jets
with different $R$ for
$60~{}\textrm{GeV}/c<p_{\textrm{T}}<80~{}\textrm{GeV}/c$ (right). Lines
indicate calculations with the PYTHIA event generator using the Perugia 0 tune
[17].

Fig. 3 (left) shows the $z_{g}$\- differential jet yield for jets with R = 0.4
for $30~{}\textrm{GeV}/c<p_{\textrm{T}}<40~{}\textrm{GeV}/c$,
$60~{}\textrm{GeV}/c<p_{\textrm{T}}<80~{}\textrm{GeV}/c$, and
$160~{}\textrm{GeV}/c<p_{\textrm{T}}<180~{}\textrm{GeV}/c$. Results agree
within the uncertainties, indicating no $p_{\textrm{T}}$-dependence of the
$z_{\textrm{g}}$ for jets with large resolution parameter. In the middle and
right panel the $z_{\textrm{g}}$-differential yield of jets is plotted for
different $R$ from $R~{}=~{}0.2$ to $R~{}=~{}0.4$ for jets with
$30~{}\textrm{GeV}/c<p_{\textrm{T}}<40~{}\textrm{GeV}/c$ (middle) and
$60~{}\textrm{GeV}/c<p_{\textrm{T}}<80~{}\textrm{GeV}/c$ (right). For jets
with lower $p_{\textrm{T}}$ a difference in the $z_{\textrm{g}}$-differential
yield for different $R$ is observed, tending towards more symmetric splitting
for jets with smaller $R$. For jets with higher $p_{\textrm{T}}$ no dependence
of the $z_{\textrm{g}}$ on $R$ is observed. The trend is well reproduced by
PYTHIA6 using the Perugia 0 tune.

Figure 4: $z_{\textrm{g}}$-differential yield of track-based jets with
$80~{}\textrm{GeV}/c<p_{\textrm{T}}<120~{}\textrm{GeV}/c$ and $R~{}=~{}0.4$.
[18] Left: jets with $\Delta R<0.1$. Right: jets with $\Delta\textrm{R}>0.2$.
Red bands indicating the distributions from PYTHIA events embedded into heavy-
ion collisions.

In Pb-Pb collisions the $z_{\textrm{g}}$-differential yield of track based
jets was measured at $\sqrt{s_{\textrm{NN}}}$ = 2.76 TeV for jets with
$R~{}=~{}0.4$ and $80~{}\textrm{GeV}/c<p_{\textrm{T}}<120~{}\textrm{GeV}/c$
[18]. Fig. 4 shows the $z_{\textrm{g}}$\- differential yields for strongly
collimated jets with $\Delta R<0.1$, where $\Delta R$ is the distance between
the subjets, and jets with $\Delta\textrm{R}>0.2$. For the comparison to the
vacuum PYTHIA events were embedded into PbPb events in order to account for
effects from the underlying background. While no modification can be observed
for very collimated subjets, for jets with larger $\Delta R$ a suppression of
jets with symmetric splittings with an enhancement of untagged jets can be
observed.

##  4 Outlook

The study of jet substructure can be generalized by varying the grooming
conditions or looking at all splittings and constructing the Lund plane [19].
The Lund plane is obtained directly from the QCD kernel. Recent studies [20]
show that different regions in the Lund plane are sensitive to different
effects, among them medium-induced radiation and coherent radiation. A precise
measurement of the Lund plane in heavy-ion collisions will allow to provide
further constraints on the different effects contributing to jet modification
in the hot and dense medium.

## References

  * [1] M. Dasgupta, F. A. Dreyer, G. P. Salam and G. Soyez, _Inclusive jet spectrum for small-radius jets_ , _JHEP_ 06 (2016) 057 [1602.01110].
  * [2] A. J. Larkoski, S. Marzani, G. Soyez and J. Thaler, _Soft Drop_ , _JHEP_ 05 (2014) 146 [1402.2657].
  * [3] J. M. Butterworth, A. R. Davison, M. Rubin and G. P. Salam, _Jet substructure as a new Higgs search channel at the LHC_ , _Phys. Rev. Lett._ 100 (2008) 242001 [0802.2470].
  * [4] M. Cacciari, G. P. Salam and G. Soyez, _The anti- $k_{t}$ jet clustering algorithm_, _JHEP_ 04 (2008) 063 [0802.1189].
  * [5] M. Cacciari, G. P. Salam and G. Soyez, _FastJet User Manual_ , _Eur. Phys. J._ C72 (2012) 1896 [1111.6097].
  * [6] ALICE collaboration, J. Adam et al., _Measurement of jet quenching with semi-inclusive hadron-jet distributions in central Pb-Pb collisions at $\sqrt{s_{\mathrm{NN}}}=2.76$ TeV_, _JHEP_ 09 (2015) 170 [1506.03984].
  * [7] T. Sjostrand, S. Mrenna and P. Z. Skands, _PYTHIA 6.4 Physics and Manual_ , _JHEP_ 05 (2006) 026 [hep-ph/0603175].
  * [8] T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten et al., _An Introduction to PYTHIA 8.2_ , _Comput. Phys. Commun._ 191 (2015) 159 [1410.3012].
  * [9] G. Soyez, G. P. Salam, J. Kim, S. Dutta and M. Cacciari, _Pileup subtraction for jet shapes_ , _Phys. Rev. Lett._ 110 (2013) 162001 [1211.2811].
  * [10] P. Berta, M. Spousta, D. W. Miller and R. Leitner, _Particle-level pileup subtraction for jets and jet shapes_ , _JHEP_ 06 (2014) 092 [1403.3108].
  * [11] S. Frixione, P. Nason and C. Oleari, _Matching NLO QCD computations with Parton Shower simulations: the POWHEG method_ , _JHEP_ 11 (2007) 070 [0709.2092].
  * [12] S. Alioli, P. Nason, C. Oleari and E. Re, _A general framework for implementing NLO calculations in shower Monte Carlo programs: the POWHEG BOX_ , _JHEP_ 06 (2010) 043 [1002.2581].
  * [13] ALICE collaboration, J. Mulligan, _Inclusive jet measurements in pp and Pb-Pb collisions with ALICE_ , _PoS_ HardProbes2018 (2019) 080 [1812.07681].
  * [14] ALICE collaboration, S. Acharya et al., _Measurement of charged jet cross section in pp collisions at ${\sqrt{s}=5.02}$ TeV_, 1905.02536.
  * [15] ALICE collaboration, S. Acharya et al., _Charged jet cross section and fragmentation in proton-proton collisions at $\sqrt{s}$ = 7 TeV_, _Phys. Rev._ D99 (2019) 012016 [1809.03232].
  * [16] R. Brun, F. Bruyant, M. Maire, A. McPherson and P. Zanarini, _GEANT3_ , .
  * [17] P. Z. Skands, _Tuning Monte Carlo Generators: The Perugia Tunes_ , _Phys. Rev._ D82 (2010) 074018 [1005.3457].
  * [18] ALICE collaboration, S. Acharya et al., _Exploration of jet substructure using iterative declustering in pp and Pb-Pb collisions at LHC energies_ , 1905.02512.
  * [19] B. Andersson, G. Gustafson, L. Lonnblad and U. Pettersson, _Coherence Effects in Deep Inelastic Scattering_ , _Z. Phys._ C43 (1989) 625.
  * [20] H. A. Andrews et al., _Novel tools and observables for jet physics in heavy-ion collisions_ , 1808.03689.