Comprehending Semantic Types in JSON Data with Graph Neural Networks
Semantic types are a more powerful and detailed way of describing data than
atomic types such as strings or integers. They establish connections between
columns and concepts from the real world, providing more nuanced and
fine-grained information that can be useful for tasks such as automated data
cleaning, schema matching, and data discovery. Existing deep learning models
trained on large text corpora have been successful at performing single-column
semantic type prediction for relational data. However, in this work, we propose
an extension of the semantic type prediction problem to JSON data, labeling the
types based on JSON Paths. Similar to columns in relational data, JSON Path is
a query language that enables the navigation of complex JSON data structures by
specifying the location and content of the elements. We use a graph neural
network to comprehend the structural information within collections of JSON
documents. Our model outperforms a state-of-the-art existing model in several
cases. These results demonstrate the ability of our model to understand complex
JSON data and its potential usage for JSON-related data processing tasks.
Shuang Wei, Michael J. Mior
# Comprehending Semantic Types in JSON Data with Graph Neural Networks

###### Abstract.

Semantic types are a more powerful and detailed way of describing data than atomic types such as strings or integers. They establish connections between columns and concepts from the real world, providing more nuanced and fine-grained information that can be useful for tasks such as automated data cleaning, schema matching, and data discovery. Existing deep learning models trained on large text corpora have been successful at performing single-column semantic type prediction for relational data. However, in this work, we propose an extension of the semantic type prediction problem to JSON data, labeling the types based on JSON Paths. Similar to columns in relational data, JSON Path is a query language that enables the navigation of complex JSON data structures by specifying the location and content of the elements. We use a graph neural network to comprehend the structural information within collections of JSON documents. Our model outperforms a state-of-the-art existing model in several cases. These results demonstrate the ability of our model to understand complex JSON data and its potential usage for JSON-related data processing tasks.

JSON data, graph neural networks, semantic type detection, deep learning +
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methodologies Machine learning

+
Footnote †: journal: Computer methods Machine learning
$.user.username refers to the "username" key within the top-level object nested under the key "user".

## 3. Feature Selection

In our work, we use the same set of features as Sherlock, which is a single-column prediction model that takes all values from a single column as input and outputs the predicted semantic type for the corresponding column. The extracted feature vectors are used to train a neural network for the detection of semantic types. In Sherlock, a total of 1,587 features are extracted from the column values across four different dimensions. These dimensions are described below.

1. **Global statistics.** This category is a set of 27 hand-crafted features, typically some high-level statistical characteristics of the column. For example, column entropy describes the uniformity of the distribution of column values. Another example is the number of values that measures the number of unique values recorded in the column.
2. **Character-level distributions.** This category contains simple statistical features of character distributions. Specifically, 10 statistical functions, i.e. any, all, mean, variance, min, max, median, sum, kurtosis, skewness, are applied on all 96 ASCII-printable characters plus the form feed character, resulting in 960 features. For example, the any function checks if any column value contains a specific character. all checks if all column values contain a character. Some other examples of features are the maximum number of appearances of a character in a single column value and the median value of appearance of a character for all column values.
3. **Word embeddings.** Sherlock uses a pre-trained GloVe embedding (Haff et al., 2015) to characterize the semantic content of column values. The GloVe model contains a 50-dimensional embedding for 400,000 English words aggregated from 6,000,000,000 tokens. Similar to word2vec (Haff et al., 2015), GloVe embeddings can be used to measure semantic similarity between words. The advantage of GloVe compared to word2vec is that it does not rely simply on local information of words, but it also incorporates global statistics such as word co-occurrence. By calculating the mean, mode, median, and variance on the 50-dimensional GloVe feature vector for all column values, a 200-dimensional feature vector is produced in this category.
4. **Paragraph vectors.** A distributed bag of words version of the paragraph vector (PV-DBOW), or the doc2vec model (Haff et al., 2015) is implemented to capture features at the "topic" level of the column. The doc2vec model forces the model to predict random words that are sampled from paragraphs in the output by ignoring context words from the input. The model is pre-trained in Sherlock using the Gensim library to extract a paragraph feature that has 400 dimensions.

## 4. Proposed Graph Model

Raw JSON data is annotated by treating each key-value pair as a data point. For instance, from the user.json file, four data points can be extracted as illustrated in Figure 1. For each key-value pair, the label is determined by annotating the JSON Path to represent the semantic meaning, while the features are extracted using Sherlock from the corresponding values at each path. As an example, consider the key-value pair "user": {"id":9171087, "id_str": "9171087", "name": "ud83c"}. Here, the label assigned is user, and the features are extracted from {"id":9171087, "id_str": "9171087", "name": "ud83c"}. At this stage, we can proceed to apply Sherlock and evaluate its performance for semantic type detection, using it as a baseline result.

The structure of our model is illustrated in Figure 2. Each JSON file is processed by first obtaining all key-value pairs, as described in the preceding section. Using the same example as in Figure 1, we can obtain four key-value pairs. Subsequently, the features for each key-value pair are computed based on their respective values, yielding features \(f_{1}\), \(f_{2}\), \(f_{3}\), and \(f_{4}\).

Following this, four graphs are generated, labeled as "id", "id_str", "name", and "user". The first three graphs, \(G_{1}\), \(G_{2}\), and \(G_{3}\), each consist of a single-node, with the node features \(f_{1}\), \(f_{2}\), and \(f_{3}\). The fourth graph, \(G_{4}\), is a multi-node graph, with a root node having node features \(f_{4}\) and three edges connecting to three other nodes, each having node features \(f_{1}\), \(f_{2}\), and \(f_{3}\).

Once we obtain the graph representations, we use graph neural networks (GNNs)(Krizhevsky et al., 2015) to perform the classification task. Graph neural networks are well-suited for our problem, as JSON documents inherently possess a tree-like structure that can be represented as a graph. GNNs excel at capturing complex dependencies and relationships within graphs by leveraging structural information encoded in edges and node features. Using GNNs, we can effectively leverage the hierarchical relationships and dependencies present in JSON documents, enabling accurate analysis and prediction tasks such as semantic type prediction or classification. GNNs exploit the rich structural information of JSON documents, making them a powerful approach to understanding and extracting insights from this graph-based representation. In our study, we used the Spektral library(Serbia et al., 2017) to implement our GNN. Specifically, we employed a two-layer GCN model, where the input graph was first fed into a GCN layer with 256 hidden units, followed by graph pooling and a dropout layer. The output was then forwarded to another GCN layer with 64 hidden units, before being connected to a dense layer for multi-class classification. To optimize the model, we adopted

Figure 1. Example of JSON Data
Adam[(10)] as our optimizer and used categorical cross-entropy as our loss function. The learning rate was set to \(2\times 10^{-4}\).

## 5. Dataset

Our study uses Twitter data and Meetup data available on push-shift.io [(1)], a large-scale archive of social media content. Due to the immense size of the dataset, we selected a representative subset of each dataset for our research. Specifically, we extracted all available data from a specific date, resulting in a total of 30,000 distinct JSON objects for Twitter data, and 20,000 distinct JSON objects for Meetup data.

Figure 3 shows an example of the Twitter dataset used in this study. The example contains labels for a single-node graph, including created_at, id, and screen_name. The profile_link_color, profile_sidebar_border_color, profile_sidebar_fill_color, and profile_text_color also belong to single-node graphs with label color. Additionally, the example comprises multi-node graphs with labels such as bounding_box and user_mentions, which are also illustrated in the figure. The above-mentioned nodes have a depth of 1 in the graph, while nodes such as type and coordinate in this example have a depth of 2. Table 1 presents the number of examples corresponding to each depth (level of nesting) for our two datasets.

All key-value pairs with a null value or a type of Boolean are ignored since these values are highly repetitive in our dataset and do not contain useful semantic information. We then annotated each JSON Path with a class label, as discussed in the previous section, resulting in a label set comprising 43 distinct classes for the Twitter dataset and 32 distinct classes for the Meetup dataset.

To prepare the data for use in our model, we processed each JSON file into a graph structure, resulting in a total of around 110,000 distinct graphs for the Twitter dataset and 600,000 distinct graphs for the Meetup dataset. Each graph in our dataset is accompanied by a set of node features, an adjacency matrix, and a class label encoded in one-hot format. The dataset is partitioned into training, validation, and test sets in a 7:3:3 ratio.

## 6. Experiment Method

In our experiment, we initially preprocess the JSON file by extracting key-value pairs based on their corresponding JSON paths.

\begin{table}
\begin{tabular}{c c c} \hline Depth & \multicolumn{2}{c}{Number of examples} \\ \cline{2-3}  & Twitter & Meetup \\ \hline
1 & 1,205 & 191,884 \\
2 & 32,158 & 304,750 \\
3 & 74,187 & 101,930 \\
4 & 3,907 & 0 \\
5 & 124 & 0 \\ \hline \end{tabular}
\end{table}
Table 1. Number of examples for each depth

Figure 3. Example of Twitter JSON data

Figure 2. Proposed model architecture
Figure 4 demonstrates the conversion of a JSON document into relational tables, where each key-value pair is represented as a separate column. Subsequently, a feature extraction process is applied to these values to obtain the feature vectors. This step enables us to utilize the Sherlock model for classification, allowing us to establish a baseline performance measure. We then proceed to process the data into graphs, following the procedures outlined in Figure 2. The classification task is then performed using a graph neural network model. We summarize the time spent on each preprocessing step in Table 2, providing an overview of the time requirements for these tasks on our two different datasets. The key-value pair extraction step takes 1,205s and 3,525s for the Twitter and Meetup datasets respectively, which is the most time-consuming preprocessing step. The feature extraction and graph processing steps take less time than the key-value pair extraction. The feature extraction step is the only preprocessing step required to train the Sherlock model.

## 7. Experiment Results

Table 3 provides a comprehensive comparison between the performance of Sherlock and the proposed model in terms of the F1 score and accuracy metrics. The evaluation is conducted for all semantic classes, which are categorized into single-node and multi-node based on the number of nodes in the graph. The table presents some selected examples from each category, including screen_name, country_code, timestamp_ms, color, and description for single-node, and bounding_box, user_mentions, retweet_status, hashtag, and full_name for multi-node. The average performance result is also presented at the bottom of the table.

In the single-node setting, the proposed model outperforms Sherlock on some labels, for example, with F1 scores of 0.95 for screen_name, 0.92 for country_code, compared to 0.92 and 0.80, respectively, for Sherlock. However, Sherlock achieves a perfect F1 score of 1.00 for the timestamp_ms label, while the proposed model only achieves a score of 0.97. The Sherlock model also exhibits higher accuracy levels for the semantic types of country_code and timestamp_ms. Sherlock also performs slightly better for the description label, with an F1 score of 0.75 compared to 0.67 for the proposed model. Based on the results, it can be suggested that the base Sherlock model might exhibit better performance when applied to single-node scenarios, which may be attributed to the fact that the Sherlock model utilizes a more complex neural network architecture compared to our network.

In the multi-node setting, the proposed model achieves a perfect F1 score of 1.00 for the bounding_box label, while Sherlock achieves a score of 0.83. The proposed model also outperforms Sherlock for the user_mentions and retweet_status labels, with F1 scores of 0.84 and 0.82 compared to 0.59 and 0.57, respectively. However, Sherlock achieves a slightly better F1 score of 0.40 for the hashtag label compared to 0.34 for the proposed model. The proposed model achieves a higher F1 score of 0.22 for the full_name label compared to 0.00 for Sherlock. In terms of accuracy, our proposed model significantly outperforms Sherlock. Specifically, for the hashtag semantic type, our model achieves an accuracy of 0.80, while Sherlock only achieves an accuracy of 0.23. Similarly, for the full_name semantic type, our model achieves an accuracy of 0.50, while Sherlock fails to classify any instance correctly. These findings demonstrate that our proposed model is more adept at predicting complex structures, thus providing greater utility for practical applications.

Table 4 presents the comparison between Sherlock and our proposed model on the meetup dataset. The JSON files within the meetup dataset exhibit highly similar structures, resulting in higher overall prediction accuracy and F1 score. In cases involving multiple nodes, our model achieves perfect accuracy of 1.00, while Sherlock performs equally well on classes such as event and category, with F1 scores of 0.99 and 0.97 respectively for group and group_photo. For single-node classes, both models demonstrate similar performance levels. The Meetup dataset consists of numerous homogeneous JSON files, many of which exhibit similar hierarchical structures. As a result, the base model Sherlock can achieve good performance in the multi-node setup, indicating that our model does not outperform Sherlock on this dataset. The reason could be that the Meetup dataset exhibits a higher level of homogeneity and has a less complex hierarchical structure for our model to take advantage of.

Table 5 shows a comparison between the training time and model size of Sherlock and the proposed model. Our proposed model takes significantly longer to train but produces a much smaller model. We

\begin{table}
\begin{tabular}{c c c} \hline \hline Preprocessing steps & Twitter & Meetup \\ \hline Key-value pair extraction & 1,205s & 3,525s \\ Feature extraction & 785s & 2,100s \\ Graph processing & 152s & 450s \\ \hline Total time & 2,142s & 7,075s \\ \hline \hline \end{tabular}
\end{table}
Table 2. Time spent on preprocessing steps

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & & \multicolumn{2}{c}{Sherlock} & \multicolumn{2}{c}{Proposed model} \\ \cline{3-6}  & Label & F1 score & Accuracy & F1 score & Accuracy \\ \hline Single-Node & screen\_- & 0.92 & 0.93 & **0.95** & 0.93 \\  & name & & & \\  & country\_- & 0.80 & **1.00** & **0.92** & 0.85 \\  & code & & & & \\  & timestamp\_- & **1.00** & **1.00** & 0.97 & 0.96 \\  & ms & & & & \\  & color & **0.99** & 0.99 & 0.98 & 0.99 \\  & description & **0.75** & **0.77** & 0.67 & 0.60 \\ \hline Multi-Node & bounding\_- & 0.83 & 1.00 & **1.00** & 1.00 \\  & box & & & & \\  & user\_- & 0.59 & 0.41 & **0.84** & **0.97** \\  & mentions & & & & \\  & retweet\_- & 0.57 & 0.48 & **0.82** & **0.86** \\  & status & & & & \\  & hashtag & 0.34 & 0.23 & **0.40** & **0.80** \\  & full\_- & 0.00 & 0.00 & **0.22** & **0.97** \\  & name & & & & \\ \hline Average & & 0.82 & 0.84 & **0.85** & **0.85** \\ \hline \hline \end{tabular}
\end{table}
Table 3. Comparison of Sherlock and the proposed model on Twitter dataset
expect that advances in graph neural network training will apply to our setting to further reduce the training time (Beng et al., 2017; Chen et al., 2018). Our proposed model exhibits a notable reduction in model size in comparison to Sherlock. This is mainly attributed to the fact that our model incorporates a smaller number of neural network layers. Consequently, this disparity may account for certain suboptimal predictions made by our model in comparison to Sherlock, particularly in single-node predictions.

Our proposed model achieves a higher average F1 score compared to Sherlock. These results suggest that our proposed model has a superior ability to learn structural information from the data.

## 8. Future Work

In our ongoing research, we aim to explore alternative subgraph representations to enhance the prediction of semantic types in our proposed model. The current subgraphs we utilize do not incorporate the parent node and its sibling nodes, thus missing out on valuable information present in those nodes. Sato (Sato, 2010) is a model that utilizes neighbor columns and table-level information for semantic prediction; however, it exhibits limited scalability in terms of training time. To address this limitation, we plan to construct subgraphs that include the parent and sibling nodes of the target node. In addition, we will assign edge weights to the graph, allowing us to emphasize the importance of the target node for prediction. By enabling additional edge features within each subgraph, we can leverage neural networks such as edge-conditioned GCN (Kipf and Welling, 2015) that are capable of utilizing edge weight information. Subsequently, we intend to combine the outputs obtained from different subgraphs using a transformer (Kipf and Welling, 2015). The transformer architecture is suitable for this purpose because of its ability to capture global dependencies and model the relationships between the outputs of different subgraphs effectively.

In addition, we plan to extend our experimentation beyond the current dataset in order to enhance the robustness and generalizability of our model. Specifically, we will train and test our model on additional datasets that are representative of real-world scenarios. Our goal is to evaluate the effectiveness of the model's predictive capabilities on JSON structures that it has not yet encountered. Moreover, we aim to examine the impact of training set size on the performance of our model, with particular attention to the detection of less frequent semantic types. By conducting such analyses, we aim to better understand the limitations and strengths of our model and further refine it for real-world applications.

## 9. Conclusion

In this paper, we proposed a model for predicting semantic types in nested JSON data that can be used for various automated data processing tasks. Existing models either predict for a single set of values at a time or are limited to non-nested relational data.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{3}{c}{Sherlock} & \multicolumn{2}{c}{Proposed model} \\ \cline{3-6}  & Label & F1 score & Accuracy & F1 score & Accuracy \\ \hline Single-Node & event\_id & 0.66 & **0.64** & **0.68** & 0.63 \\  & id & 0.86 & 0.85 & **0.91** & **0.90** \\  & member\_- & **0.95** & **0.95** & **0.95** & 0.94 \\  & name & & & & \\  & shortname & **0.99** & 0.99 & 0.98 & **1.00** \\ \hline Multi-Node & event & **1.00** & **1.00** & **1.00** & **1.00** \\  & category & **1.00** & **1.00** & **1.00** & **1.00** \\  & group & 0.99 & 0.97 & **1.00** & **1.00** \\  & group\_- & 0.97 & 0.99 & **1.00** & **1.00** \\  & photo & & & & \\ \hline Average & & 0.89 & 0.89 & **0.92** & **0.90** \\ \hline \hline \end{tabular}
\end{table}
Table 4. Comparison of Sherlock and the proposed model on the Meetup dataset

Figure 4. transformation of JSON document to relational table

\begin{table}
\begin{tabular}{c c c c} \hline \hline Model & \multicolumn{2}{c}{Training time} & Model size \\ \cline{2-4}  & Twitter & Meetup & \\ \hline Sherlock & 252s & 690s & 5.9MB \\ Proposed model & 1,230s & 3,051s & 1.9MB \\ \hline \hline \end{tabular}
\end{table}
Table 5. Training time and model size comparison
To address this limitation, we proposed an extension of the semantic type prediction problem to semi-structured JSON data with types labeled based on JSON Paths. Our proposed model annotates the semantic type of JSON data with its hierarchical structure and employs a graph neural network to predict the semantic type using the same set of features extracted by Sherlock. We demonstrate several cases where our model outperforms Sherlock, indicating its ability to comprehend complex JSON data and its potential for semi-structured data processing tasks.

Our ongoing research focuses on enhancing the prediction of semantic types in our proposed model through alternative subgraph representations. By incorporating the parent and sibling nodes into subgraphs and assigning edge weights, we can capture valuable information for accurate predictions. Leveraging additional edge features will further improve our model's performance. We plan to validate our approach on diverse datasets, ensuring its robustness and generalizability to real-world scenarios. Furthermore, we plan to investigate the impact of the size of the training set on model performance, especially for detecting less frequent semantic types.

In conclusion, our work contributes to the development of deep learning models for predicting semantic types in JSON data. Our proposed model offers a better understanding of complex data and improved performance compared to Sherlock on semi-structured data.
