ref{fig:network_relation_view} (A). In each matrix, the similarity values are linearly scaled to the cell brightness, where zero is represented by the highest brightness and one the lowest. In our initial design, all the neurons are listed in the matrix without considering their importance. However, such a matrix may be too large to be displayed in the canvas. We improved the scalability by grouping the rows and columns based on their importance in the neuron extraction step, dividing the matrix into four regions, Figure~\ref{fig:network_relation_view} (B). The top-left region shows the important neurons from both the source and the target models in the matrix form, while the other three regions are summarized into histograms of similarity value distributions. On the left and the top side of the matrices, the corresponding neuron indices are displayed with colors indicating the domain origins. 


\vspace{0.8mm} \noindent \textbf{Weight Visualization.} In Figure~\ref{fig:network_relation_view} (A), the order of the domain origins in two consecutive matrices (Layer 1 and 2) are switched alternatively, leaving the space of weights in the source and the target model on the corresponding adjacent corners. Following the same splitting strategy with the matrices, the weights are also summarized into four regions, Figure~\ref{fig:network_relation_view} (C). In Region (1), the weights joining important neurons between both layers are rendered as an adjacency matrix. The rows and columns of the small squares are aligned with the corresponding important neurons in the similarity matrices. The brightnesses of the cells are mapped to the weight values, and the inner red dot indicates the existance of the corresponding important weight in the source model. Region (2) and (3) summarize the weight values into boxplots by aggregating the weights on the side of the important neurons. For example, the left side of Region (2) is next to the non-important neurons in Layer 1, while the bottom side connects to the important neurons in Layer 2. Thus, the vertical boxes show the weight value distributions connected with the same important neuron in Layer 2. The thicknesses of the rectangles in the boxes represent the number of weights. Pie chart glyphs are attached next to the boxes to show the proportions of weights with correspondence in the source model. For the weights connecting non-important neurons on both sides, we use a histogram in Region (4) to summarize the distribution of the weight values. Note that on the source model side (e.g., the bottom left corner in Figure~\ref{fig:network_relation_view} (A)), there are no red dots in the squares or pie chart glyphs since only the correspondences from target network components are considered.


\vspace{0.8mm} \noindent \revise{\textbf{Alternative Design.} We also considered utilizing node-link diagrams where the neurons in different layers are encoded as columns of nodes with weight links between consecutive columns~\cite{Liu,Hohman2019a}. However, isolating the source and the target models into two diagrams weakens the direct comparison of the corresponding layers and weights in the two neural networks. Furthermore, visual clutter can be easily generated when large numbers of neurons and weights exist in the layers, whereas the matrix layout can increase readability by providing a non-overlapped representation of nodes and links~\cite{VonLandesberger2011}.}

\vspace{0.8mm} \noindent \textbf{Interactions.} The network relation view supports various interactions on the similarity matrices and the weights. Users can change the desired class and network layers in the title bar. Clicking on the indices will open a detail panel, Figure~\ref{fig:teaser} (C.1), which shows a feature visualization~\cite{olah2017feature} of the neuron and the top-5 most similar neurons in the other model. By clicking on a square, the details of the associated weight are shown in a pop-up panel including the indices of the connected neurons as well as the weight feature map, Figure~\ref{fig:teaser} (C.2). The visual elements, including cells, boxplots, and pie glyphs, also provide pop-up details. \revise{To simplify the displayed elements, two switches are employed to toggle the appearance of non-important regions in the similarity matrices and the source weights.}




\subsection{Feature Analysis}
Deep neural networks are always considered as feature extractors that build new features from the original input space. As such, analysts often need to inspect what features are reconstructed in the hidden layers and how they are used in the classification process (\textbf{T3}). We have designed the feature analysis module to present the feature-related information and support the exploration of informative features.

\subsubsection{Domain Discriminability}
Usually, how a feature can discriminate the instances from different domains plays an essential role in a successful transfer. The feature extractors should be carefully investigated to see if they can reconstruct important common patterns shared in both domains. Most of the feature-based transfer learning techniques seek to build new feature transformations upon the source domain to create such shared features.


\setlength\textfloatsep{1.8mm}
\begin{algorithm}[htp]
    \label{alg:a-distance}
    \SetAlgoLined
    \KwData{
        $P$ selected neurons, $\{n_1, n_2,..., n_{P}\}$; the source dataset, $X_{{\mathcal{D}_s}} = \{(\boldsymbol{x}_{{{\mathcal{D}_s}}, i}, y_{{{\mathcal{D}_s}}, i}) | i \in [1, N_{{{\mathcal{D}_s}}}] \}$; the target dataset $X_{\mathcal{D}_t} = \{(\boldsymbol{x}_{{\mathcal{D}_t}, i}, y_{{\mathcal{D}_t}, i}) | i \in [1, N_{\mathcal{D}_t}] \}$ 
    }
    
    \KwResult{The domain discriminability values for the $P$ neurons, $\{u_1, u_2,..., u_P\}$}
    
    $A \leftarrow [\textrm{empty matrix}]_{(N_{\mathcal{D}_s} + N_{\mathcal{D}_t}) \times P}$ \tcp{Attribution values}

    $\boldsymbol{l} \leftarrow [\textrm{empty vector}]$ \tcp{Domain labels}

    \For{$i = 1:N_{\mathcal{D}_s}$}{ \label{alg-source-start}
        \For{$j = 1:P$}{
            $\boldsymbol{A}_{i, j} \leftarrow \textrm{LayerConductance}(n_j, \boldsymbol{x}_{{\mathcal{D}_s}, i})$
        }\label{alg-source-end}
        $\boldsymbol{l}_i \leftarrow 0$ \tcp{Label 0 for source instances} \label{alg-domain-label-1}
    }

    \For{$i = 1:N_{\mathcal{D}_t}$}{ \label{alg-target-start}
        \For{$j = 1:P$}{
            $\boldsymbol{A}_{i + N_{{\mathcal{D}_s}}, j} \leftarrow \textrm{LayerConductance}(n_j, \boldsymbol{x}_{{\mathcal{D}_t}, i})$
        }\label{alg-target-end}
        $\boldsymbol{l}_{i + N_{{\mathcal{D}_s}}} \leftarrow 1$ \tcp{Label 1 for target instances} \label{alg-domain-label-2}
    }

    $C = \textrm{LinearSVM}(A, \boldsymbol{l}, \textrm{10-fold cross validation})$ \label{alg-classifier}

   

    $\boldsymbol{u} \leftarrow [\textrm{feature weights of}$ $C]$ \tcp{Use the weights from the classifier as domain discriminability values}

    \caption{Computing the domain discriminability values.}

\end{algorithm}

To support the diagnosis of the transferred knowledge, we propose \textit{domain discriminability} to measure whether a learned feature on the neuron is domain-invariant or domain-dependent. This measure is inspired by the widely-used $\mathcal{A}$-distance~\cite{ben2007analysis} which estimates the differences of two data distributions with a linear classifier between the two groups of data. The computation of the domain discriminability for neurons is described in Algorithm~\ref{alg:a-distance}. First, we compute the Layer Conductance values for each selected neuron on the source and the target datasets, respectively (line~\ref{alg-source-start}-\ref{alg-source-end} and~\ref{alg-target-start}-\ref{alg-target-end}). In the matrix $A$, each row represents the attribution values for an instance on the selected neurons. Then, a domain label list $\boldsymbol{l}$ is prepared to specify the domain origins of each row (line~\ref{alg-domain-label-1} and~\ref{alg-domain-label-2}). Finally, a linear classifier $C$ is trained on the stacked attribution matrix and the domain label list (line~\ref{alg-classifier}). Acting as a feature selection method, the coefficients from the trained $C$ present how important a column in $A$, i.e., a selected neuron, can be in discriminating the domain origins of the data instances.


One potential limitation in computing the domain discriminability values is that the total number of neurons in the entire