 semantics conditioned feature modeling network is trained, a large number ($K$) of point features for each object class could be generated conditioned on the corresponding semantic feature $\mathbf{e}_{y}$ and $K$ different random noises $\{\mathbf{z}_{k}\}_{k=1}^{K}$ sampled from the standard Gaussian distribution $\mathcal{N}(\mathbf{0},\mathbf{I})$. Specifically, we generate point features according to:
\begin{equation}
	\bar{\mathbf{x}} = G(\mathbf{e}_{y}, \mathbf{z})
	\label{eq3}
\end{equation}
where $\bar{\mathbf{x}}$ is the generated point feature. In the following, we describe the feature generation and classifier learning in three different tasks, i.e. C3DS, Z3DS, and GZ3DS.

\subsubsection{Conventional 3D Scene Semantic Segmentation}
In C3DS, the testing points are assumed to come from only seen classes, hence, we generate a large number of point features for each seen class in $Y^{S}$ conditioned on seen-class semantic features according to (\ref{eq3}). The set of generated point features and corresponding labels is denoted by $\bar{\mathcal{X}}^{S}$. Then, we train a semantics enhanced classifier $f_{s}(\cdot)$ with $\bar{\mathcal{X}}^{S}$ as follows:
\begin{equation}
	L_{s} = E_{(\bar{\mathbf{x}},y) \in \bar{\mathcal{X}}^{S}} [\mathbb{C}(\mathbb{S}(f_{s}(\bar{\mathbf{x}})),y)]
	\label{eq4}
\end{equation}
where $\mathbb{C}(\cdot)$ and $\mathbb{S}(\cdot)$ are a cross-entropy loss function and a softmax function respectively. Note that $f_{s}(\cdot)$ could be any classifier, not constrained to linear classifier as done in existing segmentation networks~\cite{2018dgcnn, 2020randlanet, fan2021scf}. After training the classifier, given a real testing point feature $\mathbf{x}$, we predict its label $y$ by:
\begin{equation}
	y = \arg \max_{y \in \mathcal{Y^{S}}} \mathbb{S}(f_{s}(\mathbf{x}))
	\label{eq5}
\end{equation}
Finally, for a given testing 3D point cloud, we achieve point cloud semantic segmentation by classifying the feature of each 3D point extracted by the backbone network via the learned semantics enhanced feature classifier.

\subsubsection{Zero-Shot 3D Scene Semantic Segmentation}
Thanks to the language-level semantics conditioned point feature modeling, the proposed framework has the flexible ability to segment novel-class objects in 3D point cloud scenes (including Z3DS and GZ3DS) if their corresponding semantics are available, which is an impossible function for existing segmentation networks. In Z3DS, the task is to classify unseen-class points. To this end, we sample a large number of point features for each unseen class in $Y^{U}$ from the learned conditional distributions conditioned on the unseen-class semantic features according to (\ref{eq3}). That's to say, the semantic feature $\mathbf{e}_{y}$ is sampled from unseen-class label set $Y^{U}$, which is different from those in C3DS where the semantic feature $\mathbf{e}_{y}$ is sampled from seen-class label set $Y^{S}$. Then we train a semantics enhanced classifier $f_{u}(\cdot)$ in a similar way to (\ref{eq4}), and classify the real testing unseen-class points as done in (\ref{eq5}). 

\subsubsection{Generalized Zero-Shot 3D Scene Semantic Segmentation}
In GZ3DS, the testing points could be from either seen classes or unseen classes. Hence, according to (\ref{eq3}), we generate a large number of point features for every class in $Y$ conditioned on all semantic features, and the semantics enhanced classifier $f_{g}(\cdot)$ training and the point feature classifying are similar to those in C3DS and Z3DS. The only difference among C3DS, Z3DS and GZ3DS is that their conditions and classification space are different, which in turn demonstrates the flexibility of our proposed framework.


\section{Experiment}
\label{experiments}

\subsection{Experimental Setup}
\subsubsection{Backbone Networks and Datasets}
In C3DS, we evaluate the proposed SeCondPoint framework with $3$ typical 3D point cloud segmentation backbone networks: DGCNN~\cite{2018dgcnn}, RandLA-Net~\cite{2020randlanet}, and SCF-Net~\cite{fan2021scf}. In this work, the architectures of the used networks are totally the same with those in their original papers since we directly use their public codes and models. We choose the three networks not only because their codes and models are public, but also because they are representative networks. Specifically, DGCNN is a graph-based architecture and usually used to process block-size small-scale point clouds. While RandLA-Net and SCF-Net are two different point-based networks, which can be directly used to deal with large-scale point clouds. Two public 3D point cloud scene datasets, S3DIS~\cite{2016s3dis} and ScanNet~\cite{2017scannet}, are used to evaluate the proposed SeCondPoint framework. The details about the two datasets are provided in the supplemental materials due to the limited space. The semantic feature of each given class is determined simply by 1) recording its class name and 2) retrieving its semantic feature among all word2vec embeddings~\cite{MikolovSCCD13} according to its name, which is represented by a $300$-D vector. 

In Z3DS, we evaluate the proposed SeCondPoint framework with DGCNN and RandLA-Net being the network backbones. Other backbones could be adapted seamlessly to Z3DS in the same way. We construct two benchmarks by re-organizing S3DIS~\cite{2016s3dis} and ScanNet~\cite{2017scannet}. In S3DIS, we consider the $12$ semantic categories as valid categories and ignore the `clutter' class, which are split into seen classes and unseen classes in $3$ different manners, i.e. 10/2, 8/4, 6/6 as seen classes/unseen classes respectively. In ScanNet, we choose $19$ semantic categories as valid categories, which are also split in $3$ different ways, i.e. 16/3, 13/6, 10/9 splits. More details are introduced in the supplemental materials due to the limited space. For the sake of clarity, we denote the re-organized S3DIS and ScanNet as S3DIS-0 and ScanNet-0 respectively. The semantic features are the same with those in C3DS.

\iffalse
Two public 3D point cloud scene datasets, S3DIS~\cite{2016s3dis} and ScanNet~\cite{2017scannet} are used to evaluate the proposed SeCondPoint framework.S3DIS includes 3D scans of $272$ rooms distributed in $6$ areas from $3$ different buildings. For each point in the 3D scans, it is represented with X-Y-Z Euclidean coordinates and R-G-B color features, and it is annotated with one of the semantic labels from $13$ categories, including ceiling, floor, wall, beam, column, window, door, chair, table, bookcase, sofa, board, and clutter. ScanNet includes $1513$ scanned and reconstructed point cloud scenes. Among them, $1201/312$ point clouds are used for training/testing respectively. Each point in a 3D point cloud is represented with X-Y-Z Euclidean coordinates and is labeled with one of $21$ semantic categories where $20$ semantic categories are used for evaluation and the remaining one (`unannotated' class) is often ignored. For small-scale point cloud based networks (e.g. DGCNN), a whole 3D point cloud is usually split into several blocks of area size 1m*1m in the X-Y plane firstly, and then $4096$ points are randomly sampled from each block as the network inputs. For large-scale point cloud based networks (e.g. RandLA-Net and SCF-Net), the original 3D point clouds are directly processed after a downsampling pre-processing. The semantic features are provided by word2vec embeddings~\cite{MikolovSCCD13} pre-trained on the Google News dataset, which can be directly download from the Internet. The semantic feature of each given class is determined simply by 1) recording its class name and 2) retrieving its semantic feature among all word2vec embeddings according to its name, which is represented by a $300$-D vector.

For Z3DS, we validate the ability of the proposed SeCondPoint framework to perform zero-shot semantic segmentation with DGCNN and RandLA-Net being the network backbones. Other network backbones could be adapted seamlessly to Z3DS in the same way. We construct two benchmarks by re-organizing S3DIS~\cite{2016s3dis} and ScanNet~\cite{2017scannet}. In S3DIS, we consider the $12$ semantic categories as valid categories and ignore the `