§ ABSTRACT

In the first half this paper, we generalize the theory of layer points <cit.> to the more general context of $\vec{v}$-hierarchical clusterings <cit.>.
Layer points provide a compressed description of a hierarchical clustering by recording only the points where a cluster changes.  For multi-parameter hierarchical clusterings we consider both a global notion of layer points and layer points in the direction of a single parameter.  An interleaving of hierarchical clusterings of the same set induces an interleaving of global layer points.  In the particular, we consider cases where a hierarchical clustering of a finite metric space, $Y$, is interleaved with a hierarchical clustering of some sample $X \subseteq Y$.

In the second half, we focus on the hierarchical clustering $\pi_0 L_{-,k}(Y)$ for some finite metric space $Y$.  When $X \subseteq Y$ satisfies certain conditions guaranteeing $X$ is well dispersed in $Y$ and the points of $Y$ are dense around $X$, there is an interleaving of layer points for $\pi_0 L_{-,k}(Y)$ and a truncated version of $L_{-,0}(X) = V_{-}(X)$.  Under stronger conditions, this interleaving defines a retract from the layer points for $\pi_0 L_{-,k}(Y)$ to the layer points for $\pi_0 L_{-,0}(X)$.

§ INTRODUCTION

At its most basic level, a clustering of a data set is any collection of its disjoint subsets.  Of course, in practice, these subsets should indicate some relationship between data points and there are a wide variety of methods that attempt to form clusters in a meaningful way.
If a clustering method depends on a choice of parameter, we can consider how the clustering changes as the parameter varies.  Heuristically, in a (covariant) hierarchical clustering, clusters appear or merge together as the parameter increases.  (Whereas, in a contravariant hierarchical clustering, clusters break apart or disappear as the parameter increases.)

Rather than a single parameter, a hierachical clustering may be indexed instead by the elements of any partially ordered set.  In particular,   we will utilize the $\vec{v}$-hierarchial clusterings of Rolle and Scoccola <cit.> as a framework for multi-parameter hierarchical clusterings.
Consider $H$, a $\vec{v}$-hierarchical clustering of a set $Z$ for some $\vec{v} \in \{-1,1\}^n$.  Such a hierarchical clustering is indexed by the elements of $R^{\vec{v}}$, which is the set $\RR^{n}$ with a partial order that depends on $\vec{v}$ (see Section <ref>).
The set
\[\Gamma(H) = \{ (\vec{s}, S) | \vec{s} \in R^{\vec{v}}, S \in H(\vec{s})\},\]

has a partial order
where $(\vec{s}, S) \leq (\vec{t}, T)$ if $\vec{s} \leq \vec{t}$ and $S \subseteq T$.  This is a sub-poset of $R^{\vec{v}} \times \mathcal{P}(Z)$ where $\mathcal{P}(Z)$ is the power set of $Z$ with the partial order given by inclusion.

Consider the equivalence relation on $\Gamma(H)$ generated by setting $(\vec{t}, T) \sim (\vec{s}, S)$ whenever $\vec{t} \leq \vec{s}$ and $S=T$.  The equivalence classes of this equivalence relation are called the layers of $\Gamma(H)$.
Alternately, layers can be viewed as the path components of a graph whose vertices are the elements of $\Gamma(H)$ with edges $(\vec{s}, S) \to (\vec{t}, S)$ for all $\vec{s} \leq \vec{t}$ with $S \in H(\vec{s}) \cap H(\vec{t})$.
This graph is the component graph defined in <cit.>.

Layer points are distinguished points on the boundary of a layer, and we consider two types of them.
A global layer point for a $\vec{v}$-hierarchical clustering is some $(\vec{s}, S)$ in $\Gamma(H)$ where $S$ does not appear in $H(\vec{t})$ for any $\vec{t} < \vec{s}$.
An $i^{th}$-parameter layer point of a multiparameter hierarchical clustering, $H$, is a global layer point of a single-parameter slice of $H$.
In both cases, a layer point marks a first occurrence of a cluster, keeping in mind that when considering multiple parameters the “first” occurrence may not be unique.  In order to guarantee the existence of such points, we restrict our attention to $\vec{v}$-hierarchical clusterings that are bounded below (see Definition <ref>).

Taking maximal lower bounds within layers defines a (not necessarily unique) map from $\Gamma(H)$ to layer points.  This leads to our first stability result.
The layer points of single parameter hierarchical clusterings are stable in the sense that an interleaving of hierarchical clusterings induces a homotopy interleaving of layer points (Theorem <ref>).  In particular, we will be interested in these interleavings of layer points for approximations of a hierarchical clustering by a hierarchical clustering of a subsample of points.

A standard example of a 2-parameter hierarchical clustering is given by taking the path components of the Lesnick, or degree-Rips, complex associated to a subset of a metric space. The Lesnick complexes, $L_{s,k}(Z)$ filter the Vietoris-Rips complex, $V_s (X)$. (See Section <ref> for a brief introduction to these complexes.)  This is the context in which layer points were originally introduced by Jardine in <cit.>.  We consider the single parameter slices of this clustering where the density parameter is constant.

Let $X$ be a sample (subset) of a finite metric space $Y$.  We define the hierarchical clustering $L_0 X[c]$, a truncated version of $\pi_0 L_{-, 0}(X)=\pi_0 V_{-}(X)$, as an approximation for $L_k Y = \pi_0 L_{-,k}(Y)$, the $2^{nd}$-parameter slice of $\pi_0 L_{-,-}(Y)$.  This approximation is stable—if $X$ and $Y$ satisfy an appropriate notion of closeness, then there is an interleaving of $L_0 X[c]$ and $L_k Y$ (Lemma <ref>).

To define what it means for $X$ and $Y$ to be close, we consider two directional notions of distance between $X$ and $Y$.  The first is the directional Hausdorff distance, $h(Y, X)$.
The second is a directional distance adjusted for density, $N_k(Y, X)$.
The value of $N_k(X, Y)$ gives a threshold for the parameter at which all points in $X$ have $k$ neighbours in $Y$. The stability results presented in this paper apply to sets $X \subseteq Y$ where $h(Y, X)$ and $N_k(X,Y)$ are small.

Let $s_0 < s_1 < \ldots < s_M$ be the phase change numbers of $X$, which are the distinct distances between elements of $X$.
Our strongest stability result states that if
$2 h(Y, X) + N_k(X,Y) < s_{i+1} -s_i$
for all $i$, then the poset of branch points of $\pi_0 L_{-, 0}(X)=V_{-}(X)$ is a retract of the poset of layer points of $\pi_0 L_{-,k}(Y)$ (Corollary <ref>).
We show this by constructing an interleaving of the form

\[\begin{tikzcd}
\Lambda(L_0 X) \ar[r, "id"] \ar[d] & \Lambda(L_0 X) \ar[d] \\
\Lambda(L_k Y) \ar[r] \ar[ur] & \Lambda(L_k Y)
\end{tikzcd}\]

where the upper triangle commutes and the lower triangle commutes up to homotopy.
The existence of this diagram follows from a more general result, which requires the truncated version of the hierarchical clustering, $L_0 X$.

The hierarchical clustering $L_0 X[c]$ is defined so $L_0 X[c](s)$ is empty if $s<c$, but is otherwise equal to $L_0 X(s)$.
In Theorem <ref>, we prove the existence of a homotopy commutative diagram of posets,

\[\begin{tikzcd}[column sep= large]
\Lambda(L_0 X[c]) \ar[d, "i_{\epsilon}"] \ar[r, "(id_Y)_{\epsilon+\delta}"]& \Lambda(L_0 X[c]) \ar[d, "i_{\epsilon}"]\\
\Lambda(L_k Y) \ar[r, "(id_Y)_{\epsilon+\delta}"] \ar[ur, "\theta_\delta"] & \Lambda(L_k Y),
\end{tikzcd}\]

where the maps depend on a choice of nonnegative numbers $\epsilon$, $\delta$, and $c$ satisfying $\delta \geq 2 h(Y, X)$, and $N_k(X,Y)-\epsilon \leq c \leq \delta$.

If these numbers satisfy the further condition

\[ c+\epsilon+\delta < s_{i+1} - s_i \]

for all $i$, then the map $\theta_{\delta}i_{\epsilon}: \Lambda(L_0 X[c]) \to \Lambda(L_0 X[c])$ is the identity.  This gives a retract,

\[ \theta_\delta: \Lambda(L_k Y) \to \Lambda(L_0 X[c]). \]

Consequently, if $N_k(X,Y)+2h(Y,X) < s_{i+1}-s_i$ for all $i$, there is a retract

\[ \Lambda(L_k Y) \to \Lambda(L_0 X). \]

An alternate proof of this corollary (Corollary <ref>) can be given by showing
\[\Lambda(L_0 X[c]) \cong \Lambda(L_0 X)\]
for all $c$ satisfying $c< s_{i+1}-s_i$ for all $i$.
The maps in the interleaving of $\Lambda(L_0 X)$ and $\Lambda(L_k Y)$ of Corollary <ref> could thus be given piecewise without defining the truncated hierarchical clustering, $L_0 X[c]$.  However, these piecewise maps are still essentially passing to the truncated clustering, and using $L_0 X[c]$ allows for the weaker statement in Theorem <ref>, where $\Lambda(L_0 X[c])$ and $\Lambda(L_0 X)$ are not necessarily isomorphic.

§ LAYER POINTS

§.§ Layers

Let $Z$ be any set.  A clustering of $Z$ is a collection of disjoint, non-empty subsets of $Z$, called clusters.  The clusters need not cover $Z$, and, in particular, a clustering may be empty. Let $C(Z)$ be the set of clusterings of $Z$ with the partial order where $A \leq B$ if each cluster in $A$ is contained in a (unique) cluster of $B$.

We will consider families of clusterings indexed by partially ordered sets of a particular form.

Take $\vec{v} \in \{-1, 1\}^{n}$ for some $n \geq 1$ and let $R^{\vec{v}}$ be the product poset
\[R_{v_1} \times R_{v_2}  \times \cdots \times R_{v_n}\]
where $R_1 = \RR$ and $R_{-1}= \RR^{op}$.
A $\vec{v}$-hierarchical clustering  of $Z$ is an order preserving map

\[ H : R^{\vec{v}} \to C(Z), \]

such that $H(\vec{s})=\emptyset$ for some $\vec{s} \in R^{\vec{v}}$.

Equivalently, if the posets $\RR^n$ and $C(Z)$ are regarded as categories,  $H$ is a functor from $\RR^n$ to $C(Z)$ that is covariant in the $i^{th}$ parameter if $v_i=1$ and contravariant in the $i^{th}$ parameter if $v_i=-1$.

This is slightly different from the original definition of a $\vec{v}$-hierarchical clustering in <cit.>, where the domain of each parameter is $\RR_{>0}$, not $\RR$.  Since we require $H(\vec{s})=\emptyset$ for some $\vec{s}$ (and hence $H(\vec{t})=\emptyset$ for all $\vec{t} \leq {\vec{s}}$), these definitions are equivalent up to a shift in parameters.

For any $\vec{v}$-hierarchical clustering, $H$, we define a poset, $\Gamma(H)$, as in <cit.>.
The elements of $\Gamma(H)$ are all pairs $(\vec{s}, S)$ with $\vec{s} \in R^{\vec{v}}$ and $S$ a cluster in $H(\vec{s})$.
There is a comparison $(\vec{s}, S) \leq (\vec{t}, T)$ if $\vec{s} \leq \vec{t}$ and $S \subseteq T$.
This is the restriction of the product partial order on $R^{\vec{v}} \times \mathcal{P}(Z)$.

Let $\sim$ be the equivalence relation on $\Gamma(H)$ generated by the relation $(\vec{s}, S) \sim (\vec{t}, T)$ if $\vec{s} \leq \vec{t}$ and $S=T$.
An equivalence class of $\sim$ is called a layer of $\Gamma(H)$.

Each layer
is of the form

\[ \gamma= \{ (\vec{s}, S) | \vec{s} \in P\} \]

for some $P \subseteq R^{\vec{v}}$.
We call the set $P \subseteq R^{\vec{v}}$ the support of $\gamma$ and may write $\gamma=(P, S)$.

Supports of layers must satisfy the following property.

If $P \subseteq R^{\vec{v}}$ is the support of a layer for some $\vec{v}$-hierarchical clustering, then $P$ is equal to the intersection of the up-set of $P$,

\[ U_P := \bigcup_{\vec{p} \in P} \{ \vec{r} \in R^{\vec{v}} | \vec{r} \geq \vec{p} \} \]

and the down-set of $P$,

\[ D_P := \bigcup_{\vec{p} \in P} \{ \vec{r} \in R^{\vec{v}} | \vec{r} \leq \vec{p} \}. \]

Suppose $(P, S)$ is a layer in $\Gamma(H)$ for some $\vec{v}$-hierarchical clustering, $H$.
By definition, $U_P$ and $D_P$ both contain all elements of $P$.   If $\vec{r} \in U_P \cap D_P$, then there exist $\vec{p}, \vec{q} \in P$ such that $\vec{p} \leq \vec{r} \leq \vec{q}$.
Consequently, $H(\vec{p}) \leq H(\vec{r}) \leq H(\vec{q})$. Since $S \in H(\vec{p})$, this means there exist unique clusters $S' \in H(\vec{r})$ and $S'' \in H(\vec{q})$ satisfying

\[ S \subseteq S' \subseteq S'' \]

However, $S \in H(\vec{q})$, so $S''$ must be equal to $S$, and therefore $S'$ is equal $S$ as well.  This proves $(\vec{r}, S) \in \Gamma(H)$ exists and $(\vec{p}, S) \sim (\vec{r}, S)$.  So, $(\vec{r},S) \in (P,S)$ and hence $\vec{r} \in P$.

When the clustering has a single parameter, the supports of layers are all intervals.
A non-example and an example of sets that could potentially support a layer of a two-parameter clustering are depicted in Figure <ref>.

[gray!20] (-6,1) circle (1cm);
[gray!20] (1,1) arc
start angle=0,
end angle=90,
x radius=1cm,
y radius =1cm
] ;
[gray!20] (0,2) arc
start angle=180,
end angle=270,
x radius=1cm,
y radius =1cm
] ;
[gray!20] (2,-1)–(2,0)–(1,0)–(3,0)–(3,-.5)–(4,-0.5)–(4,-1);
[dashed] (1,1) arc
start angle=0,
end angle=90,
x radius=1cm,
y radius =1cm
] ;
(0,2) arc
start angle=180,
end angle=270,
x radius=1cm,
y radius =1cm
] ;
[below] at (4,-1.5)$R_1$;
[left] at (-1,3)$R_2$;
at (0.5,1.5)$P_0$;
at (2.5,-.5)$P_1$;
[below] at (-4,-1.5)$R_1$;
[left] at (-8,3)$R_2$;
[gray] (-5,1) arc
start angle=0,
end angle=90,
x radius=1cm,
y radius =1cm
] ;
[gray] (-5,1)–(-5,-1.7);
[gray] (-6,2)–(-8.2,2);
[left, gray] at (-8.2,2) $\partial D_Q$;
(-7,1) arc
start angle=180,
end angle=270,
x radius=1cm,
y radius =1cm
] ;
(-7, 1)–(-7,3);
[above] at (-7,3) $\partial U_Q$;
at (-6, 1) $Q$;
[white] (2,-1) circle (.09cm);
(2,-1) circle (.09cm);

The disk $Q$ cannot support a layer, as $U_Q \cap D_Q$ contains points outside of $Q$.
The shaded sets in the right diagram, $P_0$ and $P_1$, each satisfy the requirements to support a layer.

Note that there may be layers of the form $(P, S)$ and $(Q, S)$ for some distinct $P$ and $Q$.  In this case, any $p \in P$ and $q \in Q$ are incomparable.   (Otherwise, $(\vec{p}, S)$ and $(\vec{q}, S)$ would be in the same equivalence class.) This implies the following corollary of Proposition <ref>.

Let $H$ be a $\vec{v}$-hierarchical clustering of $Z$.  For any $S \subseteq Z$, let

\[ Supp(S) = \{ \vec{s} \in R^{\vec{v}} | S \in H(\vec{s}) \}. \]

Then $Supp(S) = U_{Supp(S)} \cap D_{Supp(S)}$.

The set $Supp(S)$ is the union of a family of disjoint sets $\{P_i\}_{i \in I}$ where each $P_i$ is the support of a distinct layer, $(P_i, S)$. For any $\vec{r} \in U_{P_i} \cap D_{P_j}$, there exist $\vec{p} \in P_i$ and $\vec{q} \in P_j$ such that $\vec{p} \leq \vec{r} \leq \vec{q}$.  If $i \neq j$, no $\vec{p} \in P_i$ and $\vec{q} \in P_j$ are comparable, so $U_{P_i} \cap D_{P_j} $ is empty.  Hence,

\[ Supp(S) = \bigcup_{i \in I} P_i = \bigcup_{i \in I} (U_{P_i} \cap D_{P_i}) =  \left(\bigcup_{i \in I} U_{P_i} \right) \cap  \left(\bigcup_{j \in I} D_{P_j} \right) = U_{Supp(S)} \cap D_{Supp(S)}. \]

In Figure <ref>, $P_0$ and $P_1$ could potentially support layers associated to the same cluster.
Not every $P \subseteq R^{\vec{v}}$ satisfying $P = U_P \cap D_P$ will necessarily be equal to $Supp(S)$ for some cluster, $S$, of an arbitrary $\vec{v}$-hierarchical clustering.  However, for any such $P$, if $Z$ has a proper subset $W$, then there exists at least one $\vec{v}$-hierarchical clustering of $Z$ where $P=Supp(W)$.
One such hierarchical clustering is given by

\[ H(\vec{s}) =
\begin{cases}
\emptyset &= \vec{s} \in R^{\vec{v}} \setminus U_P \\
\big\{W\big\} &=  \vec{s} \in P \\
\big\{Z\big\} &= \vec{s} \in U_P \setminus P. \\
\end{cases}\]

From here on, we restrict to $\vec{v}$-hierarchical clusterings satisfying the following condition.

Consider $R^{\vec{v}}$ as the topological space $\RR^n$, and let $\partial X$ denote the boundary of $X \subseteq R^{\vec{v}}$.
We define the lower boundary of $X$ to be

\[ D_X \cap \partial X. \]

A set $X \subseteq R^{\vec{v}}$ is closed below if the lower boundary of $X$ is contained in $X$.

A $\vec{v}$-hierarchical clustering, $H$, is closed below if the support of every layer of $\Gamma(H)$ is closed below.

For example, in Figure <ref>, $P_0$ is closed below, but $P_1$ is not.

Restricting to clusterings that are closed below will be used to guarantee the existence of minimal points within a layer.  These minima give the locations of layer points.

Let $H$ be a $\vec{v}$-hierarchical clustering of a set $Z$.  An element $(\vec{s}, S)$ of the poset $\Gamma(H)$ is a (global) layer point if
it satisfies one of the following conditions:

* there is no $(\vec{t}, T) \in \Gamma(H)$ with $(\vec{t}, T) < (\vec{s}, S)$, or

* for all $(\vec{t}, T) \in \Gamma(H)$ with $(\vec{t}, T) < (\vec{s}, S)$, $|T| < |S|$.

The set of global layer points in $\Gamma(H)$ is denoted $\Lambda(H)$.

If $(\vec{t}, T) \leq (\vec{s}, S)$, then $T \subseteq S$. In the second case, the condition $|T|<|S|$ is therefore equivalent to $T \neq S$.
So, the layer points are precisely the points in $\Gamma(H)$ where a cluster changes.  The poset $\Gamma(H)$, or equivalently the graph it defines, thus provides a condensed version of the information needed to define the $\vec{v}$-hierarchical clustering, $H$.

A layer point can also be described as an element $(\vec{s}, S) \in (P,S)\subseteq \Gamma(H)$ where $D_{\{\vec{s}\}} \cap P = \{\vec{s}\}$.  For every layer point $(\vec{s}, S)$, $\vec{s}$ lies on the lower boundary of the support of the associated layer.

Suppose there is a layer, $(P, S)$ in $\Gamma(H)$ where $P$ is the shaded region in
Figure <ref>.  An element $(\vec{s}, S) \in (P,S)$ is a layer point if $\vec{s}$ is $\vec{s}_2$ or any point on the portion of the lower boundary of $P$ between $\vec{s}_0$ and $\vec{s_1}$ (shown in bold).

[gray!20] (0.7,2.5) arc
start angle=180,
end angle=270,
x radius=1cm,
y radius =1cm
] ;
[gray!20] (0.7,3)–(0.7, 2.5)–(1.7, 1.5)–(2.5, 1.5)–(2.5,0.5)–(3,0.5)–(3,3);
[<->] (-0.2,0)–(3,0);
[below] at (3,0) $R_1$;
[<->] (0, -0.2)–(0,3);
[left] at (0,3) $R_2$;
at (2,2.2) $P$;
[very thick] (0.7,2.5) arc
start angle=180,
end angle=270,
x radius=1cm,
y radius =1cm
] ;
(0.7, 2.5)–(0.7, 3);
(2.5,0.5) circle (2pt);
(1.7,1.5) circle (2pt);
(0.7, 2.5) circle (2pt);
[left] at (0.7, 2.5) $\vec{s}_0$;
[below] at (1.7,1.5) $\vec{s}_1$;
[left] at (2.5,0.5) $\vec{s}_2$;

The shaded area, $P$, supports a layer, $(P, S)$. The bold points indicate the locations of global layer points associated to $S$.

An antichain is a subset of a poset that contains no pairs of comparable elements.
By definition, any two global layer points associated to the same cluster are incomparable.  So, the set of global layer points in a single layer form an antichain in $\Gamma(H)$.
This defines a map from $\Gamma(H)$ to the collection of antichains in $\Gamma(H)$ that takes $(\vec{s}, S)$ to the set of global layer points in the layer containing $(\vec{s}, S)$.
If any one of these global layer points is comparable to $(\vec{s}, S)$, then it is a maximal layer point below $(\vec{s}, S)$.
That is, if $(\vec{r}, S)$ is a global layer point associated to $S$ and $(\vec{r}, S)$ is comparable to some $(\vec{s}, S)$, then $(\vec{r}, S) \leq (\vec{s}, S)$ and for any other layer point $(\vec{t}, T) \in \Lambda(H)$ comparable to $(\vec{s}, S)$, $(\vec{t}, T) \not > (\vec{r}, S)$.
In the following section, we will show that there is at least one global layer point comparable to $(\vec{s}, S)$.

For a single parameter clustering, the lower boundary of each layer is a point, so there is exactly one layer point corresponding to each layer.  Furthermore, this layer point is comparable to all points in the layer. Thus, there is a unique maximal layer point below any element of $\Gamma(H)$.   This defines a map

\[ m:\Gamma(H) \to \Lambda(H). \]

In Section <ref>, we will use this map to induce interleavings of layer points from interleavings of clusterings.

§.§ Slice Clusterings

Single parameter clusterings can be extracted from a given multi-parameter clustering by taking one-dimensional slices.
Recall that for any $\vec{v} \in \{-1,1\}^n$, $R^{\vec{v}} = \prod\limits_{i=1}^n R_{v_i}$ where
\[R_{v_i} = \begin{cases} \RR & v_i = 1 \\ \RR^{op} & v_i=-1. \end{cases}\]

For any $1 \leq i \leq n$, let $C_i = R_{v_1} \times \cdots \times R_{v_{i-1}} \times R_{v_{i+1}} \times \cdots \times R_n$.
Let $p_i: R^{\vec{v}} \to C_i$ be the projection map,
\begin{align*}
(r_1, \ldots, r_n) & \mapsto (r_1, \ldots, r_{i-1}, r_{i+1}, \ldots, r_n)
\end{align*}
and let $\phi_i: R_{v_i} \times C_i \to R^{\vec{v}}$ be the insertion map,
\begin{align*}
\big(t, (c_1, \ldots, c_{n-1})\big) & \mapsto (c_1, \ldots, c_{i-1}, t, c_{i}, \ldots, c_{n-1}).
\end{align*}

By construction, these maps satisfy the properties $\phi_i(r_i, p_i \vec{r}) = \vec{r}$ for all $\vec{r} \in R^{\vec{v}}$ and $p_i(\phi_i(t, \vec{c}))=\vec{c}$ for all $(t, \vec{c}) \in R_{v_i} \times C_i$.

For any $1 \leq i \leq n$ and any $\vec{w} \in C_i$ we define the $i^{th}$-parameter slice of $H$ at $\vec{w} \in C_i$ to be the hierarchical clustering

\[ H_{\vec{w}}^i : R_{v_i} \to C(Z) \]

that takes $s$ to $H(\phi_i(s, \vec{w}))$.
Take $\Gamma_{\vec{w}}^i(H)$ to be the subset of $\Gamma(H)$ consisting of all $(\vec{s}, S) \in \Gamma(H)$ with $p_i(\vec{s})=\vec{w}$.
There is an isomorphism of posets,
\[ \Phi: \Gamma_{\vec{w}}^i(H)  \to \Gamma(H_{\vec{w}}^i)\]

induced by the projection of $R^{\vec{v}}$ onto $R_{v_i}$.

We can consider the layer points of the slice clusterings within the larger context of the multiparameter clustering.

Let $H$ be a $\vec{v}$-hierarchical clustering of a set $Z$.  An element $(\vec{t}, T)$ of the poset $\Gamma(H)$ is a $i^{th}$-parameter layer point if it is the image of the inclusion

\[ \Lambda(H_{\vec{s}}^i) \to \Gamma(H_{\vec{s}}^i) \xrightarrow{\cong} \Gamma_{\vec{s}}^i(H) \to \Gamma(H)  \]

for some $\vec{s}$.  We denote the set of all $i^{th}$-parameter layer points by $\Lambda^i(H)$.

For any layer, $(P_i, S)$, in $\Gamma_{\vec{w}}^i(H)$, let $(P, S)$ be the layer associated to $S$ in $\Gamma(H)$.  Then $P_i = P \cap p_i^{-1}(\vec{w})$.  This means if  $(\vec{s}, S)$ is the (unique) $i^{th}$-parameter layer point associated to $S$ in $\Gamma_{\vec{w}}^i(H)$, then $\vec{s}$ lies on the lower boundary of $P$.

Figure <ref> gives an example of $i^{th}$-parameter layer points.  Comparing with Figure <ref>, we see that the global layer points associated to the layer supported by this region are the points that are both $1^{st}$- and $2^{nd}$-parameter layer points.  In fact, for any number of parameters, the global layer points can be found by taking the intersection of the $i^{th}$-parameter layer points for all $i$.

[gray!20] (0.7,2.5) arc
start angle=180,
end angle=270,
x radius=1cm,
y radius =1cm
] ;
[gray!20] (0.7,3)–(0.7, 2.5)–(1.7, 1.5)–(2.5, 1.5)–(2.5,0.5)–(3,0.5)–(3,3);
[<->] (-0.2,0)–(3,0);
[below] at (3,0) $R_1$;
[<->] (0, -0.2)–(0,3);
[left] at (0,3) $R_2$;
[very thick] (0.7,2.5) arc
start angle=180,
end angle=270,
x radius=1cm,
y radius =1cm
] ;
[very thick] (0.7, 2.5)–(0.7, 3);
[very thick] (2.5,1.5)–(2.5,0.5);
(0.7, 2.5)–(0.7, 3);
(2.5,0.5) circle (2pt);
[white] (2.5, 1.5) circle (2pt);
[thick] (2.5, 1.5) circle (2pt);
(1.7,1.5) circle (2pt);
[below] at (1.7,1.5) $\vec{s}_1$;
[left] at (2.5,0.5) $\vec{s}_2$;
at (2,2.2) $P$;

[gray!20] (0.7,2.5) arc
start angle=180,
end angle=270,
x radius=1cm,
y radius =1cm
] ;
[gray!20] (0.7,3)–(0.7, 2.5)–(1.7, 1.5)–(2.5, 1.5)–(2.5,0.5)–(3,0.5)–(3,3);
[<->] (-0.2,0)–(3,0);
[below] at (3,0) $R_1$;
[<->] (0, -0.2)–(0,3);
[left] at (0,3) $R_2$;
[very thick] (0.7,2.5) arc
start angle=180,
end angle=270,
x radius=1cm,
y radius =1cm
] ;
[very thick] (1.7, 1.5)–(2.5,1.5);
(0.7, 2.5)–(0.7, 3);
(2.5,0.5) circle (2pt);
[white] (2.5, 1.5) circle (2pt);
[thick] (2.5,1.5) circle (2pt);
(0.7, 2.5) circle (2pt);
[left] at (0.7, 2.5) $\vec{s}_0$;
[left] at (2.5,0.5) $\vec{s}_2$;
[very thick] (2.5, 0.5)–(3,0.5);
at (2,2.2) $P$;
The shaded area supports a layer, $(P, S)$. The bold points in the left diagram indicate the locations of 1st-parameter layer points associated to $S$.  The bold points in the right diagram indicate the locations of 2nd-parameter layer points associated to $S$.

An element $(\vec{s}, S) \in \Gamma(H)$ is a global layer point if and only if it is an $i^{th}$-parameter layer point for all $i$.

Suppose $(\vec{s}, S)$ is a global layer point in $\Gamma(H)$.  We will show $(s_i, S)$ is a layer point for the $i^{th}$-parameter slice of $H$ at $p_i \vec{s}$.
For any $(t, T) \leq (s_i, S)$ in $\Gamma(H_{p_i \vec{s}}^i)$,
\[ \phi_i (t, p_i\vec{s}) \leq \phi_i (s_i, p_i \vec{s}) = \vec{s} \]

and $T \in H_{p_i\vec{s}}^i (t) = H(\phi_i(t, p_i\vec{s}))$ with $T \subseteq S$, so

\[  (\phi_i (t, p_i\vec{s}), T) \leq (\vec{s}, S).  \]

If $t \neq s_i$, then $\phi_i (t, p_i\vec{s}) \neq \vec{s}$, so $|T| < |S|$.
Since this is true for any  $(t, T) < (s_i, S)$, the point $(s_i, S) \in \Gamma(H_{\vec{s}}^i)$ is a layer point.
The image of $(s_i, S)$ under the inclusion of $\Lambda(H_{p_i\vec{s}}^i)$ into $\Gamma(H)$ is $\left(\phi_i (s_i, p_i \vec{s}), S \right)=(\vec{s}, S)$, so $(\vec{s}, S)$ is an $i^{th}$-parameter layer point.

Now suppose $(\vec{s}, S)$ is an $i^{th}$-parameter layer point for all $i$.  For any $(\vec{t}, T) \leq (\vec{s}, S)$ with $\vec{t} \neq \vec{s}$, there is at least one $i$ such that $t_i < s_i$.
\[ \vec{t} \leq \phi_i(t_i, p_i\vec{s}) < \vec{s}. \]

Let $\overline{T}$ be the cluster in $H(\phi_i(t_i, p_i\vec{s}))$ containing $T$.  Then, in $\Gamma(H_{p_i \vec{s}}^i)$,

\[(t_i, \overline{T} ) < (s_i, S).\]

Since $(\vec{s}, S)$ is an $i^{th}$-parameter layer point, this implies $\lvert\overline{T}\rvert < |S|$, and consequently, $|T|<|S|$.  Hence, $(\vec{s}, S)$ is a global layer point.

Recall that the maximal global layer point below some $(\vec{s}, S) \in \Gamma(H)$ is not necessarily unique in the multiparameter case.
Instead, the global layer points in the layer associated to $S$ form an antichain in $\Gamma(H)$.
However, since the slice clusterings each have a single parameter, there is a unique maximal $i^{th}$-parameter layer point below $(\vec{s}, S)$.  This defines a map

\begin{align*}
m_i:\Gamma(H) &\to \Lambda^i(H)
\end{align*}

for each $i$ that takes $(\vec{s}, S)$ to the maximal $i^{th}$-parameter layer point below $(\vec{s}, S)$.

These maps to $i^{th}$-parameter layer points are compatible in that the maximal $j^{th}$-parameter layer point below an $i^{th}$-parameter layer point is also an $i^{th}$-parameter layer point.

If $(\vec{s}, S) \in \Lambda^i(H)$ then $m_j (\vec{s}, S) \in \Lambda^i(H)$ for all $1 \leq j \leq n$.

Let $(\vec{t}, S) := m_j(\vec{s}, S)$.
For any $\vec{w} \leq \vec{t}$ in $R^{\vec{v}}$ with $p_i \vec{w} = p_i \vec{t}$, we define $\vec{q}=\phi_i(w_i, p_i \vec{s})$.  (The positions of these four points in $R^{\vec{v}}$ are depicted in Figure <ref>.) Then we have

\begin{align*}
p_j \vec{q} &= p_j\vec{w} \\
q_j = s_j &\geq t_j = w_j \\
p_i\vec{q} &= p_i \vec{s} \\
q_i = w_i &\leq t_i = s_i.
\end{align*}

For any $W \in H(\vec{w})$ with $W \subseteq S$, let $Q$ be the cluster in $H(\vec{q})$ containing $W$.

Suppose $W = S$.  Then $S \subseteq Q$.  Since $(\vec{s}, S)$ is an $i^{th}$-parameter layer point, $p_i\vec{s}=p_i\vec{q}$, and $q_i \leq s_i$, this implies $(\vec{q}, Q) = (\vec{s}, S)$.  So, $w_i = q_i = s_i = t_i$.  Since $p_i \vec{w} = p_i \vec{t}$, $(\vec{w}, W)=(\vec{t}, S)$.

Hence, if $(\vec{w}, W) \in \Gamma_{p_i \vec{t}} \ (H)$ and $(\vec{w}, W) < (\vec{t}, S)$, then $W \neq S$.
That is, $m_j(\vec{s}, S)=(\vec{t}, S)$ is an $i^{th}$-parameter layer point.

[<->] (-0.4, 0)–(4,0);
[<->] (0, -0.4)–(0,4);
[below] at (4,0) $R_{v_i}$;
[left] at (0,4) $R_{v_j}$;
(1,1) circle (3pt);
[below left] at (1,1) $\vec{w}$;
(3,1) circle (3pt);
[below right] at (3,1) $\vec{t}$;
(1,3) circle (3pt);
[above left] at (1,3) $\vec{q}$;
(3,3) circle (3pt);
[above right] at (3,3) $\vec{s}$;
(-0.2, 1)–(.2,1);
[left] at (-.2, 1) $t_j$;
(-0.2, 3)–(.2,3);
[left] at (-.2, 3) $s_j$;
[below] at (1,-.2) $w_i$;
[below] at (3,-.2) $s_i$;

The points $\vec{s}$, $\vec{t}$, $\vec{w}$, and $\vec{q}$ form a square in the 2-parameter slice of $R^{\vec{v}}$, $R_{v_i} \times R_{{v_j}}$.

Composing the maps $m_i$ for all $i$ therefore gives a retract

\[ m: \Gamma(H) \to \Lambda(H). \]

However, the composition of a given $m_i$ and $m_j$ is not necessarily commutative, so this map depends on a choice of order.  Additionally, even when the order of the maps is fixed, the global layer point selected is not necessarily the same for two elements in the same layer.

§.§ Branch Points

For a single parameter clustering, the layer set contains the branch points, as defined in <cit.>.  Heuristically, branch points indicate where clusters originate or combine, but not where clusters grow in size without merging with another cluster.
The branch points can been seen as indicating the births and deaths of persistent clusters (where persistent clusters are defined as in <cit.>).

Let $H$ be a single parameter hierarchical clustering of a set $Z$.  An element $(s, S)$ of the poset $\Gamma(H)$ is a branch point if it satisfies one of the following conditions:

* there is no $(t, T) \in \Gamma(H)$ with $(t, T) < (s, S)$, or

* there exist distinct $(t, T_0)$ and $(t, T_1)$ in $\Gamma(H)$ such that $(t, T_0) \leq (r, R)$ and $(t, T_1) \leq (r, R)$ if and only if $(r, R) \geq (s, S)$.

For a $\vec{v}$-hierarchical clustering, $H$, $i^{th}$-parameter branch points are defined as the branch points of the slice hierarchical clusterings, in the same way as $i^{th}$-parameter layer points were defined as the layer points of the slice clusterings.  These appear in <cit.> as vertical and horizontal branch points.

Unlike with layer points, an element of $\Gamma(H)$ does not need to be an $i^{th}$ parameter branch point for all $i$ to satisfy a generalized version of Definition 6.
This results in some ambiguity with regard to what a global branch point should be, but we propose the following definition.

An element of $\Gamma(H)$ is a global branch point if it is an $i^{th}$-parameter branch point for all $i$.

Unfortunately, the analogue of Lemma <ref> does not hold for $i^{th}$-parameter branch points, so composing maps to slice branch points will not necessarily yield a global branch point. We do not focus on the theory of branch points here, but we will utilize them in Section <ref> where the layer points of a particular hierarchical clustering are all branch points.

§.§ Summary

We summarize the relationships between the various types of layer and branch points in the following commuative diagram of inclusions, which exists for for all $1 \leq i \leq n$ and all $\vec{s} \in C_i$.

\[ \begin{tikzcd}[column sep=1.5em,/tikz/column 2/.style={column sep=0.5em}, /tikz/column 4/.style={column sep=0.5em}]
Br_{\vec{s}}^i(H) \ar[r] \ar[d] & \displaystyle\bigcup_{\vec{w} \in C_i} Br_{\vec{w}}^i(H) \ar[r, equal] & Br^i(H) \ar[d] & \displaystyle\bigcap_{1 \leq i \leq n} Br^i(H) \ar[r, equal] \ar[l]  & Br(H) \ar[d]
\\
\Lambda_{\vec{s}}^i(H) \ar[r] \ar[d] & \displaystyle\bigcup_{\vec{w} \in C_i} \Lambda_{\vec{w}}^i(H) \ar[r, equal] & \Lambda^i(H) \ar[d]  & \displaystyle\bigcap_{1 \leq i \leq n} \Lambda^i(H) \ar[r, equal] \ar[l] & \Lambda(H)  \\
\Gamma_{\vec{s}}^i(H) \ar[r]  & \displaystyle\bigcup_{\vec{w} \in C_i} \Gamma_{\vec{w}}^i(H) \ar[r, equal] &  \Gamma(H)
\end{tikzcd}\]

§.§ Interleavings

Suppose $Z$ is a set, $f$ is a self-map on $Z$, and $H$ and $E$ are $\vec{v}$-hierarchical clusterings of $Z$.
As in <cit.>, we put $\vec{s}+\vec{v}\vec{\epsilon}$ to mean $(s_1+v_1 \epsilon_1, \ldots, s_n + v_n\epsilon_n)$ for any $\vec{\epsilon} \in \RR_{\geq 0}^n$.
If, there exists, for all $S \in H(\vec{s})$, some $T \in E(\vec{s}+\vec{v}\vec{\epsilon})$ such that $f(S) \subseteq T$, then there exists an induced map
\[f_{\scaleto{\vec{\epsilon}}{7pt}}:H(\vec{s}) \to E(\vec{s}+\vec{v}\vec{\epsilon})\]
that takes a cluster $S$ to the cluster containing $f(S)$, which we denote by $\overline{f(S)}$.  When $f_{\scaleto{\vec{\epsilon}}{7pt}}$ exists, it is natural in $\vec{s}$.

We say there is an $(\vec{\epsilon}, \vec{\delta})$-interleaving of the pair $(H, E)$ given by $(f,g)$ if, for every $\vec{s} \in R^{\vec{v}}$, all maps in the following diagrams exist and the diagrams commute for all $\vec{s}$:

\[\adjustbox{scale=0.99}{\begin{tikzcd}[column sep=tiny, scale=0.5]
H(\vec{s}) \ar[rr, "{(id_Y)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}}"] \ar[dr, "{f_{\scaleto{\vec{\epsilon}}{6pt}}}", swap]
& & H(\vec{s}+\vec{v}(\vec{\epsilon}+\vec{\delta}))   \\
& E(\vec{s}+\vec{v}\vec{\epsilon}) \ar[ur,"{g_{\scaleto{\vec{\delta}}{7pt}}}", swap]  &
\end{tikzcd}  \begin{tikzcd}[column sep=tiny, scale=0.5]
& H(\vec{s}+\vec{v}\vec{\delta}) \ar[dr, "{f_{\scaleto{\vec{\epsilon}}{6pt}}}"] & \\
E(\vec{s}) \ar[ur,"{g_{\scaleto{\vec{\delta}}{7pt}}}"] \ar[rr, "{(id_Y)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}}", swap] & & E(\vec{s}+\vec{v}(\vec{\epsilon}+{\delta})).
\end{tikzcd}}\]

This is a special case of the interleavings of generalized persistence modules in <cit.>.
In the case where $\vec{\epsilon}=\vec{\delta}$ and $f=g=id_Y$, this coincides with the notion of an $\vec{\epsilon}$-interleaving in <cit.>.

An $(\vec{\epsilon}, \vec{\delta})$-interleaving of $(H, E)$ induces a commutative diagram of posets,

\[\begin{tikzcd}[column sep=large]
\Gamma(H) \ar[d, "{f_{\scaleto{\vec{\epsilon}}{6pt}}}", swap] \ar[r, "{(id_Z)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}}"] & \Gamma(H)\ar[d, "{f_{\scaleto{\vec{\epsilon}}{6pt}}}"] \\
\Gamma(E) \ar[r, "{( id_Z)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}}", swap] \ar[ur, "{g_{\scaleto{\vec{\delta}}{7pt}}}"] & \Gamma(E)
\end{tikzcd}\]

\begin{align*}
f_{\scaleto{\vec{\epsilon}}{6pt}}(\vec{s}, S) &= \left( \vec{s}+\vec{v}\vec{\epsilon}, \ \overline{f(S)}\right) \\
g_{\scaleto{\vec{\delta}}{7pt}}(\vec{s}, S) &= \left( \vec{s}+\vec{v}\vec{\delta}, \ \overline{g(S)}\right) \\
(id_Z)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}(\vec{s}, S) &= \left(\vec{s}+\vec{v}\left(\vec{\epsilon} + \vec{\delta}\right), \ \overline{S}\right).
\end{align*}

In general, interleavings of $(H,E)$ do not induce interleavings of the slice clusterings, $(H_{\vec{s}}^i, E_{\vec{s}}^i)$, since the maps in the interleaving may move between slices.
However, if $\vec{\epsilon}=\phi_i(\epsilon, \vec{0})$ and $\vec{\delta}=\phi_i(\delta, \vec{0})$ for some $i$, then the maps restrict to $i^{th}$-directional slices.  In that case, there is an $(\vec{\epsilon}, \vec{\delta})$-interleaving of $(H, E)$ given by $(f,g)$ if and only if there is an $(\epsilon, \delta)$-interleaving of $(H_{\vec{s}}^i, E_{\vec{s}}^i)$ given by $(f,g)$ for all $\vec{s} \in C_i$.

An interleaving of $\vec{v}$-hierarchical clusterings induces a homotopy interleaving of layer points.  In the multi-parameter case, this depends on a choice of maps $m_H$ and $m_E$.

An $(\vec{\epsilon}, \vec{\delta})$-interleaving of a pair of $\vec{v}$-hierarchical clusterings, $(H, E)$, given by $(f,g)$ induces a homotopy commutative diagram,

\[  \begin{tikzcd}[column sep = large]
\Lambda(H) \ar[d, "f_{\scaleto{\vec{\epsilon}}{6pt}}", swap] \ar[r, "(id_Z)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}"] & \Lambda(H)\ar[d, "f_{\scaleto{\vec{\epsilon}}{6pt}}"]\\
\Lambda(E) \ar[r, "(id_Z)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}", swap] \ar[ur, "g_{\scaleto{\vec{\delta}}{6pt}}"] & \Lambda(E).\end{tikzcd}\]

Recall there exists a (non-unique) map to maximal layer points,
\begin{align*}m_H:& \Gamma(H) \to \Lambda(H).\end{align*}

Let $i_H: \Lambda(H) \to \Gamma(H)$ be the inclusion.  Then

\begin{align*}
m_H \circ i_H = id_{\Lambda(H)} \text{ and,} \\
i_H \circ m_H \leq id_{\Lambda(H)}.
\end{align*}

The same properties hold for the retract $m_E$ and inclusion $i_E$.
An $(\vec{\epsilon}, \vec{\delta})$-interleaving of $(H, E)$ given by $(f,g)$ therefore defines a diagram

\[\begin{tikzcd}[column sep=large]
\Lambda(H) \ar[dr] & &  \Lambda(H) & \\
& \Gamma(H) \ar[d, "{f_{\scaleto{\vec{\epsilon}}{6pt}}}", swap] \ar[r, "{(id_Z)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}}"] & \Gamma(H)\ar[d, "{f_{\scaleto{\vec{\epsilon}}{6pt}}}"] \ar[u, "m_H", swap] & \Lambda(H) \ar[l, "i_H"] \ar[ul, equal] \\
\Lambda(E) & \Gamma(E) \ar[r, "{( id_Z)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}}", swap] \ar[ur, "{g_{\scaleto{\vec{\delta}}{7pt}}}"] \ar[l, "m_E"]& \Gamma(E) \ar[rd, "m_E"] &  \\
& \Lambda(E) \ar[u, "i_E"] \ar[ul, equal] &  & \Lambda(E)
\end{tikzcd}\]

where the center squares commutes and the outer triangles commute up to a homotopy of poset maps.
We collapse this diagram to obtain a homotopy commutative diagram,

\[  \begin{tikzcd}[column sep = large]
\Lambda(H) \ar[d, "f_{\scaleto{\vec{\epsilon}}{6pt}}", swap] \ar[r, "(id_Z)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}"] & \Lambda(H)\ar[d, "f_{\scaleto{\vec{\epsilon}}{6pt}}"] \\
\Lambda(E) \ar[r, "(id_Z)_{\scaleto{\vec{\epsilon}+\vec{\delta}}{9pt}}", swap] \ar[ur, "g_{\scaleto{\vec{\delta}}{6pt}}"] & \Lambda(E)
\end{tikzcd}\]

where, for simplicity, we omit the compositions with $i$ and $m$ in the map notation. (So, for example we put $f_{\scaleto{\vec{\epsilon}}{7pt}}$ rather than $m_E \circ f_{\scaleto{\vec{\epsilon}}{7pt}} \circ i_H$.)

For branch points, there is a unique maximal branch point below any element of $\Gamma(H)$ when $H$ has a single parameter.  So, an interleaving of single parameter hierarchical clusterings also induces a homotopy interleaving of branch points.
However, in the multi-parameter case, the situation is even worse than it is for layer points, since we have not defined a map from $\Gamma(H)$ to the branch points of $H$.
In the remaining sections, we primarily focus on interleavings of layer points for single parameter hierarchical clusterings.

§.§ Approximations of Clusterings

Let $X \subseteq Y$ be finite metric spaces.  The directional Hausdorff distance, $h$, from $Y$ to $X$ is given by

\[  h(Y, X) = \sup_{y \in Y} \  \inf_{x \in X} d(y, x). \]

Let $H$ be a $\vec{v}$-hierarchical clustering of $X$ and let $E$ be a $\vec{v}$-hierarchical clustering of $Y$. Since $X \subseteq Y$, $H$ is also a $\vec{v}$-hierarchical clustering of $Y$.

We say $(X, H)$ is an $(\vec{\epsilon}, \vec{\delta})$-approximation of $(Y, E)$
if there exists an $(\vec{\epsilon}, \vec{\delta})$-interleaving of the pair $(H, E)$ given by $(i, \theta)$ where $i$ is the inclusion of $X$ into $Y$ and $\theta: Y \to X$ is some map satisfying $d(y, \theta(y)) \leq h(Y,X)$.

Note that the existence of $i_{\vec{\epsilon}}$ in the $(\vec{\epsilon}, \vec{\delta})$-interleaving of $(H,E)$ is equivalent to requiring that each cluster in $H(\vec{s})$ is contained in a cluster of $E(\vec{s}+\vec{\epsilon})$.
An $(\vec{\epsilon}, \vec{\delta})$-approximation of $(Y, E)$ by $(X, H)$ gives a homotopy commutative interleaving diagram of layer points,

\[\begin{tikzcd}[column sep = large]
\Lambda(H) \ar[r, "(id_Y)_{\scaleto{\vec{\epsilon}+ \vec{\delta}}{8pt}}"] \ar[d, "i_{\scaleto{\vec{\epsilon}}{6pt}}", swap] & \Lambda(H) \ar[d, "i_{\scaleto{\vec{\epsilon}}{6pt}}"] \\
\Lambda(E) \ar[r, "(id_Y)_{\scaleto{\vec{\epsilon}+ \vec{\delta}}{8pt}}", swap] \ar[ur, "\theta_{\scaleto{\vec{\delta}}{7pt}}"] & \Lambda(E)
\end{tikzcd}\]

as in the previous section. This relies on the fact that $H$ and $E$ are both $\vec{v}$-clusterings of $Y$.

§ LAYER STABILITY FOR LESNICK CLUSTERINGS

§.§ Approximating Lesnick Clusterings
In the remaining sections, we focus on the $\vec{v}$-hierarchical clusterings given by taking path components of Lesnick complexes, which are subcomplexes of the Vietoris-Rips complex.

The Vietoris-Rips complex, $V_s(Z)$, is the simplicial complex with $n$-simplices of the form $[z_0, z_1, \ldots, z_n]$ for any points $z_i \in Z$ with $d(z_i, z_j)$ for all $0 \leq i, j \leq n$.

An $s$-neighbour of a point $z \in Z$ is a point $x \in Z$ with $x \neq z$ and $d(x,z) \leq s$.  The Lesnick complexes, or degree Rips complexes, $L_{s,k}(Z)$ filter the Vietoris-Rips complex, $V_s(Z)$, according to density, as measured by number of neighbours.
The Lesnick complex, $L_{s,k}(Z)$
has as $n$-simplices the simplices $[z_0, z_1, \ldots, z_n] \in \left(V_s(Z)\right)_n$ where each $z_i$ has at least $k$ distinct $s$-neighbours.

For all $s$, note $L_{s,0}(Z)=V_{s}(Z)$.

Let $ L:\RR \times \RR^{op} \to C(Z)$ be the map

\[ L(s,t) =  \begin{cases}  \pi_0 L_{s, \lceil t\rceil }(Z) & (s,t) \in [0, \infty) \times [0, \infty)^{op} \\ \emptyset & \text{otherwise.}   \end{cases} \]

We choose to take $\lceil t \rceil$ to obtain an integer number of neighbours rather than $\lfloor t \rfloor$ in order to ensure that that $L$ is closed below.
The second parameter slice $L_k^2$ will be denoted hereafter by $L_k Z$.

We note that each cluster $S$ in $L_k Z(s)$ for some $s \in \RR_{\geq 0}$ is the path component of some $z \in \big(L_{s,k}(Z)\big)_0 \subseteq Z$.  For any $(t, T) \in \Gamma(Z; L_k Z)$ with $(s, S) \leq (t, T)$, $T$ is the path component of $z$ in $L_{t,k}(Z)\supseteq L_{s,k}(Z)$.  We put $S= [z]_s$ and $T= [z]_t$, unless the location of $[z]$ is clear from context.

As in <cit.>, we use the following notation for the $n^{th}$ unordered configuration space of $Y$.

For any positive integer $n$, let $Y_{dis}^n$ be the set of all subsets of $Y$ consisting of exactly $n$ distinct elements.

Using $Y_{dis}^{k+1}$, we can describe the density of $Y$ around the points of $X$.

For any nonnegative integer, $k$, let

\[ N_k(X, Y) = \sup_{x \in X} \  \inf_{S \in Y_{dis}^{k+1}} \ \sup_{y \in S} \ d(x,y). \]

Then $N_k(X,Y) \leq p$ if each $x \in X$ has at least $k$ distinct $p$-neighbours in $Y$.
If $N_k(X, Y) \leq p$ then $N_\ell (X, Y) \leq q$ for all $\ell \leq k$ and $q \geq p$.

Let $L_0 X[c]$ be the hierarchical clustering of $X$ given by

\[ L_0 X[c] (s) = \begin{cases}
\pi_0 L_{s,0}(X) & s \geq c \\ \emptyset & \text{otherwise}
\end{cases} = \begin{cases}
\pi_0 V_s (X) & s \geq c \\ \emptyset & \text{otherwise.}
\end{cases}\]

For any $c \geq 0$, the poset, $\Gamma(L_0 X[c])$ is a subset of $\Gamma(\L_0 X)$, although $\Lambda(L_0 X[c])$ is not necessarily a subset of $\Lambda(L_0 X)$.  All layer points of $L_0 X = \pi_0 V_\bullet (X)$ are branch points and $L_0 X[c]$ inherits this property.

This truncated clustering can be used, under certain conditions, to approximate the clustering $L_k Y$.
Suppose $c \geq 0$. Then $(X, L_0 X[c])$ is an $(\epsilon, \delta)$-approximation for $(Y, L_k Y)$ for all $\epsilon, \delta \geq 0$ satisfying

\[N_k(X, Y) - \epsilon \leq c \leq \delta,\]
\[\delta \geq 2h(Y,X).\]

Let $s \geq c$. Since
\[N_k(X, Y) \leq c+\epsilon \leq s+\epsilon,\]
each $x \in X$ has at least $k$ $(s+\epsilon)$-neighbours in $Y$.  So, for any $[x]_s$ in $L_0 X[c](s)$, the cluster $[x]_{s+\epsilon}$ exists in $L_k Y(s+\epsilon)$. Since $s+\epsilon \geq s$, $[x]_{s+\epsilon}$ must contain $i([x]_s)$.

Fix any map $\theta:Y \to X$ with $d\big(y, \theta(y)\big) \leq h(Y,X)$ for all $y \in Y$.  For convenience, we will assume $\theta(x)=x$ for all $x \in X$, but the result holds even if this assumption is dropped. For any $[y]_s \in L_k Y(s)$ and any $y_0 \in [y]_{s}$, there exists a sequence of points in $Y$,

\[ y_0, y_1, y_2, \ldots, y_{n-1}, y_n=y \]

satisfying $d(y_i, y_{i+1}) \leq s$ for all $0 \leq i < n$.  For any such $i$,

\begin{align*} d\big(\theta(y_i), \theta(y_{i+1}) \big) &\leq d(\theta(y_i), y_i) + d(y_i, y_{i+1}) + d(y_{i+1}, \theta(y_{i+1})) \\ &\leq s+2h(Y,X) \\ &\leq s+\delta. \end{align*}

So, $\theta([y]_s)$ is contained in the cluster, $[\theta(y)]_{s+\delta}$ in $L_k X(s+\delta)$.

Thus $i_{\epsilon}$ and $\theta_\delta$ are well defined in the diagrams

\[\adjustbox{scale=0.9}{\begin{tikzcd}[column sep=tiny, scale=0.5, every label/.append style = {font = \normalsize}]
L_0 X[c](s) \ar[rr, "(id_Y)_{\scaleto{{\epsilon}+\delta}{7pt}}"] \ar[dr, "i_{\scaleto{\epsilon}{5pt}}", swap]
& & L_0 X[c](s+\epsilon+\delta)   \\
& L_k Y(s+\epsilon) \ar[ur,"\theta_{\scaleto{\delta}{6pt}}", swap]  &
\end{tikzcd}  \begin{tikzcd}[column sep=tiny, scale=0.5, every label/.append style = {font = \normalsize}]
& L_0 X[c](s+\delta) \ar[dr, "i_{\scaleto{\epsilon}{5pt}}"] & \\
L_k Y(s) \ar[ur,"\theta_{\scaleto{\delta}{6pt}}"] \ar[rr, swap,  "(id_Y)_{\scaleto{\epsilon+\delta}{7pt}}"] & & L_k Y(s+\epsilon+\delta).
\end{tikzcd}}\]

for all $s$.
It remains to show the diagrams commute.

For any $[y]_{s} \in L_k Y(s)$,

\[ (id_Y)_{\epsilon+\delta}\left([y]_{s+\epsilon}\right) = [y]_{s+2\epsilon+\delta} = [\theta(y)]_{s+2\epsilon+\delta} = i_\epsilon \theta_\delta [y]_{s+\epsilon},  \]

since $d(y,\theta(y)) \leq h(Y,X) \leq \frac{1}{2} \delta \leq s+\epsilon+\delta$.

\[ (id_Y)_{\scaleto{\epsilon+\delta}{6pt}} \left([x]_s \right)= [x]_{s+\epsilon+\delta} = [\theta(x)]_{s+\epsilon+\delta} = \theta_\delta i_\epsilon [x]_s  \]

for all $[x]_s \in L_0 X [c](s)$.

There is therefore a homotopy commutative diagram

\[\begin{tikzcd}[column sep=large]
\Lambda(L_0 X[c]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"] \ar[r, "(id_Y)_{\scaleto{\epsilon+\delta}{6pt}}"]& \Lambda(L_0 X[c]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"]\\
\Lambda(L_k Y) \ar[r, "(id_Y)_{\scaleto{\epsilon +\delta}{6pt}}", swap] \ar[ur, "\theta_{\scaleto{\delta}{5pt}}"] & \Lambda(L_k Y).
\end{tikzcd}\]

For all ${\epsilon}, {\delta} \geq 0$ satisfying $N_k (X, X) - \epsilon \leq c \leq \delta$, $(X, L_0 X [c])$ is an $(\epsilon, \delta)$-approximation for $(X, L_k X)$.  In particular, $(X, L_0 X [c])$ is an $\left(N_k(X,X), 0\right)$-approximation for $(X, L_k X)$.

This follows from Lemma <ref> by taking $Y=X$, which implies $h(Y,X)=0$.  Note that $N_k(X,X)$ gives the greatest lower bound on the distance parameter, $s$, for which the vertex set of $L_{k, s} X$ is equal to $X$.

§.§ Retracts of Lesnick Clusterings

As in <cit.>, we call the distances between elements of $X$ the phase change numbers of $X$.  Since $X$ is finite, there are a finite number of phase change numbers,

\[ s_0 < s_1 < s_2 < \cdots < s_U. \]

Fix some number $M$ such that $\pi_0 L_{s, 0}(X) = \{X\}$ for all $s \geq s_{M}$.  It always works to take $M = U$, but choosing a smaller value of $M$, if possible, will give a less restrictive condition on $X$ in Theorem <ref>.

The distance parameter associated to any layer point of $(X ,L_0 X[c])$ must be a phase change number and must be no larger than $s_M$.  Using this fact, we prove the following theorem.

Suppose $c, \epsilon, \delta$ are nonnegative numbers satisfying $\delta \geq 2h(Y,X)$ and $N_k(X,Y) - \epsilon \leq c \leq \delta$.  If
\[c+\epsilon+\delta< s_{i+1}-s_i\]
for all $0 \leq i < M$, then $\Lambda(L_0 X[c])$
is a retract of $\Lambda(L_k Y)$.

In fact, the condition $c+\epsilon+\delta < s_{i+1}-s_i$ can be replaced with the weaker condition that $s - t > \epsilon + \delta$ for all $(t, T) < (s, S) \in \Lambda(L_0 X[c])$.  However, checking the condition $s_{i+1} - s_i > c+\epsilon+\delta$ does not require computing the set of layer points.

To prove Theorem <ref>, we will show that in the diagram

\[\begin{tikzcd}[column sep=large]
\Lambda(L_0 X[c]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"] \ar[r, "(id_Y)_{\scaleto{\epsilon+\delta}{6pt}}"]& \Lambda(L_0 X[c]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"]\\
\Lambda(L_k Y) \ar[r, "(id_Y)_{\scaleto{\epsilon +\delta}{6pt}}", swap] \ar[ur, "\theta_{\scaleto{\delta}{5pt}}"] & \Lambda(L_k Y).
\end{tikzcd}\]

the composition $\theta_\delta i_\epsilon : \Lambda(L_0 X[c]) \to \Lambda(L_0 X[c])$ is the identity.  We begin by bounding the shift of parameter in the map $i_\epsilon$.

Let $(t, [x]):= i_\epsilon (s, [x])$ for some $(s, [x]) \in \Lambda(L_0 X[c])$ with $s>c$.  Then
\[s-2h(Y,X)\leq t \leq s+\epsilon.\]

Since $(t, [x])$ is a layer point below $(s+\epsilon,[x]) \in \Gamma(L_k Y)$, $t \leq s +\epsilon$.

To prove $s - 2h(Y,X) \leq t$, we use the fact that the layer points of $L_0 X = \pi_0 V_\bullet (X)$ are all branch points.  This is true for the truncated clustering as well, so for any layer point $(s, [x]) \in \Lambda(L_0 X[c])$, either $s=c$ or, for some $x_0, x_1 \in [x]_s$,  $[x_0]_{w} \neq [x_1]_{w}$ for all $w < s$.

We assume $s>c$, so there must exist $x_0, x_1 \in [x]_s$ satisfying the second condition.  Since $[x_0] =  [x_1]$ in $L_0 X[c](s) =\pi_0 L_{s,0}(X)=\pi_0 V_s (X)$ and $X \subseteq Y$, $[x_0] = [x_1]$ in $L_k Y(s) = \pi_0 L_{s,k}(Y)$ as well.

If $(t, [x_0]):= m(s, [x_0])$ in $\Lambda(L_k Y)$, then $|[x_0]_t|=|[x_0]_s|$.  Hence, $[x_0]=[x]=[x_1]$ in $L_k Y(t)$.
Then there exists a sequence of points in $Y$,

\[ x_0=y_0, y_1, \ldots, y_n = x_1 \]

where each $y_i$ has at least $k$ $t$-neighbours and $d(y_i, y_{i+1})\leq t$.

\[ d\big(\theta(y_i), \theta(y_{i+1})\big) \leq d\big(\theta(y_i), y_i\big) + d(y_i, y_{i+1}) + d\big( y_{i+1}, \theta(y_{i+1})\big) \leq t+2h(Y,X) \]

for all $i$. Since $\theta(y_0)=x_0$ and $\theta(y_n)=x_n$, this implies $x_0$ and $x_1$ are in the same path component
in $L_0 X[c]\big(t+2h(Y,X)\big)$.  Thus, $t+2h(Y,X) \geq s$.

This bound allows us to prove the main theorem.

Suppose $c, \epsilon, \delta$ are nonnegative numbers satisfying $\delta \geq 2h(Y,X)$, $N_k(X,Y) - \epsilon \leq c \leq \delta$, and
\[c+\epsilon+\delta< s_{i+1}-s_i\]
for all $0 \leq i < M$.
Consider the homotopy commutative diagram

\[\begin{tikzcd}[column sep=large]
\Lambda(L_0 X[c]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"] \ar[r, "(id_Y)_{\scaleto{\epsilon+\delta}{6pt}}"]& \Lambda(L_0 X[c]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"]\\
\Lambda(L_k Y) \ar[r, "(id_Y)_{\scaleto{\epsilon +\delta}{6pt}}", swap] \ar[ur, "\theta_{\scaleto{\delta}{5pt}}"] & \Lambda(L_k Y).
\end{tikzcd}\]

If $(s, [x]) \in \Lambda(L_0 X[c])$, then $s=c$ or $s$ is a phase change number of $X$.
The first phase change number, $s_0$, is zero, so,
\[s_1 = s_1-s_0 > c+\epsilon+\delta.\]

In the case where $s$ is a phase change number, we have for $(q, [x]) :=  \theta_\delta i_\epsilon (s, [x])$,
\[q \leq c+\epsilon+\delta.\]

\[ s_0=0 < c \leq q \leq c+\epsilon+\delta < s_1. \]

Since $q$ is therefore not a phase change number, $q = c$.

In the case where $s>c$, the distance from $s$ to the next phase change number is greater than $c+\epsilon+\delta$.
Lemma <ref> implies that, for $(t, [x]):= i_\epsilon (s,[x])$,
\[s-2h(Y,X) \leq t  \leq s+\epsilon. \]

Since $2h(Y,X) \leq \delta$,

\[ s \leq s+\delta-2h(Y,X) \leq t+\delta \leq s+\epsilon+\delta.  \]

Because $(q, [x]):= \theta_\delta i_\epsilon(s, [x])$ is the maximal branch point below $(t+\delta, [x])$ and $(s, [x])$ is a branch point,
\[s \leq q \leq s+\epsilon+\delta.\]

The distance from $s$ to the next phase change number is greater than $\epsilon+\delta$, hence $q=s$.

Under the assumptions of Proposition 15,
\[(id_Y)_{\epsilon+\delta}: \Lambda(L_0 X[c]) \to \Lambda(L_0 X[c])\]
is the identity. So in this case, the upper triangle in the diagram of layer points commutes on the nose, not just up to homotopy.

§ SPECIAL CASES OF LAYER STABILITY

In this section we consider several special cases of Theorem <ref>.  Suppose $\epsilon$, $\delta$, and $c$ are nonnegative numbers satisfying $N_k(X,Y)-\epsilon \leq c \leq \delta$ and $\delta \geq 2h(Y,X)$.  The map

\begin{align*}  i_\epsilon : \Lambda(L_0 X[c]) &\to \Lambda(L_k Y)
\end{align*}

exists as long as $\epsilon+c \geq N_k(X,Y)$.  So, in order to minimize the term $c+\epsilon+\delta$, we can start by recognizing that $\epsilon$ never needs to be greater than $N_k(X,Y)$ to define the interleaving of layer points.  For any fixed value $\epsilon$ between 0 and $N_k(X,Y)$, the minimal values for $c$ and $\delta$ are $N_k(X,Y)-\epsilon$ and $\max\{2h(Y,X), N_k(X,Y)-\epsilon\}$, respectively.
Choosing these values for $c$ and $\delta$ gives a homotopy commutative diagram for any $0 \leq \epsilon \leq N_k(X,Y)$,

\begin{align} \label{diagram}
\begin{tikzcd}[column sep=huge, ampersand replacement=\&]
\Lambda(L_0 X[N_k(X,Y)-\epsilon]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"] \ar[r, "(id_Y)_{\scaleto{\epsilon+n}{6pt}}"] \& \Lambda(L_0 X[N_k(X,Y)-\epsilon]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"]\\
\Lambda(L_k Y) \ar[r, "(id_Y)_{\scaleto{\epsilon +n}{6pt}}", swap] \ar[ur, "\theta_{\scaleto{n}{3pt}}"] \& \Lambda(L_k Y)
\end{tikzcd}
\end{align}

where $n = \max\{ 2h(Y,X), N_k(X,Y)-\epsilon \}$.
\[N_k(X,Y)+n=\max\{ N_k(X,Y)+2h(Y,X), 2N_k(X,Y)-\epsilon \}< s_{i+1}-s_i\]
for all $0 \leq i < M$, then
\[\theta_n i_\epsilon = (id_Y)_{n+\epsilon} = id_{\Lambda(L_0 X[N_k(X,Y)-\epsilon])}.\]

Note that if $N_k(X,Y)+2h(Y,X) \geq s_{i+1}-s_i$ for some $i$, there is no choice of $\epsilon$ that will give direct commutativity in the upper triangle. However, if
\[N_k(X,Y)+2h(Y,X) < s_{i+1}-s_i\]
for all $i$ then any nonnegative $\epsilon$ between $N_k(X,Y)-2h(Y,X)$ and $N_k(X,Y)$ will guarantee direct commutativity.

Setting $\epsilon=N_k(X,Y)$ in (<ref>) gives the following result.

If $N_k(X,Y)+2h(Y,X) < s_{i+1}-s_i$ for all $0 \leq i < M$, then there is a diagram,

\[\begin{tikzcd}
\Lambda(L_0 X) \ar[d] \ar[r, "id"]& \Lambda(L_0 X) \ar[d]\\
\Lambda(L_k Y) \ar[r] \ar[ur] & \Lambda(L_k Y)
\end{tikzcd}\]

where the bottom triangle commutes up to homotopy and the top triangle commutes directly, defining a retract

\[ \Lambda(L_k Y) \to \Lambda(L_0 X). \]

We can also consider the case where $X=Y$, implying $h(Y, X)=0$.  If $N_k(X, X) \leq p$, then for any nonnegative $c,\epsilon,$ and $ \delta$ satisfying $N_k(X,Y)-\epsilon \leq c \leq \delta$, Corollary <ref> implies the existence of a homotopy commutative diagram of posets,

\[\begin{tikzcd}[column sep=large]
\Lambda(L_0 X[c]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"] \ar[r, "(id_X)_{\scaleto{\epsilon+\delta}{6pt}}"]& \Lambda(L_0 X[c]) \ar[d, "i_{\scaleto{\epsilon}{3.5pt}}"]\\
\Lambda(L_k X) \ar[r, "(id_X)_{\scaleto{\epsilon +\delta}{6pt}}", swap] \ar[ur, "\theta_{\scaleto{\delta}{5pt}}"] & \Lambda(L_k X).
\end{tikzcd}\]

However, the condition

\[ c+\epsilon+\delta < s_{i+1}-s_i \]

for all $i$, which would guarantee $\theta_\delta$ is a retract,
can only be met for all $0 \leq i < M$ if $k=0$, and thus $N_k(X,X)=0$.

To see that $c+\epsilon+\delta < s_{i+1}-s_i$ for all $i$ implies $k=0$, suppose $k>0$ and therefore $N_k(X,X)>0$.
Take any $x \in X$ and $z \neq x$, an $N_k(X,Y)$-neighbour of $x$.  Then $d(x,z) \leq N_k(X,Y)$.  Since $d(x, z)$ is a positive phase change number,

\[  s_1 - s_0 = s_1 - 0 \leq d(x, z) \leq N_k(X,Y). \]

\[ N_k(X,Y) \leq N_k(X,Y)+\delta \leq c+\epsilon+\delta, \]

so, $s_1 - s_0 \leq c+\epsilon+\delta$.

In the case where $k=0$, the only choice for $\epsilon$ in (<ref>) is $\epsilon=0=N_k(X,Y)$, which results in the rather unhelpful diagram

\[\begin{tikzcd}
\Lambda(L_0 X) \ar[d] \ar[r]& \Lambda(L_0 X) \ar[d, ]\\
\Lambda(L_0 X) \ar[r, ] \ar[ur, ] & \Lambda(L_0 X)
\end{tikzcd}\]

where all the maps are the identity.

A slightly more interesting example can be constructed for any $c$ which is less than $s_{i+1} - s_i$ for all $i$, by setting $\epsilon=0$ and $\delta=c$

For any nonnegative number, $c$, satisfying $c< s_{i+1}-s_i$ for all $0 \leq i < M$, there exists a commutative diagram,

\[\begin{tikzcd}\label{cor:X=Y_layers}
\Lambda(L_0 X[c]) \ar[d, "i_0"] \ar[r, "id"]& \Lambda(L_0 X[c]) \ar[d, "i_0"]\\
\Lambda(L_0 X) \ar[r, "id"] \ar[ur, "\theta_c"] & \Lambda(L_0 X).
\end{tikzcd}\]

In this case, both triangles commute directly, not just up to homotopy, so this gives an isomorphism $i_0: \Lambda(L_0 X[c]) \to \Lambda(L_0 X)$.
That is, truncating the poset $\Gamma(L_0 X)$ by a small amount does not change structure of the layer point poset.
This is unsurprising, and one can directly show the stronger result that there is
an isomorphism $\Lambda(L_0 X[c]) \to \Lambda(L_0 X)$ for any $c < s_1$.
This provides an alternate argument for Corollary <ref>, as in the interleaving diagram of Theorem <ref> the poset $\Lambda(L_0 X[c])$ can be replaced by $\Lambda(L_0 X)$ when $c< s_{i+1}-s_i$ for all $0 \leq i < M$.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

journaltitleFoundations of Computational Mathematics
titleMetrics for generalized persistence modules

titleData and homotopy types

titleBranch Points and Stability

journaltitleCanadian Mathematical Bulletin
titleStable Components and Layers

titleStable and Consistent Density-Based Clustering


#  Liouville theorem for a class semilinear elliptic problem on Heisenberg
group

Xi-Nan Ma  Department of Mathematics
University of Science and Technology of China
Hefei, 230026, Anhui Province, China<EMAIL_ADDRESS>and  Qianzhong Ou
School of Mathematics and Statistics
Guangxi Normal University
Guilin, 541004, Guangxi Province, China<EMAIL_ADDRESS>
###### Abstract.

We obtain an entire Liouville type theorem to the classical semilinear
subcritical elliptic equation on Heisenberg group. A pointwise estimate near
the isolated singularity was also proved. The soul of the proofs is an a
priori integral estimate, which deduced from a generalized formula of that
found by Jerison and Lee.

MSC 2020: Primary 35J61; Secondary 32V20.
Key Words: semilinear elliptic equation; Heisenberg group; CR Yamabe problem;
Liouville theorem.
The research of the first author was supported by NSFC 11871255 and NSFC
11721101. Research of the second author was supported by NSFC 11861016.

##  1\. Introduction

In this paper, we study the following equation

(1.1) $-\triangle_{\mathbb{H}^{n}}u=2n^{2}u^{q}\quad\text{in}\quad\Omega,$

where $\Omega$ is a domain in Heisenberg group $\mathbb{H}^{n}$, and $u$ is a
smooth, nonnegative real function defined in $\Omega$, while
$\triangle_{\mathbb{H}^{n}}u=u_{\alpha\overline{\alpha}}+u_{\overline{\alpha}\alpha}$
is the Heisenberg laplacian of $u$. Let $Q=2n+2$ be the homogeneous dimension
of $\mathbb{H}^{n}$. Denote $q_{*}=\frac{Q}{Q-2}$ and $q^{*}=\frac{Q+2}{Q-2}$.
We will deduce an entire Liouville type theorem and a point wise estimate near
the isolated singularity for solutions to (1.1). Precisely, we have

######  Theorem 1.1.

Let $\Omega=\mathbb{H}^{n}$ be the whole space and $1<q<q^{*}$, then the
equation (1.1) has no positive solution, namely, any nonnegative entire
solution of (1.1) must be the trivial one.

######  Theorem 1.2.

Let $\Omega=B_{1}(0)\backslash\\{0\\}$ be the punctured unit ball in
$\mathbb{H}^{n}$ and $1<q<q^{*}$, then any positive solution $u$ of (1.1)
satisfies:

(1.2) $u(\xi)\leq
C|\xi|^{\frac{-2}{q-1}}\quad\text{for}\,\,\xi\,\,\text{near}\,\,0,$

with some positive constant $C$ depending only on $n$ and $q$.

The soul of the proofs of theorem 1.1,1.2 is an integral estimate, which may
be interested itself. In fact we shall prove the following

######  Theorem 1.3.

Let $1<q<q^{*}$, $B_{4r}(\xi_{0})\subset\Omega$ be any ball centered at
$\xi_{0}$ with radio $4r$. Then any positive solution $u$ of (1.1) satisfies:

(1.3) $\int_{B_{r}(\xi_{0})}u^{3q-q^{*}}\leq
C\,r^{Q-2\times\frac{3q-q^{*}}{q-1}},$

with some positive constant $C$ depending only on $n$ and $q$.

For $1<q<q^{*}$, we see $Q-2\times\frac{3q-q^{*}}{q-1}<0$. So if $u$ be a
positive solution of (1.1) with $\Omega=\mathbb{H}^{n}$, taking
$r\rightarrow+\infty$ in (1.3) we have

(1.4) $\int_{\mathbb{H}^{n}}u^{3q-q^{*}}\leq 0.$

This contradiction signifies directly the conclusion of theorem 1.1. Also, we
will prove theorem 1.2 by using (1.3) combining with the Harnack inequality
deduced by Capogna-Danielli-Garofalo ( see Theorem 3.1 in [5]).

The equation (1.1) had been studied intensively by many authors in decades. In
fact, it comes from the CR Yamabe problem on $\mathbb{H}^{n}$. Let
$\mathbf{\Theta}$ be the standard contact form on $\mathbb{H}^{n}$. Consider
another smooth contact form $\theta=u^{\frac{2}{n}}\mathbf{\Theta}$. Then the
pseudo-Hermitian scalar curvature associated to the Fefferman metric of
($\mathbb{H}^{n}$,$\theta$) is $R=4n(n+1)u^{q-q^{*}}$ while $u$ satisfies the
equation (1.1). Especially, for $q=q^{*}$, the pseudo-Hermitian scalar
curvature $R$ is a constant and it is called CR Yamabe problem to find such a
contact form $\theta$. Accordingly, for $q=q^{*}$, the equation (1.1) is
called the CR Yamabe equation. The constant $1+q^{*}=\frac{2Q}{Q-2}$ is the CR
Sobolev embedding exponent and, for the Yamabe equation, there is nontrivial
solution as follows

(1.5)
$u(z,t)=C\big{|}t+\sqrt{-1}z\cdot\overline{z}+z\cdot\mu+\lambda\big{|}^{-n}$

for some $C>0$, $\lambda\in\mathbf{C}$, Im($\lambda$)$>|\mu|^{2}/4$, and
$\mu\in\mathbf{C}^{n}$, which is also the only extremals of the CR Yamabe
functional, or the CR Sobolev inequality, on $\mathbb{H}^{n}$. So our theorem
1 also confirms that $q^{*}$ is really critical. The equation (1.1) had
catched many mathematician’s attention since it raised in the CR Yamabe
problem. The CR Yamabe problem had been initiated and studied by David Jerison
and John M. Lee in their series fundamental works (see [10]-[13]). For
compact, strictly pseudovonvex CR manifold, the CR Yamabe problem had been
solved in case of not locally CR equivalent to sphere $\mathbf{S}^{2n+1}$ by
Jerison-Lee [13] for $n\geq 2$ and Gamara [7] for $n=1$, and in case of
locally CR equivalent to $\mathbf{S}^{2n+1}$ by Gamara-Yacoub [8] for all
$n\geq 1$. The CR Yamabe problem on closed Einstein pseudohermitian manifold
was also studied by Wang [16].

On $\mathbb{H}^{n}$, the uniqueness of CR Yamabe solutions was also obtained
by Jerison-Lee [11] for the case of finite volume, i.e., $u\in
L^{\frac{2Q}{Q-2}}(\mathbb{H}^{n})$, and by Garofalo-Vassilev [6] for the case
of cylindrically symmetry on groups of Heisenberg type. For the subcritical
case $1<q\leq q_{*}$, Birindelli-Dolcetta [3] proved that the only nonnegative
entire solution of (1.1) is the trivial one, where they also showed that
$q=q_{*}$ is sharp for the nonexistence of the inequality

(1.6) $-\triangle_{\mathbb{H}^{n}}u\geq
2n^{2}u^{q}\quad\text{in}\quad\mathbb{H}^{n}.$

For the subcritical case $q_{*}<q<q^{*}$, the classification of solutions to
the equations (1.1) is still open, except for some partial results, such as
the solutions are cylindrical or decay at infinity in [4], and as
$n>1,\,1<q\leq q^{*}-\frac{1}{(Q-2)(Q-1)^{2}}$ in [17].

There are analogous results in the Euclidean case. In the splendid paper [9],
B. Gidas and J. Spruck proved that, for $1<q<\frac{n+2}{n-2}$, the following
equation (1.7) has no positive entire solution in the n-dimension Euclidean
space $\mathbb{R}^{n}$:

(1.7) $-\triangle u=u^{q}.$

The method used by Gidas-Spruck [9] is the integral estimate, as here in our
paper. Later, Chen-Li [3] also got the same result by using the method of
moving plane. Gidas-Spruck [9] also gave a singularity estimate, precisely,
they proved that for $\frac{n}{n-2}<q<\frac{n+2}{n-2}$, the positive solution
of (1.7) in the punctured unit ball, with a nonremovable singularity at the
origin, must satisfies

(1.8) $|x|^{\frac{2}{q-1}}u(x)\rightarrow C_{0}\quad\text{as}\,\,x\rightarrow
0.$

Also as in the Euclidean case, the Liouville type result in theorem 1.1 may be
useful in resolving the Dirichlet problem of the same equation, via the blow-
up analysis, .

To get the integral estimate (1.3), there are usually two difficulties to be
overcome in a noncompact domain. One is to find a suitable identity, and the
other is to estimate the “tail” terms after integrating by part of the
identity multiplied suitable cut-off function. When they studied the CR Yamabe
problem in their splendid work [11], Jerison and Lee had found several
remarkable identities with the help of a computer program. The idea of
Jerison-Lee [11] was originally due to Obata in his classic work [15]. Roughly
speaking, the main idea is to find an identity to express some suitable
nonnegative terms (usually with associate geometry data) in a divergence form.
Then integrating both sides of the identity to get useful results as one
desired. But there is a pity that the identity in case of Heisenberg group
$\mathbb{H}^{n}$ given by Jerison-Lee(see (4.2) for example in [11]) is in a
so complicated form that, one must suffer an awful long and tedious
computation to check it and, may hardly to make any generalization.
Nevertheless, based on our new observation, we would generalize the Jerison-
Lee’s identity to a new form and gave a transparent proof, so that it can be
used in dealing with the subcritical case of the equation (1.1).

The paper is organized as follows. In section 2, we introduced some notations
and proved a generalization of the Jerison-Lee’s identity. Then, using this
generalized identity, we proved theorem 3 in section 3. The proof of theorem 2
shall be presented in section 4.

##  2\. Generalization of Jerison-Lee’s identity

In this Section we discuss the generalization of a remarkable Jerison-Lee’s
identity from [11] on Heissenberg group $\mathbb{H}^{n}$. We adopt notations
as in [11].

We shall first give a brief introduction to the Heissenberg group
$\mathbb{H}^{n}$ and some notations. We consider $\mathbb{H}^{n}$ as the set
$\mathbb{C}^{n}\times\mathbb{R}$ with coordinates ($z,\,t$) and group law
$\circ$:

$(z,t)\circ(\xi,t)=\big{(}z+\xi,\,t+s+2\mathbf{Im}z^{\alpha}\overline{\xi}^{\alpha}\big{)}\quad\text{for}\,\,(z,t),\,(\xi,t)\in\mathbb{C}^{n}\times\mathbb{R},$

where and in the sequel, the repeated indices are sum form $1$ to $n$. The CR
structure of $\mathbb{H}^{n}$ is given by the bundle $\mathcal{H}$ spanned by
the left-invariant vector fields $Z_{\alpha}=\partial/\partial
z^{\alpha}+\sqrt{-1}\overline{z}^{\alpha}\partial/\partial t$,
$\alpha=1,\cdots,n$. The standard (left-invariant) contact form on
$\mathbb{H}^{n}$ is
$\mathbf{\Theta}=dt+\sqrt{-1}(z^{\alpha}d\overline{z}^{\alpha}-\overline{z}^{\alpha}dz^{\alpha})$.
With respect to the standard holomorphic frame $\\{Z_{\alpha}\\}$ and dual
admissible coframe $\\{dz^{\alpha}\\}$, the
$Levi\,\,forms\,\,h_{\alpha\overline{\beta}}=2\delta_{\alpha\overline{\beta}}$.
Accordingly, for a smooth function $f$ on $\mathbb{H}^{n}$, denote its
derivatives by $f_{\alpha}=Z_{\alpha}f$,
$f_{\alpha\overline{\beta}}=Z_{\overline{\beta}}(Z_{\alpha}f)$ ,
$f_{0}=\frac{\partial f}{\partial t}$, $f_{0\alpha}=Z_{\alpha}(\frac{\partial
f}{\partial t})$, etc. We would also indicate the derivatives of functions or
vector fields with indices preceded by a comma, to avoid confusion. Then we
have the following commutative formulae:

$f_{\alpha\beta}-f_{\beta\alpha}=0,\quad
f_{\alpha\overline{\beta}}-f_{\overline{\beta}\alpha}=2\sqrt{-1}\delta_{\alpha\overline{\beta}}\,f_{0},\quad
f_{0\alpha}-f_{\alpha 0}=0,$ $f_{\alpha\beta 0}-f_{\alpha 0\beta}=0,\qquad
f_{\alpha\beta\overline{\gamma}}-f_{\alpha\overline{\gamma}\beta}=2\sqrt{-1}\delta_{\beta\overline{\gamma}}\,f_{\alpha
0},\,\cdots.$

Now we are at the point to give the generalized identity for positive solution
of the equation (1.1). Let $u>0$ solves (1.1). Take $e^{f}=u^{\frac{1}{n}}$
and $q=q^{*}+\frac{p}{n}$, then $f$ satisfies the following equation

(2.1) $\mathbf{Re}f_{\alpha\overline{\alpha}}=-n|\partial f|^{2}-ne^{(2+p)f},$

where $|\partial f|^{2}=f_{\alpha}f_{\overline{\alpha}}$. Define the tensors

(2.2)
$\begin{split}D_{\alpha\beta}=&f_{\alpha\beta}-2f_{\alpha}f_{\beta},\qquad\qquad\qquad
D_{\alpha}=D_{\alpha\beta}f_{\overline{\beta}},\\\
E_{\alpha\overline{\beta}}=&f_{\alpha\overline{\beta}}-\frac{1}{n}f_{\gamma\overline{\gamma}}\delta_{\alpha\overline{\beta}},\qquad\quad\qquad
E_{\alpha}=E_{\alpha\overline{\beta}}f_{\beta},\\\
G_{\alpha}=&\sqrt{-1}f_{0\alpha}-\sqrt{-1}f_{0}f_{\alpha}+e^{(2+p)f}f_{\alpha}+|\partial
f|^{2}f_{\alpha}.\end{split}$

Denote the function $g=|\partial f|^{2}+e^{(2+p)f}-\sqrt{-1}f_{0}$. Then we
can rewrite the equation (2.1) as

(2.3) $f_{\alpha\overline{\alpha}}=-ng.$

Moreover, we observe that

(2.4)
$\begin{split}\,&E_{\alpha\overline{\beta}}=f_{\alpha\overline{\beta}}+g\delta_{\alpha\overline{\beta}},\qquad\,\,\qquad
E_{\alpha}=f_{\alpha\overline{\beta}}f_{\beta}+gf_{\alpha},\\\
\,&D_{\alpha}=f_{\alpha\beta}f_{\overline{\beta}}-2|\partial
f|^{2}f_{\alpha},\qquad
G_{\alpha}=\sqrt{-1}f_{0\alpha}+gf_{\alpha},\end{split}$

and by

(2.5) $(|\partial
f|^{2})_{,\overline{\alpha}}=D_{\overline{\alpha}}+E_{\overline{\alpha}}+\overline{g}f_{\overline{\alpha}}-2f_{\overline{\alpha}}e^{(2+p)f},$

we find

(2.6)
$\begin{split}g_{\overline{\alpha}}=&D_{\overline{\alpha}}+E_{\overline{\alpha}}+G_{\overline{\alpha}}+pf_{\overline{\alpha}}e^{(2+p)f}.\end{split}$

In view of the above observation, now we give the crucial identity as follows

######  Proposition 2.1.

(2.7)
$\begin{split}\,&\mathbf{Re}Z_{\overline{\alpha}}\Big{\\{}e^{2(n-1)f}\Big{[}\big{(}g+3\sqrt{-1}f_{0}\big{)}E_{\alpha}\\\
\,&\hskip
77.0pt+\big{(}g-\sqrt{-1}f_{0}\big{)}D_{\alpha}-3\sqrt{-1}f_{0}G_{\alpha}-\frac{p}{4}f_{\alpha}|\partial
f|^{4}\Big{]}\Big{\\}}\\\
=&\,e^{(2n+p)f}\big{(}|E_{\alpha\overline{\beta}}|^{2}+|D_{\alpha\beta}|^{2}\big{)}\\\
\,&+e^{2(n-1)f}\big{(}|G_{\alpha}|^{2}+|G_{\alpha}+D_{\alpha}|^{2}+|G_{\alpha}-E_{\alpha}|^{2}+|D_{\alpha\beta}f_{\overline{\gamma}}+E_{\alpha\overline{\gamma}}f_{\beta}|^{2}\big{)}\\\
\,&+e^{(2n-2)f}\mathbf{Re}\big{(}D_{\alpha}+E_{\alpha}\big{)}f_{\overline{\alpha}}\big{(}pe^{(2+p)f}-\frac{p}{2}|\partial
f|^{2}\big{)}\\\ \,&-p(2n-1)|\partial
f|^{2}e^{2(n+1+p)f}-\frac{p}{4}(7n-6)|\partial f|^{4}e^{(2n+p)f}\\\
\,&-\frac{p}{4}n|\partial
f|^{6}e^{2(n-1)f}-3np|f_{0}|^{2}e^{(2n+p)f}.\end{split}$

######  Remark 2.2.

Note that for $p=0$, then (2.7) is exactly a remarkable identity found by
Jerison and Lee (see (4.2) in [11]).

$\mathbf{Proof\,\,of\,\,proposition\,\,\ref{Pro-1}}$ Denote

$\mathcal{L}=\mathcal{L}_{1}+\mathcal{L}_{2}+\mathcal{L}_{3}+\mathcal{L}_{4},$

with

$\quad\mathcal{L}_{1}=Z_{\overline{\alpha}}\Big{\\{}\big{(}g+3\sqrt{-1}f_{0}\big{)}E_{\alpha}e^{2(n-1)f}\Big{\\}},$

$\quad\mathcal{L}_{2}=Z_{\overline{\alpha}}\Big{\\{}\big{(}g-\sqrt{-1}f_{0}\big{)}D_{\alpha}e^{2(n-1)f}\Big{\\}},$

$\mathcal{L}_{3}=Z_{\overline{\alpha}}\Big{\\{}-3\sqrt{-1}f_{0}G_{\alpha}e^{2(n-1)f}\Big{\\}},$

$\mathcal{L}_{4}=Z_{\overline{\alpha}}\Big{\\{}-\frac{p}{4}f_{\alpha}|\partial
f|^{4}e^{2(n-1)f}\Big{\\}}.$

First we compute $\mathcal{L}_{3}$. We have, by (2.4) and the commutative
formulae,

(2.8)
$\begin{split}G_{\alpha,\overline{\alpha}}=&\,\sqrt{-1}f_{0\alpha\overline{\alpha}}+g_{\overline{\alpha}}f_{\alpha}+gf_{\alpha\overline{\alpha}}\\\
=&\,\sqrt{-1}f_{\alpha\overline{\alpha}0}+g_{\overline{\alpha}}f_{\alpha}+g(f_{\overline{\alpha}\alpha}+2n\sqrt{-1}f_{0})\\\
=&\,f_{\alpha}g_{\overline{\alpha}}-n\sqrt{-1}g_{0}-n|g|^{2}+2n\sqrt{-1}f_{0}g.\end{split}$

There for

(2.9)
$\begin{split}e^{-2(n-1)f}\mathcal{L}_{3}=&\,e^{-2(n-1)f}Z_{\overline{\alpha}}\Big{\\{}-3\sqrt{-1}f_{0}G_{\alpha}e^{2(n-1)f}\Big{\\}}\\\
=&\,-3\sqrt{-1}f_{0}G_{\alpha,\overline{\alpha}}-3\sqrt{-1}f_{0\overline{\alpha}}G_{\alpha}-6(n-1)\sqrt{-1}f_{0}f_{\overline{\alpha}}G_{\alpha}\\\
=&\,-3\sqrt{-1}f_{0}\big{(}f_{\alpha}g_{\overline{\alpha}}-n\sqrt{-1}g_{0}-n|g|^{2}+2n\sqrt{-1}f_{0}g\big{)}\\\
\,&\,+3\sqrt{-1}(G_{\overline{\alpha}}-\overline{g}f_{\overline{\alpha}})G_{\alpha}-6(n-1)\sqrt{-1}f_{0}f_{\overline{\alpha}}G_{\alpha}\\\
=&\,3|G_{\alpha}|^{2}-3(\overline{g}+2(n-1)\sqrt{-1}f_{0})f_{\overline{\alpha}}G_{\alpha}\\\
\,&\,-3\sqrt{-1}f_{0}f_{\alpha}g_{\overline{\alpha}}-3nf_{0}g_{0}+3n\sqrt{-1}f_{0}|g|^{2}+6n|f_{0}|^{2}g.\end{split}$

Next we compute $\mathcal{L}_{1}$. Also by (2.4) and the commutative formulae,

(2.10)
$\begin{split}E_{\alpha,\overline{\alpha}}=&\,f_{\alpha\overline{\beta}\overline{\alpha}}f_{\beta}+f_{\alpha\overline{\beta}}f_{\beta\overline{\alpha}}+g_{\overline{\alpha}}f_{\alpha}+gf_{\alpha\overline{\alpha}}\\\
=&\,f_{\alpha\overline{\alpha}\overline{\beta}}f_{\beta}+f_{\alpha\overline{\beta}}(f_{\overline{\alpha}\beta}+2\sqrt{-1}f_{0}\delta_{\beta\overline{\alpha}})+g_{\overline{\alpha}}f_{\alpha}+gf_{\alpha\overline{\alpha}}\\\
=&\,-ng_{\overline{\beta}}f_{\beta}+f_{\alpha\overline{\beta}}f_{\overline{\alpha}\beta}+2\sqrt{-1}f_{0}f_{\alpha\overline{\alpha}}+g_{\overline{\alpha}}f_{\alpha}+gf_{\alpha\overline{\alpha}}\\\
=&\,(1-n)f_{\alpha}g_{\overline{\alpha}}+(E_{\alpha\overline{\beta}}-g\delta_{\alpha\overline{\beta}})(E_{\overline{\alpha}\beta}-\overline{g}\delta_{\overline{\alpha}\beta})-n|g|^{2}\\\
=&\,|E_{\alpha\overline{\beta}}|^{2}+(1-n)f_{\alpha}g_{\overline{\alpha}}.\end{split}$

There for

(2.11)
$\begin{split}e^{-2(n-1)f}\mathcal{L}_{1}=&\,e^{-2(n-1)f}Z_{\overline{\alpha}}\Big{\\{}\big{(}g+3\sqrt{-1}f_{0}\big{)}E_{\alpha}e^{2(n-1)f}\Big{\\}}\\\
=&\,\big{(}g+3\sqrt{-1}f_{0}\big{)}E_{\alpha,\overline{\alpha}}\\\
\,&\,+\big{(}g_{\overline{\alpha}}+3\sqrt{-1}f_{0\overline{\alpha}}\big{)}E_{\alpha}+2(n-1)\big{(}g+3\sqrt{-1}f_{0}\big{)}f_{\overline{\alpha}}E_{\alpha}\\\
=&\,\big{(}g+3\sqrt{-1}f_{0}\big{)}\big{(}|E_{\alpha\overline{\beta}}|^{2}+(1-n)f_{\alpha}g_{\overline{\alpha}}\big{)}\\\
\,&\,+g_{\overline{\alpha}}E_{\alpha}+3\big{(}-G_{\overline{\alpha}}+\overline{g}f_{\overline{\alpha}}\big{)}E_{\alpha}+2(n-1)\big{(}g+3\sqrt{-1}f_{0}\big{)}f_{\overline{\alpha}}E_{\alpha}\\\
=&\,\big{(}g+3\sqrt{-1}f_{0}\big{)}|E_{\alpha\overline{\beta}}|^{2}+\big{(}g_{\overline{\alpha}}-3G_{\overline{\alpha}}\big{)}E_{\alpha}\\\
\,&\,+\big{(}3\overline{g}+2(n-1)(g+3\sqrt{-1}f_{0})\big{)}f_{\overline{\alpha}}E_{\alpha}\\\
\,&\,+(1-n)(g+3\sqrt{-1}f_{0})f_{\alpha}g_{\overline{\alpha}}.\end{split}$

Now we compute $\mathcal{L}_{2}$. Using the commutative formulae, we compute

(2.12)
$\begin{split}f_{\alpha\beta\overline{\alpha}}=&\,f_{\alpha\overline{\alpha}\beta}+2\sqrt{-1}f_{0\alpha}\delta_{\beta\overline{\alpha}}\\\
=&\,(f_{\overline{\alpha}\alpha}+2n\sqrt{-1}f_{0})_{,\beta}+2\sqrt{-1}f_{0\beta}\\\
=&\,-n\overline{g}_{\beta}+2(n+1)(G_{\beta}-gf_{\beta})\\\
=&\,2(n+1)G_{\beta}-n\overline{g}_{\beta}-2(n+1)f_{\beta}g.\end{split}$

By this and (2.4), (2.5) we deduce

(2.13)
$\begin{split}D_{\alpha,\overline{\alpha}}=&\,f_{\alpha\beta\overline{\alpha}}f_{\overline{\beta}}+f_{\alpha\beta}f_{\overline{\beta}\overline{\alpha}}-2(|\partial
f|^{2})_{,\overline{\alpha}}f_{\alpha}-2|\partial
f|^{2}f_{\alpha\overline{\alpha}}\\\
=&\,\big{(}2(n+1)G_{\beta}-n\overline{g}_{\beta}-2(n+1)f_{\beta}g\big{)}f_{\overline{\beta}}\\\
\,&\,+(D_{\alpha\beta}+2f_{\alpha}f_{\beta})(D_{\overline{\alpha}\overline{\beta}}+2f_{\overline{\alpha}}f_{\overline{\beta}})\\\
\,&\,-2\big{(}D_{\overline{\alpha}}+E_{\overline{\alpha}}+\overline{g}f_{\overline{\alpha}}-2f_{\overline{\alpha}}e^{(2+p)f}\big{)}f_{\alpha}+2n|\partial
f|^{2}g\\\
=&\,|D_{\alpha\beta}|^{2}+2f_{\overline{\alpha}}D_{\alpha}-2f_{\alpha}E_{\overline{\alpha}}+2(n+1)f_{\overline{\alpha}}G_{\alpha}-nf_{\overline{\alpha}}\overline{g}_{\alpha}.\end{split}$

There for

(2.14) $\begin{split}\,&e^{-2(n-1)f}\mathcal{L}_{2}\\\
=&\,\,e^{-2(n-1)f}Z_{\overline{\alpha}}\Big{\\{}\big{(}g-\sqrt{-1}f_{0}\big{)}D_{\alpha}e^{2(n-1)f}\Big{\\}}\\\
=&\,\,\big{(}g-\sqrt{-1}f_{0}\big{)}D_{\alpha,\overline{\alpha}}\\\
\,&\,\,+\big{(}g_{\overline{\alpha}}-\sqrt{-1}f_{0\overline{\alpha}}\big{)}D_{\alpha}+2(n-1)\big{(}g-\sqrt{-1}f_{0}\big{)}f_{\overline{\alpha}}D_{\alpha}\\\
=&\,\,\big{(}g-\sqrt{-1}f_{0}\big{)}\big{(}|D_{\alpha\beta}|^{2}+2f_{\overline{\alpha}}D_{\alpha}-2f_{\alpha}E_{\overline{\alpha}}+2(n+1)f_{\overline{\alpha}}G_{\alpha}-nf_{\overline{\alpha}}\overline{g}_{\alpha}\big{)}\\\
\,&\,\,+\big{(}g_{\overline{\alpha}}+G_{\overline{\alpha}}-\overline{g}f_{\overline{\alpha}}\big{)}D_{\alpha}+2(n-1)\big{(}g-\sqrt{-1}f_{0}\big{)}f_{\overline{\alpha}}D_{\alpha}\\\
=&\,\,\big{(}g-\sqrt{-1}f_{0}\big{)}|D_{\alpha\beta}|^{2}+(g_{\overline{\alpha}}+G_{\overline{\alpha}})D_{\alpha}\\\
\,&\,\,+(2ng-\overline{g}-2n\sqrt{-1}f_{0})f_{\overline{\alpha}}D_{\alpha}-2(g-\sqrt{-1}f_{0})f_{\alpha}E_{\overline{\alpha}}\\\
\,&\,\,+2(n+1)(g-\sqrt{-1}f_{0})f_{\overline{\alpha}}G_{\alpha}-n(g-\sqrt{-1}f_{0})f_{\overline{\alpha}}\overline{g}_{\alpha}.\end{split}$

Finally, for $\mathcal{L}_{4}$, by (2.3) and (2.5), a direct computation shows

(2.15)
$\begin{split}e^{-2(n-1)f}\mathcal{L}_{4}=&e^{-2(n-1)f}Z_{\overline{\alpha}}\Big{\\{}-\frac{p}{4}f_{\alpha}|\partial
f|^{4}e^{2(n-1)f}\Big{\\}}\\\
=&-\frac{p}{2}\big{(}E_{\overline{\alpha}}+D_{\overline{\alpha}}\big{)}f_{\alpha}|\partial
f|^{2}\\\ \,&\,\,-\frac{p}{4}n|\partial f|^{6}+\frac{p}{4}(n+2)|\partial
f|^{4}e^{(2+p)f}-\frac{p}{4}(n+2)\sqrt{-1}f_{0}|\partial f|^{4}.\end{split}$

By (2.9), (2.11), (2.14) and (2.15), noticing
$f_{\alpha}E_{\overline{\alpha}}=f_{\overline{\alpha}}E_{\alpha}$ (this also
implies it’s real), we obtain

(2.16)
$\begin{split}\,&e^{-2(n-1)f}(\mathcal{L}_{1}+\mathcal{L}_{2}+\mathcal{L}_{3}+\mathcal{L}_{4})\\\
=&\,\,\big{(}g-\sqrt{-1}f_{0}\big{)}|D_{\alpha\beta}|^{2}+\big{(}g+3\sqrt{-1}f_{0}\big{)}|E_{\alpha\overline{\beta}}|^{2}+3|G_{\alpha}|^{2}\\\
\,&\,\,+(g_{\overline{\alpha}}+G_{\overline{\alpha}})D_{\alpha}\,+\big{(}g_{\overline{\alpha}}-3G_{\overline{\alpha}}\big{)}E_{\alpha}\\\
\,&\,\,+\Big{(}2ng-\overline{g}-\frac{p}{2}|\partial
f|^{2}-2n\sqrt{-1}f_{0}\Big{)}f_{\overline{\alpha}}D_{\alpha}\\\
\,&\,\,+\Big{(}2(n-2)g+3\overline{g}-\frac{p}{2}|\partial
f|^{2}+(6n-4)\sqrt{-1}f_{0})\Big{)}f_{\overline{\alpha}}E_{\alpha}\\\
\,&\,\,+\Big{(}2(n+1)g-3\overline{g}-(8n-4)\sqrt{-1}f_{0}\Big{)}f_{\overline{\alpha}}G_{\alpha}\\\
\,&\,\,-\big{(}(n-1)g+3n\sqrt{-1}f_{0}\big{)}f_{\alpha}g_{\overline{\alpha}}\,-n(g-\sqrt{-1}f_{0})f_{\overline{\alpha}}\overline{g}_{\alpha}-3nf_{0}g_{0}\\\
\,&\,\,+3n\sqrt{-1}f_{0}|g|^{2}+6n|f_{0}|^{2}g\\\
\,&\,\,-\frac{p}{4}n|\partial f|^{6}+\frac{p}{4}(n+2)|\partial
f|^{4}e^{(2+p)f}-\frac{p}{4}(n+2)\sqrt{-1}f_{0}|\partial f|^{4}.\end{split}$

Straight calculations show

(2.17)
$\begin{split}g_{0}=&\,\,\sqrt{-1}f_{\alpha}G_{\overline{\alpha}}-\sqrt{-1}f_{\overline{\alpha}}G_{\alpha}\\\
\,&\,\,+2f_{0}|\partial
f|^{2}+(2+p)f_{0}e^{(2+p)f}-\sqrt{-1}f_{00}.\end{split}$

By this and virtue of (2.6), we finally reach

(2.18)
$\begin{split}\,&e^{-2(n-1)f}(\mathcal{L}_{1}+\mathcal{L}_{2}+\mathcal{L}_{3}+\mathcal{L}_{4})\\\
=&\,\,\big{(}g-\sqrt{-1}f_{0}\big{)}|D_{\alpha\beta}|^{2}+\big{(}g+3\sqrt{-1}f_{0}\big{)}|E_{\alpha\overline{\beta}}|^{2}+3|G_{\alpha}|^{2}\\\
\,&\,\,+(D_{\overline{\alpha}}+E_{\overline{\alpha}}+2G_{\overline{\alpha}})D_{\alpha}+\big{(}D_{\overline{\alpha}}+E_{\overline{\alpha}}-2G_{\overline{\alpha}}\big{)}E_{\alpha}\\\
\,&\,\,+p\big{(}f_{\overline{\alpha}}-\frac{1}{2}f_{\overline{\alpha}}|\partial
f|^{2}\big{)}\big{(}D_{\alpha}+E_{\alpha}\big{)}\\\
\,&\,\,+(n-1)\big{(}|\partial
f|^{2}+e^{(2+p)f}\big{)}(f_{\overline{\alpha}}G_{\alpha}-f_{\alpha}G_{\overline{\alpha}})\\\
\,&\,\,+(5n+1)\sqrt{-1}f_{0}(f_{\overline{\alpha}}G_{\alpha}+f_{\alpha}G_{\overline{\alpha}})\\\
\,&\,\,-p(2n-1)|\partial f|^{2}e^{2(2+p)f}-\frac{p}{4}(7n-6)|\partial
f|^{4}e^{(2+p)f}\\\ \,&\,\,-\frac{p}{4}n|\partial
f|^{6}-3np|f_{0}|^{2}e^{(2+p)f}\\\
\,&\,\,-\frac{p}{4}(n+2)\sqrt{-1}f_{0}|\partial
f|^{4}-p\sqrt{-1}f_{0}|\partial f|^{2}e^{(2+p)f}\\\
\,&\,\,+3n\sqrt{-1}f_{0}|g|^{2}-6n\sqrt{-1}f_{0}|f_{0}|^{2}+3n\sqrt{-1}f_{0}f_{00}.\end{split}$

From this one can check (2.7) easily and complete the proof of proposition
2.1. ∎

##  3\. Proof of Theorem 1.3

$\mathbf{Proof\,\,of\,\,Theorem\,\,\ref{Thm3}}$ Let $f$ satisfy the equation
(2.3) and hence the identity (2.7). Then by $q=q^{*}+\frac{p}{n}$, the
subcritical exponent $1<q<q^{*}$ corresponding to $-2<p<0$ and (1.3) is
equivalent to

(3.1)
$\int_{B_{r}(\xi_{0})}e^{(2n+4+3p)f}\lesssim\,\,Cr^{2n+2-2\times\frac{2n+4+3p}{2+p}}.$

Note that (2.7) can be rewritten as

(3.2)
$\begin{split}\mathcal{M}=\mathbf{Re}Z_{\overline{\alpha}}\Big{\\{}\Big{[}&\big{(}D_{\alpha}+E_{\alpha})(|\partial
f|^{2}+e^{(2+p)f})\\\
\,&-\sqrt{-1}f_{0}\big{(}2D_{\alpha}-2E_{\alpha}+3G_{\alpha}\big{)}-\frac{p}{4}f_{\alpha}|\partial
f|^{4}\Big{]}e^{2(n-1)f}\Big{\\}},\end{split}$

with

(3.3)
$\begin{split}\mathcal{M}=&\big{(}|E_{\alpha\overline{\beta}}|^{2}+|D_{\alpha\beta}|^{2}\big{)}e^{(2n+p)f}+\Big{(}|G_{\alpha}|^{2}+|D_{\alpha\beta}f_{\overline{\gamma}}+E_{\alpha\overline{\gamma}}f_{\beta}|^{2}\Big{)}e^{2(n-1)f}\\\
\,&+s\Big{(}|G_{\alpha}+D_{\alpha}|^{2}+|G_{\alpha}-E_{\alpha}|^{2}\Big{)}e^{2(n-1)f}\\\
\,&+(1-s)\Big{(}|G_{\alpha}+D_{\alpha}|^{2}+|G_{\alpha}-E_{\alpha}|^{2}\Big{)}e^{2(n-1)f}\\\
\,&+pe^{2(n-1)f}\mathbf{Re}[f_{\overline{\alpha}}(G_{\alpha}+D_{\alpha})]\big{(}e^{(2+p)f}-\frac{1}{2}|\partial
f|^{2}\big{)}\\\
\,&+pe^{2(n-1)f}\mathbf{Re}[f_{\overline{\alpha}}(E_{\alpha}-G_{\alpha})]\big{(}e^{(2+p)f}-\frac{1}{2}|\partial
f|^{2}\big{)}\\\ \,&-p(2n-1)|\partial
f|^{2}e^{2(n+1+p)f}-\frac{p}{4}(7n-6)|\partial f|^{4}e^{(2n+p)f}\\\
\,&-\frac{p}{4}n|\partial
f|^{6}e^{2(n-1)f}-3np|f_{0}|^{2}e^{(2n+p)f}.\end{split}$

So for $-2<p<0$, we shall choose suitable $0<s<1$ such that $\mathcal{M}\geq
0$.

Now we rewrite $\mathcal{M}$ as

(3.4)
$\begin{split}\mathcal{M}=&\big{(}|E_{\alpha\overline{\beta}}|^{2}+|D_{\alpha\beta}|^{2}\big{)}e^{(2n+p)f}+\Big{(}|G_{\alpha}|^{2}+|D_{\alpha\beta}f_{\overline{\gamma}}+E_{\alpha\overline{\gamma}}f_{\beta}|^{2}\Big{)}e^{2(n-1)f}\\\
\,&+s\Big{(}|G_{\alpha}+D_{\alpha}|^{2}+|G_{\alpha}-E_{\alpha}|^{2}\Big{)}e^{2(n-1)f}\\\
\,&+e^{2(n-1)f}\Big{|}\sqrt{1-s}(G_{\alpha}+D_{\alpha})+\frac{p}{2\sqrt{1-s}}f_{\alpha}\big{(}e^{(2+p)f}-\frac{1}{2}|\partial
f|^{2}\big{)}\Big{|}^{2}\\\
\,&+e^{2(n-1)f}\Big{|}\sqrt{1-s}(E_{\alpha}-G_{\alpha})+\frac{p}{2\sqrt{1-s}}f_{\alpha}\big{(}e^{(2+p)f}-\frac{1}{2}|\partial
f|^{2}\big{)}\Big{|}^{2}\\\ \,&-\frac{p^{2}}{2(1-s)}e^{2(n-1)f}|\partial
f|^{2}\Big{[}e^{2(2+p)f}+\frac{1}{4}|\partial f|^{4}-e^{(2+p)f}|\partial
f|^{2}\Big{]}\\\ \,&-p(2n-1)|\partial
f|^{2}e^{2(n+1+p)f}-\frac{p}{4}(7n-6)|\partial f|^{4}e^{(2n+p)f}\\\
\,&-\frac{p}{4}n|\partial
f|^{6}e^{2(n-1)f}-3np|f_{0}|^{2}e^{(2n+p)f}.\end{split}$

Then we treat the terms in the last three lines and get

(3.5)
$\begin{split}\mathcal{M}=&\big{(}|E_{\alpha\overline{\beta}}|^{2}+|D_{\alpha\beta}|^{2}\big{)}e^{(2n+p)f}+\Big{(}|G_{\alpha}|^{2}+|D_{\alpha\beta}f_{\overline{\gamma}}+E_{\alpha\overline{\gamma}}f_{\beta}|^{2}\Big{)}e^{2(n-1)f}\\\
\,&+s\Big{(}|G_{\alpha}+D_{\alpha}|^{2}+|G_{\alpha}-E_{\alpha}|^{2}\Big{)}e^{2(n-1)f}\\\
\,&+e^{2(n-1)f}\Big{|}\sqrt{1-s}(G_{\alpha}+D_{\alpha})+\frac{p}{2\sqrt{1-s}}f_{\alpha}\big{(}e^{(2+p)f}-\frac{1}{2}|\partial
f|^{2}\big{)}\Big{|}^{2}\\\
\,&+e^{2(n-1)f}\Big{|}\sqrt{1-s}(E_{\alpha}-G_{\alpha})+\frac{p}{2\sqrt{1-s}}f_{\alpha}\big{(}e^{(2+p)f}-\frac{1}{2}|\partial
f|^{2}\big{)}\Big{|}^{2}\\\
\,&-p\big{[}\frac{n}{4}+\frac{p}{8(1-s)}\big{]}|\partial
f|^{6}e^{2(n-1)f}-\frac{p}{4}\big{[}7n-6-\frac{2p}{1-s}\big{]}|\partial
f|^{4}e^{(2n+p)f}\\\ \,&-p\big{[}2n-1+\frac{p}{2(1-s)}\big{]}|\partial
f|^{2}e^{2(n+1+p)f}-3np|f_{0}|^{2}e^{(2n+p)f}.\end{split}$

Now we take $0<s=s_{0}=\frac{1}{2}+\frac{p}{4n}<1$, then

(3.6)
$\begin{split}\mathcal{M}=&\big{(}|E_{\alpha\overline{\beta}}|^{2}+|D_{\alpha\beta}|^{2}\big{)}e^{(2n+p)f}+\Big{(}|G_{\alpha}|^{2}+|D_{\alpha\beta}f_{\overline{\gamma}}+E_{\alpha\overline{\gamma}}f_{\beta}|^{2}\Big{)}e^{2(n-1)f}\\\
\,&+s_{0}\Big{(}|G_{\alpha}+D_{\alpha}|^{2}+|G_{\alpha}-E_{\alpha}|^{2}\Big{)}e^{2(n-1)f}\\\
\,&+e^{2(n-1)f}\Big{|}\sqrt{1-s_{0}}(G_{\alpha}+D_{\alpha})+\frac{p}{2\sqrt{1-s_{0}}}f_{\alpha}\big{(}e^{(2+p)f}-\frac{1}{2}|\partial
f|^{2}\big{)}\Big{|}^{2}\\\
\,&+e^{2(n-1)f}\Big{|}\sqrt{1-s_{0}}(E_{\alpha}-G_{\alpha})+\frac{p}{2\sqrt{1-s_{0}}}f_{\alpha}\big{(}e^{(2+p)f}-\frac{1}{2}|\partial
f|^{2}\big{)}\Big{|}^{2}\\\ \,&-p\frac{n(2n+p)}{4(2n-p)}|\partial
f|^{6}e^{2(n-1)f}-\frac{p}{4}\big{[}7n-6-\frac{8np}{2n-p}\big{]}|\partial
f|^{4}e^{(2n+p)f}\\\ \,&-p\frac{4n^{2}-2n+p}{2n-p}|\partial
f|^{2}e^{2(n+1+p)f}-3np|f_{0}|^{2}e^{(2n+p)f},\end{split}$

and clearly all the coefficients in above are positive for $-2<p<0$ and
$\mathcal{M}\geq 0$.

Since $B_{4r}\subset\Omega$, we can take a real smooth cut off function $\eta$
such that

(3.7) $\begin{cases}\eta\equiv 1&in\,\,B_{r},\\\ 0\leq\eta\leq
1&in\,\,B_{2r},\\\ \eta\equiv 0&in\,\,\Omega\backslash B_{2r},\\\
|\partial\eta|\lesssim\frac{1}{r}&in\,\,\Omega,\end{cases}$

where we use “$\lesssim$” , “$\cong$” to replace “$\leq$” and “$=$”
respectively, to drop out some positive constants independent of $r$ and $f$.

Take a real $s>0$ big enough. Multiply both sides of (3.2) by $\eta^{s}$ and
integrate over $\Omega$ we have

(3.8) $\begin{split}\,&\int_{\Omega}\eta^{s}\mathcal{M}\\\
=&\int_{\Omega}\eta^{s}\mathbf{Re}Z_{\overline{\alpha}}\Big{\\{}\big{[}\big{(}D_{\alpha}+E_{\alpha})(|\partial
f|^{2}+e^{(2+p)f})\\\ \,&\hskip
68.0pt-\sqrt{-1}f_{0}\big{(}2D_{\alpha}-2E_{\alpha}+3G_{\alpha}\big{)}-\frac{p}{4}f_{\alpha}|\partial
f|^{4}\big{]}e^{2(n-1)f}\Big{\\}}.\end{split}$

Integrating by part and using (3.7) we get

(3.9) $\begin{split}\,&\int_{\Omega}\eta^{s}\mathcal{M}\\\
=&-s\int_{\Omega}\eta^{s-1}\mathbf{Re}\eta_{\overline{\alpha}}\Big{\\{}\big{[}\big{(}D_{\alpha}+E_{\alpha})(|\partial
f|^{2}+e^{(2+p)f})\\\ \,&\hskip
99.0pt-\sqrt{-1}f_{0}\big{(}2D_{\alpha}-2E_{\alpha}+3G_{\alpha}\big{)}-\frac{p}{4}f_{\alpha}|\partial
f|^{4}\big{]}e^{2(n-1)f}\Big{\\}}\\\
\lesssim&\frac{1}{r}\int_{\Omega}\eta^{s-1}\Big{\\{}|D_{\alpha}+E_{\alpha}|(|\partial
f|^{2}+e^{(2+p)f})e^{2(n-1)f}\\\ \,&\hskip
55.0pt+|f_{0}|\big{|}2D_{\alpha}-2E_{\alpha}+3G_{\alpha}\big{|}e^{2(n-1)f}+|\partial
f|^{5}e^{2(n-1)f}\Big{\\}}\end{split}$

Since

$|D_{\alpha}+E_{\alpha}|\leq|D_{\alpha}+G_{\alpha}|+|E_{\alpha}-G_{\alpha}|,$

$\big{|}2D_{\alpha}-2E_{\alpha}+3G_{\alpha}\big{|}\leq
2|D_{\alpha}+G_{\alpha}|+2|E_{\alpha}-G_{\alpha}|+|G_{\alpha}|,$

using the Young’s inequality $ab\leq\epsilon a^{2}+\frac{C}{\epsilon}b^{2}$ in
(3.9) we obtain

(3.10)
$\begin{split}\int_{\Omega}\eta^{s}\mathcal{M}\lesssim&\epsilon\int_{\Omega}\eta^{s}\big{(}|D_{\alpha}+G_{\alpha}|^{2}+|E_{\alpha}-G_{\alpha}|^{2}+|G_{\alpha}|^{2}\big{)}e^{2(n-1)f}\\\
\,&\hskip 13.0pt+\frac{1}{\epsilon
r^{2}}\int_{\Omega}\eta^{s-2}\big{(}|\partial
f|^{4}+e^{2(2+p)f}+|f_{0}|^{2}\big{)}e^{2(n-1)f}\\\ \,&\hskip
113.0pt+\frac{1}{r}\int_{\Omega}\eta^{s-1}|\partial
f|^{5}e^{2(n-1)f}.\end{split}$

This implies, by taking $\epsilon$ small, that

(3.11)
$\begin{split}\int_{\Omega}\eta^{s}\mathcal{M}\lesssim&\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}\big{(}|\partial
f|^{4}+e^{2(2+p)f}+|f_{0}|^{2}\big{)}e^{2(n-1)f}\\\ \,&\hskip
86.0pt+\frac{1}{r}\int_{\Omega}\eta^{s-1}|\partial
f|^{5}e^{2(n-1)f}.\end{split}$

To go forward, we need the following lemmas, that will be proved at the end of
this section.

######  Lemma 3.1.

(3.12)
$\begin{split}\int_{\Omega}\eta^{s-2}|f_{0}|^{2}e^{2(n-1)f}\lesssim&\,\epsilon
r^{2}\int_{\Omega}\eta^{s}\mathcal{M}+\int_{\Omega}\eta^{s-2}|\partial
f|^{4}e^{2(n-1)f}\\\ \,&+\int_{\Omega}\eta^{s-2}|\partial
f|^{2}e^{(2n+p)f}+\frac{1}{r^{2}}\int_{\Omega}\eta^{s-4}|\partial
f|^{2}e^{2(n-1)f}.\end{split}$

######  Lemma 3.2.

(3.13)
$\int_{\Omega}\eta^{s}e^{(2n+4+3p)f}\lesssim\,\int_{\Omega}\eta^{s}|\partial
f|^{2}e^{2(n+1+p)f}+\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}e^{(2n+2+2p)f}.$

Now plugging (3.12) into (3.11) with small $\epsilon$ we get

(3.14)
$\begin{split}\int_{\Omega}\eta^{s}\mathcal{M}\lesssim&\,\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}e^{2(n+1+p)f}\\\
\,&+\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}|\partial
f|^{4}e^{2(n-1)f}+\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}|\partial
f|^{2}e^{(2n+p)f}\\\ \,&+\frac{1}{r^{4}}\int_{\Omega}\eta^{s-4}|\partial
f|^{2}e^{2(n-1)f}+\frac{1}{r}\int_{\Omega}\eta^{s-1}|\partial
f|^{5}e^{2(n-1)f}.\end{split}$

For the last term in above, using Young’s inequality one get

(3.15) $\begin{split}\frac{1}{r}\int_{\Omega}\eta^{s-1}|\partial
f|^{5}e^{2(n-1)f}\lesssim&\epsilon\int_{\Omega}\eta^{s}|\partial
f|^{6}e^{2(n-1)f}+\frac{1}{r^{6}}\int_{\Omega}\eta^{s-6}e^{2(n-1)f}.\end{split}$

Similarly, one has

(3.16) $\begin{split}\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}|\partial
f|^{4}e^{2(n-1)f}\lesssim&\epsilon\int_{\Omega}\eta^{s}|\partial
f|^{6}e^{2(n-1)f}+\frac{1}{r^{6}}\int_{\Omega}\eta^{s-6}e^{2(n-1)f},\end{split}$

(3.17) $\begin{split}\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}|\partial
f|^{2}e^{(2n+p)f}\lesssim&\epsilon\int_{\Omega}\eta^{s}|\partial
f|^{4}e^{(2n+p)f}+\frac{1}{r^{4}}\int_{\Omega}\eta^{s-4}e^{(2n+p)f},\end{split}$

and

(3.18) $\begin{split}\frac{1}{r^{4}}\int_{\Omega}\eta^{s-4}|\partial
f|^{2}e^{2(n-1)f}\lesssim&\epsilon\int_{\Omega}\eta^{s}|\partial
f|^{6}e^{2(n-1)f}+\frac{1}{r^{6}}\int_{\Omega}\eta^{s-6}e^{2(n-1)f}.\end{split}$

Inserting these into (3.14) and taking $\epsilon$ small yields

(3.19)
$\begin{split}\int_{\Omega}\eta^{s}\mathcal{M}\lesssim&\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}e^{2(n+1+p)f}\\\
\,&+\frac{1}{r^{4}}\int_{\Omega}\eta^{s-4}e^{(2n+p)f}+\frac{1}{r^{6}}\int_{\Omega}\eta^{s-6}e^{2(n-1)f}.\end{split}$

Combining this with (3.13) we arrive at

(3.20) $\begin{split}\,&\int_{\Omega}\eta^{s}e^{(2n+4+3p)f}\\\
\lesssim&\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}e^{2(n+1+p)f}\\\
\,&+\frac{1}{r^{4}}\int_{\Omega}\eta^{s-4}e^{(2n+p)f}+\frac{1}{r^{6}}\int_{\Omega}\eta^{s-6}e^{2(n-1)f}\\\
\lesssim&\epsilon\int_{\Omega}\eta^{s}e^{(2n+4+3p)f}+r^{-2\times\frac{2n+4+3p}{2+p}}\int_{\Omega}\eta^{s-2\times\frac{2n+4+3p}{2+p}},\end{split}$

where in the last step, the Young’s inequality has been used three time with
different exponent pairs. Note that $0\leq\eta\leq 1$ in $\Omega$ and $\eta=1$
in $B_{r}(\xi_{0})\subset\Omega$. Therefor, by choosing $s>0$ big enough and
$\epsilon$ small, we finally obtain

(3.21)
$\int_{B_{r}(\xi_{0})}e^{(2n+4+3p)f}\lesssim\,\,r^{2n+2-2\times\frac{2n+4+3p}{2+p}}.$

This is (3.1), and hence theorem 1.3 is proved.∎

To complete this section, now we give the proofs of lemma 3.1,3.2.

$\mathbf{Proof\,\,of\,\,lemma\,\,\ref{lem-1}}$

Since $f$ satisfies the equation (2.3), a straight calculation shows

(3.22)
$e^{-kf}\mathbf{Re}Z_{\overline{\alpha}}\Big{(}\sqrt{-1}f_{0}f_{\alpha}e^{kf}\Big{)}=-\mathbf{Re}G_{\overline{\alpha}}f_{\alpha}-n|f_{0}|^{2}+|\partial
f|^{4}+|\partial f|^{2}e^{(2+p)f}.$

Multiply both sides of (3.22) by $\eta^{s-2}e^{kf}$ with $k=2(n-1)$ and
integrate over $\Omega$ we have

(3.23)
$\begin{split}\,&\int_{\Omega}\eta^{s-2}\mathbf{Re}Z_{\overline{\alpha}}\Big{(}\sqrt{-1}f_{0}f_{\alpha}e^{2(n-1)f}\Big{)}\\\
=&\,\int_{\Omega}\eta^{s-2}\Big{(}-\mathbf{Re}G_{\overline{\alpha}}f_{\alpha}-n|f_{0}|^{2}+|\partial
f|^{4}+|\partial f|^{2}e^{(2+p)f}\Big{)}e^{2(n-1)f}.\end{split}$

Integrating by part, using (3.7) and arranging the terms yields

(3.24)
$\begin{split}n\int_{\Omega}\eta^{s-2}|f_{0}|^{2}e^{2(n-1)f}=&\,\int_{\Omega}\eta^{s-2}\big{(}|\partial
f|^{4}+|\partial f|^{2}e^{(2+p)f}\big{)}e^{2(n-1)f}\\\
\,&\quad-\int_{\Omega}\eta^{s-2}\mathbf{Re}G_{\overline{\alpha}}f_{\alpha}e^{2(n-1)f}\\\
\,&\quad+(s-2)\int_{\Omega}\eta^{s-3}\mathbf{Re}\eta_{\overline{\alpha}}\Big{(}\sqrt{-1}f_{0}f_{\alpha}e^{2(n-1)f}\Big{)}\\\
\lesssim&\,\int_{\Omega}\eta^{s-2}\big{(}|\partial f|^{4}+|\partial
f|^{2}e^{(2+p)f}\big{)}e^{2(n-1)f}\\\
\,&\quad+\int_{\Omega}\eta^{s-2}|G_{\overline{\alpha}}||\partial
f|e^{2(n-1)f}\\\ \,&\quad+\frac{1}{r}\int_{\Omega}\eta^{s-3}|f_{0}||\partial
f|e^{2(n-1)f}.\end{split}$

For the above last two terms, Young’s inequality implies

(3.25)
$\begin{split}\,&\int_{\Omega}\eta^{s-2}|G_{\overline{\alpha}}||\partial
f|e^{2(n-1)f}+\frac{1}{r}\int_{\Omega}\eta^{s-3}|f_{0}||\partial
f|e^{2(n-1)f}\\\ \leq&\,\epsilon
r^{2}\int_{\Omega}\eta^{s}|G_{\alpha}|^{2}e^{2(n-1)f}+\epsilon\int_{\Omega}\eta^{s-2}|f_{0}|^{2}e^{2(n-1)f}\\\
\,&\hskip 106.0pt+\frac{C}{\epsilon r^{2}}\int_{\Omega}\eta^{s-4}|\partial
f|^{2}e^{2(n-1)f}.\end{split}$

Submitting this into (3.24) with small $\epsilon$ we get

(3.26)
$\begin{split}\int_{\Omega}\eta^{s-2}|f_{0}|^{2}e^{2(n-1)f}\lesssim&\,\epsilon
r^{2}\int_{\Omega}\eta^{s}\mathcal{M}+\int_{\Omega}\eta^{s-2}|\partial
f|^{4}e^{2(n-1)f}\\\ \,&+\int_{\Omega}\eta^{s-2}|\partial
f|^{2}e^{(2n+p)f}+\frac{1}{r^{2}}\int_{\Omega}\eta^{s-4}|\partial
f|^{2}e^{2(n-1)f}.\end{split}$

This is just (3.12).∎

$\mathbf{Proof\,\,of\,\,lemma\,\,\ref{lem-2}}$

Multiply both sides of the equation (2.3) by $-\eta^{s}e^{2(n+1+p)f}$ and
integrate over $\Omega$ we have

(3.27)
$\begin{split}n\int_{\Omega}\eta^{s}ge^{2(n+1+p)f}=&-\int_{\Omega}\eta^{s}f_{\alpha\overline{\alpha}}e^{2(n+1+p)f}\\\
=&2(n+1+p)\int_{\Omega}\eta^{s}|\partial f|^{2}e^{2(n+1+p)f}\\\
\,&+s\int_{\Omega}\eta^{s-1}f_{\alpha}\eta_{\overline{\alpha}}e^{2(n+1+p)f}.\end{split}$

Using (3.7) and arranging the terms yields

(3.28)
$\begin{split}\int_{\Omega}\eta^{s}e^{(2n+4+3p)f}\lesssim&\int_{\Omega}\eta^{s}|\partial
f|^{2}e^{2(n+1+p)f}+\frac{1}{r}\int_{\Omega}\eta^{s-1}|\partial
f|e^{2(n+1+p)f}\\\ \lesssim&\int_{\Omega}\eta^{s}|\partial
f|^{2}e^{2(n+1+p)f}+\frac{1}{r^{2}}\int_{\Omega}\eta^{s-2}e^{2(n+1+p)f},\end{split}$

where in the last step, the Cauchy-Schwarz inequality has been used, and this
is (3.13) as desired. ∎

##  4\. Proof of theorem 1.2

Before the proof of theorem 1.2, we give the following Harnack inequality,
which is a special case of that given by Capogna-Danielli-Garofalo (see
Theorem 3.1 in [5]),

######  Lemma 4.1.

Let $0\leq u\in C^{2}(\Omega)$ satisfies

(4.1) $\triangle_{\mathbb{H}^{n}}u+h(\xi)u=0\quad\text{in}\quad\Omega.$

with $h(\xi)\in L^{s}_{loc}(\Omega)$ for some $s>\frac{Q}{2}$. Then there
exist constants $C_{0}>0$, $r_{0}>0$, such that for any $B_{r}(\xi)$, with
$B_{4r}(\xi)\subset\Omega$, and $r<r_{0}$,

(4.2) $\max_{B_{r}(\xi)}u\leq C_{0}\min_{B_{r}(\xi)}u.$

$\mathbf{Proof\,\,of\,\,Theorem\,\,\ref{Thm2}}.$

Rewrite the equation (1.1) as

(4.3) $\triangle_{\mathbb{H}^{n}}u+h(\xi)u=0\quad\text{in}\quad
B_{1}(0)\backslash\\{0\\},$

with $h(\xi)=2n^{2}u^{q-1}$. For any $\xi_{0}\in B_{1}\backslash\\{0\\}$, take
$r=\frac{1}{4}|\xi_{0}|$. Denote $\big{|}B_{r}(\xi_{0})\big{|}$ the volume of
the ball $B_{r}(\xi_{0})$. Using the estimate (1.3) we have

(4.4)
$\int_{B_{r}(\xi_{0})}h^{s}=(2n^{2})^{s}\int_{B_{r}(\xi_{0})}u^{3q-q^{*}}\leq
C\,r^{Q-2\times\frac{3q-q^{*}}{q-1}}.$

with $s=\frac{3q-q^{*}}{q-1}>\frac{Q}{2}$ for $1<q<q^{*}$. This implies
$h(\xi)\in L^{s}_{loc}(\Omega)$ for some $s>\frac{Q}{2}$ and hence $u$
satisfies the Harnack inequality (4.2). So for $|\xi_{0}|$ small enough,
combining this Harnack inequality with (1.3) we finally obtain

(4.5)
$\frac{1}{C}|\xi_{0}|^{Q}u(\xi_{0})^{3q-q^{*}}\leq|B_{r}(\xi_{0})|\Big{[}\frac{u(\xi_{0})}{C_{0}}\Big{]}^{3q-q^{*}}\leq\int_{B_{r}(\xi_{0})}u^{3q-q^{*}}\leq
C\,r^{Q-2\times\frac{3q-q^{*}}{q-1}}.$

This implies (1.2) and the proof of theorem 1.2 goes to the end.∎

######  Remark 4.2.

In the following paper [14], we shall generalize this methods to a class
semilinear elliptic equation on CR manifold and get a rigidity results.

Acknowledgement Partial research of the second author was done while he was
visiting The Chinese University of Hong Kong. He would like to thank the
Institute of Mathematical Sciences in The Chinese University of Hong Kong for
its warm hospitality.

## References

  * [1] H. Berestycki, I. Capuzzo-Dolcetta & L. Nirenberg, Superlinear indefinite elliptic problems and nonlinear Liouville theorems, Topological Methods in Nonlinear Analysis, 4 (1995), 59-78.
  * [2] M. F. Bidaut-Veron & S. Pohozaev, Nonexistence results and estimates for some nonlinear elliptic problems, J. Anal. Math. 84 (2001), 1-49.
  * [3] I. Birindelli, I. Capuzzo-Dolcetta & A. Cutri, Liouville theorems for semilinear equations on the Heisenberg group, Ann. Inst. Henri Poincaré-Analyse non linéaire, 14 (1997), 295-308.
  * [4] I. Birindelli & J. Prajapat, Nonlinear Liouville theorems in the Heisenberg group via the moving plane, Comm. Partial Differential Equations, 24 (1999), 1875-1890.
  * [5] Luca Capogna, Donatella Danielli & Nicola Garofalo, An embedding theorem and the harnack inequality for nonlinear subelliptic equations, Communications in Partial Differential Equations, 18:9-10 (1993), 1765-1794.
  * [6] N. Garofalo & D. Vassilev, Symmetry properties of positive entire solutions of Yamabe type equations on the groups of Heisenberg type, Duke Math. J., 106  (2001), 411-448.
  * [7] N. Gamara, CR Yamabe conjecture, the case n = 1, J. Eur. Math. Soc., 3 (2001), 105-137.
  * [8] N. Gamara Abdelmoula & R. Yacoub: CR Yamabe conjecture. The case conformally flat. Pac. J. Math., 201 (2001), No. 1, 121-175.
  * [9] B. Gidas & J. Spruck, Global and local behavior of positive solutions of nonlinear elliptic equations, Comm. Pure Appl. Math. 34 (1981), 525-598.
  * [10] D.S. Jerison & J.M. Lee, The Yamabe Problem on CR manifolds, J. Differential Geom. 25 (1987), 167-197.
  * [11] D. Jerison & J. M. Lee, Extremals for the Sobolev inequality on the Heisenberg group and the CR Yamabe problem, J. Amer. Math. Soc., 1 (1988), 1-13.
  * [12] D. Jerison & J. M. Lee, Pseudo-Einstein structures on CR manifolds, Amer. J. Math., 110 (1988), 157-178.
  * [13] D. Jerison & J.M. Lee: Intrinsic CR normal coordinates and the CR Yamabe problem. J. Diff. Geometry, 29 (1989), 303-343.
  * [14] X.N. Ma & Q.Z. Ou: Rigidity results for a class semilinear elliptic on compact CR manifold, preprint.
  * [15] M. Obata. The conjecture on conformal transformations of Riemannian manifolds, J. Differential Geom., 6 (1971), 247-258.
  * [16] X. Wang, On a remarkable formula of Jerison and Lee in CR geometry, Math. Res. Lett. 22 (2015), no. 1, 279-299.
  * [17] L. Xu, Semi-linear Liouville theorems in the Heisenberg group via vector field methods, J. Diff. Eq., 247 (2009), 2799-2820.