# Smooth Anosov Katok Diffeomorphisms With Generic Measure

Divya Khurana

###### Abstract

We construct a plethora of Anosov-Katok diffeomorphisms with non-ergodic
generic measures and various other mixing and topological properties. We also
construct an explicit collection of the set containing the generic points of
the system with interesting values of its Hausdorff dimension.

##  1 Introduction

In 1970 Anosov and Katok introduced the so called approximation by conjugation
method (also known as the Anosov-Katok or the AbC method) to construct
examples of transformations satisfying a pre-specified set of topological
and/or measure theoretic properties. In the realm of smooth (or in some cases,
real-analytic or even symplectic) zero entropy diffeomorphisms, this technique
till date remains one of the rare methods that one can use to explore the
possibility of the existence of diffeomorphisms satisfying such a set of
properties. Such transformations or diffeomorphisms often are important in
their own right. However, more interestingly, in recent years, there have been
situations where they have been able to exhibit connections, such as that of
rotation number at the boundary with the dynamical behaviour of a
diffeomorphism [FS]. This method has gained further momentum with the body of
work produced by Foreman and Weiss [FW04],[FW1] establishing anti-
classification theorems for smooth diffeomorphisms.

In this article, we wish to explore the construction of various types of
Anosov-Katok diffeomorphisms which supports non-ergodic generic measures. For
a probability preserving dynamical system $(M,\mathcal{B},\mu,T)$, we define
the set of $\mu$-generic points

$\displaystyle L_{\mu}=\\{x\in
M:\lim_{n\to\infty}\frac{1}{n}\sum_{i=0}^{n}f(T^{i}x)=\int_{X}fd\mu\;\forall\;f\in
C_{c}(M)\\}$

where $C_{c}(M)$ is the set of all compactly supported real valued continuous
functions. The measure $\mu$ is called a generic measure if
$L_{\mu}\neq\emptyset$. The celebrated Birkhoff ergodic theorem asserts that
$\mu(L_{\mu})=1$ for an $\mu$-ergodic transformation.

There has been a considerable amount of interest regarding the existence of
generic measures, particularly in the realm of interval exchange
transformations. Chaika and Masur [Cha] showed that there exists a minimal
non-uniquely ergodic interval exchange transformation on $6$ intervals with
$2$ ergodic measures, which also has a non-ergodic generic measure. Later, Cyr
and Kra [Cy19] found a criterion for establishing upper bounds on the number
of distinct non atomic generic measures for subshifts based on complexity, and
as a consequence, they showed that for $k>2$, a minimal exchange of $k$
intervals has at most $k-2$ generic measures. On the other hand, Gelfert and
Kwietniak [Gel18] gave an example of a topologically mixing subshift that can
have exactly two ergodic measures, none of whose convex combination is
generic.

Anosov and Katok, in [AK], constructed examples of smooth measure preserving
diffeomorphism, which is weakly mixing in the space
$\mathcal{A}(M)=\overline{\\{h\circ S_{t}\circ
h^{-1}:t\in\mathbb{T}^{1},h\in\text{Diff}^{\infty}(M,\mu)\\}}^{C^{\infty}}$,
on any manifold admitting a non-trivial $\mathbb{T}^{1}$ action. Later Fayad
and Saprykina produced the smooth weakly mixing diffeomorphism in the
restricted space $\mathcal{A}_{\alpha}(M)=\overline{\\{h\circ S_{\alpha}\circ
h^{-1}:h\in\text{Diff}^{\infty}(M,\mu)\\}}^{C^{\infty}}$ for any Liouville
number $\alpha$, i.e. for each $n$, there exist integers $p>0$ and $q>1$ such
that $0<|\alpha-\frac{p}{q}|<\frac{1}{q^{n}}$. Both the above constructions
are built using the approximation by conjugation method: The diffeomorphism is
obtained as the limits of sequences $T_{n}=H_{n}S_{\alpha_{n+1}}H_{n}^{-1}$
where $\alpha_{n+1}\in\mathbb{Q}$ and $H_{n}=h_{1}\ldots h_{n}$ where $h_{n}$
is a measure preserving diffeomorphism satisfying $S_{{\alpha}_{n}}\circ
h_{n}=h_{n}\circ S_{{\alpha}_{n}}$. For the diffeomorphism $T_{n}$ to converge
in the space $\mathcal{A}_{\alpha}(M)$, for any $\alpha$, it needs
construction of more explicit conjugation maps $h_{n}$ and very precise norm
estimates and is generally difficult when compared to convergence in the space
$\mathcal{A}(M)$.

In general, a uniquely ergodic measure preserving transformation on a compact
metric space is minimal on the support of the measure, but the converse is not
true. Markov produced the first counterexample. Further, Windsor, in ([Win]),
constructed a minimal measure preserving diffeomorphism in
$\mathcal{A}_{\alpha}(M)$ with the finite number of ergodic measures.
Afterwards, Banerjee and Kunde([BaKu]) produced a similar result for the real
analytic category on $\mathbb{T}^{2}$.

It is well known that the Anosov-Katok constructions allow great flexibility,
and we present several results in this article that explore the existence of
non-ergodic generic measures in this setup. We also note that our
constructions will be smooth and, in some cases, even real-analytic. Hereby we
extend the above results to produce more compelling examples with different
measure-theoretical and topological dynamical properties.

######  Theorem A.

For any natural number $r$, and any Liouvillian number $\alpha$, there exists
a minimal $T\in\mathcal{A}_{\alpha}(\mathbb{T}^{2})$ such that the Lebesgue
measure is a generic measure for $T$, and there exists $r$ absolutely
continuous w.r.t. to Lebesgue measures $\mu_{1},\mu_{2},\ldots,\mu_{r}$ such
that $T$ is weakly mixing w.r.t. each of these measures.

In fact the approximation by conjugation method on $\mathbb{T}^{2}$ offers
enough flexibility to repeat the construction using block-slide type of maps
([BaKu], Theorem E) and get the result in the analytic set-up.

######  Theorem B.

For any natural number $r$, there exist a minimal real-analytic
$T\in\text{Diff }^{\omega}(\mathbb{T}^{2},\mu)$ constructed by the
approximation by conjugation method, such that the Lebesgue measure is a
generic measure for $T$, and there exists $r$ absolutely continuous w.r.t. to
Lebesgue measures $\mu_{1},\mu_{2},\ldots,\mu_{r}$ such that $T$ is weakly
mixing w.r.t. each of these measures.

One of this paper’s objectives is to examine the generic points and try to
estimate their size. Here, instead of measuring the set of generic points by
the Lebesgue measure, we produce more interesting values of their Hausdorff
dimension.

######  Theorem C.

There exist a smooth diffeomorphism $T\in\text{Diff
}^{\infty}({\mathbb{T}}^{2},\mu)$ constructed by the approximation of
conjugation method, such that the set $\mathrm{B}$ containing all the generic
points of $T$ has

$\log_{3}2\leq\text{dim}_{H}(\mathrm{B})\leq 1+\log_{3}2,$

and $\mu(\mathrm{B})=0.$

We can generalize the above result by choosing the generalized Cantor set
(p-series Cantor set [CC]) instead of the Cantor set in the above setup and
construct the generic sets of different Hausdorff dimensions as

######  Theorem D.

For any $1<\alpha<2,$ there exist a smooth diffeomorphism $T\in\text{Diff
}^{\infty}(\mathbb{T}^{2},\mu)$ constructed by the approximation of
conjugation method, such that the set $\mathrm{B}_{\alpha}$ containing all the
generic points of $T$ has

$\alpha-1\leq\text{dim}_{H}(\mathrm{B}_{\alpha})\leq\alpha,$

and $\mu(\mathrm{B}_{\alpha})=0.$

In $\cite[cite]{[\@@bibref{}{ZBR}{}{}]},$ Theorem- 2.3.1, the author presented
a variational type formula for the full-shift on an alphabet of two symbols
$(\Omega,\sigma)$. But in our set-up, it appears that this theorem does not
hold. For example, if $f:\mathbb{T}^{2}\to\mathbb{R}$ is any continuous
function and $\alpha=\int fd\mu$ with $<1\alpha<2$ and $\mu$ being the usual
Lebesgue measure, then the Hausdorff dimension of $E_{f}(\alpha)$ is greater
than $zero$ (see theorem D) where $E_{f}(\alpha)=\\{x\in\mathbb{T}^{2}\
:\lim\limits_{N\longrightarrow\infty}\frac{1}{N}\sum\limits_{n=1}^{N}f(T^{n}x)=\alpha\\}$.
Whereas, according to the theorem A in [FFW] and theorem 2.3.1 in [ZBR], this
number should be zero as topological entropy and all measure theoretic
entropy, in our case, is always zero.

For an ergodic transformation, the set of non-generic points has measure zero
but can have more exciting values of its Hausdorff dimension. Precisely, one
can obtain the analogue result of theorem D for the set of non-generic points
for the case of ergodic measure with the appropriate choice of combinatorics.

######  Theorem E.

For any $1<\alpha<2,$ there exist a smooth ergodic diffeomorphism
$T\in\text{Diff }^{\infty}(\mathbb{T}^{2},\mu)$ constructed by the
approximation of conjugation method, such that the set $\mathrm{B}_{\alpha}$
containing all the non-generic points of $T$ has

$\alpha-1\leq\text{dim}_{H}(\mathrm{B}_{\alpha})\leq\alpha,$

and $\mu(\mathrm{B}_{\alpha})=0.$

######  Remark 1.

The diffeomorphism produced in the above Theorem-C, D, and E could be made
minimal by following the same construction as in theorem A.

##  2 Preliminaries

This section explains some basic definitions and standard techniques that we
use throughout the paper.

###  2.1 Basics of ergodic theory

Consider $(X,d)$ be a $\sigma$-compact metric space, $\mathcal{B}$ is a
$\sigma$ algebra, $\mu$ is a measure and $T:X\longrightarrow X$ is a measure
preserving transformation(mpt) i.e. $\mu(T^{-1}(A))=\mu(A)\ \forall
A\in\mathcal{B}$.

######  Definition 2.1.

A mpt $(X,\mathcal{B},\mu,T)$ is called ergodic if every invariant set
$E\in\mathcal{B}$ satisfies $\mu(E)=0\ or\ \mu(X\backslash E)=0$. We say $\mu$
is ergodic measure.


# Multi-Task Learning of Query Intent and Named Entities using Transfer
Learning

Shalin Shah<EMAIL_ADDRESS>
Ryan Siskind<EMAIL_ADDRESS>

AI Sciences
Target Corporation
Sunnyvale, CA 94086, USA

###### Abstract

Named entity recognition (NER) has been studied extensively and the earlier
algorithms were based on sequence labeling like Hidden Markov Models (HMM) and
conditional random fields (CRF). These were followed by neural network based
deep learning models. Recently, BERT has shown new state of the art accuracy
in sequence labeling tasks like NER. In this short article, we study various
approaches to task specific NER. Task specific NER has two components -
identifying the intent of a piece of text (like search queries), and then
labeling the query with task specific named entities. For example, we consider
the task of labeling Target store locations in a search query (which could be
entered in a search box or spoken in a device like Alexa or Google Home).
Store locations are highly ambiguous and sometimes it is difficult to
differentiate between say a location and a non-location. For example, "pickup
my order at orange store" has "orange" as the store location, while "buy
orange at target" has "orange" as a fruit. We explore this difficulty by doing
multi-task learning which we call global to local transfer of information. We
jointly learn the query intent (i.e. store lookup) and the named entities by
using multiple loss functions in our BERT based model and find interesting
results.

_Keywords_ : named entity recognition, multi-task learning, BERT, deep
learning, sequence labeling, task specific NER

##  1 Introduction

We use BERT [1] for multi-task learning. We jointly learn the intent of the
search query (store lookup) and also the named entities (store locations). We
find that this method performs better in terms of precision (which is more
important than recall). The false positive rate improved significantly through
the global query intent loss function combined with a tagging loss function.
As far as we know, this is the first attempt to jointly learn task specific
NER along with a text classification loss function. We found one other piece
of work that does something similar using LSTM networks [2]. A good overview
of multi-task learning can be found in [3] and [4]. This page does something
very similar using BERT but its not completely clear on whether their work
uses multi-task learning.

Also, a distinguishing feature of our work is that each word has its own
intent, rather than just one binary output for the intent (which appears
similar to the ideas in [5]). Several intents can be learnt together with the
respective named entities, like navigational store hours lookup (store name is
the entity), product lookup (the product is the entity), add to cart (product
is the entity), store pickup (the store is the entity), page navigation (the
page location is the entity, like navigating to a category page etc.), product
styles (in which the style and the product would be the entities). All intents
have their own token level loss functions for intent along with a common named
entity recognition engine in a multi-task setting (This page has several
interesting ideas about multi-task learning. In particular they use the term
negative transfer in which one loss function cannibalizes the effect of the
other and both then do poorly).

BERT [1] is a very effective pre-trained model which can be fine tuned for
various tasks. BERT learns a masked language model in which a few words of a
piece of text are masked and the model learns how to predict them. It is
bidirectional and uses attention [5], which includes interactions of a word
with other words following it or preceding it. By adding task specific loss
functions and learning on a comparatively smaller set of data, the model can
be fine tuned. For example, the final layer of the model could be a Softmax
layer which can do text classification or tagging. Or it could possibly be a
linear layer.

We use BERT with two added loss functions - one that learns NER tags and the
other that learns a classification problem. The NER task learns store
locations and the classification task learns the query intent (i.e. store
lookup). More in section 4.

##  2 Named Entity Recognition and Sequence Labeling

Sequence labeling is a task in which a piece of text has to be tagged with
information like names, parts of speech tags, and other tags. Named entity
recognition could be a task that follows POS tagging and NP chunking, but it
does not have to be.

The earliest NER models were based on sequence labeling, like hidden Markov
models and conditional random fields. The NLTK package [6] and the Stanford
NLP package [7] both use CRF for sequence tagging. SpaCy [8] [9] uses
convolutional neural networks and recurrent neural networks to do named entity
recognition.

Recently, BERT has shown to obtain state of the art (SOTA) results on named
entity tasks. The HuggingFace library [10] provides a transformers python
package which includes pre-trained models for various language tasks like
named entity recognition. We use this package in our work along with BERT [1]
and pyTorch [11].

This git page does a comprehensive survey of SOTA algorithms for various NLP
tasks including named entity recognition.

##  3 Multi-task Learning

Multi-task learning has multiple objective functions which makes it a multi-
objective learning problem. The most common approach to multi-objective
learning is to just do a convex combination of the multiple objectives with
weights for each objective while sharing the neural network representation for
each objective. This has shown to improve generalization as described in this
book [12].

It is natural that learning the intent of a query and then doing NER will
result in a hierarchical model which will reduce the false positive rate.
Learning the query intent jointly with NER will change the representation
(neural network) for NER tagging as well and might result in a better
precision.

In a multi-task or a multi-objective problem, the inductive bias of the
classifier is shared between multiple objectives [12]. This results in an
amortized classifier, which, for multiple-objectives, only differs in the
final layers. This results in an improved performance because of the global to
local transfer.

In our case, the global information for a query (the query intent), and the
word specific local information (the NER labels) are jointly learnt, while
sharing the entire BERT neural network model.

##  4 The NER Algorithm

As figures 1 and 2 show, there are two distinct loss functions which share a
common neural network.

Figure 1: NER and Intent Example

One loss function is the Softmax loss which classifies each word of a query
into an entity type (store location name). This is a commonly used loss
function for multi-label classification with known derivatives. The Softmax
function is preceded by linear nodes and the Softmax is given as under:

$Softmax_{i}=\frac{exp(x_{i})}{\sum_{j=1}^{n}exp(x_{j})}$

The other loss function uses shared labels. If the number of words in a query
is $n$, and if the intent of the query is store lookup, there are $n$ ones in
the target labels. If the intent is not a store lookup, there are $n$ zeros in
the target labels. The reason for having $n$ identical labels is that during
inference, each word would have its own intent (which appears similar to the
ideas in [5]). But the intent is shared by all the words of a query and it
helps learning and improves the precision as well as the F-1 score, as table 1
shows. We compare our algorithm BERT-Multitask with BERT-base for the store
lookup NER task.

So, there are two distinct forward and backward passes for the two cost
functions. The passes are done separately and sequentially for the two losses.
The learning rate for the global intent loss is reduced by a factor of $0.1$.

As we noted, the intent loss is shared during the backward pass, but during
the forward pass, each word of the query has its own intent score. In our
opinion, this facilitates learning and as the results show, it improves the
precision and F-1 score.

Figure 2: Joint Learning of Intent and NER

##  5 Results

The results of our algorithm as compared to BERT-base are shown in table 1.
Our algorithm is able to improve precision by $3\%$ by just doing an argmax.
The goal of our algorithm is indeed, to improve precision and reduce false
positives. At Target corporation, we deal with millions of distinct
search/voice queries and a precision increase of $3\%$ is considered very
significant and desirable.

Moreover, in our opinion, with algorithms improving the SOTA of NER constantly
being developed, and with the SOTA reaching the $90^{\prime}s$, a $3\%$
increase in precision is difficult to achieve.

As seen in figure 1, the intent scores for the first query ("orange target
store") are high for all three words. And for the second query ("buy an
orange"), the intent scores are all negative. It is worth noting that the
score for "orange" is significantly different between the two queries. So,
what ends up happening is that if the intent is "store lookup", the scores for
the entire query will be higher, but the interactions between intent and NER
will push the scores for entities even higher. Working together, the intent
and NER losses improve the results significantly.

We run both BERT-base and BERT-Multitask with the same random seed, to avoid
variations because of the random number generators.

Table 1: Results on a 20% held out test set (same random seed) Algorithm |  Precision |  Recall |  F-1 Score
---|---|---|---
BERT-Multitask |  94% |  79% |  0.86
BERT-base |  91% |  80% |  0.85

##  6 Conclusion

In this paper, we presented a multi-tasking BERT based named entity
recognition algorithm which does task specific NER. Task specific NER, in our
domain, means identifying store lookup intent with the store name. By using
two loss functions, and different forward and backward passes, we find that
the precision is improved.

Moreover, by having a separate intent score for each word in the query,
thresholds can be computed (on a validation set) and classification can be
done based on these threshold. But we haven’t explored this yet.

Our algorithm is applicable to any task in which text classification precedes
tagging. For instance, learning the intent of gifting and then tagging the
gift in the query is another important application. There are numerous such
applications.

Future work could include using the generated thresholds and see if the
precision improves further. Another opportunity is to experiment with hyper-
parameters (like the weight of the intent loss).

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [2] Mingbo Ma, Kai Zhao, Liang Huang, Bing Xiang, and Bowen Zhou.  Jointly trained sequential labeling and classification by sparse attention neural networks.  arXiv preprint arXiv:1709.10191, 2017.
  * [3] Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  arXiv preprint arXiv:1706.05098, 2017.
  * [4] Yu Zhang and Qiang Yang.  A survey on multi-task learning.  arXiv preprint arXiv:1707.08114, 2017.
  * [5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [6] Edward Loper and Steven Bird.  Nltk: the natural language toolkit.  arXiv preprint cs/0205028, 2002.
  * [7] Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky.  The stanford corenlp natural language processing toolkit.  In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55–60, 2014.
  * [8] Matthew Honnibal and Ines Montani.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear, 2017.
  * [9] Bhargav Srinivasa-Desikan.  Natural Language Processing and Computational Linguistics: A practical guide to text analysis with Python, Gensim, spaCy, and Keras.  Packt Publishing Ltd, 2018.
  * [10] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  ArXiv, pages arXiv–1910, 2019.
  * [11] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in neural information processing systems, pages 8026–8037, 2019.
  * [12] Rich Caruana.  Multitask learning.  Machine learning, 28(1):41–75, 1997.
  * [13] Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang.  Named entity recognition through classifier combination.  In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003, pages 168–171, 2003.
  * [14] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  In Advances in Neural Information Processing Systems, pages 527–538, 2018.