propose a local differentially private model without requiring specific
trajectory semantics. It is evaluated with the Gowalla LBSN dataset as well as
a taxi dataset and outperforms Ngram based on query error, frequent pattern
similarity and distance error.

##  6\. Discussion and Conclusion

Generating synthetic urban mobility data holds much potential, but it is a
complex task with no one-size-fits-all solution that could serve this
heterogeneous field. This is also reflected in the diversity of the reviewed
models. However, the absence of a clear definition for high utility in
synthetic datasets makes model comparison challenging. There are no
straightforward downstream tasks suited for evaluation; instead, mobility data
is used for a variety of non-standardized analyses and models that pose
different requirements to the data, reflected by different statistical
distributions, potentially entailing complex interactions. Thus, a single
valid definition of high utility can hardly be established. In addition,
varying dataset formats make standardized evaluations even more difficult. For
example, a temporal representation is either omitted completely, modeled as a
start timestamp, or based on fixed time windows.

It is likely impossible to create a model that preserves all distributions
equally well while also maintaining privacy. We suggest that researchers
developing new models clearly state their intended use cases and provide
meaningful evaluations covering all relevant mobility characteristics with at
least one measure each. Details on implementations of utility evaluations
would be desirable, such as utilized grid resolutions for spatial aggregations
and histogram bin sizes for all other aggregations. Such choices are often not
stated or can only implicitly be deduced from the implementation. This
facilitates the results’ interpretation, as the utility can easily be
increased by the choice of a coarser resolution (for example, the spatial
distribution might easily be maintained well on a $2km$ x $2km$ grid but not
nearly as well on a $250m$ x $250m$ grid). Without respective details,
practitioners do not have sufficient information to decide whether the
reported utility satisfies their needs. A granularity standard can hardly be
given in general as it highly depends on the intended use case. Also, the
provision of the model’s source code and generated synthetic datasets would
facilitate the evaluation and comparison with existing models. Additionally,
this field of research would benefit from real-world test cases of
organizations that provide data and use synthesized trajectories for their
actual purposes, which would reveal shortcomings in practical settings. This
could also offer valuable insights into the necessary levels of similarity
across different mobility characteristics in varying contexts.

While there are specific limitations for each model, there are (so far) also
global limitations: many models entirely omit the temporal dimension. If
included, it is only considered as the time of day (potentially also the day
of the week). This means that the data does not reflect information on
mobility behavior differences over the course of the year, e.g., seasonal
differences, event-related mobility behavior (e.g., a sports event), or
weather-related behavior adaptions.

None of the publications provides evaluations in relation to the dataset size.
This is especially relevant for deep learning-based models that commonly need
a lot of data to properly learn latent distributions. Generally, not all
publications provide sufficient information about the origin, format, size,
and availability of the utilized datasets. Potential biases or dataset-
specific characteristics are rarely stated by authors, even though they might
impact the generalizability of the results. More careful considerations of
trajectory semantics, such as the type (i.e., waypoints or staypoints) or the
length of trajectories, and how many trajectories a user contributes, would be
desirable, as this information is crucial for determining the suitability of
models for specific contexts.

Privacy is one of the main motivations for synthetic data generation. Yet,
many models lack privacy guarantees or privacy evaluations. As recent
approaches tend to rely more on deep learning models, they also tend to omit
privacy guarantees (see Figure 4). This might not be surprising, as
differentially private deep learning has only recently been provided as part
of open-source libraries (e.g., Tensorflow Privacy in
2019888https://github.com/tensorflow/privacy). Two deep learning models
already make use of differentially private stochastic gradient descent methods
(Yu, 2019; Lestyán et al., 2022) and likely more models will follow.

Differential privacy is no silver bullet and it comes with certain
limitations. Protecting the privacy of individuals leads to a utility
reduction of data from minority groups (Pujol and Machanavajjhala, 2021). For
example, a dataset might only include a small share of children who have a
different mobility behavior than, say, commuters, who typically constitute a
large subpopulation in mobility data. In such a case, a differentially private
synthetic dataset might well resemble commuters’ mobility behavior but not the
children’s movements. It should be noted that deep learning models without any
additional privacy guarantee mechanisms likely do not properly learn minority
group behavior either (Blanzeisky and Cunningham, 2021). Additionally,
differential privacy provides guarantees on the level of items (trips in this
context) or users, and not on the level of groups. For example, the well-known
incident of the released aggregated Strava data which revealed secret military
bases (Hern, 2018) would not have been prevented by differential privacy.
Thus, the publication of differentially private synthetic data still requires
careful thought about the information that is still entailed.

In summary, for the contribution of meaningful new models, we consider the
following aspects as valuable parts of the respective publication: Information
about applicable use cases; a detailed description of the input and output
data format of the proposed model; datasets and configurations of measures
used for evaluation; openly available source code as well as the created
synthetic dataset(s). We see the potential for future research in the
following aspects: A comprehensive comparison of different models based on a
heterogeneous set of datasets and similarity measures would foster use case
specific recommendations. Standardized benchmarking tools, such as the
Synthetic Data Vault (Patki et al., 2016) that provides a set of standard
datasets and evaluations for tabular data or time series synthetic data
generation models, could contribute to better comparability, accelerate the
development of new models and create confidence in practitioners’ decision
processes. We have developed a Python package dp_mobility_report999Package
code repository: https://anonymous.4open.science/r/dp_mobility_report-A35C/
capable of computing a comprehensive set of measures for various mobility
characteristics (so far only considering staypoints), allowing custom
configurations of histogram bin sizes and tessellations. This could be
extended to provide standard test cases comprising datasets and
configurations. Moreover, comparing deep learning and traditional methods
would provide meaningful insights, with special regard to the achieved utility
concerning the input dataset size. Finally, recent developments in the fusion
of differential privacy and deep learning open the way for further research.

## References

  * (1)
  * uni (2008) 2008\.  Global Positioning System Standard Positioning Service Performance Standard - 4th Edition.
  * Abadi et al. (2016) Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016.  Deep Learning with Differential Privacy. In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ _(CCS ’16)_. Association for Computing Machinery, New York, NY, USA, 308–318.  https://doi.org/10.1145/2976749.2978318
  * Aktay et al. (2020) Ahmet Aktay, Shailesh Bavadekar, Gwen Cossoul, John Davis, Damien Desfontaines, Alex Fabrikant, Evgeniy Gabrilovich, Krishna Gadepalli, Bryant Gipson, Miguel Guevara, Chaitanya Kamath, Mansi Kansal, Ali Lange, Chinmoy Mandayam, Andrew Oplinger, Christopher Pluntke, Thomas Roessler, Arran Schlosberg, Tomer Shekel, Swapnil Vispute, Mia Vu, Gregory Wellenius, Brian Williams, and Royce J. Wilson. 2020\.  Google COVID-19 Community Mobility Reports: Anonymization Process Description (Version 1.1).  _arXiv: 2004.04145_ (2020).
  * Alatrista-Salas et al. (2022) Hugo Alatrista-Salas, Peter Montalvo-Garcia, Miguel Nunez-del-Prado, and Julián Salas. 2022\.  Geolocated Data Generation and Protection Using Generative Adversarial Networks. In _Modeling Decisions for Artificial Intelligence: Proceedings of the 19th International Conference, MDAI 2022, Sant Cugat, Spain, August 30 – September 2, 2022_. Springer-Verlag, Berlin, Heidelberg, 80–91.  https://doi.org/10.1007/978-3-031-13448-7_7
  * Anda et al. (2021) Cuauhtemoc Anda, Sergio A. Ordonez Medina, and Kay W. Axhausen. 2021\.  Synthesising Digital Twin Travellers: Individual Travel Demand from Aggregated Mobile Phone Data.  _Transportation Research Part C: Emerging Technologies_ 128 (2021), 103118\.  https://doi.org/10.1016/j.trc.2021.103118
  * Assefa et al. (2021) Samuel A. Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E. Tillman, Prashant Reddy, and Manuela Veloso. 2021.  Generating Synthetic Data in Finance: Opportunities, Challenges and Pitfalls. In _Proceedings of the First ACM International Conference on AI in Finance_ (New York, New York) _(ICAIF ’20)_. Association for Computing Machinery, New York, NY, USA, Article 44, 8 pages.  https://doi.org/10.1145/3383455.3422554
  * Badu-Marfo et al. (2022) Godwin Badu-Marfo, Bilal Farooq, and Zachary Patterson. 2022\.  Composite Travel Generative Adversarial Networks for Tabular and Sequential Population Synthesis.  _IEEE Transactions on Intelligent Transportation Systems_ 23, 10 (2022), 17976–17985.  https://doi.org/10.1109/TITS.2022.3168232
  * Benaim et al. (2020) Anat Reiner Benaim, Ronit Almog, Yuri Gorelik, Irit Hochberg, Laila Nassar, Tanya Mashiach, Mogher Khamaisi, Yael Lurie, Zaher S. Azzam, Johad Khoury, Daniel Kurnik, and Rafael Beyar. 2020\.  Analyzing Medical Research Results Based on Synthetic Data and Their Relation to Real Data Results: Systematic Comparison From Five Observational Studies.  _JMIR Medical Informatics_ 8, 2 (2020), e16492.  https://doi.org/10.2196/16492
  * Benarous et al. (2022) Maya Benarous, Eran Toch, and Irad Ben-Gal. 2022.  Synthesis of Longitudinal Human Location Sequences: Balancing Utility and Privacy.  _ACM Transactions on Knowledge Discovery from Data_ (April 2022), 3529260\.  https://doi.org/10.1145/3529260
  * Berke et al. (2022) Alex Berke, Ronan Doorley, Kent Larson, and Esteban Moro. 2022.  Generating Synthetic Mobility Data for a Realistic Population with RNNs to Improve Utility and Privacy. In _Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing_ (Virtual Event) _(SAC ’22)_. Association for Computing Machinery, New York, NY, USA, 964–967.  https://doi.org/10.1145/3477314.3507230
  * Bindschaedler and Shokri (2016) Vincent Bindschaedler and Reza Shokri. 2016.  Synthesizing Plausible Privacy-Preserving Location Traces. In _2016 IEEE Symposium on Security and Privacy (SP)_. 546–563.  https://doi.org/10.1109/SP.2016.39
  * Bindschaedler et al. (2017) Vincent Bindschaedler, Reza Shokri, and Carl A. Gunter. 2017\.  Plausible Deniability for Privacy-Preserving Data Synthesis.  _Proc. VLDB Endow._ 10, 5 (jan 2017), 481–492.  https://doi.org/10.14778/3055540.3055542
  * Blanco-Justicia et al. (2022) Alberto Blanco-Justicia, Najeeb Moharram Jebreel, Jesús A. Manjón, and Josep Domingo-Ferrer. 2022\.  Generation of Synthetic Trajectory Microdata from Language Models. In _Privacy in Statistical Databases_ _(Lecture Notes in Computer Science)_ , Josep Domingo-Ferrer and Maryline Laurent (Eds.). Springer International Publishing, Cham, 172–187.  https://doi.org/10.1007/978-3-031-13945-1_13
  * Blanzeisky and Cunningham (2021) William Blanzeisky and Pádraig Cunningham. 2021.  Algorithmic Factors Influencing Bias in Machine Learning. In _Machine Learning and Principles and Practice of Knowledge Discovery in Databases_. Springer International Publishing, Cham, 559–574.
  * Brinkhoff (2002) Thomas Brinkhoff. 2002\.  A Framework for Generating Network-Based Moving Objects.  _GeoInformatica_ 6, 2 (2002), 153–180.  https://doi.org/10.1023/A:1015231126594
  * Bwambale et al. (2021) Andrew Bwambale, Charisma F. Choudhury, Stephane Hess, and Md. Shahadat Iqbal. 2021.  Getting the Best of Both Worlds: A Framework for Combining Disaggregate Travel Survey Data and Aggregate Mobile Phone Data for Trip Generation Modelling.  _Transportation_ 48, 5 (2021), 2287–2314.  https://doi.org/10.1007/s11116-020-10129-5
  * Cao and Li (2021) Chu Cao and Mo Li. 2021\.  Generating Mobility Trajectories with Retained Data Utility. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ (Virtual Event, Singapore) _(KDD ’21)_. Association for Computing Machinery, New York, NY, USA, 2610–2620.  https://doi.org/10.1145/3447548.3467158
  * Chekol and Fufa (2022) Ayele Gobezie Chekol and Marta Sintayehu Fufa. 2022.  A Survey on next Location Prediction Techniques, Applications, and Challenges.  _EURASIP Journal on Wireless Communications and Networking_ 2022, 1 (March 2022), 29.  https://doi.org/10.1186/s13638-022-02114-6
  * Chen et al. (2012a) Rui Chen, Gergely Acs, and Claude Castelluccia. 2012a.  Differentially Private Sequential Data Publication via Variable-Length n-Grams. In _Proceedings of the 2012 ACM Conference on Computer and Communications Security - CCS ’12_. 638–649.  https://doi.org/10.1145/2382196.2382263
  * Chen et al. (2012b) Rui Chen, Benjamin C.M. Fung, Bipin C. Desai, and Nériah M. Sossou. 2012b.  Differentially Private Transit Data Publication: A Case Study on the Montreal Transportation System. In _Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (Beijing, China) _(KDD ’12)_. Association for Computing Machinery, New York, NY, USA, 213–221.  https://doi.org/10.1145/2339530.2339564
  * Chen et al. (2021) Richard J. Chen, Ming Y. Lu, Tiffany Y. Chen, Drew F. K. Williamson, and Faisal Mahmood. 2021\.  Synthetic Data in Machine Learning for Medicine and Healthcare.  _Nature Biomedical Engineering_ 5, 6 (June 2021), 493–497.  https://doi.org/10.1038/s41551-021-00751-8
  * Chen et al. (2020) Si Chen, Anmin Fu, Jian Shen, Shui Yu, Huaqun Wang, and Huaijiang Sun. 2020\.  RNN-DP: A New Differential Privacy Scheme Base on Recurrent Neural Network for Dynamic Trajectory Privacy Protection.  _Journal of Network and Computer Applications_ 168 (2020), 102736\.  https://doi.org/10.1016/j.jnca.2020.102736
  * Chiesa and Taraglio (2022) Stefano Chiesa and Sergio Taraglio. 2022.  Traffic Request Generation through a Variational Auto Encoder Approach.  _Computers_ 11, 5 (2022), 71.  https://doi.org/10.3390/computers11050071
  * Cho et al. (2011) Eunjoon Cho, Seth A. Myers, and Jure Leskovec. 2011\.  Friendship and Mobility: User Movement in Location-Based Social Networks. In _Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (San Diego, California, USA) _(KDD ’11)_. Association for Computing Machinery, New York, NY, USA, 1082–1090.  https://doi.org/10.1145/2020408.2020579
  * Choi et al. (2021) Seongjin Choi, Jiwon Kim, and Hwasoo Yeo. 2021.  TrajGAIL: Generating Urban Vehicle Trajectories Using Generative Adversarial Imitation Learning.  _Transportation Research Part C: Emerging Technologies_ 128 (2021), 103091\.  https://doi.org/10.1016/j.trc.2021.103091
  * Creutzig (2021) Felix Creutzig. 2021\.  From Smart City to Digital Urban Commons: Institutional Considerations for Governing Shared Mobility Data.  _Environmental Research: Infrastructure and Sustainability_ 1, 2 (Sept. 2021), 025004.  https://doi.org/10.1088/2634-4505/ac0a4e
  * Culnane et al. (2019) Dr Chris Culnane, A/Prof Benjamin I. P. Rubinstein, and A/Prof Vanessa Teague. 2019\.  Stop the Open Data Bus, We Want to Get Off.  arXiv:1908.05004 [cs]
  * Dandekar et al. (2016) Ashish Dandekar, Stéphane Bressan, Talel Abdessalem, Huayu Wu, and Wee Siong Ng. 2016\.  Trajectory Simulation in Communities of Commuters. In _2016 International Conference on Advanced Computer Science and Information Systems (ICACSIS)_. 39–42.  https://doi.org/10.1109/icacsis.2016.7872718
  * Dash et al. (2019) Saloni Dash, Ritik Dutta, Isabelle Guyon, Adrien Pavao, Andrew Yale, and Kristin P. Bennett. 2019\.  Synthetic Event Time Series Health Data Generation.  https://doi.org/10.48550/ARXIV.1911.06411
  * de Montjoye et al. (2013) Yves-Alexandre de Montjoye, César A. Hidalgo, Michel Verleysen, and Vincent D. Blondel. 2013.  Unique in the Crowd: The Privacy Bounds of Human Mobility.  _Scientific Reports_ 3, 1 (March 2013), 1376\.  https://doi.org/10.1038/srep01376
  * de Transport de Montréal (2022) Société de Transport de Montréal. 1997-2022.  Société de Transport de Montréal.  https://www.stm.info/en
  * Deldar and Abadi (2020) Fatemeh Deldar and Mahdi Abadi. 2020.  Enhancing Spatial and Temporal Utilities in Differentially Private Moving Objects Database Release.  _International Journal of Information Security_ 20, 4 (July 2020), 511–533.  https://doi.org/10.1007/s10207-020-00516-5
  * Dilmegani (2022) Cem Dilmegani. 2022\.  Synthetic Data Generation: Techniques, Best Practices & Tools.  https://research.aimultiple.com/synthetic-data-generation/
  * Du et al. (2023) Yuntao Du, Yujia Hu, Zhikun Zhang, Ziquan Fang, Lu Chen, Baihua Zheng, and Yunjun Gao. 2023.  LDPTrace: Locally Differentially Private Trajectory Synthesis.  _arXiv_ (Feb. 2023), 13.  https://doi.org/10.48550/arXiv.2302.06180
  * Dwork and Roth (2013) Cynthia Dwork and Aaron Roth. 2013.  The Algorithmic Foundations of Differential Privacy.  _Foundations and Trends in Theoretical Computer Science_ 9, 3-4 (2013), 211–407.  https://doi.org/10.1561/0400000042
  * Emam et al. (2020) Khaled El Emam, Lucy Mosquera, and Richard Hoptroff. 2020\.  _Practical Synthetic Data Generation: Balancing Privacy and the Broad Availability of Data_.  O’Reilly Media, Inc.
  * Erdemir et al. (2020) Ecenaz Erdemir, Pier Luigi Dragotti, and Deniz Gündüz. 2020\.  Privacy-Aware Time-Series Data Sharing with Deep Reinforcement Learning.  _IEEE Transactions on Information Forensics and Security_ 16 (2020), 389–401.  https://doi.org/10.1109/TIFS.2020.3013200
  * Errounda and Liu (2019) Fatima Zahra Errounda and Yan Liu. 2019.  An Analysis of Differential Privacy Research in Location Data. In _2019 IEEE 5th Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing, (HPSC) and IEEE Intl Conference on Intelligent Data and Security (IDS)_. 53–60.  https://doi.org/10.1109/BigDataSecurity-HPSC-IDS.2019.00021
  * Feng et al. (2020) Jie Feng, Zeyu Yang, Fengli Xu, Haisu Yu, Mudan Wang, and Yong Li. 2020\.  Learning to Simulate Human Mobility. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 3426–3433.  https://doi.org/10.1145/3394486.3412862
  * Fiore et al. (2020) Marco Fiore, Panagiota Katsikouli, Elli Zavou, Mathieu Cunche, Françoise Fessant, Dominique Le Hello, Ulrich Aivodji, Baptiste Olivier, Tony Quertier, and Razvan Stanica. 2020\.  Privacy in Trajectory Micro-Data Publishing: A Survey.  _Transactions on Data Privacy_ 13 (2020), 91.  https://doi.org/hal-02968279
  * Gao et al. (2020) Song Gao, Jinmeng Rao, Yuhao Kang, Yunlei Liang, and Jake Kruse. 2020.  Mapping County-Level Mobility Pattern Changes in the United States in Response to COVID-19.  _SIGSPATIAL Special_ 12, 1 (June 2020), 16–26.  https://doi.org/10.1145/3404111.3404115
  * Ghane et al. (2020) Soheila Ghane, Lars Kulik, and Kotagiri Ramamohanarao. 2020\.  TGM: A Generative Mechanism for Publishing Trajectories With Differential Privacy.  _IEEE Internet of Things Journal_ 7, 4 (2020), 2611–2621.  https://doi.org/10.1109/jiot.2019.2943719
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Networks.  (2014).  arXiv:1406.2661
  * Gursoy et al. (2019) Mehmet Emre Gursoy, Ling Liu, Stacey Truex, and Lei Yu. 2019\.  Differentially Private and Utility Preserving Publication of Trajectory Data.  _IEEE Transactions on Mobile Computing_ 18, 10 (2019), 2315–2329.  https://doi.org/10.1109/tmc.2018.2874008
  * Gursoy et al. (2018) Mehmet Emre Gursoy, Ling Liu, Stacey Truex, Lei Yu, and Wenqi Wei. 2018.  Utility-Aware Synthesis of Differentially Private and Attack-Resilient Location Traces. In _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_. 196–211.  https://doi.org/10.1145/3243734.3243741
  * Gursoy et al. (2020) M. Emre Gursoy, Vivekanand Rajasekar, and Ling Liu. 2020\.  Utility-Optimized Synthesis of Differentially Private Location Traces. In _2020 Second IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA)_. 30–39.  https://doi.org/10.1109/TPS-ISA50397.2020.00015
  * Han et al. (2012) Jiawei Han, Micheline Kamber, and Jian Pei. 2012.  2 - Getting to Know Your Data.  In _Data Mining (Third Edition)_ , Jiawei Han, Micheline Kamber, and Jian Pei (Eds.). Morgan Kaufmann, Boston, 39–82.  https://doi.org/10.1016/B978-0-12-381479-1.00002-2
  * He et al. (2015) Xi He, Graham Cormode, Ashwin Machanavajjhala, Cecilia M. Procopiuc, and Divesh Srivastava. 2015\.  DPT: Differentially Private Trajectory Synthesis Using Hierarchical Reference Systems.  _Proceedings of the VLDB Endowment_ 8, 11 (2015), 1154–1165.  https://doi.org/10.14778/2809974.2809978
  * Hern (2018) Alex Hern. 2018\.  Fitness Tracking App Strava Gives Away Location of Secret US Army Bases.  https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases
  * Hess et al. (2015) Andrea Hess, Karin Anna Hummel, Wilfried N. Gansterer, and Günter Haring. 2015.  Data-Driven Human Mobility Modeling: A Survey and Engineering Guidance for Mobile Networking.  _ACM Comput. Surv._ 48, 3, Article 38 (dec 2015), 39 pages.  https://doi.org/10.1145/2840722
  * Ho et al. (2021) Stella Ho, Youyang Qu, Bruce Gu, Longxiang Gao, Jianxin Li, and Yong Xiang. 2021\.  DP-GAN: Differentially private consecutive data publishing using generative adversarial nets.  _Journal of Network and Computer Applications_ 185 (2021), 103066\.  https://doi.org/10.1016/j.jnca.2021.103066
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Comput._ 9, 8 (nov 1997), 1735–1780.  https://doi.org/10.1162/neco.1997.9.8.1735
  * Huang et al. (2019) Dou Huang, Xuan Song, Zipei Fan, Renhe Jiang, Ryosuke Shibasaki, Yu Zhang, Haizhong Wang, and Yugo Kato. 2019\.  A Variational Autoencoder Based Generative Model of Urban Human Mobility. In _2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)_. 425–430.  https://doi.org/10.1109/MIPR.2019.00086
  * Isaacman et al. (2012) Sibren Isaacman, Richard Becker, Ramón Cáceres, Margaret Martonosi, James Rowland, Alexander Varshavsky, and Walter Willinger. 2012\.  Human Mobility Modeling at Metropolitan Scales. In _Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services - MobiSys ’12_. ACM Press, Low Wood Bay, Lake District, UK, 239.  https://doi.org/10.1145/2307636.2307659
  * Jiang et al. (2021) Hongbo Jiang, Jie Li, Ping Zhao, Fanzi Zeng, Zhu Xiao, and Arun Iyengar. 2021\.  Location Privacy-preserving Mechanisms in Location-based Services: A Comprehensive Survey.  _Comput. Surveys_ 54, 1 (Jan. 2021), 4:1–4:36.  https://doi.org/10.1145/3423165
  * Jiang et al. (2023) Wenjun Jiang, Wayne Xin Zhao, Jingyuan Wang, and Jiawei Jiang. 2023.  Continuous Trajectory Generation Based on Two-Stage GAN.  arXiv:2301.07103 [cs]
  * Jordon et al. (2022) James Jordon, Lukasz Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N. Cohen, and Adrian Weller. 2022.  Synthetic Data – What, Why and How?  (2022).  arXiv:2205.03257
  * Kang et al. (2021) Xu Kang, Liang Liu, Dong Zhao, and Huadong Ma. 2021\.  TraG: A Trajectory Generation Technique for Simulating Urban Crowd Mobility.  _IEEE Transactions on Industrial Informatics_ 17, 2 (2021), 820–829.  https://doi.org/10.1109/tii.2020.2976777
  * Kiukkonen et al. (2010) Niko Kiukkonen, Jan Blom, Olivier Dousse, Daniel Gatica-Perez, and Juha Laurila. 2010.  Towards Rich Mobile Phone Datasets: Lausanne Data Collection Campaign.  _Proc. ICPS, Berlin_ 68, 7 (2010).
  * Kopp et al. (2014) Christine Kopp, Bruno Kochan, Michael May, Luca Pappalardo, Salvatore Rinzivillo, Daniel Schulz, and Filippo Simini. 2014.  Evaluation of Spatio-Temporal Microsimulation Systems.  In _Data Science and Simulation in Transportation Research_ , Davy Janssens, Ansar-Ul-Haque Yasar, and Luk Knapen (Eds.). IGI Global, 141–166.  https://doi.org/10.4018/978-1-4666-4920-0.ch008
  * Kulkarni and Garbinato (2017) Vaibhav Kulkarni and Benoît Garbinato. 2017.  Generating Synthetic Mobility Traffic Using RNNs. In _Proceedings of the 1st Workshop on Artificial Intelligence and Deep Learning for Geographic Knowledge Discovery_. 1–4.  https://doi.org/10.1145/3149808.3149809
  * Kulkarni et al. (2018) Vaibhav Kulkarni, Natasa Tagasovska, Thibault Vatter, and Benoit Garbinato. 2018.  Generative Models for Simulating Mobility Trajectories.  (2018).  arXiv:1811.12801
  * Kullback and Leibler (1951) S. Kullback and R. A. Leibler. 1951.  On Information and Sufficiency.  _The Annals of Mathematical Statistics_ 22, 1 (March 1951), 79–86.  https://doi.org/10.1214/aoms/1177729694
  * Lestyán et al. (2022) Szilvia Lestyán, Gergely Ács, and Gergely Biczók. 2022.  In Search of Lost Utility: Private Location Data.  _Proceedings on Privacy Enhancing Technologies_ 2022, 3 (2022), 354–372.  https://doi.org/10.56553/popets-2022-0076
  * Levina and Bickel (2001) E. Levina and P. Bickel. 2001.  The Earth Mover’s Distance Is the Mallows Distance: Some Insights from Statistics. In _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , Vol. 2. 251–256 vol.2.  https://doi.org/10.1109/ICCV.2001.937632
  * Li et al. (2016a) Jiapeng Li, Wei Chen, An Liu, Zhixu Li, and Lei Zhao. 2016a.  FTS: A Practical Model for Feature-Based Trajectory Synthesis.  In _Web Technologies and Applications_. 28–40.  https://doi.org/10.1007/978-3-319-45814-4_3
  * Li et al. (2016b) Meng Li, Liehuang Zhu, Zijian Zhang, and Rixin Xu. 2016b.  Differentially Private Publication Scheme for Trajectory Data. In _2016 IEEE First International Conference on Data Science in Cyberspace (DSC)_. IEEE, 596–601.  https://doi.org/10.1109/DSC.2016.64
  * Lian and Zhang (2018) Jing Lian and Lin Zhang. 2018.  One-Month Beijing Taxi GPS Trajectory Dataset with Taxi IDs and Vehicle Status. In _Proceedings of the First Workshop on Data Acquisition To Analysis_ (Shenzhen, China) _(DATA ’18)_. Association for Computing Machinery, New York, NY, USA, 3–4.  https://doi.org/10.1145/3277868.3277870
  * Liu et al. (2019) Huichuan Liu, Yong Zeng, Zhihong Liu, Jianfeng Ma, and Xiaoyan Zhu. 2019.  Adaptive N-gram Model Based Differentially Private Trajectory Synthesis Method. In _2019 International Conference on Networking and Network Applications (NaNA)_. 356–361.  https://doi.org/10.1109/nana.2019.00068
  * Liu et al. (2018) Xi Liu, Hanzhou Chen, and Clio Andris. 2018.  trajGANs: Using Generative Adversarial Networks for Geo-Privacy Protection of Trajectory Data (Vision Paper). In _Location Privacy and Security Workshop_. 1–7.
  * Luca et al. (2021) Massimiliano Luca, Gianni Barlacchi, Bruno Lepri, and Luca Pappalardo. 2021.  A Survey on Deep Learning for Human Mobility.  _Acm Computing Surveys_ 55, 1, Article 7 (Nov. 2021).  https://doi.org/10.1145/3485125
  * Martín-Martín et al. (2021) Alberto Martín-Martín, Mike Thelwall, Enrique Orduna-Malea, and Emilio Delgado López-Cózar. 2021\.  Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations’ COCI: A Multidisciplinary Comparison of Coverage via Citations.  _Scientometrics_ 126, 1 (Jan. 2021), 871–906.  https://doi.org/10.1007/s11192-020-03690-4
  * Mauro et al. (2022) Giovanni Mauro, Massimiliano Luca, Antonio Longa, Bruno Lepri, and Luca Pappalardo. 2022\.  Generating Synthetic Mobility Networks with Generative Adversarial Networks.  (2022).  arXiv:2202.11028
  * May Petry et al. (2020) Lucas May Petry, Camila Leite Da Silva, Andrea Esuli, Chiara Renso, and Vania Bogorny. 2020\.  MARC: A Robust Method for Multiple-Aspect Trajectory Classification via Space, Time, and Semantic Embeddings.  _International Journal of Geographical Information Science_ 34, 7 (July 2020), 1428–1450.  https://doi.org/10.1080/13658816.2019.1707835
  * Mir et al. (2013) Darakhshan J Mir, Sibren Isaacman, Ramón Cáceres, Margaret Martonosi, and Rebecca N Wright. 2013\.  Dp-Where: Differentially Private Modeling of Human Mobility. In _2013 IEEE International Conference on Big Data_. IEEE, 580–588.  https://doi.org/10.1109/BigData.2013.6691626
  * Mokbel et al. (2013) Mohamed F Mokbel, Louai Alarabi, Jie Bao, Ahmed Eldawy, Amr Magdy, Mohamed Sarwat, Ethan Waytas, and Steven Yackel. 2013\.  MNTG: An Extensible Web-Based Traffic Generator. In _International Symposium on Spatial and Temporal Databases_. Springer, 38–55.
  * Mokhtar et al. (2017) Sonia Ben Mokhtar, Antoine Boutet, Louafi Bouzouina, Patrick Bonnel, Olivier Brette, Lionel Brunie, Mathieu Cunche, Stephane D ’Alu, Vincent Primault, Patrice Raveneau, Herve Rivano, and Razvan Stanica. 2017\.  PRIVA’MOV: Analysing Human Mobility Through Multi-Sensor Datasets. In _NetMob 2017_.
  * Moreira-Matias et al. (2013) Luis Moreira-Matias, João Gama, Michel Ferreira, João Moreira, and Luís Damas. 2013\.  Predicting Taxi-Passenger Demand Using Streaming Data.  _IEEE Transactions on Intelligent Transportation Systems_ 14 (Sept. 2013), 1393–1402.  https://doi.org/10.1109/TITS.2013.2262376
  * Naboulsi et al. (2016) Diala Naboulsi, Marco Fiore, Stephane Ribot, and Razvan Stanica. 2016.  Large-Scale Mobile Traffic Analysis: A Survey.  _IEEE Communications Surveys & Tutorials_ 18, 1 (2016), 124–161.  https://doi.org/10.1109/COMST.2015.2491361
  * Ouyang et al. (2018) Kun Ouyang, Reza Shokri, David S. Rosenblum, and Wenzhuo Yang. 2018\.  A Non-Parametric Generative Model for Human Trajectories. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence_. 3812–3817.  https://doi.org/10.24963/ijcai.2018/530
  * Pang et al. (2020) Yanbo Pang, Takehiro Kashiyama, Takahiro Yabe, Kota Tsubouchi, and Yoshihide Sekimoto. 2020\.  Development of People Mass Movement Simulation Framework Based on Reinforcement Learning.  _Transportation Research Part C: Emerging Technologies_ 117 (2020), 102706\.  https://doi.org/10.1016/j.trc.2020.102706
  * Pang et al. (2017) Yanbo Pang, Kota Tsubouchi, Takahiro Yabe, and Yoshihide Sekimoto. 2017.  Modeling and Reproducing Human Daily Travel Behavior from GPS Data. In _Proceedings of the 1st ACM SIGSPATIAL Workshop on Prediction of Human Mobility_. 1–9.  https://doi.org/10.1145/3152341.3152347
  * Pang et al. (2018) Yanbo Pang, Kota Tsubouchi, Takahiro Yabe, and Yoshihide Sekimoto. 2018.  Replicating Urban Dynamics by Generating Human-like Agents from Smartphone GPS Data. In _Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems_ (Seattle, Washington) _(SIGSPATIAL ’18)_. Association for Computing Machinery, New York, NY, USA, 440–443.  https://doi.org/10.1145/3274895.3274935
  * Pappalardo and Simini (2018) Luca Pappalardo and Filippo Simini. 2018.  Data-Driven Generation of Spatio-Temporal Routines in Human Mobility.  _Data Mining and Knowledge Discovery_ 32, 3 (2018), 787–829.  https://doi.org/10.1007/s10618-017-0548-4
  * Patki et al. (2016) N. Patki, R. Wedge, and K. Veeramachaneni. 2016.  The Synthetic Data Vault. In _2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA)_. 399–410.  https://doi.org/10.1109/DSAA.2016.49
  * Pelletier et al. (2011) Marie-Pier Pelletier, Martin Trépanier, and Catherine Morency. 2011.  Smart Card Data Use in Public Transit: A Literature Review.  _Transportation Research Part C: Emerging Technologies_ 19, 4 (Aug. 2011), 557–568.  https://doi.org/10.1016/j.trc.2010.12.003
  * Pesavento et al. (2020) John Pesavento, Andy Chen, Rayan Yu, Joon-Seok Kim, Hamdi Kavak, Taylor Anderson, and Andreas Züfle. 2020.  Data-Driven Mobility Models for COVID-19 Simulation. In _Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Advances in Resilient and Intelligent Cities_ _(ARIC ’20)_. Association for Computing Machinery, New York, NY, USA, 29–38.  https://doi.org/10.1145/3423455.3430305
  * Piorkowski et al. (2009) Michal Piorkowski, Natasa Sarafijanovic-Djukic, and Matthias Grossglauser. 2009\.  CRAWDAD Dataset Epfl/Mobility (v. 2009-02-24).  https://doi.org/10.15783/C7J010
  * Polonetsky and Renieris (2020) Jules Polonetsky and Elizabeth Renieris. 2020.  _Privacy 2020. 10 Privacy Risks and 10 Privacy Enhancing Technologies to Watch in the next Decade_.  Technical Report.
  * Pujol and Machanavajjhala (2021) David Pujol and Ashwin Machanavajjhala. 2021.  Equity and Privacy: More Than Just a Tradeoff.  _IEEE Security & Privacy_ 19, 6 (Nov. 2021), 93–97.  https://doi.org/10.1109/MSEC.2021.3105773
  * Qiu et al. (2021) Guoying Qiu, Deke Guo, Yulong Shen, Guoming Tang, and Sheng Chen. 2021.  Mobile Semantic-Aware Trajectory for Personalized Location Privacy Preservation.  _IEEE Internet of Things Journal_ 8, 21 (2021), 16165–16180.  https://doi.org/10.1109/JIOT.2020.3016466
  * Rao et al. (2021) Jinmeng Rao, Song Gao, Yuhao Kang, and Qunying Huang. 2021\.  LSTM-TrajGAN: A Deep Learning Approach to Trajectory Privacy Protection. In _11th International Conference on Geographic Information Science, GIScience 2021, September 27-30, 2021, Poznań, Poland - Part I_ _(LIPIcs, Vol. 177)_ , Krzysztof Janowicz and Judith Anne Verstegen (Eds.). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 12:1–12:17.  https://doi.org/10.4230/LIPIcs.GIScience.2021.I.12
  * Roy et al. (2016) Harichandan Roy, Murat Kantarcioglu, and Latanya Sweeney. 2016\.  Practical Differentially Private Modeling of Human Movement Data.  In _Data and Applications Security and Privacy XXX_. 170–178.  https://doi.org/10.1007/978-3-319-41483-6_13
  * Sakuma et al. (2021) Yuiko Sakuma, Thai P Tran, Tomomu Iwai, Akihito Nishikawa, and Hiroaki Nishi. 2021.  Trajectory Anonymization through Laplace Noise Addition in Latent Space. In _2021 Ninth International Symposium on Computing and Networking (CANDAR)_. IEEE, 65–73.  https://doi.org/10.1109/CANDAR53791.2021.00016
  * Science (2018) Digital Science. 2018\.  _Dimensions [Software]_.  Retrieved September 2022 from https://app.dimensions.ai
  * Shah et al. (2017) Arsalan Shah, Petr Belyaev, Borja Ramis Ferrer, Wael M. Mohammed, and Jose L. Martinez Lastra. 2017\.  Processing Mobility Traces for Activity Recognition in Smart Cities. In _IECON 2017 - 43rd Annual Conference of the IEEE Industrial Electronics Society_. 8654–8661.  https://doi.org/10.1109/IECON.2017.8217520
  * Shin et al. (2020) Seungjae Shin, Hongseok Jeon, Chunglae Cho, Seunghyun Yoon, and Taeyeon Kim. 2020.  User Mobility Synthesis Based on Generative Adversarial Networks: A Survey. In _2020 22nd International Conference on Advanced Communication Technology (ICACT)_. 94–103.  https://doi.org/10.23919/icact48636.2020.9061335
  * Shokri et al. (2011) Reza Shokri, George Theodorakopoulos, Jean-Yves Le Boudec, and Jean-Pierre Hubaux. 2011.  Quantifying Location Privacy. In _2011 IEEE Symposium on Security and Privacy_. IEEE, Oakland, CA, USA, 247–262.  https://doi.org/10.1109/SP.2011.18
  * Singh et al. (2021) Vivek Kumar Singh, Prashasti Singh, Mousumi Karmakar, Jacqueline Leta, and Philipp Mayr. 2021\.  The Journal Coverage of Web of Science, Scopus and Dimensions: A Comparative Analysis.  _Scientometrics_ 126, 6 (June 2021), 5113–5142.  https://doi.org/10.1007/s11192-021-03948-5
  * Smolak et al. (2020) Kamil Smolak, Witold Rohm, Krzysztof Knop, and Katarzyna Siła-Nowicka. 2020.  Population Mobility Modelling for Mobility Data Simulation.  _Computers, Environment and Urban Systems_ 84 (2020), 101526.  https://doi.org/10.1016/j.compenvurbsys.2020.101526
  * Song et al. (2010) Chaoming Song, Tal Koren, Pu Wang, and Albert-László Barabási. 2010.  Modelling the Scaling Properties of Human Mobility.  _Nature Physics_ 6, 10 (Oct. 2010), 818–823.  https://doi.org/10.1038/nphys1760
  * Sun et al. (2023) Xinyue Sun, Qingqing Ye, Haibo Hu, Yuandong Wang, Kai Huang, Tianyu Wo, and Jie Xu. 2023.  Synthesizing Realistic Trajectory Data With Differential Privacy.  _IEEE Transactions on Intelligent Transportation Systems_ (2023), 1–14.  https://doi.org/10.1109/TITS.2023.3241290
  * Tamura et al. (2022) Naoki Tamura, Kenta Urano, Shunsuke Aoki, Takuro Yonezawa, and Nobuo Kawaguchi. 2022.  Synthetic People Flow: Privacy-Preserving Mobility Modeling from Large-Scale Location Data in Urban Areas.  In _Mobile and Ubiquitous Systems: Computing, Networking and Services_. 553–567.  https://doi.org/10.1007/978-3-030-94822-1_36
  * Trotter (2014) J.K. Trotter. 2014\.  Public NYC Taxicab Database Lets You See How Celebrities Tip.  https://gawker.com/the-public-nyc-taxicab-database-that-accidentally-track-1646724546
  * Wang and Xie (2018) Cheng Wang and Zhiyang Xie. 2018.  Artificial Impostors for Location Privacy Preservation.  (2018).  arXiv:1801.06827
  * Wang et al. (2022) Haiming Wang, Zhikun Zhang, Tianhao Wang, Shibo He, Michael Backes, Jiming Chen, and Yang Zhang. 2022.  PrivTrace: Differentially Private Trajectory Synthesis by Adaptive Markov Model.  arXiv:2210.00581 [cs]
  * Wang and Osaragi (2022) Weiying Wang and Toshihiro Osaragi. 2022.  Daily Human Mobility: A Reproduction Model and Insights from the Energy Concept.  _ISPRS International Journal of Geo-Information_ 11, 4 (April 2022), 219.  https://doi.org/10.3390/ijgi11040219
  * Wang et al. (2021) Xingrui Wang, Xinyu Liu, Ziteng Lu, and Hanfang Yang. 2021\.  Large Scale GPS Trajectory Generation Using Map Based on Two Stage GAN.  _Journal of Data Science_ 19, 1 (2021), 126–141.  https://doi.org/10.6339/21-JDS1004
  * Wei et al. (2021) Hua Wei, Dongkuan Xu, Junjie Liang, and Zhenhui (Jessie) Li. 2021\.  How Do We Move: Modeling Human Movement with System Dynamics.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 35, 5 (May 2021), 4445–4452.  https://doi.org/10.1609/aaai.v35i5.16571
  * Xiong et al. (2023) Gang Xiong, Zhishuai Li, Meihua Zhao, Yu Zhang, Qinghai Miao, Yisheng Lv, and Fei-Yue Wang. 2023.  TrajSGAN: A Semantic-Guiding Adversarial Network for Urban Trajectory Generation.  _IEEE Transactions on Computational Social Systems_ (2023), 1–11.  https://doi.org/10.1109/TCSS.2023.3235923
  * Xu et al. (2017) Fengli Xu, Zhen Tu, Yong Li, Pengyu Zhang, Xiaoming Fu, and Depeng Jin. 2017\.  Trajectory Recovery From Ash: User Privacy Is NOT Preserved in Aggregated Mobility Data. In _Proceedings of the 26th International Conference on World Wide Web_ _(WWW ’17)_. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1241–1250.  https://doi.org/10.1145/3038912.3052620
  * Xu et al. (2021) Nan Xu, Loc Trinh, Sirisha Rambhatla, Zhen Zeng, Jiahao Chen, Samuel Assefa, and Yan Liu. 2021.  Simulating Continuous-Time Human Mobility Trajectories. In _Workshop on Deep Learning for Simulation_.
  * Yang et al. (2013a) Dingqi Yang, Daqing Zhang, Zhiyong Yu, and Zhu Wang. 2013a.  A Sentiment-Enhanced Personalized Location Recommendation System. In _Proceedings of the 24th ACM Conference on Hypertext and Social Media_ _(HT ’13)_. Association for Computing Machinery, New York, NY, USA, 119–128.  https://doi.org/10.1145/2481492.2481505
  * Yang et al. (2013b) Dingqi Yang, Daqing Zhang, Zhiyong Yu, and Zhiwen Yu. 2013b.  Fine-Grained Preference-Aware Location Search Leveraging Crowdsourced Digital Footprints from LBSNs. In _Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing_ _(UbiComp ’13)_. Association for Computing Machinery, New York, NY, USA, 479–488.  https://doi.org/10.1145/2493432.2493464
  * Yang et al. (2014) Dingqi Yang, Daqing Zhang, Zhiyong Yu, Zhiwen Yu, and Djamal Zeghlache. 2014.  SESAME: Mining User Digital Footprints for Fine-Grained Preference-Aware Social Media Search.  _ACM Transactions on Internet Technology_ 14, 4 (Dec. 2014), 28:1–28:24.  https://doi.org/10.1145/2677209
  * Yang et al. (2015) Dingqi Yang, Daqing Zhang, Vincent W. Zheng, and Zhiyong Yu. 2015.  Modeling User Activity Preference by Leveraging User Spatial Temporal Characteristics in LBSNs.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ 45, 1 (Jan. 2015), 129–142.  https://doi.org/10.1109/TSMC.2014.2327053
  * Yang et al. (2022) Jianyu Yang, Xiang Cheng, Sen Su, Huizhong Sun, and Changju Chen. 2022.  Collecting Individual Trajectories under Local Differential Privacy. In _2022 23rd IEEE International Conference on Mobile Data Management (MDM)_. 99–108.  https://doi.org/10.1109/mdm55031.2022.00035
  * Yu (2019) Lei Yu. 2019.  _Reconciling Data Privacy and Utility in the Era of Big Data_.  Ph. D. Dissertation. Georgia Institute of Technology.
  * Yuan et al. (2011) Jing Yuan, Yu Zheng, Xing Xie, and Guangzhong Sun. 2011\.  Driving with Knowledge from the Physical World. In _Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ _(KDD ’11)_. Association for Computing Machinery, New York, NY, USA, 316–324.  https://doi.org/10.1145/2020408.2020462
  * Yuan et al. (2010) Jing Yuan, Yu Zheng, Chengyang Zhang, Wenlei Xie, Xing Xie, Guangzhong Sun, and Yan Huang. 2010.  T-Drive: Driving Directions Based on Taxi Trajectories. In _Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems_ (San Jose, California) _(GIS ’10)_. Association for Computing Machinery, New York, NY, USA, 99–108.  https://doi.org/10.1145/1869790.1869807
  * Zhan et al. (2022) Yuting Zhan, Hamed Haddadi, Alex Kyllo, and Afra Mashhadi. 2022.  Privacy-Aware Human Mobility Prediction via Adversarial Networks. In _2022 2nd International Workshop on Cyber-Physical-Human System Design and Implementation (CPHS)_. 7–12.  https://doi.org/10.1109/CPHS56133.2022.9804533
  * Zhao et al. (2019) Ping Zhao, Hongbo Jiang, Jie Li, Fanzi Zeng, Xiao Zhu, Kun Xie, and Guanglin Zhang. 2019\.  Synthesizing Privacy Preserving Traces: Enhancing Plausibility With Social Networks.  _IEEE/ACM Transactions on Networking_ 27, 6 (Dec. 2019), 2391–2404.  https://doi.org/10.1109/TNET.2019.2947452
  * Zheng (2011) Yu Zheng. 2011.  Location-Based Social Networks: Users.  In _Computing with Spatial Trajectories_ , Yu Zheng and Xiaofang Zhou (Eds.). Springer New York, New York, NY, 243–276.  https://doi.org/10.1007/978-1-4614-1629-6_8
  * Zheng et al. (2010) Yu Zheng, Xing Xie, Wei-Ying Ma, et al. 2010\.  GeoLife: A Collaborative Social Networking Service among User, Location and Trajectory.  _IEEE Data Eng. Bull._ 33, 2 (2010), 32–39.
  * Zhou et al. (2021) Fan Zhou, Xin Liu, Kunpeng Zhang, and Goce Trajcevski. 2021\.  Toward Discriminating and Synthesizing Motion Traces Using Deep Probabilistic Generative Models.  _IEEE Transactions on Neural Networks and Learning Systems_ 32, 6 (2021), 2401–2414.  https://doi.org/10.1109/tnnls.2020.3005325
  * Zhou et al. (2018) Yuren Zhou, Billy Pik Lik Lau, Chau Yuen, Bige Tuncer, and Erik Wilhelm. 2018.  Understanding Urban Human Mobility through Crowdsensed Data.  _IEEE Communications Magazine_ 56, 11 (Nov. 2018), 52–59.  https://doi.org/10.1109/MCOM.2018.1700569

##  7\. Appendix

##  8\. Links to Publically Available Model Source Codes

Model Name/Authors |  Source Code Link
---|---
AdaTrace (Gursoy et al., 2018) |  https://bit.ly/3SEbdoG
DP-Star (Gursoy et al., 2019) |  https://bit.ly/3fmMYwR
TSG (Wang et al., 2021) |  https://bit.ly/3SphvJ0
LSTM-TrajGAN (Rao et al., 2021) |  https://bit.ly/3Ckmpld
MoveSim (Feng et al., 2020) |  https://bit.ly/3Sr5YZY
STULIG (Zhou et al., 2021) |  https://bit.ly/3SD6wvX
DITRAS (Pappalardo and Simini, 2018) |  https://bit.ly/3LRDaak
3W (Smolak et al., 2020) |  https://bit.ly/3Sp4eAm
Berke et al. (Berke et al., 2022) |  https://bit.ly/3RrNDKF
Kulkarni and Garbinato (Kulkarni et al., 2018) |  https://bit.ly/3RBIVdL
DP-Loc (Lestyán et al., 2022) |  http://bit.ly/3Zm3oHq
TrajGAIL (Choi et al., 2021) |  http://bit.ly/3EIgJ52
PrivTrace (Wang et al., 2022) |  http://bit.ly/3Zn65bW
LDPTrace (Du et al., 2023) |  http://bit.ly/3mnYJ9P

##  9\. Links to Publically Available Datasets

Dataset Name |  Download Link
---|---
Bike NYC |  https://bit.ly/3E3P3rV
Bike Washington |  https://bit.ly/3zvH3wo
Boston Hubway Bike |  https://bit.ly/3y1rDjf
GeoLife (Beijing) |  https://bit.ly/3Rqbjz8
Porto Taxi |  https://bit.ly/3RkGWtR
San Francisco Taxi |  https://bit.ly/40DaY1B
Microsoft T-Drive (Beijing 2011) |  https://bit.ly/3CjbYOE
Lausanne Data Collection Campaign (LDCC) |  https://bit.ly/3BXztvr
Foursquare Weekly |  https://bit.ly/3ULvhHv
NYC Restaurant Rich Dataset |  https://bit.ly/3ULvhHv
NYC and Tokyo Dataset |  https://bit.ly/3ULvhHv
Brightkite |  https://stanford.io/3Cjc7BG
Gowalla |  https://stanford.io/3SDcT1K
Brinkhoff (Oldenburg) |  https://bit.ly/3y0YHYD
Survey Montreal |  https://bit.ly/3dUnWVx
MTL Trajet |  https://bit.ly/3IP35yt


# Low-Resource Sequence Labeling via Unsupervised Multilingual Contextualized
Representations

Zuyi Bao‡, Rui Huang♢, Chen Li‡ Corresponding Author. Kenny Q. Zhu
‡Alibaba Group
♢Zhejiang University, China
†Shanghai Jiao Tong University, Shanghai, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Kenny Q. Zhu was partially
supported by NSFC grant 91646205 and Alibaba visiting scholar program.

###### Abstract

Previous work on cross-lingual sequence labeling tasks either requires
parallel data or bridges the two languages through word-by-word matching. Such
requirements and assumptions are infeasible for most languages, especially for
languages with large linguistic distances, e.g., English and Chinese. In this
work, we propose a Multilingual Language Model with deep semantic Alignment
(MLMA) to generate language-independent representations for cross-lingual
sequence labeling. Our methods require only monolingual corpora with no
bilingual resources at all and take advantage of deep contextualized
representations. Experimental results show that our approach achieves new
state-of-the-art NER and POS performance across European languages, and is
also effective on distant language pairs such as English and Chinese.111The
code is released at https://github.com/baozuyi/MLMA.

##  1 Introduction

Sequence labeling tasks such as named entity recognition (NER) and part-of-
speech (POS) tagging are fundamental problems in natural language processing
(NLP). Recent sequence labeling models achieve state-of-the-art performance by
combining both character-level and word-level information Chiu and Nichols
(2016); Ma and Hovy (2016); Lample et al. (2016). However, these models
heavily rely on large-scale annotated training data, which may not be
available in most languages. Cross-lingual transfer learning is proposed to
address the label scarcity problem by transferring annotations from high-
resource languages (source languages) to low-resource languages (target
languages). In this scenario, a major challenge is how to bridge interlingual
gaps with modest resource requirements.

There is a large body of work exploring cross-lingual transfer through
language-independent features, such as morphological features and universal
POS tags for cross-lingual NER Tsai et al. (2016) and dependency parsers
McDonald et al. (2011). However, these approaches require linguistic knowledge
for language-independent feature engineering, which is expensive in low-
resource settings. Other work relies on bilingual resources to transfer
knowledge from source languages to target languages. Parallel corpora are
widely used to project annotations from the source to the target side Yarowsky
et al. (2001); Ehrmann et al. (2011); Kim et al. (2012); Wang and Manning
(2014). These methods could achieve strong performance with a large amount of
bilingual data, which is scarce in low-resource settings.

Recent research leverages cross-lingual word embeddings (CLWEs) to establish
inter-lingual connections and reduce the requirements of parallel data to a
small lexicon or even no bilingual resource Ni et al. (2017); Fang and Cohn
(2017); Xie et al. (2018). However, word embedding spaces may not be
completely isomorphic due to language-specific linguistic properties, and
therefore cannot be perfectly aligned. For example, different from English,
Chinese nouns do not distinguish singular and plural forms, while Spanish
nouns distinguish masculine and feminine.

On the other hand, NER tags such as person names, organizations, and locations
are shared across different languages. Language-independent frameworks such as
universal conceptual cognitive annotation Abend and Rappoport (2013),
universal POS Petrov et al. (2011a), and universal dependencies Nivre et al.
(2016) are defined to represent different languages in a unified formation.
These work serves as our motivation to assume that the semantic meanings of
words from different languages can be roughly aligned at a conceptual level
and it is more reasonable to align deep semantic representations instead of
shallow word embeddings. Meanwhile, monolingual contextualized embeddings
derived from language models have shown to be effective for extracting
semantic information and have achieved significant improvement on several NLP
tasks Peters et al. (2018).

In this paper, we propose a Multilingual Language Model with deep semantic
Alignment (MLMA). We train MLMA on monolingual corpora from each language and
align its internal states across different languages. Then MLMA is utilized to
generate language-independent representations and to bridge the gaps between
high-resource and low-resource languages. For evaluation, we conduct extensive
experiments on the NER and POS benchmark datasets under cross-lingual
settings. The experiment results show that our methods achieve substantial
improvements comparing to previous state-of-the-art methods in European
languages. We also validate our approaches on a distant language pair,
English-Chinese, and the results are competitive with previous methods which
use large-scale parallel corpora. Our contributions are as follows:

  1. 1.

Instead of word-level alignment, we propose MLMA that uses contextualized
representations to bridge the inter-lingual gaps.

  2. 2.

We propose three methods to align contextualized representations without any
bilingual resource.

  3. 3.

Our methods achieve new state-of-the-art performance on cross-lingual NER and
POS tasks in European languages, and very competitive results for English-
Chinese NER, where previous work uses large parallel data.

##  2 Approach

Our approach belongs to the model transfer (Section 5.2) and mainly consists
of three steps:

  1. 1.

Training a multilingual language model with alignment (MLMA) using monolingual
corpora of the source and the target languages. (Section 2.1, 2.2 and 2.3)

  2. 2.

Building a cross-lingual sequence labeling model based on the language-
independent representations from the MLMA. (Section 2.4 and 2.5)

  3. 3.

Learning the cross-lingual sequence labeling model (with MLMA fixed) on the
annotated data of source languages and directly applying it to the target
languages.

The architecture of MLMA is shown in Figure 1. In the following sections, we
focus on introducing the Step 1 and 2. We first present the architecture of
MLMA and describe how to build the unsupervised multilingual alignment. Next,
we propose effective methods for collapsing the multi-layer hidden states from
MLMA into a single representation. Finally, we introduce the sequence labeling
model used in the experiments.

###  2.1 Language Model Architecture

Figure 1: The architecture of MLMA consists of language-specific embedding
layers and language-agnostic Transformer layers. MLMA is jointly learned
through language modeling loss and alignment loss, and its internal
representations are utilized to bridge the gap between source and target
languages.

MLMA is a language model with multi-head self-attention mechanism Vaswani et
al. (2017). The architecture is similar to Radford et al. (2018), except that
we combine both a forward and a backward Transformer decoder to build a
bidirectional language model. Take the forward direction as an example, given
a sentence with $N$ tokens $W=[w_{1},w_{2},\cdots,w_{N}]^{T}$ as input, we
first map the sequence of tokens $W$ to token embeddings
$\overrightarrow{H}_{0}\in\mathbb{R}^{N\times d}$:

$\overrightarrow{H}_{0}=WE_{e}+E_{p}$ (1)

where $E_{e}$ and $E_{p}$ are the embedding matrix and the positional encoding
matrix, and $d$ is the dimension of embeddings and hidden states.

Then $n$ blocks of transformer layers are stacked above the token embeddings.
Each block contains a masked multi-head self-attention and a position-wise
feedforward layer. The detailed implementation is the same as Vaswani et al.
(2017).

$\overrightarrow{H}_{l}={\rm TransformerLayer}(\overrightarrow{H}_{l-1})$ (2)

where $\overrightarrow{H}_{l}$ refers to the output of the $l$-th transformer
block. Finally, the output distribution over the next tokens is calculated
through a softmax function with tied embedding matrix.

$\overrightarrow{P}={\rm softmax}(\overrightarrow{H}_{n}E_{e}^{T})$ (3)

For the backward direction, we calculate $\overleftarrow{H_{l}}$ and
$\overleftarrow{P}$ in an analogous way. Finally, we jointly minimize the
negative log likelihood of the forward and backward directions:

$\displaystyle NLL=-\sum_{t=1}^{N}($ $\displaystyle\log
p(w_{t}|w_{1},\ldots,w_{t-1})$ (4) $\displaystyle+$ $\displaystyle\log
p(w_{t}|w_{t+1},\ldots,w_{N}))$

In a multilingual setting, we share all parameters in Transformer layers
across different languages to facilitate language-agnostic representations,
except that we adopt an individual embedding matrix $E_{e}$ for each language.

###  2.2 Unsupervised Distribution Alignment

We find that only sharing Transformer layers is not enough for forcing hidden
representations from different languages into a common space, as suggested in
experiments (Section 3.4). Therefore, we propose three methods to build cross-
lingual representations based on identical strings, mean/variance, and average
linkage.

To simplify the description, we take the alignment between two languages $s$
and $t$ as an example, but our methods can be directly extended to a scenario
with multiple languages by adding the alignment between each pair of
languages.

#### Notation

For the language model, given a sentence with $N$ tokens, the forward internal
representation $\overrightarrow{H}_{l}$ in Eq (2) can be expanded as
$\overrightarrow{H}_{l}=[\overrightarrow{h}_{l,1},\cdots,\overrightarrow{h}_{l,k},\cdots,\overrightarrow{h}_{l,N}]^{T}$,
where $\overrightarrow{h}_{l,k}$ refers to the forward hidden representation
of the $k$-th token in the sentence. Then we concatenate the forward and
backward hidden representations for each token,
$h_{l,k}=\overrightarrow{h}_{l,k}\oplus\overleftarrow{h}_{l,k}$.

We denote the collection of the token representations ${h}_{l,k}$ at layer $l$
from the whole corpora of language $s$ as $C_{l}^{s}$, which can be regarded
as a sampling from the deep semantic space of language $s$. Similarly,
$C_{l}^{t}$ is used for language $t$.

#### Identical Strings

Similar language pairs such as English and Spanish have a large number of
identical strings shared between their vocabularies, which are utilized as the
seed dictionary for embedding alignment in previous work Smith et al. (2017).
Similarly, we treat identical strings as explicit supervision signals and
align the embeddings of identical strings between different languages. The
matching of the embeddings from different languages will lead to an implicit
alignment of internal representations. In the experiments, we directly
minimize the Euclidean distance between the embeddings of each identical
string across different languages:

$L_{iden}=\dfrac{\lambda^{iden}}{|W_{iden}^{(s,t)}|}\sum_{w\in
W_{iden}^{(s,t)}}||e_{w}^{s}-e_{w}^{t}||$

where $W_{iden}^{(s,t)}$ is the set of identical strings between the
vocabulary of language $s$ and language $t$, and $|W_{iden}^{(s,t)}|$ refers
to the number of members in $W_{iden}^{(s,t)}$. $\lambda^{id}$ is a scaling
weight, and $e_{w}^{s}$ ($e_{w}^{t}$) is the embedding of word $w$ from
embedding matrix $E_{e}^{s}$ ($E_{e}^{t}$) of language $s$ ($t$).

#### Mean and Variance

In this section, we propose another approach to directly align the
distributions of internal representations between different languages. In
particular, we leverage the mean and variance of internal distributions for
alignment. We denote the mean and variance of $C_{l}^{s}$ as $m_{l}^{s}$ and
$v_{l}^{s}$. Similarly, $m_{l}^{t}$ and $v_{l}^{t}$ refer to the mean and
variance of $C_{l}^{t}$. We minimize the Euclidean distance between the mean
and variance of language $s$ and language $t$ for all layers:

$L_{mv}=\sum_{l=0}^{n}(\lambda_{l}^{m}\cdot\dfrac{||m_{l}^{s}-m_{l}^{t}||}{|m_{l}^{s}|+|m_{l}^{t}|}+\lambda_{l}^{v}\cdot\dfrac{||v_{l}^{s}-v_{l}^{t}||}{|v_{l}^{s}|+|v_{l}^{t}|})$

where $\lambda_{l}$ is a scaling weight, and $|\cdot|$ is the L1 norm of a
vector. Without the denominators, the model could escape this regularization
by learning a mean and variance with low absolute values.

In practice, rather than calculating the mean and variance over the whole
source and target corpora, we use the mean and variance of the source and
target inner states ${h}_{l,k}$ in the current mini-batch as an approximation.

#### Average Linkage

In this method, we employ another metric, average linkage, to perform a more
precise point-wise matching. The average linkage is a widely used metric for
calculating the similarity of clusters and networks Yim and Ramdeen (2015);
Seifoddini (1989); Newman (2012); Moseley and Wang (2017). It is sensitive to
the shape, thus serves as a better choice than mean and variance. The average
linkage measures the similarity of two sets $X$ and $Y$ by calculating the
averaged distance between all members of each set:

$avl(X,Y)=\dfrac{1}{n_{X}\cdot n_{Y}}\sum_{x\in X}\sum_{y\in Y}f(x,y)$

where $n_{X}$ ($n_{Y}$) is the number of members in $X$ ($Y$), and $f$ is a
distance function. We take Euclidean distance as the distance function $f$ and
minimize the average linkage between $C_{l}^{s}$ and $C_{l}^{t}$:

$\displaystyle L_{avl}=\sum_{l=0}^{n}$
$\displaystyle\lambda_{l}^{avl}\cdot[2\cdot avl(C_{l}^{s},C_{l}^{t})$
$\displaystyle-avl(C_{l}^{s},C_{l}^{s})-avl(C_{l}^{t},C_{l}^{t})]$

Similarly, the terms $avl(C_{l}^{s},C_{l}^{s})$ and $avl(C_{l}^{t},C_{l}^{t})$
are used to prevent the model from escaping this regularization. In practice,
we calculate $L_{avl}$ between the source and target inner states ${h}_{l,k}$
inside the mini-batch as an approximation.

The regularization $L_{avl}$ is similar to the maximum mean discrepancy (MMD),
which is often employed in domain adaptation Tzeng et al. (2014); Long et al.
(2015) and style transfer Li et al. (2017) for images. However, different from
MMD, our method directly uses Euclidean distance instead of the kernel
function.

###  2.3 Training of MLMA

During the training stage of MLMA, we sample equivalent number of sentences
from the monolingual corpora of each language for each mini-batch. Then MLMA
is optimized through a combination of the language modeling loss $L_{lm}$ and
the alignment regularization loss $L_{reg}$. For each alignment method, we use
its corresponding alignment loss:

$\displaystyle L=$ $\displaystyle L_{lm}+L_{reg}$ $\displaystyle{\rm where}$
$\displaystyle L_{lm}=\sum_{i\in\\{s,t\\}}\lambda_{i}^{lm}\cdot NLL_{i},$
$\displaystyle L_{reg}\in\\{L_{id},L_{mv},L_{avl}\\}$

where $\lambda_{i}^{lm}$ is used for balancing the convergence speed of
different languages. $NLL_{i}$ is the negative log likelihood of language $i$
in Eq (4).

###  2.4 Cross-lingual Representations

After the MLMA is trained, we fix its parameters and extract the hidden states
as cross-lingual contextualized representations (CLCRs). In this section, we
propose two effective strategies for integrating these multi-layer high-
dimensional representations into downstream models.

Self-Weighted Sum For each token, we concatenate all layers of hidden states
and feed them into a multi-layer perceptron (MLP) to calculate a
$(n+1)$-dimensional weight vector, $s={\rm softmax}({\rm
MLP}(h_{0,k}\oplus\cdots\oplus h_{n,k}))$. Then we calculate a weighted sum of
these layers according to the weight vector,
$\textbf{CLCR}_{k}=\sum_{l=0}^{n}s_{l}\cdot h_{l,k}$.

Fully-Weighted Sum We introduce a weight matrix, $F\in\mathbb{R}^{(n+1)\times
2d}$, with separate weights for each hidden dimension. The weight matrix $F$
is softmaxed by column and used to calculate a weighted sum of all layers for
each hidden dimension, $\textbf{CLCR}_{k}=\sum_{l=0}^{n}F_{l}\odot h_{l,k}$,
where $\odot$ is the element-wise product.

The parameters of the ${\rm MLP}$ and $F$ are trained during the learning of
sequence labeling model.

###  2.5 Sequence Labeling Model

The sequence labeling model is then built on the CLCRs. For both NER and POS
tasks, we use an LSTM-CRF model following Lample et al. (2016), which consists
of a character-level LSTM, a word-level LSTM, and a linear-chain CRF.

More specifically, given a sequence of words as $[w_{1},w_{2},\ldots,w_{N}]$,
where $w_{k}$ is composed of a sequence of characters
$[c_{k,1},c_{k,2},\ldots,c_{k,m}]$. First, for each word $w_{k}$, the
character-level LSTM takes its character sequence
$[c_{k,1},c_{k,2},\ldots,c_{k,m}]$ as input and outputs a vector $e_{k}$ to
represent this word. Then the pre-trained $\textbf{CLCR}_{k}$ is concatenated
with $e_{k}$ to form a word-level embedding $x_{k}$. Finally, the sequence of
word-level embeddings $[x_{1},x_{2},\ldots,x_{N}]$ are fed into the word-level
LSTM, and the linear-chain CRF are employed to predict the probability
distribution for all possible output label sequences.

##  3 Experiments

We first introduce the datasets used in the experiment and then the
implementation details of our models, before presenting the results on NER and
POS tasks.

###  3.1 Datasets

For cross-lingual NER, we evaluate the proposed approaches on CoNLL 2002/2003
datasets Tjong Kim Sang (2002); Tjong Kim Sang and De Meulder (2003), which
contain four European languages, English (en), Spanish (es), Dutch (nl),
German (de) and four entity types (person, location, organization, and MISC).
We also evaluate a distant language pair, English-Chinese, on OntoNotes(v4.0)
dataset Hovy et al. (2006). We adopt the same dataset split and four valid
entity types (person, location, organization, and GPE) as described in Wang
and Manning (2014).

For cross-lingual POS, we use the Danish (da), Dutch (nl), German (de), Greek
(el), Italian (it), Portuguese (pt), Spanish (es) and Swedish (sv) portion
from CoNLL 2006/2007 dataset Buchholz and Marsi (2006); Nivre et al. (2007).
Following previous work Fang and Cohn (2017), we train the sequence labeling
model on Penn Treebank data and adopt the universal POS tagset Petrov et al.
(2011b).

In all cases, the sequence labeling model is trained on the source language
(English) training data and is tested on the target language test data.

###  3.2 Details of MLMA

We adopt a 6-layer bi-directional Transformer decoder with 8 attention heads.
The dimension size of hidden states and inner states are 512 and 2048,
respectively. The dropout rates after attention and residual connection are
both 0.1. We use the Adam optimization scheme Kingma and Ba (2014) with a
learning rate of 0.0001 and a gradient clip norm of 5.0. The vocabulary size
of each language is 200,000, and we train the model with a sampled softmax
Jean et al. (2015) of 8192 samples. We only keep the sentences containing less
than 200 tokens for training and group them into batches by length. Each batch
contains around 4096 tokens for each language. The language modeling weight
$\lambda_{i}^{lm}$ is set to be 1.0 for each language. For alignment,
$\lambda_{l}^{m}$, $\lambda_{l}^{v}$, $\lambda_{l}^{al}$ are set to be 0.1,
0.01 and 1.0 for every layer $l$, and $\lambda^{iden}$ is set to be 100.

For languages except English, the latest dump of Wikipedia is used as
monolingual corpora. For English, we use 1B Word Benchmark Chelba et al.
(2013) to reduce the effects of potential internal alignment in Wikipedia
Zirikly and Hagiwara (2015); Tsai et al. (2016).

All characters are preprocessed to lowercase, and Chinese text are converted
into the simplified version through OpenCC222https://github.com/BYVoid/OpenCC.
The corpora of European languages are tokenized by nltk Loper and Bird (2002)
and Chinese text is segmented using Ltp333https://github.com/HIT-SCIR/pyltp.

###  3.3 Details of Sequence Labeling Model

In our experiments, we set the hidden size of word-level LSTM and character-
level LSTM to be 300 and 100, respectively. The character embedding size is
set to be 100. We apply dropout at both the input and the output of word-level
LSTM to prevent overfitting. The dropout rate is set to be 0.5. We train the
sequence labeling model for 20 epochs using Adam optimizer with a batch size
of 20 and perform an early stopping when there is no improvement for 3 epochs.
We set the initial learning rate to be 0.001 and decay the learning rate by
0.1 for each epoch. We do not update the pre-trained cross-lingual deep
representations from MLMA during training. For each model, we run it five
times and report the mean and standard deviation. We disable the character-
level LSTM in English-German and English-Chinese NER as they have a different
character pattern from English. For POS, we disable the character-level LSTM
following Fang and Cohn (2017).

###  3.4 Results for NER

Model | es | nl | de | Extra Resources
---|---|---|---|---
MLM (w/o alignment) + s.w.s. | 21.16 ± 1.40 | 33.97 ± 1.49 | 15.46 ± 1.21 | None
MLM (w/o alignment) + f.w.s. | 23.61 ± 2.33 | 32.94 ± 1.62 | 16.38 ± 1.09 | None
Täckström et al. (2012) | 59.30 | 58.40 | 40.40 | parallel corpus
Nothman et al. (2013) | 61.00 | 64.00 | 55.80 | Wikipedia
Wang and Manning (2014) | - | - | 60.00 | parallel corpus
Tsai et al. (2016) | 60.55 | 61.60 | 48.10 | Wikipedia
Ni et al. (2017) | 65.10 | 65.40 | 58.50 | Wikipedia, parallel corpus, 5K dict.
Mayhew et al. (2017) | 65.95 | 66.50 | 59.11 | Wikipedia, 1M dict.
Xie et al. (2018) | 72.37 | 71.25 | 57.76 | None
MUSE* | 66.17 ± 1.15 | 65.52 ± 0.78 | 55.46 ± 0.59 | None
Multilingual BERT* | 66.42 ± 1.15 | 69.21 ± 0.48 |  70.78 ± 0.36 | None
MLMA-Iden + s.w.s. | 69.45 ± 0.91 | 68.82 ± 0.82 | 55.75 ± 1.64 | None
MLMA-Iden + f.w.s. | 67.10 ± 0.78 | 68.15 ± 0.67 | 55.25 ± 1.29 | None
MLMA-Mv + s.w.s. | 73.81 ± 0.83 | 70.61 ± 1.79 | 57.70 ± 0.71 | None
MLMA-Mv + f.w.s. | 74.12 ± 1.00 | 71.72 ± 0.70 | 57.84 ± 0.80 | None
MLMA-Avl + s.w.s. | 75.01 ± 0.79 | 76.22 ± 0.42 | 60.98 ± 1.00 | None
MLMA-Avl + f.w.s. | 74.43 ± 0.50 | 76.02 ± 0.55 | 60.50 ± 0.43 | None
MLMA-Avl (init) + s.w.s. | 75.72 ± 0.80 |  76.90 ± 0.30 | 63.01 ± 0.83 | None
MLMA-Avl (init) + f.w.s. | 76.30 ± 0.76 | 76.85 ± 0.43 | 62.85 ± 0.47 | None
MLMA-Avl (multi) + s.w.s. |  79.36 ± 0.57 | 74.89 ± 0.28 | 65.93 ± 0.32 | None
MLMA-Avl (multi) + f.w.s. | 79.34 ± 0.35 | 74.74 ± 0.40 | 66.53 ± 0.35 | None
Table 1:  NER F1 scores on test sets of European languages. For previous work which reports multiple results, we only list their best performance on each language. Results of methods with mark * are obtained by running their released source code or models. The results of MUSE embeddings are produced by using them for direct model transfer. “MLM” denotes our multilingual language model without alignment. The three alignment methods, Iden = identical strings, Mv = mean and variance, Avl = average linkage, respectively. “s.w.s” and “f.w.s.” are self-weighted sum and fully-weighted sum. “init” represents using MUSE to initialize the embedding matrices in the MLMA. “multi” refer to the multi-source transfer. Model | zh
---|---
Wang and Manning (2014)$\diamond$ | 64.40
MUSE* | 35.35 ± 0.84
Xie et al. (2018)* | 44.13 ± 1.49
Our methods |
MLMA-Iden + s.w.s. | 11.08 ± 0.89
MLMA-Iden + f.w.s. | 11.17 ± 0.69
MLMA-Avl + s.w.s. | 50.11 ± 1.51
MLMA-Avl + f.w.s. | 45.88 ± 2.49
MLMA-Avl (init) + s.w.s | 60.33 ± 1.39
MLMA-Avl (init) + f.w.s | 58.92 ± 1.22
Table 2:  NER F1 scores on test sets for Chinese. The notations are the same as Table 1. Methods with mark $\diamond$ require parallel corpora. Model | es | nl | de | da | el | it | pt | sv | Avg.
---|---|---|---|---|---|---|---|---|---
Das and Petrov (2011)$\diamond$ | 84.2 | 79.5 | 82.8 | 83.2 | 82.5 | 86.8 | 87.9 | 80.5 | 83.31
Fang and Cohn (2017)$\dagger$ | 68.40 | 64.50 | 65.90 | 73.5 | 65.5 | 64.8 | 67.8 | 66.0 | 67.05
Fang and Cohn (2017)$\dagger\ddagger$ | 81.20 | 82.30 | 78.90 | 81.9 | 80.1 | 81.9 | 82.1 | 78.1 | 80.81
Xie et al. (2018)* | 73.25 | 75.46 | 80.72 | 29.75 | 71.65 | 71.19 | 76.48 | 64.36 | 67.86
MUSE* | 78.30 | 80.84 | 81.10 | 73.99 | 63.16 | 80.63 | 82.79 | 66.38 | 75.90
Multilingual BERT* | 83.86 | 84.79 | 87.16 | 83.77 | 82.27 | 88.39 | 87.86 | 81.07 | 84.90
Our methods |  |  |  |  |  |  |  |  |
MLMA-Avl + s.w.s. | 81.60 | 85.10 | 84.10 | 83.38 | 77.04 | 84.46 | 86.93 | 80.78 | 82.92
MLMA-Avl + f.w.s. | 81.20 | 85.54 | 84.92 | 83.45 | 77.48 | 84.80 | 87.43 | 80.72 | 83.19
MLMA-Avl (init) + s.w.s. | 82.73 | 85.79 | 85.76 | 82.44 | 80.55 | 86.76 | 85.99 | 79.75 | 83.72
MLMA-Avl (init) + f.w.s. | 82.27 | 85.97 | 86.37 | 82.25 | 81.31 | 86.28 | 86.53 | 80.00 | 83.87
Table 3:  POS accuracy on test sets of European languages. The notations are
consistent with Table 1. Fang and Cohn (2017) report different results
according to different resource requirements. We only list their best results
in each setting. Methods with mark $\diamond$, $\dagger$, $\ddagger$ require
parallel corpora, bilingual lexicons, and training data respectively.

We first train a multilingual language model without alignment (MLM) and
report its performance of cross-lingual NER in Table 1. The poor performance
demonstrates that only sharing part of the parameters in a language model is
far from enough for cross-lingual transfer.

As shown in Table 1, the mean/variance alignment strategy (MLMA-Mv) is
competitive with previous work which utilizes extra bilingual resources
(Section 5.1). The average linkage strategy (MLMA-Avl) performs a more precise
alignment and gains a further improvement. We conducted experiments of using
all three alignments together, and results show no significant improvement
over average linkage alone. These results agree with our statements that
average linkage performs a more precise matching, and thus, carries the
benefits brought by the other methods.

To demonstrate the strengths of the proposed cross-lingual contextualized
representations (CLCRs) over cross-lingual word embeddings (CLWEs), we also
report the results of using CLWEs for direct model transfer in Table 1.
Specifically, we compare with the unsupervised method MUSE from Conneau et al.
(2017). The experiment results demonstrate its effectiveness for cross-lingual
sequence labeling. The alignment method using identical strings (MLMA-Iden)
outperforms MUSE, suggesting that the contextual-level representations are
more effective than the word-level ones. The other proposed methods (MLMA-Mv
and MLMA-Avl) achieve significant improvement over MUSE and MLMA-Iden, which
shows the benefit of directly aligning the contextualized representations.

Combination with CLWEs We further demonstrate that CLWEs are compatible with
our methods by using MUSE embeddings to initialize the embedding layer of our
multilingual language model. The results of MLMA-Avl (init) shown in Table 1
indicate that the CLWEs lead to a better initialization and improved
performance.

Multi-source Transfer We conduct experiments of multi-source transfer based on
method MLMA-Avl and report the performance as MLMA-Avl (multi) in Table 1. The
experiment settings largely follow the previous work Mayhew et al. (2017).
They employ two source languages for each target language and use syntactic
features to choose the related source languages. For Spanish and German, we
use English and Dutch as source languages. English and Spanish are adopted for
Dutch. The multi-source transfer leads to a significant improvement for
Spanish and German, but a slight decline for Dutch. In the follow-up
experiment, we find that the Spanish training set achieves a poor cross-
lingual performance on Dutch. Similar results are observed in the experiments
of Spanish to English and Dutch to English. These results suggest that the
cross-lingual transfer may be directional, and we leave this issue for future
work.

Comparison with BERT We also compare the performance of our MLMA with the
released multilingual BERT Devlin et al. (2018). As shown in Table 1, our
MLMA-Avl achieves a better performance on Spanish and Dutch. For German, BERT
achieves a high performance as it employs effective subword information
through BPE. The architecture of BERT also performs better than LSTM.

It is worth mentioning that, in previous work and this work, the corpora used
in the experiments are limited to the source and the target language. In
contrast, the multilingual BERT is jointly learned on Wikipedia of 102
languages and may benefit from a multi-hop transfer. BERT employs a shared BPE
vocabulary for different languages, which implicitly performs a subword
alignment similar to MLMA-Iden. Meanwhile, the proposed MLMA-Mv and MLMA-Avl
methods are compatible with BERT and can be used to align the inner states of
BERT.

MUSE
---
brown: oliváceo (olive), negruzcas (blackish), negruzco (blackish), marrón
(brown), ocráceo (ochraceous)
chair: vicepresidenta (vice president), vicedecano (vice dean), cátedra
(chair), vicedecana (vice dean), catedrático (professor)
MLMA-Avl
[Brown]’s office told news outlets of his visit to Afghanistan … |  [Neira] escapó meses después rumbo a Miami para …
(Neira escaped months later heading to Miami to …)
Wearing a [brown] suit with matching hat, … |  La corona y vientre del macho son de un [verde] esmeralda brillante iridiscente, …
(The crown and belly of the male are of an iridescent bright emerald green, …)
Sweden currently holds the EU [chair]. |  Tras tomar posesión de su [asiento] , Lois decide limpiar el lago para empezar, …
(After taking possession of her seat, Lois decides to clean the lake to begin,
… )
It’s an honor to be asked to [chair] the Man Booker Prize, … |  …, y fue la primera mujer en [presidir] un sindicato AFL-CIO.
(…, and was the first woman to preside over an AFL-CIO union.)
Table 4: English words and their nearest Spanish words according to MUSE and
MLMA-Avl.

###  3.5 A Case Study of Chinese NER

We conduct experiments and evaluate our approaches on a distant language pair,
English-Chinese. The experiment results are shown in Table 2. Wang and Manning
(2014) utilize 80K parallel sentences for annotation projection and report a
strong performance. As Chinese and English do not share the alphabet, the
number of identical strings is significantly smaller than similar languages
pairs such as English-Spanish. Therefore, the MLMA-Iden achieves a lower
result comparing to MUSE which uses adversarial training. The MLMA-Avl method
performs a direct alignment of internal representations and achieves a
significant improvement over the word-level methods. The initialization from
CLWEs also proves its effectiveness for distant language pairs by gaining
further improvement and reaching a comparable result with Wang and Manning
(2014). This experiment suggests that cross-lingual transfer is still
challenging between distant language pairs.

###  3.6 Results for POS

We evaluate our methods on another sequence labeling task POS, and the results
are shown in Table 3. We compare with previous studies using unsupervised
cross-lingual clustering Fang and Cohn (2017) and large-scale parallel corpora
Das and Petrov (2011). As shown in Table 3, our models with deep semantic
alignment outperform previous lexicon-based cross-lingual clustering by a
large margin. When comparing to the previous method with a small amount of
training data, the MLMA-Avl method obtains an improved accuracy without
training data in the target languages. For further comparison, We also list
the performance of applying the method from Xie et al. (2018) and multilingual
BERT to the POS task.444The poor performance of Xie et al. (2018) on en-da is
due to the low quality of word translation pairs generated by their method.

POS mainly relies on the information of each single word, and parallel corpora
providing word alignment are effective for cross-lingual POS. Thus, previous
annotation projection methods through parallel corpora are strong approaches
for cross-lingual POS and often achieve a significantly better performance
against previous unsupervised methods. The experimental results show that the
proposed CLCRs are competitive and even achieve better average accuracy.

###  3.7 Self-Weighted v.s. Fully-Weighted Sum

As shown in Table 1, 2 and 3, we observe that Self-Weighted Sum (SWS)
generally outperforms Fully-Weighted Sum (FWS) in NER tasks, while the
opposite is true for POS tasks. SWS allows weights to vary at each position in
a sequence, while FWS imposes adaptive weights for each hidden dimension. We
hypothesize that NER is more context-sensitive and requires models to adapt to
different context information, which makes SWS a better option. On the other
hand, the POS of words is more independent across different context, but
certain feature dimensions in contextualized representations may be critical
for making a judgment. Therefore, FWS has the edge over SWS for its ability to
select out these dimensions.

##  4 What is Connected during Alignment?

In this section, we dive into the MLMA and investigate the question of what is
connected between different languages during the alignment. From English 1B
and Spanish Wikipedia, we randomly select 1,000 sentences for each language
and extract their cross-lingual contextual representations using our MLMA-Avl
model. We calculate the nearest neighbors in cosine distance for each word,
and some of them are listed in Table 4.

In these cases, the MLMA can disambiguate word senses according to context
information. For example, for the word brown in English, the MLMA groups color
brown with verde (green), and name Brown with Neira (a person name in Spanish)
in the Spanish corpus. The proposed method is different from unsupervised
translation in that, instead of learning a precise matching between English
and Spanish words, the CLCRs establishes a high-level semantic connection
between the source and the target language. The next example demonstrates that
the MLMA is able to distinguish the part-of-speech of words. It connects an
English verb chair with a Spanish verb presidir (preside), while a noun chair
with a noun asiento (seat) in Spanish. To compare with unsupervised cross-
lingual word embeddings, we list the top 5 similar words calculated using
MUSE. As shown in Table 4, MUSE successfully groups the English word brown
with Spanish words that are related to colors. However, without the help of
contextual information, its ability of word sense disambiguation is limited.

##  5 Related Work

Previous work in cross-lingual transfer learning can be roughly divided into
two main branches: annotation projection and model transfer.

###  5.1 Annotation Projection

In annotation projection approaches, parallel or comparable corpora are
commonly used Yarowsky et al. (2001); Ehrmann et al. (2011); Das and Petrov
(2011); Li et al. (2012); Täckström et al. (2013); Wang and Manning (2014); Ni
et al. (2017). The source language sentences of parallel corpora are first
annotated either manually or by a pre-trained tagger. Then, annotations on the
source side are projected to the target side through word alignment to
generate distantly supervised training data. Finally, a model of the target
language is trained on the generated data. Wikipedia contains multilingual
articles for various topics and can thus be used to generate
parallel/comparable corpora or even weakly annotated target language sentences
Kim et al. (2012).

However, parallel corpora and Wikipedia can be rare for true low-resource
languages. Mayhew et al. (2017) reduce the resource requirement by proposing a
cheap translation method, which “translates” the training data from the source
to the target language word by word through a bilingual lexicon. While Xie et
al. (2018) reduce the requirement of bilingual lexicons by an unsupervised
word-by-word translation through CLWEs.

###  5.2 Model Transfer

Model transfer methods train a model on the source language with language-
independent features. Thus, the trained model can be directly applied to the
target language.

McDonald et al. (2011) design a cross-lingual parser based on delexicalized
features like universal POS tags. Täckström et al. (2012) reveal that cross-
lingual word cluster features induced using large parallel corpora are useful.
Lexicon and Wikipedia also demonstrate effectiveness for language-independent
feature engineering. Zirikly and Hagiwara (2015) generate multilingual
gazetteers from the source language gazetteers and comparable corpus. Page
categories and linkage information to entries from Wikipedia are extracted as
strong language-independent features (wikifier features) Tsai et al. (2016).
Bharadwaj et al. (2016) facilitate the cross-lingual transfer through phonetic
features, which work well between languages like Turkish, Uzbek, and Uyghur,
but are not strictly language independent. Recently, CLWEs are used as
language-invariant representations for direct model transfer in NER Ni et al.
(2017) and POS Fang and Cohn (2017).

Some of the previous work also proposes sequence labeling models with shared
parameters between languages for performing cross-lingual knowledge transfer
Lin et al. (2018); Cotterell and Duh (2017); Yang et al. (2017); Ammar et al.
(2016); Kim et al. (2017). However, these models are usually obtained through
joint learning and require annotated data from the target language.

##  6 Conclusion

In this paper, we focused on a low-resources cross-lingual setting and
proposed transfer learning methods based on the alignment of deep semantic
spaces between different languages. The proposed multilingual language model
bridges different languages by automatically learning cross-lingual
disambiguated representations. Abundant NER and POS experiments are conducted
on the benchmark datasets. Experimental results show that our approaches using
only monolingual corpora achieve improved performance comparing to previous
strong cross-lingual studies with extra resources.

## References

  * Abend and Rappoport (2013) Omri Abend and Ari Rappoport. 2013.  Universal conceptual cognitive annotation (UCCA).  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 228–238, Sofia, Bulgaria. Association for Computational Linguistics.
  * Ammar et al. (2016) Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah Smith. 2016\.  Many languages, one parser.  _Transactions of the Association for Computational Linguistics_ , 4:431–444.
  * Bharadwaj et al. (2016) Akash Bharadwaj, David Mortensen, Chris Dyer, and Jaime Carbonell. 2016.  Phonologically aware neural model for named entity recognition in low resource transfer settings.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1462–1472. Association for Computational Linguistics.
  * Buchholz and Marsi (2006) Sabine Buchholz and Erwin Marsi. 2006.  Conll-x shared task on multilingual dependency parsing.  In _Proceedings of the Tenth Conference on Computational Natural Language Learning_ , CoNLL-X ’06, pages 149–164, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Chelba et al. (2013) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013.  One billion word benchmark for measuring progress in statistical language modeling.  _arXiv preprint arXiv:1312.3005_.
  * Chiu and Nichols (2016) Jason Chiu and Eric Nichols. 2016.  Named entity recognition with bidirectional lstm-cnns.  _Transactions of the Association for Computational Linguistics_ , 4:357–370.
  * Conneau et al. (2017) Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017.  Word translation without parallel data.  _arXiv preprint arXiv:1710.04087_.
  * Cotterell and Duh (2017) Ryan Cotterell and Kevin Duh. 2017.  Low-resource named entity recognition with cross-lingual, character-level neural conditional random fields.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 91–96. Asian Federation of Natural Language Processing.
  * Das and Petrov (2011) Dipanjan Das and Slav Petrov. 2011.  Unsupervised part-of-speech tagging with bilingual graph-based projections.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1_ , HLT ’11, pages 600–609, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Ehrmann et al. (2011) Maud Ehrmann, Marco Turchi, and Ralf Steinberger. 2011.  Building a multilingual named entity-annotated corpus using annotation projection.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing 2011_ , pages 118–124. Association for Computational Linguistics.
  * Fang and Cohn (2017) Meng Fang and Trevor Cohn. 2017.  Model transfer for tagging low-resource languages using a bilingual dictionary.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 587–593. Association for Computational Linguistics.
  * Hovy et al. (2006) Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006.  Ontonotes: The 90% solution.  In _Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers_ , NAACL-Short ’06, pages 57–60, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Jean et al. (2015) Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015.  On using very large target vocabulary for neural machine translation.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1–10. Association for Computational Linguistics.
  * Kim et al. (2017) Joo-Kyung Kim, Young-Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. 2017.  Cross-lingual transfer learning for pos tagging without cross-lingual resources.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2832–2838. Association for Computational Linguistics.
  * Kim et al. (2012) Sungchul Kim, Kristina Toutanova, and Hwanjo Yu. 2012.  Multilingual named entity recognition using parallel data and metadata from wikipedia.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 694–702. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 260–270. Association for Computational Linguistics.
  * Li et al. (2012) Qi Li, Haibo Li, Heng Ji, Wen Wang, Jing Zheng, and Fei Huang. 2012.  Joint bilingual name tagging for parallel corpora.  In _Proceedings of the 21st ACM International Conference on Information and Knowledge Management_ , CIKM ’12, pages 1727–1731, New York, NY, USA. ACM.
  * Li et al. (2017) Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. 2017.  Demystifying neural style transfer.  _arXiv preprint arXiv:1701.01036_.
  * Lin et al. (2018) Ying Lin, Shengqi Yang, Veselin Stoyanov, and Heng Ji. 2018.  A multi-lingual multi-task architecture for low-resource sequence labeling.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 799–809. Association for Computational Linguistics.
  * Long et al. (2015) Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan. 2015.  Learning transferable features with deep adaptation networks.  _arXiv preprint arXiv:1502.02791_.
  * Loper and Bird (2002) Edward Loper and Steven Bird. 2002.  Nltk: The natural language toolkit.  In _Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics-Volume 1_ , pages 63–70. Association for Computational Linguistics.
  * Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016.  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1064–1074. Association for Computational Linguistics.
  * Mayhew et al. (2017) Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017.  Cheap translation for cross-lingual named entity recognition.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2536–2545. Association for Computational Linguistics.
  * McDonald et al. (2011) Ryan McDonald, Slav Petrov, and Keith Hall. 2011.  Multi-source transfer of delexicalized dependency parsers.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 62–72. Association for Computational Linguistics.
  * Moseley and Wang (2017) Benjamin Moseley and Joshua Wang. 2017.  Approximation bounds for hierarchical clustering: Average linkage, bisecting k-means, and local search.  In _Advances in Neural Information Processing Systems_ , pages 3094–3103.
  * Newman (2012) Mark EJ Newman. 2012.  Communities, modules and large-scale structure in networks.  _Nature physics_ , 8(1):25.
  * Ni et al. (2017) Jian Ni, Georgiana Dinu, and Radu Florian. 2017.  Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1470–1480. Association for Computational Linguistics.
  * Nivre et al. (2007) Joakim Nivre, Johan Hall, Sandra Kübler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007.  The conll 2007 shared task on dependency parsing.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_.
  * Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016.  Universal dependencies v1: A multilingual treebank collection.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_.
  * Nothman et al. (2013) Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R. Curran. 2013\.  Learning multilingual named entity recognition from wikipedia.  _Artif. Intell._ , 194:151–175.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_.
  * Petrov et al. (2011a) Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011a.  A universal part-of-speech tagset.  _arXiv preprint arXiv:1104.2086_.
  * Petrov et al. (2011b) Slav Petrov, Dipanjan Das, and Ryan T. McDonald. 2011b.  A universal part-of-speech tagset.  _CoRR_ , abs/1104.2086.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Seifoddini (1989) Hamid K Seifoddini. 1989.  Single linkage versus average linkage clustering in machine cells formation applications.  _Computers & Industrial Engineering_, 16(3):419–426.
  * Smith et al. (2017) Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017.  Offline bilingual word vectors, orthogonal transformations and the inverted softmax.  _arXiv preprint arXiv:1702.03859_.
  * Täckström et al. (2013) Oscar Täckström, Dipanjan Das, Slav Petrov, Ryan McDonald, and Joakim Nivre. 2013.  Token and type constraints for cross-lingual part-of-speech tagging.  _Transactions of the Association for Computational Linguistics_ , 1:1–12.
  * Täckström et al. (2012) Oscar Täckström, Ryan McDonald, and Jakob Uszkoreit. 2012.  Cross-lingual word clusters for direct transfer of linguistic structure.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 477–487. Association for Computational Linguistics.
  * Tjong Kim Sang (2002) Erik F. Tjong Kim Sang. 2002.  Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition.  In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * Tsai et al. (2016) Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016.  Cross-lingual named entity recognition via wikification.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 219–228. Association for Computational Linguistics.
  * Tzeng et al. (2014) Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. 2014.  Deep domain confusion: Maximizing for domain invariance.  _arXiv preprint arXiv:1412.3474_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 5998–6008.
  * Wang and Manning (2014) Mengqiu Wang and Christopher D. Manning. 2014.  Cross-lingual projected expectation regularization for weakly supervised learning.  _Transactions of the Association for Computational Linguistics_ , 2:55–66.
  * Xie et al. (2018) Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A Smith, and Jaime Carbonell. 2018\.  Neural cross-lingual named entity recognition with minimal resources.  _arXiv preprint arXiv:1808.09861_.
  * Yang et al. (2017) Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. 2017.  Transfer learning for sequence tagging with hierarchical recurrent networks.  _arXiv preprint arXiv:1703.06345_.
  * Yarowsky et al. (2001) David Yarowsky, Grace Ngai, and Richard Wicentowski. 2001.  Inducing multilingual text analysis tools via robust projection across aligned corpora.  In _Proceedings of the First International Conference on Human Language Technology Research_ , HLT ’01, pages 1–8, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Yim and Ramdeen (2015) Odilia Yim and Kylee T Ramdeen. 2015.  Hierarchical cluster analysis: comparison of three linkage measures and application to psychological data.  _The quantitative methods for psychology_ , 11(1):8–21.
  * Zirikly and Hagiwara (2015) Ayah Zirikly and Masato Hagiwara. 2015.  Cross-lingual transfer of named entity recognizers without parallel corpora.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 390–396. Association for Computational Linguistics.