One of the big challenges in machine learning applications is that training data can be different from the real-world data faced by the algorithm. In language modeling, users’ language (e.g. in private messaging) could change in a year and be completely different from what we observe in publicly available data. At the same time, public data can be used for obtaining general knowledge (i.e. general model of English). We study approaches to distributed fine-tuning of a general model on user private data with the additional requirements of maintaining the quality on the general data and minimization of communication costs. We propose a novel technique that significantly improves prediction quality on users’ language compared to a general model and outperforms gradient compression methods in terms of communication efficiency. The proposed procedure is fast and leads to an almost 70% perplexity reduction and 8.7 percentage point improvement in keystroke saving rate on informal English texts. Finally, we propose an experimental framework for evaluating differential privacy of distributed training of language models and show that our approach has good privacy guarantees. Two common problems arising after deployment of a machine learning model on user devices are discrepancy between training data and actual data stored on user devices, and the need of regular model updates. In the case of language modeling, it corresponds to the difference between language and style of the training corpus mined in the Internet and messages of the user, which account for most of the text generated on the device. Even if the training corpus includes a substantial part of informal texts (tweets, forum threads, etc.), real user data can be very different. This is a challenge for word prediction algorithms in software keyboard applications. The most general approach to improvement of customer experience in typing is integrating a separate user language model trained on device in an on-line fashion. In the simplest case it is a smoothed n-gram (e.g. Kneser-Ney n-gram model BID6 )).In BID26 continuously learned personalized language model based on LSTM was proposed but as far as each user generates only a small portion of textual data, such data by itself cannot be used for updates of the general model. Thus , for a model update, a collection of potentially sensitive data from many users is needed. As shown in , collecting data for training may be avoided. We propose a similar approach for distributed fine-tuning of language models on private data. In this sense our method can be considered as "federated fine-tuning" but we prefer to take more traditional term. In this setting we start with a language model trained on a large text corpus representing the general language. This model G will be updated continuously on user devices but with an additional requirement that the model must not go too far from the general language model, i.e. we don't overfit on user data.We pursue two goals: 1) to develop an algorithm of distributed fine-tuning that is fast, communication efficient and doesn't need collecting sensitive user data; and 2) to prevent the language model from forgetting "general English". Besides , we provide analysis of possibility of privacy violation After each round the server model G t+1 is sent to the next K elements. in our model. BID8 ) demonstrated an attack on distributed training algorithm leading to information leakage. This means that privacy analysis in necessary for such algorithms.Our main contributions are: 1) we propose an efficient procedure of distributed fine-tuning of language models immune to the problem of catastrophic forgetting BID3 ), 2) we provide experimental evaluation of on-device training time, communication costs and convergence rates of the general language model in realistic conditions, 3) we compare two most popular strategies of improving communication efficiency in the context of distributed learning, and 4) we propose an experimental framework for evaluation of differential privacy of distributed training of language models, and using this framework, we evaluate privacy guarantees of our approach.In our research we are focused on improvement of keystroke saving rate (see section 2.4) because this metric reflects customer typing experience more directly than perplexity or BLEU. We use LSTM architecture for our language model as described in BID27 and evaluate ondevice training time for this architecture. We show that the on-device training time is reasonably small, thus demonstrating the feasibility of the whole approach. We have presented our results in distributed fine-tuning of neural language models. We paid special attention to preventing a catastrophic forgetting of the general language after a model fine-tuning on the user devices. Our experiments showed that the performance of an initial model of the general English on user data can be improved significantly almost without a performance degradation on the standard English training data. We found that a combination of on-device training with random rehearsal and server-side model averaging provides the best performance for such distributed finetuning. Users' models were trained for the whole epoch that reduced communication costs while at the same time being quite fast -it took less than 3 minutes with a realistic assessment of volume of the available user data. Finally, we provided an experimental evaluation of differential privacy of our method and showed that the method has a reasonable level of differential privacy compared to other solutions. We still have to note that we provided an empirical estimation of differential privacy which holds with some high probability but not almost surely.This statistic doesn't converge to the Kolmogorov distribution as shown in W. Lilliefors (1969) . It converges to the distribution with smaller critical values at the same significance levels because we overfit on the sample data when the estimator r is plugged in. We chose a 5% significance level and critical value for it is 1.08. In 19 cases out of 20 the Lilliefors test failed to reject the null hypothesis at a 5% significance level. TAB4 provides exact values obtained during the application of the statistical test. Relying on these values along with data visualization in 3 we can state that random variable c(s) has tails that decrease like the Pareto distribution tails.The hypothesis that we accepted suggests that the cumulative distribution function of c(s) is given by the formula (8). It means that the tail distribution function for all x > x 0 is given by DISPLAYFORM0 We chose x 0 = c (k) n , so F (x 0 ) is just the ratio k/n. Thus, C can be estimated by DISPLAYFORM1 Values of C are given in the TAB4 . Finally, from formula (11) and proposition 1 it is easy to derive that (ε, δ)-differential privacy is provided by the values ε, δ that satisfy DISPLAYFORM2
We propose a method of distributed fine-tuning of language models on user devices without collection of private data This paper deals with improving language models on mobile equipments based on small portion of text that the user has inputted by employing a linearly interpolated objectives between user specific text and general English. 