We demonstrate a low effort method that unsupervisedly constructs task-optimized embeddings from existing word embeddings to gain performance on a supervised end-task. This avoids additional labeling or building more complex model architectures by instead providing specialized embeddings better fit for the end-task(s). Furthermore, the method can be used to roughly estimate whether a specific kind of end-task(s) can be learned form, or is represented in, a given unlabeled dataset, e.g. using publicly available probing tasks. We evaluate our method for diverse word embedding probing tasks and by size of embedding training corpus -- i.e. to explore its use in reduced (pretraining-resource) settings. Unsupervisedly pretrained word embeddings provide a low-effort, high pay-off way to improve the performance of a specific supervised end-task by exploiting Transfer learning from an unsupervised to the supervised task. Additionally, recent works indicate that universally best embeddings are not yet possible, and that instead embeddings need to be tuned to fit specific end-tasks using inductive bias -i.e. semantic supervision for the unsupervised embedding learning process BID1 BID13 . This way, embeddings can be tuned to fit a specific Single-task (ST) or Multi-task (MT: set of tasks) semantic BID16 BID7 . Hence the established notion, that in order to fine-tune embeddings for specific end-tasks, labels for those endtasks a required. However, in practice, especially in industry applications, labeled dataset are often either too small, not available or of low quality and creating or extending them is costly and slow.Instead, to lessen the need for complex supervised (Multi-task) fine-tuning, we explore using unsupervised fine-tuning of word embeddings for either a specific end-task (ST) or a set of desired end-tasks (MT). By taking pretrained word embeddings and unsupervisedly postprocessing (finetuning) them, we evaluate postprocessing performance changes on publicly available probing tasks developed by BID6 1 to demonstrate that widely used word embeddings like Fasttext and GloVe can either: (a) be unsupervisedly specialized to better fit a single supervised task or, (b) can generally improve embeddings for multiple supervised end-tasks -i.e. the method can optimize for single and Multi-task settings. As in standard methodology, optimal postprocessed embeddings can be selected using multiple proxy-tasks for overall improvement or using a single endtask's development split -e.g. on a fast baseline model for further time reduction. Since most embeddings are pretrained on large corpora, we also investigate whether our method -dubbed MORTY -benefits embeddings trained on smaller corpora to gauge usefulness for low-labeling-resource domains like biology or medicine. We demonstrate the method's application for Single-task, Multitask, small and large corpus-size setting in the evaluation section 3. Finally, MORTY (sec. 2), uses very little resources 2 , especially regarding recent approaches that exploit unsupervised pretaining to boost end-task performance by adding complex pretraining components like ELMo, BERT BID14 BID2 which may not yet be broadly usable due to their hardware and processing time requirements. As a result, we demonstrate a simple method, that allows further pretraining exploitation, while requiring minimum extra effort, time and compute resources. We demonstrated a low-effort method to unsupervisedly construct task-optimized word embeddings from existing ones to gain performance on a (set of) supervised end-task(s). Despite its simplicity, MORTY is able to produces significant performance improvements for Single and Multi-task supervision settings as well as for a variety of desirable word encoding properties -even on smaller corpus sizes -while forgoing additional labeling or building more complex model architectures.
Morty refits pretrained word embeddings to either: (a) improve overall embedding performance (for Multi-task settings) or improve Single-task performance, while requiring only minimal effort.