Model training remains a dominant financial cost and time investment in machine learning applications. Developing and debugging models often involve iterative training, further exacerbating this issue. With growing interest in increasingly complex models, there is a need for techniques that help to reduce overall training effort. While incremental training can save substantial time and cost by training an existing model on a small subset of data, little work has explored policies for determining when incremental training provides adequate model performance versus full retraining. We provide a method-agnostic algorithm for deciding when to incrementally train versus fully train. We call this setting of non-deterministic full- or incremental training ``Mixed Setting Training". Upon evaluation in slot-filling tasks, we find that this algorithm provides a bounded error, avoids catastrophic forgetting, and results in a significant speedup over a policy of always fully training. The recent explosion in machine learning interest has led to substantial societal impact. However, with ever-growing model complexity comes corresponding growth in training time and cost. Recent models such as BERT (Devlin et al., 2019) can cost thousands of dollars and days to complete training, and similarly complex models consume staggering amounts of energy during training (Strubell et al., 2019) . Approaches for reducing the training burden are necessary for enabling continued growth in the machine learning community. Techniques such as incremental training can reduce the total time and financial investments associated with training models, while also enabling model refinement when original training data is unavailable. By using newly-added data as a basis for training, incremental training can lead to dramatic train-time speedups with marginal losses in accuracy (Ade & Deshmukh, 2013) . However, little work has explored developing policies that inform users when incremental training can be used versus a full retraining phase of a model. Previous research/findings in incremental training have been limited to a single setting in which the model either incrementally trains, or fully trains. In this paper, we describe an algorithm that chooses whether to train a model fully after a new observation of data, or simply leave the model incrementally trained. We refer to this kind of training as "Mixed Setting Training", as the model will have a decision rule of whether to train incrementally or not. Our goals for this method are as follows: • Training speedup. Training represents a substantial burden financially and with respect to time. We seek a technique that saves training time overall. • Bounded errors. We seek to apply incremental training such that a model's predictive performance is retained. We must provide a guarantee about how much error we introduce, and fully train instead if we cannot meet that guarantee. • Prevents catastrophic forgetting. Approaches to saving on training time should avoid catastrophic forgetting. We propose a new method, called 'Parameterized Incremental Training', to meet these goals. Our design provides a substantial speedup over always fully training, and guarantees by design that there will be an upper bound on error ( %). A hyperparameter, N, describes the size of the reservoir and thus provides predictable space requirements. Constant checks over an unbiased sample of the dataset prevent catastrophic forgetting. Incremental training is an important technique when training models quickly with limited access to data. Research has been done previously on techniques, and our data shows a policy to decide when to incrementally train. We find that on slot-filling tasks on the ATIS and Opentable datasets that One Shot SGD and reservoir sampling methods have roughly equivalent accuracy on the training set. This is in contrast to the findings in Golmant et al. (2017) , where they found a significant accuracy increase for reservoir sampling. We suspect that slot-filling tasks differ architecturally from classification tasks such that one shot SGD provides better accuracy. We hope that this paper will open a field of inquiry into decision algorithms for training in a mixed setting.
We provide a method-agnostic algorithm for deciding when to incrementally train versus fully train and it provides a significant speedup over fully training and avoids catastrophic forgetting This paper proposes an approach for deciding when to incrementally vs. fully retrain a model in the setting of iterative model development in slot filling tasks.