The applicability domain (AD) (for both chemistry and machine learning) of a QSAR model is the physico-chemical, structural or biological space, knowledge or information on which the training set of the model has been developed, and for which it is applicable to make predictions for new compounds.
The purpose of AD is to state whether the model's assumptions are met, and for which chemicals the model can be reliably applicable. In general, this is the case for interpolation rather than for extrapolation. Up to now there is no single generally accepted algorithm for determining the AD: a comprehensive survey can be found in a Report and Recommendations of ECVAM Workshop 52. There exists a rather systematic approach for defining interpolation regions. The process involves the removal of outliers and a probability density distribution method using kernel-weighted sampling. 
Another widely used approach for the structural AD of the regression QSAR models is based on the leverage calculated from the diagonal values of the hat matrix of the modeling molecular descriptors. 
A recent rigorous benchmarking study of several AD algorithms identified standard-deviation of model predictions as the most reliable approach.
To investigate the AD of a training set of chemicals one can directly analyse properties of the multivariate descriptor space of the training compounds or more indirectly via distance (or similarity) metrics. When using distance metrics care should be taken to use an orthogonal and significant vector space. This can be achieved by different means of feature selection and successive principal components analysis.