Assume a simple clinical study with N=200. Half of the participants are men and half of the participants are women. The hemoglobin of the participants ranges between 80 and 150. There's also several other variables.

I would like to split the data into training and tests for a classification task, in a way that the gender and hemoglobin levels would be balanced in each set.

It is easy to pick 50 male/female to each set, but having simultaneously similar hemoglobin levels is difficult. I guess ideally I should check that the mean and s.d. are on about the same range. How should I go about doing this?

If this has been considered in an article, refs. would be great!

EDIT: to clarify, I want to exclude the possibility of gender or hemoglobin inbalance from affecting the classifier result. It should only depend on the other data I have.


There is no reason to make this a classification task unless you have carefully-elicited patient-specific utilities. Ordinarily you would want to use risk prediction instead. And in my experience the minimum sample size needed to make data-splitting a reliable procedure for both model development and precise validation is in the neighborhood of 10,000-20,000 subjects. So I suggest that you develop a strategy to form a risk prediction model and use the optimism bootstrap for estimating the amount of overfitting of that process. Then both model development and model validation can use the whole sample. (It remains to be seen whether your sample size is adequate for the task.) Details are at http://biostat.mc.vanderbilt.edu/CourseBios330 under Handouts.

Please let us know what is $Y$ and what is its distribution.

  • $\begingroup$ Sorry, my question was not maybe very clearly written. So I'm afraid that any inbalance in the training and test sets with respect to gender or hemoglobin level would negatively affect the classifier performance. With stratification I want to exclude any such possibility. Or did I understand your answer correctly? Could please try to write in simpler terms, I'm still very new ML. $\endgroup$ – learner Oct 30 '14 at 15:54
  • $\begingroup$ Imbalance is the least of your worries. The sample size is very inadequate to split the data, and perhaps even for fitting a model on the whole dataset. $\endgroup$ – Frank Harrell Oct 30 '14 at 16:45

Your Answer

