Because of the limited specificity of diagnostic imaging, many breast lesions referred for biopsy turn out to be benign. The objective of this study was to evaluate whether diffusion tensor MRI (DTI) parametric maps can be used to safely avoid biopsy of breast lesions. Individuals referred for breast biopsy based on mammogram (MG), ultrasound (US), and/or contrast enhanced (CE)-MRI were recruited. Scans consisting of T2-weighted and DTI sequences were performed. Multiple DTI-derived parametric color maps were evaluated semi-quantitatively to characterize lesions as “definitely benign,” “not definitely benign,” or “suspicious.” All patients subsequently underwent biopsy. In this moderately-sized prospective study, 21 out of 47 pathologically proven benign lesions were characterized by both readers as “definitely benign,” which would have precluded the need for biopsy. Biopsy was recommended for 11 out of 13 cancers that were characterized as “suspicious.” In the remaining two cancers and 26 of 47 benign lesions, the scans were characterized as “not definitely benign” and hence required biopsy. The main causes for “not definitely benign” scans were small lesion sizes and noise. The results suggest that in appropriately selected patients, DTI may be used to safely reduce the number of unnecessary breast biopsies.