Deep learning (DL) methods have shown great promise in auto-segmentation problems. However, for head and neck cancer, we show that DL methods fail at the axial edges of the gross tumor volume (GTV) where the segmentation is dependent on information closer to the center of the tumor. These failures may decrease trust and usage of proposed auto-contouring systems. To increase performance at the axial edges, we propose the spatially adjusted recurrent convolution U-Net (SARC U-Net). Our method uses convolutional recurrent neural networks and spatial transformer networks to push information from salient regions out to the axial edges. On average, our model increased the Sørensen–Dice coefficient (DSC) at the axial edges of the GTV by 11% inferiorly and 19.3% superiorly over a baseline 2D U-Net, which has no inherent way to capture information between adjacent slices. Over all slices, our proposed architecture achieved a DSC of 0.613, whereas a 3D and 2D U-Net achieved a DSC of 0.586 and 0.540, respectively. SARC U-Net can increase accuracy at the axial edges of GTV contours while also increasing accuracy over baseline models, creating a more robust contour.