Address violation fix
Follow up to https://github.com/pandas-dev/pandas/pull/56000 looks like this still is an address violation when the array is empty, since `arr[0]` on an empty array is out of bounds
Thanks @WillAyd 