CI: Correctly set PANDAS_CI on ubuntu builds
This looks to be not correctly set for all ubuntu builds post https://github.com/pandas-dev/pandas/pull/51640
Looks to be correctly set now for the non matrix builds. Merging