CI: Don't rebuild pandas before running tests
It appears some unit test jobs are rebuilding pandas before running `pytest`. Meson might be rebuilding pandas because an optional dependency installed pandas before the initial build, so trying to uninstall pandas before the initial build

cc @lithomas1 
Sweet! Merging