Backport PR #33102 on branch 1.0.x (PERF: fix performance regression in memory_usage(deep=True) for object dtype)
Backport PR #33102: PERF: fix performance regression in memory_usage(deep=True) for object dtype
if we decide to do a 1.0.4 we can see what is backportable.