Your last job sounds terrible.
The people rowing the boat are slaves --- their conditions seem dire and their job terrible. But, even the rowers take pity on the businessman, who they judge to have an even more terrible job previously. This surprising conclusion is joke is about how modern jobs are cushy, but might still feel like slavery in some ways (even though they aren't really comparable).