soon the congressmen and the councilmen returned to joe 's bagmen with `` no , '' or `` we 'll need more time to consider all the variables , '' though joe noted the one thing they did n't return with was his money .