Both.  It depends on the context.  Someone can say, "things are good but only going to go downhill from here" meaning things are going to get worse.  Or they can say "that was easy, and it's only goes downhill from here" meaning things will be easier.