The output of the amplifier will not drift from zero when there is no input. The capacitance (C) of the capacitor and the input and output resistances of the stages form an RC circuit. This acts as a crude high-pass filter.
When no signal is present at the input, the output of the amplifier will remain stable at zero. This stability is due to the combination of capacitance (C) in the circuit and the input and output resistances of the stages forming an RC filter that acts as a high-pass filter.