If you add acid to the solution, the concentration of H+ ions will increase; to keep equilibrium a small number of ions will be combined (forming salt and reducing the concentration of H+ ion in the solution). If you add base the concentration of H+ ion will reduce (by consumption or combining) and so a small amount of salt will break into ions and maintain the pH.