The term  is known as the asymmetry term (or Pauli term).
The theoretical justification for this term is more complex. The Pauli exclusion principle states that no two identical fermions can occupy exactly the same quantum state in an atom. At a given energy level, there are only finitely many quantum states available for particles. What this means in the nucleus is that as more particles are "added", these particles must occupy higher energy levels, increasing the total energy of the nucleus (and decreasing the binding energy). Note that this effect is not based on any of the fundamental forces (gravitational, electromagnetic, etc.), only the Pauli exclusion principle.
Protons and neutrons, being distinct types of particles, occupy different quantum states. One can think of two different "pools" of states one for protons and one for neutrons. Now, for example, if there are significantly more neutrons than protons in a nucleus, some of the neutrons will be higher in energy than the available states in the proton pool. If we could move some particles from the neutron pool to the proton pool, in other words, change some neutrons into protons, we would significantly decrease the energy. The imbalance between the number of protons and neutrons causes the energy to be higher than it needs to be, for a given number of nucleons. This is the basis for the asymmetry term.
The actual form of the asymmetry term can again be derived by modeling the nucleus as a Fermi ball of protons and neutrons. Its total kinetic energy is
where  and  are the Fermi energies of the protons and neutrons. Since these are proportional to  and  respectively, one gets
:  for some constant C.
The leading terms in the expansion in the difference  are then
At the zeroth order in the expansion the kinetic energy is just the overall Fermi energy  multiplied by . Thus we get
The first term contributes to the volume term in the semi-empirical mass formula, and the second term is minus the asymmetry term (remember, the kinetic energy contributes to the total binding energy with a negative sign).
 is 38 MeV, so calculating  from the equation above, we get only half the measured value. The discrepancy is explained by our model not being accurate: nucleons in fact interact with each other and are not spread evenly across the nucleus. For example, in the shell model, a proton and a neutron with overlapping wavefunctions will have a greater strong interaction between them and stronger binding energy. This makes it energetically favourable (i.e. having lower energy) for protons and neutrons to have the same quantum numbers (other than isospin), and thus increase the energy cost of asymmetry between them.
One can also understand the asymmetry term intuitively as follows. It should be dependent on the absolute difference , and the form  is simple and differentiable, which is important for certain applications of the formula. In addition, small differences between Z and N do not have a high energy cost. The A in the denominator reflects the fact that a given difference  is less significant for larger values of A.