The second explanation is genetic. Some of the homozygous genes will be "deleterious recessives" which would normally be shielded by a dominant allele. Also, there are some loci where the heterozygote is inherently fitter than either homozygote. If so, inbreeding will automatically cause a loss of vigour.