One can obtain a more accurate preconditioner by allowing some level of extra fill in the factorization. A common choice is to use the sparsity pattern of A2 instead of A; this matrix is appreciably more dense than A, but still sparse over all.
In the context of preconditioners, allowing for some extra data can lead to a more accurate result. One common strategy is to use the sparsity pattern of A2, which is more dense than A but still sparse overall.