The difficulties of a high Isp OTV include:
Long transfer times (radiation damage from VanAllen belts for both
 the spacecraft and OTV
Arcjets or Xenon thrusters require huge amounts of power so you have
to have either nuclear power source (messy, dangerous and source of
radiation damage) or BIG solar arrays (sensitive to radiation, or heavy)
that make attitude control and docking a big pain.

If you go solar, you have to replace the arrays every trip, with
current technology. Nuclear power sources are strongly restricted
by international treaty.

Refueling (even for very high Isp like xenon) is still required and]
turn out to be a pain.

You either have to develop autonomous rendezvous or long range teleoperation
to do docking or    ( and refueling) .

You still can't do much plane change because the deltaV required is so high!

The Air Force continues to look at doing things this way though. I suppose
they are biding their time till the technology becomes available and
the problems get solved.  Not impossible in principle, but hard to
do and marginally cheaper than one shot rockets, at least today.

Just a few random thoughts on high Isp OTV's. I designed one once...