Everyone thinks that the Pope only became infallible in 1870. Now there I have made the mistake you made by saying "everyone thinks", you said "everyone agrees", same thing. I should have said that, some, many, or most people think. 
If the only two infallible doctrines are the Immaculate Conception and the Assumption, what about the rest of the Church's dogmas and doctrines, are they all fallible, liable to change and or revocation?  Come to think of it, is the teaching of Vatican I on Papal Infallibility itself, infallible? If not then how can we say that the two Marian dogmas are themselves infallible? We could go on and on, is the Canon of Scripture set in stone or liable to revision?
You consider my position absurd, fair enough. I cannot understand your position, not so much on OS but on deciding which teachings you accept or reject and why. It smacks to me too much like the 'cafeteria', tailoring one's religion to suit one's lifestyle rather than the other way round.