I have been eating fresh and brined olives for years. I like the olives with the pits and usually consume the whole pit without removing it. It was recently pointed out to me that this is not a healthy practice, and even though I have been doing this for some time, I began to wonder whether it is 1)abnormal to eat the whole olive, pit and all, and 2) will the pits have a long term adverse effect? I have never met anyone who ate olive pits, and I find the with-pit olives to be the best, so I thought I would ask. Note that I do not grind up and chew the pits, as I am sure that my teeth would not like that! 
In our culture we eat olive pits (especially Kalamata) for generations and no one has ever had a problem with eating them in small quantities (3-4 max).  They seem to be digestible (never noticed any of them coming out! :) 