Pedestrians aren't vehicles and we aren't predictable enough to mix with other traffic.  The "right of way" on sidewalks means pedestrians can stop at any time or suddenly go in any direction.  I was once hit by a bike because I didn't know he was coming up behind me when I stepped to the side to avoid a big hole in the sidewalk.  Had another near miss when someone jumped a curb just as I pivoted away from looking in a shop window.  

It is perfectly legal to walk your bike on a sidewalk, and I'd suggest that is what you should do when there are any pedestrians nearby if you really respect their rights and safety.   And if it helps tweak your self-interest, remember that if you do really hurt somebody, they can sue you into the dirt.