but he had refused , saying that 's why he bought all the lots around ours in the first place , to prevent more neighbors like us , and that he suspected that if we owned them we 'd build on the lots ourselves .