It is not the height, I am 5'1 and have dated men that were my exact height.  You may need to broaden your horizons and join the gym or a local group of fitness people.  I am also 39 and I value my appearance, I don't have a house full of kids nor do I engage in drugs or the like.  Keep looking buddy, we are out there....