He divorced his wife Sarah and in 1875, Baldwin moved to Southern California and bought the Rancho Santa Anita in the fertile San Gabriel Valley from Harris Newmark. Baldwin offered $150,000, but Newmark wanted $200,000. Baldwin at first refused to pay the premium price, but his lawyer Reuben Lloyd told him he should buy it before the price rose further. Baldwin met with Newmark again, carrying a tin-box containing several million dollars, and withdrew $12,500 cash from it as a first payment. Baldwin invested in the Temple and Workman Bank, and when it failed, he acquired even more land through default. Baldwin eventually bought over of land. Baldwin made additions to the Hugo Reid Adobe house by Lake. In 1940 the home became a California Historic Landmark.