Smith experienced multiple financial and personal setbacks. He bought an enslaved Black man for 400 dollars. However, he wanted to return to his old enslaver, so Smith released him. Venture Smith purchased another enslaved Black man for 25 pounds. After his daughter, Hannah was married, she came down with an illness. However, her husband did not pay adequate attention to her health. Although Smith nursed her, she died. In addition, two people he enslaved ran away. In his early sixties, he was unjustly charged with the loss of a white man's property and charged for ten pounds while visiting New London. Although being absent from the scene with witnesses, he was still prosecuted. Smith then carried this matter to other courts, claiming his innocence, but the judgment was never reversed. This misleading judgment was made out of discrimination against Black people. Remembering this experience, he said: "Captain Hart was a "white gentleman", and I a "poor African, therefore it was all right, and good enough for the black dog". "