Goldstein was born in Portland, Victoria. Her family moved to Melbourne in 1877 when she was around eight years old, where she would attend Presbyterian Ladies' College. Goldstein followed her mother into the women's suffrage movement and soon became one of its leaders, becoming known both for her public speaking and as an editor of pro-suffrage publications. Despite her efforts, Victoria was the last Australian state to implement equal voting rights, with women not granted the right to vote until 1908.