The victims are said to be from an ultra-Orthodox Jewish family who were walking to the Western Wall.
The attacker is said to have stolen a gun from one of the victims and fired a number of shots before being shot dead by Israeli police.
The two injured victims, a woman and a young child, are in hospital.
It comes two days after an Israeli couple, who were in a car with their four children, were shot dead in the West Bank.
The latest attack took place just after the end of the Jewish Sabbath close to Lion's Gate in the Old City.  The victims are said to have been passing near the entrance of the al-Aqsa Mosque compound, en-route to the Western Wall.
Jerusalem police chief Moshe Edri said the Palestinian man had "stabbed an Israeli man, his wife and the toddler" several times in an alleyway before stabbing another man.
Both Israeli men had died of their wounds, he said, adding that the mother and the toddler were in a serious condition.
Police spokeswoman Luba Samri said the Palestinian attacker had taken a gun from one of the wounded men and opened fire at police and tourists. He was then shot and killed by an Israeli police officer who had rushed to the scene.
Police later identified the attacker as a 19-year-old from al-Bireh, near Ramallah in the West Bank.
Palestinian militant group Islamic Jihad has issued a statement claiming him as one of its members.
There has been a recent flare-up in tensions between Israel and Palestinians, with violent confrontations between security forces and Palestinian youths in a compound holy to both Jews and Muslims in East Jerusalem.
Earlier this week, Palestinian leader Mahmoud Abbas told the United Nations General Assembly that Israel risked creating "an explosive situation" in Jerusalem and the West Bank with its use of "brutal force".
In his address to the UN chamber on Thursday, Israeli Prime Minister Netanyahu called on Mr Abbas to "stop spreading lies" about Israel and return to peace talks.
Two Israelis have been killed and another two injured in a knife attack by a Palestinian man in the Old City of Jerusalem, police say.