Mr Harper spoke at a funeral for Cpl Nathan Cirillo, 24, who was shot by Michael Zehaf-Bibeau while standing guard unarmed at Canada's war memorial.
Zehaf-Bibeau, 32, then entered Canada's parliament and fired a number of shots.
Mr Harper choked back tears as he addressed Cirillo's five-year-old son, Marcus Daniel Cirillo.
"May time ease the searing pain of today," Mr Harper said. "And may [Marcus] some day find comfort in the fact that our entire country looks up to his dad with pride, with gratitude [and] with deep abiding respect."
Separately, US Secretary of State John Kerry has paid his respects in his first trip to Canada as America's top diplomat, laying a wreath at the National War Memorial in honour of the Canadian soldier.
He told reporters the US expresses "deep solidarity with our country's closest ally... [and] we offer our heartfelt condolences and prayers".
On Tuesday, a procession of 4,500 people walked along closed streets in Cpl Cirillo's hometown of Hamilton, Ontario, as a crowd lined the route to the funeral.
At the funeral, Mr Harper said Cpl Cirillo died "protecting and preserving" the memorial to fallen soldiers.
"Freedom is never free, it has been earned by the soldier then donated to all of us... Our hearts are broken at his loss, but our spirits are grateful for his memory," Mr Harper said.
Police are continuing to analyse a video that Zehaf-Bibeau made prior to the attack and are investigating several individuals in contact with the gunman before the incident.
Zehaf-Bibeau, a drifter with a record of petty crime and drug use, was killed by the House of Commons sergeant-at-arms inside the parliament building.
Authorities said that in the period leading up to the attack he had sought to obtain a Canadian passport for travel to Syria.
On Monday, Royal Canadian Mounted Police (RCMP) Commissioner Bob Paulson told the Canadian Senate that more must be done to prevent home grown radicals from receiving militant training overseas.
Canadian Prime Minister Stephen Harper told thousands of mourners the nation was deeply grateful to a soldier killed by a gunman in Ottawa last week.