Their discussion was not focused on the curtains, but on a political move that left jaws in Westminster on the floor - the return of Mr Cameron to frontline politics after a seven-year hiatus.
It has prompted praise from some quarters of the Conservative party, anger from others.
Concern about the accountability of an unelected foreign secretary and claims from opposition parties that Rishi Sunak has holed his own pitch as the change candidate beneath the waterline.
So, what was the thinking behind it, and how did it come about?
Despite speculation, the former foreign secretary William Hague dismissed the suggestion that he'd brokered the arrangement.
"No!" he told Times Radio when asked if it was down to him.
"I knew about it a few days before and spoke to David Cameron to brief him about my views on foreign affairs and the Foreign Office, but it wasn't my idea.
"Sometimes in politics things are simpler than they look - sometimes someone just asks someone round for a chat and says why don't you do this, and they say OK fine, and it doesn't need any intermediary, they just sort it out for themselves and that happened in this case."
Rishi Sunak has, according to Downing Street sources, spoken to Lord Cameron from time-to-time since he became prime minister, with Mr Cameron being a "helpful sounding board" on issues.
One former aide to Lord Cameron suggested the conversations intensified after Mr Sunak's conference speech, in which he claimed politics had failed for 30 years and announced the cancellation of the northern leg of HS2.
That prompted a relatively rare intervention from the then Mr Cameron, who publicly criticised Mr Sunak's decision on HS2.
His former aide said: "It was an unusual thing for him to step in and say something, and he wouldn't have done it lightly.
"No 10 realised they might have upset somebody who should be an ally, and who is not usually critical. There was a meeting, and discussions came out of that."
They said the actions of Suella Braverman - who had increasingly become a thorn in Downing Street's side with public statements that distracted from the government's planned agenda - were the "catalyst" for the move. Though the prospect of a cabinet reshuffle had been in the offing for some time.
Last week, Lord Cameron met Mr Sunak in Downing Street and was asked if he would become foreign secretary.
On Sunday, when they stood together in the line-up of prime ministers at the Remembrance Day ceremony at the cenotaph, there was no sign that both had already agreed on the political bombshell that would drop the very next day.
It is rare that a secret is so well kept in Whitehall.
One current government advisor said they were "totally blindsided" by the move - and they are not the only one, suggesting it was kept to a tight circle.
Figures in government and beyond have expressed incredulity the news did not leak in advance, as so often happens in Westminster.
But if the timing and nature of Lord Cameron's return to frontline politics shocked, his decision to do so did not come entirely out of the blue, according to some who know him.
One former advisor, who worked with Lord Cameron in Downing Street, said he was "surprised but not surprised".
"He was really acutely aware of not being that ex-prime minister sniping from side-lines" he said.
"Any intervention he made was not gratuitous.  But the paradox was he did want to come back to public service life.
"There was definitely an itch that needed scratching."
Allies of Lord Cameron suggest the job of foreign secretary is one that would have particularly appealed, where he can reach into his global contacts book and bring his experience to bear within a complex brief on the world stage.
The Sun newspaper even claimed it was an ambition of his as long as five years ago.
Not all Conservative MPs are thrilled with the appointment though, with some already expressing concern about what his return signals about the direction of the party.
And Labour spot an opportunity. One senior figure pointed out that with both the chancellor and foreign secretary - Jeremy Hunt and Lord Cameron - being figures who were in government way back in 2010 when the Conservatives first came to power - scrutinising the entire period of Tory government since then will seem more legitimate.
The prime minister, who only arrived in parliament in 2015, really cannot do the "don't blame me guv routine" anymore, they argue.
There is no doubt that when Lord Cameron strolled up Downing Street it was one of the rare moments that sent a genuine shock wave through Westminster. Let's see how it washes up.
Last Tuesday in the flat above Downing Street that used to be his family home, David Cameron met Rishi Sunak.