The military has not made public the reason for Vice Admiral Mark Norman's suspension.
Federal Defence Minister Harjit Sajjan said in a short statement that he fully supported the decision to relieve Vice Adm Norman from carrying out his military duties.
Vice Adm Norman was named to the post last August.
He was removed from of his duties "effective immediately and until further notice", according to an order from Gen Jonathan Vance, Canada's chief of the defence staff, which is dated 13 January.
Gen Vance is Canada's top soldier in uniform and commands the military. His letter did not offer any explanation for Vice Adm Norman's dismissal.
A spokesman for the general said Vice Adm Norman has been removed "from the performance of military duty. For the time being, he will not be carrying out the functions of [vice chief of the defence staff]".
He has been replaced on an interim basis by current navy commander Vice Admiral Ron Lloyd.
Vice Adm Norman is a career naval officer who has spent over three decades in Canada's military, joining in 1980 as a navy reservist.
The second highest-ranking officer in the Canadian military has been temporarily relieved from his duties.