The discovery that primary tumors condition distant organ sites of future metastasis for seeding by disseminating tumor cells through a process described as the pre-metastatic niche (PMN) formation revolutionized our understanding of cancer progression and opened new avenues for therapeutic interventions. Given the inherent inefficiency of metastasis, PMN generation is crucial to ensure the survival of rare tumor cells in the otherwise hostile environments of metastatic organs. Early on, it was recognized that preparing the “soil” of the distal organ to support the outgrowth of metastatic cells is the initiating event in PMN development, achieved through the remodeling of the organ’s extracellular matrix (ECM). Remote restructuring of ECM at future sites of metastasis under the influence of primary tumor-secreted factors is an iterative process orchestrated through the crosstalk between resident stromal cells, such as fibroblasts, epithelial and endothelial cells, and recruited innate immune cells. In this review, we will explore the ECM changes, cellular effectors, and the mechanisms of ECM remodeling throughout PMN progression, as well as its impact on shaping the PMN and ultimately promoting metastasis. Moreover, we highlight the clinical and translational implications of PMN ECM changes and opportunities for therapeutically targeting the ECM to hinder PMN formation.