Although the smart home industry is rapidly emerging, it faces the risk of privacy security that cannot be neglected. As this industry now has a complex combination system involving multiple subjects, it is difficult for the traditional risk assessment method to meet these new security requirements. In this study, a privacy risk assessment method based on the combination of system theoretic process analysis–failure mode and effect analysis (STPA–FMEA) is proposed for a smart home system, considering the interaction and control of ‘user-environment-smart home product’. A total of 35 privacy risk scenarios of ‘component-threat-failure-model-incident’ combinations are identified. The risk priority numbers (RPN) was used to quantitatively assess the level of risk for each risk scenario and the role of user and environmental factors in influencing the risk. According to the results, the privacy management ability of users and the security state of the environment have significant effects on the quantified values of the privacy risks of smart home systems. The STPA–FMEA method can identify the privacy risk scenarios of a smart home system and the insecurity constraints in the hierarchical control structure of the system in a relatively comprehensive manner. Additionally, the proposed risk control measures based on the STPA–FMEA analysis can effectively reduce the privacy risk of the smart home system. The risk assessment method proposed in this study can be widely applied to the field of risk research of complex systems, and this study can contribute to the improvement of privacy security of smart home systems.