BackgroundProstate cancer (PC) is the most common neoplasm and is the second leading cause of cancer-related deaths in men worldwide. The Hippo tumor suppressor pathway is highly conserved in mammals and plays an important role in carcinogenesis. YAP is one of major key effectors of the Hippo pathway. However, the mechanism supporting abnormal YAP expression in PC remains to be characterized.MethodsWestern blot was used to measure the protein expression of ATXN3 and YAP, while the YAP target genes were measured by real-time PCR. CCK8 assay was used to detect cell viability; transwell invasion assay was used to measure the invasion ability of PC. The xeno-graft tumor model was used for in vivo study. Protein stability assay was used to detect YAP protein degradation. Immuno-precipitation assay was used to detect the interaction domain between YAP and ATXN3. The ubiquitin-based Immuno-precipitation assays were used to detect the specific ubiquitination manner happened on YAP.ResultsIn the present study, we identified ATXN3, a DUB enzyme in the ubiquitin-specific proteases family, as a bona fide deubiquitylase of YAP in PC. ATXN3 was shown to interact with, deubiquitylate, and stabilize YAP in a deubiquitylation activity-dependent manner. Depletion of ATXN3 decreased the YAP protein level and the expression of YAP/TEAD target genes in PC, including CTGF, ANKRD1 and CYR61. Further mechanistic study revealed that the Josephin domain of ATXN3 interacted with the WW domain of YAP. ATXN3 stabilized YAP protein via inhibiting K48-specific poly-ubiquitination process on YAP protein. In addition, ATXN3 depletion significantly decreased PC cell proliferation, invasion and stem-like properties. The effects induced by ATXN3 depletion could be rescued by further YAP overexpression.ConclusionsIn general, our findings establish a previously undocumented catalytic role for ATXN3 as a deubiquitinating enzyme of YAP and provides a possible target for the therapy of PC.Video AbstractSupplementary InformationThe online version contains supplementary material available at 10.1186/s12964-023-01073-9.