at 24 $\mu$m, resulting in total infrared luminosities in the range (0.5-4.0)$\times$10$^{13}$ L$_{\sun}$. This implies the presence of either dust-enshrouded starburst activity (with SFRs of 600-4300 M$_{\sun}$ yr$^{-1}$) and/or highly-obscured active galactic nuclei (AGN). The contribution of galaxies with $M_{\rm star}>2.5\times 10^{11}$ M$_{\sun}$ to the total stellar mass budget at $3.0 \leq z < 4.0$ is $\sim8^{+13}_{-3}$%. Compared to recent estimates of the stellar mass density in galaxies with $M_{\rm star}\approx10^{9}-10^{11}$ M$_{\sun}$ at $z\sim5$ and $z\sim6$, we find an evolution by a factor of 2-7 and 3-22 from $z\sim5$ and $z\sim6$, respectively, to $z=3.5$. The previously found disagreement at the high-mass end between observed and model-predicted SMFs is now significant at the 3 $\sigma$ level when only random uncertainties are considered. However, systematic uncertainties dominate the total error budget, with errors up to a factor of $\sim8$ in the densities at the high-mass end, bringing the observed SMF in marginal agreement with the predicted SMF. Additional systematic uncertainties on the high-mass end could be potentially introduced by either 1) the intense star-formation and/or the very common AGN activities as inferred from the MIPS 24 $\mu$m detections, and/or 2) contamination by a significant population of massive, old, and dusty galaxies at 