Title: Towards Dynamic Sparsification by Iterative Prune-Grow LookAheads. Abstract: Model sparsification is a process of removing redundant connections in a neural network, making it more compact and faster. Most pruning methods start with a dense pretrained model, which is computationally intensive to train. Other pruning approaches perform compression at initialization which saves training time, however, at the cost of final accuracy as an unreliable architecture can be selected given weak feature representation. In this work, we re-formulate network sparsification as an exploitation-exploration process during initial training to enable dynamic learning of network sparsification. The exploitation phase assumes architecture stability and trains it to maximize accuracy. Whereas the exploration phase challenges the current architecture with a novel $\textit{LookAhead}$ step that reactivates pruned parameters, quickly updates them together with existing ones, and reconfigures the sparse architecture with a pruning-growing paradigm. We demonstrate that $\textit{LookAhead}$ methodology can effectively and efficiently oversee both architecture and performance during training, enabling early pruning with a capability of future recovery to correct previous poor pruning selections. Extensive results on ImageNet and CIFAR datasets show consistent improvements over the prior art by large margins, for varying networks towards both structured and unstructured sparsity. For example, our method surpasses recent work by $+1.3\%$ top-1 accuracy at the same compression ratio for ResNet50-ImageNet unstructured sparsity. Moreover, our structured sparsity results also improve upon the previous best hardware-aware pruning method by $+0.8\%$ top-1 accuracy for MobileNet-ImageNet sparsification, offering $+134$ in hardware FPS(im/s), while halving the training cost.