Thermal conductivity is a fundamental material parameter involved in various infrastructure design guides around the world. This paper developed an innovative neural network (NN) aided homogenization approach for predicting the effective thermal conductivity of various composite construction materials. The 2-D meso-structures of dense graded asphalt mixture, porous asphalt mixture, and cement concrete were generated and divided into 2n× 2nsquare elements with specific thermal conductivity values. A two-layer feed-forward neural network with sigmoid hidden neurons and linear output neurons was built to predict the effective thermal conductivity of the 2 × 2 block. The Levenberg-Marquardt backpropagation algorithm was used to train the network. By repeatedly using the neural network, the effective thermal conductivities of 2-D meso-structures were calculated. The accuracy of the above NN aided homogenization approach was validated with experiment, and various factors affecting the effective thermal conductivity were analyzed. The analysis results show that the accuracy of the NN aided approach is acceptable with relative errors of 1.92~4.34% for the dense graded asphalt mixture, 1.10~6.85% for the porous asphalt mixture, and 1.13~3.14% for the cement concrete. The relative errors for all the materials are lower than 5% when the heterogeneous structures are divided into 512 × 512 elements. Ignoring the actual material meso-structures may lead to significant errors (134.01%) in predicting the effective thermal conductivity of materials with high heterogeneity such as porous asphalt mixture. While proper simplification is acceptable for dense construction composite materials. The effective thermal conductivity of composite cement-asphalt mixtures increases with higher saturation of grouted material. However, the improvement effect of the high-conductive cement paste on the composite cement-asphalt mixtures could be significantly reduced when the cement paste concentrates at the bottom of the mixture. Cracked aggregates and segregation of material components tend to decrease the effective thermal conductivity of construction materials. The NN aided homogenization approach presented in this paper is useful for selecting the effective thermal conductivity of construction materials.