To make pollution evaluation of potentially hazardous elements in the soil more accurately, the regional geochemical baseline concentrations of eight potentially hazardous elements (Cr, Ni, Cu, Zn, As, Cd, Hg, and Pb) were established in Huilai County using cumulative frequency distribution curves. Then, the pollution load index and enrichment factor were applied to estimate the contamination levels, based on these geochemical baseline concentrations. The results suggested that topsoil was moderately polluted by potentially hazardous elements, while Cd pollution in the construction land and As pollution in the farmland was relatively severe. The possible sources of eight potentially hazardous elements were analyzed by correlation analysis, geostatistics and positive matrix factorization. Four sources have been determined and apportioned, namely industrial activities, natural sources, agricultural practices, and traffic emissions. Combining the health risk assessment with the source profiles, the health risks quantified from four sources were estimated under farmland, construction land, and woodland. The results showed that agricultural practices were the most main source of non-cancer and cancer risks under woodland and farmland for adults; industrial activities were the most main source of non-cancer and cancer risks under construction land for adults. Children's health risks, both carcinogenic risk and non-carcinogenic risk, were greater than adults, and the health risk trends of adults and children showed similarities. Therefore, agricultural practices under woodland and farmland should be controlled and managed as a priority, while industrial activities should be given priority to control and management under construction land.