Background: Fine particulate matter (PM2.5) is suspected to increase the risk of colorectal cancer, but the mechanism remains unknown. We aimed to investigate the association between PM2.5 exposure, genetic variants and colorectal cancer risk in the Prostate, Lung, Colon and Ovarian (PLCO) Cancer Screening trial. Methods: We included a prospective cohort of 139,534 cancer-free individuals from 10 United States research centers with over ten years of follow-up. We used a Cox regression model to assess the association between PM2.5 exposure and colorectal cancer incidence by calculating the hazard ratio (HR) and 95% confidence interval (CI) with adjustment for potential confounders. The polygenic risk score (PRS) and genome-wide interaction analysis (GWIA) were used to evaluate the multiplicative interaction between PM2.5 exposure and genetic variants in regard to colorectal cancer risk. Results: After a median of 10.43 years of follow-up, 1,666 participants had been diagnosed with colorectal cancer. PM2.5 exposure was significantly associated with an increased risk of colorectal cancer (HR = 1.27; 95% CI = 1.17–1.37 per 5 μg/m3 increase). Five independent susceptibility loci reached statistical significance at P < 1.22 × 10−8 in the interaction analysis. Furthermore, a joint interaction was observed between PM2.5 exposure and the PRS based on these five loci with colorectal cancer risk (P = 3.11 × 10−29). The Gene Ontology analysis showed that the vascular endothelial growth factor (VEGF) receptor signaling pathway was involved in the biological process of colorectal cancer. Conclusions: Our large-scale analysis has shown for the first time that long-term PM2.5 exposure potential increases colorectal cancer risk, which might be modified by genetic variants.