Background: Active smoking and exposure to environmental tobacco smoke may be related to cognitive function decline. We assessed the associations of urinary levels of nicotine and its metabolites with cognitive function. Methods: A total of 553 elder adults at high risk of cognitive impairment and 2212 gender- and age-matched individuals at low risk of cognitive impairment were selected at a ratio of 1: 4 from the remained individuals (n = 6771) who completed the baseline survey of the Shenzhen Ageing-Related Disorder Cohort, after excluding those with either Alzheimer’s disease, Parkinson’s syndrome or stroke as well as those with missing data on variables (including active and passive smoking status, Mini-Cog score). Urinary levels of nicotine and its metabolites and cognitive function for all individuals were measured by high-performance liquid chromatography-tandem mass spectrometry (LC-MS/MS) and assessed using the Mini-Cog test, respectively. Associations of urinary levels of nicotine and its metabolites with cognitive function were analyzed by conditional logistic regression models. Results: Individuals in the highest tertile of urinary OHCotGluc (OR: 1.52, 95%CI: 1.19–1.93) or NNO (OR: 1.50, 95%CI: 1.16–1.93) levels as well as in the second tertile of urinary ∑Nic level (OR: 1.43, 95%CI: 1.13–1.82) were at higher risk of cognitive impairment compared with those in the corresponding lowest tertile. Restricted cubic spline models revealed the non-linear dose-response relationships between urinary levels of OHCotGluc, NNO or ∑Nic and the risk of cognitive impairment. Conclusions: Urinary levels of OHCotGluc, NNO or ∑Nic exhibited a non-linear dose-response relationship with cognitive function in the urban elderly.