Soil nitrogen (N) deficiency can affect the photosynthetic N-use efficiency (PNUE), mesophyll conductance (gm), and leaf N allocation in many plant species. However, there is a lack of information about how these physiological characteristics in N-fixing trees could be affected by soil N deficiency and the difference in adaptations to N deficiency between N-fixing and non-N-fixing trees. In this study, we choose seedlings of two N-fixing (Dalbergia odorifera and Erythrophleum fordii) and two non-N-fixing trees (Castanopsis hystrix and Betula alnoides) as study objects, and we conducted a pot experiment with three levels of soil N treatments (high nitrogen, HN; medium nitrogen, MN; and low nitrogen, LN). Our results showed that soil N deficiency significantly decreased the leaf N concentration and photosynthesis ability of the two non-N-fixing trees, but it had less influence on these indices in the two N-fixing trees. The LN treatment had lower gm in D. odorifera and lower leaf N allocated to Rubisco (PR), leaf N allocated to bioenergetics (PB), and gm in B. alnoides, eventually resulting in low PNUE values. D. odorifera, B. alnoides, and C. hystrix seedling leaves showed improving leaf N allocated to the cell wall (PCW) and (or) the leaf mass per area (LMA) to adapt to a low-N soil environment. Our findings suggested that the D. odorifera and E. fordii seedlings could grow well in N-deficient soil and adding N may increase the growth rates of B. alnoides and C. hystrix seedlings and promote the growth of artificial forests.