As one of the main grain-producing areas in China, urban agglomeration in the middle reaches of the Yangtze River plays an important role in the development of agricultural production for China’s grain supply. The existing studies about agricultural production efficiency lack of regional coordination analysis at both macro and micro levels, and only few studies consider the impact of agricultural production environment pollution and other undesirable outputs. Based on the input–output index system of agricultural green production, Slacks-based model (SBM) was adopted to measure the agricultural green production efficiency of 31 prefecture level cities in the middle reaches of the Yangtze River from 2008 to 2018, and the Tobit model of panel fixed effect was used to analyze the driving effect of external factors that affect the agricultural green production efficiency of urban agglomeration in the middle reaches of the Yangtze River. At the same time, the research methods at both macro and micro levels provide ideas for the research of transregional production efficiency. The results showed that: (1) the agricultural green production efficiency of urban agglomeration in the middle reaches of the Yangtze river is relatively low, with 2009 and 2013 as the inflection points, showing a stable trend of rise and decline; (2) The green agricultural production efficiency of urban agglomeration in the middle reaches of the Yangtze River presents the spatial distribution characteristics of “high in the west and low in the east”. The regional efficiency difference is obvious, the gap gradually expands, develops from the equilibrium to the polarization; (3) Urbanization development and government intervention has a significant restraining effect on the improvement of agricultural green production efficiency, and opening to the outside world produces a remarkable influence on the improvement of agricultural green production efficiency, however, economic development and industrial structure have little impact on the improvement of agricultural green production efficiency. Therefore, it is necessary to increase investment in technological innovation, promote agricultural transformation and upgrading, promote rational factors allocation and promote coordinated development of agriculture based on regional production differences.