Urban Freight Transport (UFT) is responsible for moving goods in urban areas to meet citizens’ demands, which makes it essential for economic development. Simultaneously, UFT contributes to adverse impacts on society and the environment, including congestion and pollution. This paper assesses how the urban infrastructure and UFT externalities influence the residents’ quality of life. Three major assumptions were considered: (i) Public managers are responsible for the urban infrastructure, which is also influenced by businesses; (ii) UFT leads to negative externalities, which are influenced by government actions; and (iii) both infrastructure and externalities influence the residents’ quality of life. The analysis is based on a web-based survey conducted with residents of Brasília, Brazil. Structural Equation Modelling with Partial Least Squares was used to analyze the data. Findings showed that the residents’ quality of life is negatively influenced by UFT externalities and positively influenced by the urban infrastructure. Furthermore, both public and private management have more influence on externalities when compared to urban infrastructure. Finally, road capacity, proper loading and unloading, and supervision should be prioritized to improve citizens’ quality of life in Brasília.