Participation in the tax system of the country and tax liability itself are two of the basic characteristics of the functioning of all economically active subjects belonging to national economies. The revenues from taxes represent a substantial part of the resources in each state’s budget, so it is not easy to achieve any form of joint action by all concerned in this area. Within the revenues obtained by taxes, customs, and levies, the issue of taxes is one of the most sensitive areas. Higher efficiency of tax collection is possible to achieve by implementing not only external tools (e.g., tax control, detecting tax fraud, etc.) but also internal ones (e.g., internal motivation of taxpayers). Therefore, it is important to improve the perception of the tax system among the population and thus to influence and create a responsible approach to the collection of taxes and levies. Due to the COVID-19 pandemic, some changes and restrictions have been implemented into the system of tax and customs collection in the Slovak Republic, but very unexpectedly and purposefully. It results in a more complicated and confused system of tax and customs administration. Therefore, our research efforts are aiming to identify and analyze the key significant factors that affect the perception of tax and customs administration in the Slovak Republic from the perspective of its inhabitants, to capture differences in perception of taxpayers. The novelty of this study lies in its approach to identifying and eliminating redundancies and opacities in tax collection processes performed by institutions of tax and customs administration systems in Slovakia, namely the approach from the taxpayer point of view, to promote voluntary compliance with tax duties. Due to the lack of such oriented studies, a survey performed in three regions of Slovakia using the research tool (the questionnaire) developed by authors is presented in this article. To solve this complex problem, a solving strategy based on exploratory analysis (KMO = 0.641) and later confirmatory factor analysis is proposed to extract four key common factors from the items of the research tool (with a total value of 80.623% of the explained variability). The reliability of the used research tool, pre-tested in the pilot study, is demonstrated by the achieved Cronbach coefficient value of 0.827%. In order to evaluate the obtained data, the respondents were divided into groups by gender, residence, and occupation, in accordance with the indications and outputs of the pilot study. Finally, experimentally obtained data were analyzed and compared. The significant differences in the respondents’ perceptions of the extracted factors between selected groups are confirmed by applied analysis, both between men and women and between respondents depending on the place of residence. The main differences are observed in the questionnaire items Q10 “Administrative complexity of the tax and customs agenda”, Q7 “Ease of tax collection”, Q16 “Labor costs for staff dealing with the tax and customs agenda”, and Q5 “Helpfulness and willingness of administrative workers”. Time costs for tax processing are considered critical.