The final publication is available at Springer via http://dx.doi.org/10.1007/s11135-010-9345-5 Student evaluation of teaching (SET) questionnaires are the most common methods of evaluation used by European universities to assess the quality of teaching delivered by their lecturers. A series of multivariate statistical methods were applied to analyze the underlying structure of the SET questionnaire used by the Universidad Politecnica de Valencia (UPV) in order to develop an appropriate methodology for extracting, analyzing, and interpreting the information contained in the questionnaire. In a first step, a confirmatory factorial analysis (CFA) was developed in order to evaluate the reliability, validity and dimensionality of it, by means of two relatively new parameters commonly used in structural equation modelling: the compound reliability and extracted variance for each latent construct. In a second step, cluster analysis (CA) was used to test the ability of the questionnaire for the identification of different categories of lecturers. In the last step, a tree classification method, the chi-squared automatic interaction detector (CHAID), was used in order to characterize the different lecturer's categories obtained with CA according to all available information regarding the teaching staff and subjects. © 2010 Springer Science+Business Media B.V. Martínez Gómez, M.; Carot Sierra, JM.; Jabaloyes Vivas, JM.; Zarzo Castelló, M. (2011). A multivariate method for analyzing and improving the use of student evaluation of teaching questionnaires: a case study. Quality and Quantity. 45(6):1415-1427. doi:10.1007/s11135-010-9345-5 Bentler P.M.: Comparative fit indexes in structural models. Psychol. Bull. 107(2), 238–246 (1990) Bentler P.M., Bonett D.G.: Significance tests and goodness of fit in the analysis of covariance structures. Psychol. Bull. 88, 588–606 (1980) Berk R.A.: Survey of 12 strategies to measure teaching effectiveness. Int. J. Teach. Learn. High. Educ. 17, 48–62 (2005) Bollen, K.A. (eds): Structural Equations with Latent Variables. Wiley, New York (1989) Bollen, K.A., Long, J.S. (eds): Testing Structural Equation Models. Sage, Newbury Park (1993) Brown M.W., Cudeck R.: Alternative ways of assessing model fit. In: Bollen, K.A., Long, J.S. (eds) Testing Structural Equation Models, pp. 136–162. Sage, Newbury Park (1993) Campbell D.T., Fiske D.W.: Convergent and discriminant validation by the multitrait–multimethod matrix. Psychol. Bull. 56(2), 81–105 (1959) Cashin W.E.: Concerns about using student ratings in community colleges. N. Dir. Community Coll. 11(1), 57–65 (1983) Chiu, T., Fang, D., Chen, J., Wang, Y., Jeris, C.: A robust and scalable clustering algorithm for mixed type attributes in large database environment. In: Proceedings of Seventh ACM SIGKDD, International Conference on Knowledge Discovery and Data Mining, San Francisco, 2001 Feldman K.A.: The significance of circumstances for college student’s ratings of their teachers and courses. Res. High. Educ. 10, 149–172 (1979) Göb R., McCollin C., Fernanda Ramalhoto M.: Ordinal methodology in the analysis of likert scales. Qual. Quant. 41, 601–626 (2007) Gursoy D., Umbreit W.T.: Exploring student’s evaluations of teaching effectiveness: what factors are important? J. Hosp. Tour. Res. 29(1), 91–109 (2005) Hair, J.F., Anderson, R.E., Tatham, R.L., Black, W.C. (eds): Multivariate Data Analysis. Prentice Hall International, New York (1995) Hu L., Bentler P.M.: Cut off criteria for fit indexes in covariance. Structure analysis: conventional criteria versus new alternatives. Struct. Equ. Model. 6(1), 1–55 (1999) Jackson D., Teal C.R., Raines J.S., Nansel T.R., Force R.C., Burdsal C.A.: The dimensions of students’ perceptions of teaching effectiveness. Educ. Psychol. Meas. 59(4), 580–596 (1999) Jöreskog K.G.: Testing structural equation models. In: Bollen, K.A., Long, J.S. (eds) Testing Structural Equation Models, pp. 294–316. Sage, Newbury Park (1993) Jöreskog, K.G., Sörbom, D. (eds): LISREl V. Analysis of Linear Structural Relations by the Method of Maximum Likelihood. SPSS Publications, Chicago (1981) Jöreskog, K.G., Sörbom, D. (eds): LISREL VI. Analysis of Linear Structural Relationships by Maximum Likelihood, Instrumental Variable, and Least Squares Methods. Scientific Software, Moresville (1986) Kass G.V.: An exploratory technique for investigating large quantities of categorical data. Appl. Stat. 29(2), 119–127 (1980) Lalla M., Facchinetti G., Maestroleo G.: Ordinal scales and fuzzy set systems to measure agreement: an application to the evaluation of teaching activity. Qual. Quant. 38, 577–601 (2004) Marsh H.W.: Validity of students’ evaluations of college teaching: a multitrait multimethod analysis. J. Educ. Psychol. 74(2), 219–237 (1982) Marsh H.W.: Students’ evaluations of university teaching: dimensionality, reliability, validity, potential biases, and utility. J. Educ. Psychol. 76(5), 707–754 (1984) Marsh H.W.: Students’ evaluations of university teaching: research, findings, methodological issues and directions for future research. J. Educ. Res. 11(3), 253–388 (1987) Marsh H.W.: Weighting for the right criteria in the Instructional Development and Effectiveness Assessment (IDEA) system: global and specific ratings of teaching effectiveness and their relation to course objectives. J. Educ. Psychol. 86, 631–648 (1994) Marsh H.W., Hattie J.: The relation between research productivity and teaching effectiveness. J. High. Educ. 73, 603–641 (2002) Mulaik S.A., James L.R., Van Alstine J., Bennett N., Lind S., Stilwell C.D.: Evaluation of goodness-of-fit indices for structural equation models. Psychol. Bull. 105(3), 430–445 (1989) Ting K.F.: A multilevel perspective on student ratings of instruction: lessons from the Chinese experience. Res. High. Educ. 41, 637–661 (2000) Toland M.D., De Ayala R.J.: A multilevel factor analysis of student’s evaluations of teaching. Educ. Psychol. Meas. 65(2), 272–296 (2005)

Batista, J.M., Coenders, G. (eds): Modelos de Ecuaciones Estructurales. Cuadernos de estadística, 6. La Muralla, Madrid (2000) Batista J.M., Coenders G., Alonso J.: Análisis Factorial Confirmatorio. Su utilidad en validación de cuestionarios relacionados con la salud. Med. Clin. (Barc.) 122(1), 21–27 (2004) Cajide J.: Análisis factorial confirmatorio de las características de calidad docente universitaria (solución LISREL). Bordón 46(4), 389–405 (1994) González Such, J.: Estudio de un Instrumento para la Evaluación del Profesorado Universitario. Thesis, Universitat de València, Valencia (1997) Rodríguez Sabiote, C., Gutiérrez Pérez, J.: Debilidades de la evaluación de la calidad en la universidad española. Causas, consecuencias y propuestas de mejora. Revista Electron. de Investig. Educ. 5(1) (2003). http://redie.ens.uabc.mx/vol5no1/contenido-sabiote.html . Retrieved 20 April 2005

BHE, J.M., Coenders, G. (eds): Models of Structural Equations. Statistical notebooks, 6. La Muralla, Madrid (2000), J.M., Coenders G., Alonso J.: Confirmatory factorial analysis. Their usefulness in validating health-related questionnaires. MED. Clin. (Barc.) 122 (1), 21-27 (2004) Cajide J.: Confirmatory factorial analysis of university teaching quality characteristics (LISREL solution). Boron 46 (4), 389-405 (1994) González Such, J.: Study of an Evaluation Tool for University Teachers. Thesis, Universitat de València, Valencia (1997) Rodríguez Sabiote, C., Gutiérrez Pérez, J.: Weaknesses in quality assessment at the Spanish university. Causes, consequences and proposals for improvement. Magazine Electron. de investig. EDUC. 5 (1) (2003). http://redie.ens.uabc.mx/vol5no1/contenido-sabiote.html. Retrieved 20 April 2005

A multivariate method for analyzing and improving the use of student evaluation of teaching questionnaires: a case study

Abstract