Aporte metodológico para a avaliação dos programas de pós-graduação no Brasil: análises multivariadas aplicadas aos indicadores quantitativos da CAPES
Data
2019-03-08
Autores
Castanha, Renata Cristina Gutierres [UNESP]
Título da Revista
ISSN da Revista
Título de Volume
Editor
Universidade Estadual Paulista (Unesp)
Resumo
Destaca o papel fundamental dos programas de pós-graduação no desenvolvimento científico e socioeconômico do país e a consequente necessidade de se desenvolver sistemas de avaliação que levem em consideração as particularidades das áreas do conhecimento. Diante do fato de a CAPES utilizar indicadores de produção científica na avaliação da pós-graduação brasileira, propõe uma ferramenta para uso tanto dos Comitês de Avaliação da CAPES quanto para os próprios programas e seus coordenadores, a fim de oferecer aporte metodológico que contribua para o refinamento e assertividade da avaliação da CAPES. Mais especificamente, analisa a contribuição dos métodos estatísticos de boxplot e cluster na identificação das tendências dos indicadores normalizados fornecidos pela CAPES, referente à avaliação trienal (2010-2012), a fim de visualizar as diferenças de desempenho dos programas e identificar aqueles indicadores mais discernentes dos conceitos e valida os resultados dos agrupamentos dos programas de pós-graduação com o conceito obtido pela avaliação da CAPES no posterior quadriênio (2013-2016), a fim de visualizar se os resultados obtidos por meio das análises multivariadas coincidem com os conceitos atribuídos pela CAPES na última avaliação. Estabelece como universo de pesquisa o conjunto de programas de pós-graduação nas áreas de Ciência da Informação, Filosofia, Zootecnia/Recursos Pesqueiros, Ciências Biológicas III, Odontologia, Engenharia II e Materiais. Para cada área, extrai os indicadores quantitativos de cada programa, a saber: total de teses e dissertações, total de artigos segundo a estratificação Qualis, e total de trabalhos completos publicados em anais de eventos técnico-científicos, livros, capítulos de livro, texto integral, coletâneas, verbetes e outros e produção artística, todos relativizados em função do total de docentes permanentes e triênio, a fim de se eliminar vieses em função do tamanho dos programas. Entende que para as áreas analisadas, houve formação de grupos heterogêneos no cluster em relação ao conceito, no dendrograma, sugerindo que, embora já propiciem uma similaridade e proximidade com os conceitos atribuídos pela CAPES, os indicadores quantitativos não foram suficientes para agrupar os programas de forma idêntica à presente na avaliação, necessitando também dos indicadores qualitativos presentes nas Fichas de avaliação, em que se evidencia a trajetória, metas, colaborações científicas, inserção social, nucleação, solidariedade, internacionalização, etc. os quais apontaram a razão de um determinado programa estar agrupado com outros de diferentes conceitos. Observa que áreas mais duras do conhecimento, como a Matemática, tendem a ter a avaliação mais objetiva do que em áreas como a Filosofia, por exemplo, e, com isso, a metodologia pode ser melhor aplicada. Comprova, por meio de análise de correlação, a hipótese sobre o peso dos indicadores referentes às diferentes áreas do conhecimento. Conclui que os indicadores referentes a capítulos de livro se destacam na análise de correlação para as áreas de Ciência da Informação e Filosofia. Para a área da Zootecnia/Recursos Pesqueiros, destaca os indicadores relativos a teses e Qualis B1. Para a área de Ciências Biológicas III, destaca os indicadores relativos a teses, Qualis A2, B1 e B2. A área de Odontologia destaca o indicador Qualis A2. Para a área de Engenharia II e Materiais, destaca o indicador relativo a teses, sendo estes provavelmente os indicadores levados em maior consideração na atribuição final dos conceitos das áreas analisadas.
This research highlights the fundamental role of graduate programs in the scientific and socioeconomic development of the country and the consequent need to develop evaluation systems that take into account the particularities of the areas of knowledge. Faced with the fact that CAPES uses scholarly communication indicators in Brazilian graduate assessment, it proposes a tool for the use of both CAPES Evaluation Committees and the programs and their coordinators, in order to offer a methodological contribution that contributes to the refinement and assertiveness of the CAPES evaluation. More specifically, it analyzes the contribution of boxplot and cluster statistical methods to the identification of the trends of the indicators provided by CAPES for the triennial evaluation (2010-2012), in order to visualize the differences in performance of the programs and to identify those most discernible indicators of the concepts and validates the results of the groupings of the graduate programs with the concept obtained by the CAPES evaluation in the subsequent quadrennium (2013- 2016), in order to see if the results obtained through the multivariate analysis coincide with the concepts attributed by the CAPES in the last evaluation. It establishes as a universe of research the set of graduate programs in the areas of Information Science, Philosophy, Animal Science/Fisheries, Biological Sciences III, Dentistry, Engineering II and Materials. For each area, it extracts the quantitative indicators of each program, namely: total of theses and dissertations, average of articles according to the Qualis stratification, and average of complete papers published in annals of technicalscientific events, books, book chapters, integral text, collections, entries and others and artistic production, all normalized according to the faculty and triennium, in order to eliminate biases due to the size of the programs. For the analyzed areas, heterogeneous groups were formed in the cluster in relation to the concept in the dendrogram, suggesting that, although they already provide a similarity and proximity to the concepts attributed by CAPES, the quantitative indicators were not enough to group the qualitative indicators present in the evaluation sheets, in which the trajectory, goals, scientific collaborations, social insertion, nucleation, solidarity, internationalization, etc. are also needed, for pointing out the reason for a certain program being grouped with others of different concepts. It notes that harder areas of knowledge, such as Mathematics, tend to have more objective evaluation than in areas such as Philosophy, for example, and thus the methodology can be better applied. It proves, through correlation analysis, the hypothesis about the weight of the indicators referring to the different areas of knowledge. It concludes that the indicators referring to book chapters stand out in the analysis of correlation for the areas of Information Science and Philosophy. For the area of Animal Science/Fisheries, it highlights the indicators related to dissertations and Qualis B1. For the area of Biological Sciences III, it highlights the indicators related to dissertations, Qualis B1 and B2. The area of Dentistry highlights the Qualis A2 indicator. For the area of Engineering II and Materials, the dissertation indicator stands out, and these are probably the indicators taken into account in the final assignment of the concepts of the assessed areas.
This research highlights the fundamental role of graduate programs in the scientific and socioeconomic development of the country and the consequent need to develop evaluation systems that take into account the particularities of the areas of knowledge. Faced with the fact that CAPES uses scholarly communication indicators in Brazilian graduate assessment, it proposes a tool for the use of both CAPES Evaluation Committees and the programs and their coordinators, in order to offer a methodological contribution that contributes to the refinement and assertiveness of the CAPES evaluation. More specifically, it analyzes the contribution of boxplot and cluster statistical methods to the identification of the trends of the indicators provided by CAPES for the triennial evaluation (2010-2012), in order to visualize the differences in performance of the programs and to identify those most discernible indicators of the concepts and validates the results of the groupings of the graduate programs with the concept obtained by the CAPES evaluation in the subsequent quadrennium (2013- 2016), in order to see if the results obtained through the multivariate analysis coincide with the concepts attributed by the CAPES in the last evaluation. It establishes as a universe of research the set of graduate programs in the areas of Information Science, Philosophy, Animal Science/Fisheries, Biological Sciences III, Dentistry, Engineering II and Materials. For each area, it extracts the quantitative indicators of each program, namely: total of theses and dissertations, average of articles according to the Qualis stratification, and average of complete papers published in annals of technicalscientific events, books, book chapters, integral text, collections, entries and others and artistic production, all normalized according to the faculty and triennium, in order to eliminate biases due to the size of the programs. For the analyzed areas, heterogeneous groups were formed in the cluster in relation to the concept in the dendrogram, suggesting that, although they already provide a similarity and proximity to the concepts attributed by CAPES, the quantitative indicators were not enough to group the qualitative indicators present in the evaluation sheets, in which the trajectory, goals, scientific collaborations, social insertion, nucleation, solidarity, internationalization, etc. are also needed, for pointing out the reason for a certain program being grouped with others of different concepts. It notes that harder areas of knowledge, such as Mathematics, tend to have more objective evaluation than in areas such as Philosophy, for example, and thus the methodology can be better applied. It proves, through correlation analysis, the hypothesis about the weight of the indicators referring to the different areas of knowledge. It concludes that the indicators referring to book chapters stand out in the analysis of correlation for the areas of Information Science and Philosophy. For the area of Animal Science/Fisheries, it highlights the indicators related to dissertations and Qualis B1. For the area of Biological Sciences III, it highlights the indicators related to dissertations, Qualis B1 and B2. The area of Dentistry highlights the Qualis A2 indicator. For the area of Engineering II and Materials, the dissertation indicator stands out, and these are probably the indicators taken into account in the final assignment of the concepts of the assessed areas.
Descrição
Palavras-chave
Indicadores bibliométricos, Análise multivariada, Avaliação da pós-graduação, Programas de pós-graduação, Bibliometric indicators, Multivariate analysis, Graduate assessment, Graduate programs