Critical Thinking Performance Assessment in an Undergraduate Physical Chemistry Laboratory Course: Developing a Contextual Critical Thinking Coding Manual

Rüdiger Tiemann; Lilian Danial; Jenna Koenen

doi:10.29333/ajqr/17367

Rüdiger Tiemann ¹ ^* , Lilian Danial ¹, Jenna Koenen ²

More Detail

AM J QUALITATIVE RES, Volume 10, Issue 1, pp. 37-60

https://doi.org/10.29333/ajqr/17367

OPEN ACCESS 54 Views 26 Downloads

Download Full Text (PDF)

Abstract

Critical thinking (CT) performance assessment in authentic settings remains challenging. One reason is that the CT focus might shift depending on the context. This article describes the inductive and deductive processes of developing a contextual CT coding manual (CCTCM) and presents an exemplary analysis to demonstrate its application in assessing CT performance in an undergraduate physical chemistry laboratory course. Students’ verbal interactions during the course were used as the primary data for assessment. The CCTCM we developed comprises two components: the contextual profile of the physical chemistry laboratory and an operationalized set of CT skills. The tool enabled a contextualized analysis of CT performance. It produced data that reflected ongoing learning, accommodated quantitative analyses of small samples, and could inform curriculum design by identifying commonly applied or underused CT skills. Moreover, it allowed for monitoring individual CT performance and understanding CT skill interconnections. The CCTCM operates independently of teaching methods or students’ problem familiarity, thus preserving intuitive behaviors.

Keywords: Critical thinking, domain specific, performance assessment, undergraduate, chemistry education, science education, laboratory course, coding manual, qualitative data analysis, authentic setting, inductive, deductive, higher education

References

Andreucci-Annunziata, P., Riedemann, A., Cortés, S., Mellado, A., del Río, M. T., & Vega-Muñoz, A. (2023). Conceptualizations and instructional strategies on critical thinking in higher education: A systematic review of systematic reviews. Frontiers in Education, 8, 1-18. https://doi.org/10.3389/feduc.2023.1141686
Bagnato, S. J. (2007). Authentic assessment for early childhood intervention: Best practices. Guilford Press.
Bailin, S. (2002). Critical thinking and science education. Science & Education, 11(4), 361–375. https://doi.org/10.1023/A:1016042608621
Black, B. (2012). An overview of a program of research to support the assessment of critical thinking. Thinking Skills and Creativity, 7(2), 122–133. https://doi.org/10.1016/j.tsc.2012.04.003
Black, B., Chislett, J., Thompson, A., Thwaites, G., & Thwaites, J. (2008). Critical thinking: A definition and taxonomy for Cambridge Assessment. Cambridge Assessment. https://doi.org/10.17863/CAM.100474
Boyatzis, R. E. (1998). Transforming qualitative information: Thematic analysis and code development. Sage.
Brailas, A., Tragou, E., & Papachristopoulos, K. (2023). Introduction to qualitative data analysis and coding with QualCoder. American Journal of Qualitative Research, 7(3), 19–31. https://doi.org/10.29333/ajqr/13230
Braun, H. I., Shavelson, R. J., Zlatkin-Troitschanskaia, O., & Borowiec, K. (2020). Performance assessment of critical thinking: Conceptualization, design, and implementation. Frontiers in Education, 5, 1-10. https://doi.org/10.3389/feduc.2020.00156
Braun, V., & Clarke, V. (2024). Reporting guidelines for qualitative research: A values-based approach. Qualitative Research in Psychology, 22(2), 399–438. https://doi.org/10.1080/14780887.2024.2382244
Butler, H. A. (2024). Predicting everyday critical thinking: A review of critical thinking assessments. Journal of Intelligence, 12(2), 1-12. https://doi.org/10.3390/jintelligence12020016
Butterworth, J., & Thwaites, G. (2013). Thinking skills: Critical thinking and problem solving (2nd ed.). Cambridge University Press.
Carter, A. G., Creedy, D. K., & Sidebotham, M. (2015). Evaluation of tools used to measure critical thinking development in nursing and midwifery undergraduate students: A systematic review. Nurse Education Today, 35(7), 864–874. https://doi.org/10.1016/j.nedt.2015.02.023
Chowdhury, A., & Shil, N. C. (2021). Thinking ‘qualitative’ through a case study: Homework for a researcher. American Journal of Qualitative Research, 5(2), 190–210. https://doi.org/10.29333/ajqr/11280
Crabtree, B., & Miller, W. (1999). A template approach to text analysis: Developing and using codebooks. In B. Crabtree & W. Miller (Eds.), Doing qualitative research (pp. 163–177). Sage.
Danczak, S. M., Thompson, C. D., & Overton, T. L. (2020). Development and validation of an instrument to measure undergraduate chemistry students’ critical thinking skills. Chemistry Education Research and Practice, 21(1), 62–78. https://doi.org/10.1039/C8RP00130H
Danial, L., Koenen, J., & Tiemann, R. (2025a). Prompting and assessing critical thinking in undergraduate physical chemistry labs: Effects of instructional design and performance in-situ assessment. ChemRxiv. doi: 10.26434/chemrxiv-2025-ms8cp
Danial, L., Koenen, J., & Tiemann, R. (2025b). The Contextual Critical Thinking Coding Manual (CCTCM) - The complete coding manual. Zenodo. https://doi.org/10.5281/zenodo.16755624.
Dominguez, C. E., Dumitru, D., Bigu, D., Elen, J., Jiang, L., Railiene, A., Penkauskiene, D., Papathanasiou, I. V., Tsaras, K., Fradelos, E. C., & Ahern, A. (2018). A European collection of the critical thinking skills and dispositions needed in different professional fields for the 21st century. UTAD.
Ennis, R. H. (1985a). A logical basis for measuring critical thinking skills. Educational Leadership, 43(2), 44–48.
Ennis, R. H. (1985b). Critical thinking and the curriculum. National Forum, 65(1), 28–31.
Ennis, R. H. (1985c). Tests that could be called critical thinking tests. In A. L. Costa (Ed.), Developing minds: A resource book for teaching thinking (pp. 303–304). Association for Supervision and Curriculum Development.
Ennis, R. H. (1989). Critical thinking and subject specificity: Clarification and needed research. Educational Researcher, 18(3), 4–10. https://doi.org/10.3102/0013189X018003004
Ennis, R. H. (1990). The extent to which critical thinking is subject-specific: Further clarification. Educational Researcher, 19(4), 13–16. https://doi.org/10.3102/0013189X019004013
Ennis, R. H. (1993). Critical thinking assessment. Theory Into Practice, 32(3), 179–186. https://doi.org/10.1080/00405849309543594
Ennis, R. H. (2015). Critical thinking: A streamlined conception. In M. Davies & R. Barnett (Eds.), The Palgrave handbook of critical thinking in higher education (pp. 31–47). Palgrave Macmillan US. https://doi.org/10.1057/9781137378057_2
European Commission, Directorate-General for Education, Youth, Sport and Culture. (2019). Key competences for lifelong learning. Publications Office of the European Union. https://doi.org/10.2766/569540
Facione, P. A. (1990a). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction (The Delphi Report). California Academic Press.
Facione, P. A. (1990b). The California Critical Thinking Skills Test (CCTST): Forms A and B; and the CCTST test manual. California Academic Press.
Facione, P. A. (2000). The disposition toward critical thinking: Its character, measurement, and relationship to critical thinking skill. Informal Logic, 20(1), 61–84. https://doi.org/10.22329/il.v20i1.2254
Facione, P. A. (2015). Critical thinking: What it is and why it counts (2015 update). Insight Assessment. https://www.insightassessment.com
Facione, P. A., & Facione, N. C. (1992). The California Critical Thinking Disposition Inventory (CCTDI). California Academic Press.
Fereday, J., & Muir-Cochrane, E. (2006). Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International Journal of Qualitative Methods, 5(1), 80–92. https://doi.org/10.1177/160940690600500107
Ferreira, L. D., & Merchant, K.A. (1992). Field research in management accounting and control: A review and evaluation. Accounting, Auditing and Accountability Journal, 5(4), 3-34. https://doi.org/10.1108/09513579210019503
Fischer, S. C., Spiker, V. A., & Riedel, S. L. (2009). Critical thinking training for army officers: A model of critical thinking (Vol. 2, Technical Report). U.S. Army Research Institute.
Halpern, D. F. (2013). Thought and knowledge: An introduction to critical thinking (5th ed.). Psychology Press. https://doi.org/10.4324/9781315885278
Halpern, D. F. (2014). Critical thinking across the curriculum: A brief edition of thought and knowledge. Routledge. https://doi.org/10.4324/9781315805719
Halpin, S. N. (2024). Inter-coder agreement in qualitative coding: Considerations for its use. American Journal of Qualitative Research, 8(3), 23–43. https://doi.org/10.29333/ajqr/14887
Heard, J., Scoular, C., Duckworth, D., Ramalingam, D., & Teo, I. (2020). Critical thinking: Skill development framework. Australian Council for Educational Research. https://research.acer.edu.au/ar_misc/41
Jabali, O., Hamamra, B., & Ayyoub, A. (2024). Critical thinking, assessment, and educational policy in Palestinian universities. International Journal for Educational Integrity, 20(1), 1-23. https://doi.org/10.1007/s40979-024-00160-9
Jones, E. A., Hoffman, S., Moore, L. M., Ratcliff, G., Tibbetts, S., & Click, B. A. (1995). National assessment of college student learning: Identifying college graduates’ essential skills in writing, speech and listening, and critical thinking. Final project report. U.S. Government Printing Office. https://files.eric.ed.gov/fulltext/ED383255.pdf
Kennedy, M., Fisher, M. B., & Ennis, R. H. (1991). Critical thinking: Literature review and needed research. In L. Idol & B. F. Jones (Eds.), Educational values and cognitive instruction: Implications for reform (pp. 11–40). Lawrence Erlbaum Associates.
Ku, K. Y. (2009). Assessing students’ critical thinking performance: Urging for measurements using multi-response format. Thinking Skills and Creativity, 4(1), 70–76. https://doi.org/10.1016/j.tsc.2009.02.001
Lai, E. R. (2011). Critical thinking: A literature review. Pearson's Research Reports, 6(1), 40–41.
La Marca, P. M., Redfield, D., Winter, P. C., & Despriet, L. (2000). State standards and state assessment systems: A guide to alignment. Series on standards and assessments. Council of Chief State School Officers.
Liu, G., Long, S., & Simpson, M. E. (2001). Self-perceived gains in critical thinking and communication skills: Are there disciplinary differences? Research in Higher Education, 40, 43–60. https://doi.org/10.1023/A:1018722327398
Liu, O. L., Frankel, L., & Roohr, K. C. (2014). Assessing critical thinking in higher education: Current state and directions for next‐generation assessment. ETS Research Report Series, 2014(1), 1–23. https://doi.org/10.1002/ets2.12009
Lungu, M. (2022). The coding manual for qualitative researchers. American Journal of Qualitative Research, 6(1), 232–237. https://doi.org/10.29333/ajqr/12085
Madill, A., Jordan, A., & Shirley, C. (2000). Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. British Journal of Psychology, 91(1), 1–20. https://doi.org/10.1348/000712600161646
McPeck, J. (1981). Critical thinking and education. New York: St. Martin’s.
McPeck, J. E. (1990). Critical thinking and subject specificity: A reply to Ennis. Educational Researcher, 19(4), 10–12. https://doi.org/10.3102/0013189X019004010
Moore, T. (2011). Critical thinking: Seven definitions in search of a concept. Studies in Higher Education, 38(4), 506–522. https://doi.org/10.1080/03075079.2011.586995
Norris, S. P. (1989). Can we test validly for critical thinking? Educational Researcher, 18(9), 21–26. https://doi.org/10.3102/0013189X018009021
Norris, S. P., & Ennis, R. H. (1989). Evaluating critical thinking. Midwest Publications.
Patton, M. Q. (2002). Qualitative research and evaluation methods. Sage.
Paul, R. W., Elder, L., & Bartell, T. (1997). California teacher preparation for instruction in critical thinking: Research findings and policy recommendations. Foundation for Critical Thinking.
Pithers, R. T., & Soden, R. (2000). Critical thinking in education: A review. Educational Research, 42(3), 237–249. https://doi.org/10.1080/001318800440579
Saldaña, J. (2021). The coding manual for qualitative researchers. Sage.
Schutz, A. (1967). The phenomenology of the social world (G. Walsh & F. Lehnert, Trans.). Evanston, IL: North Western University Press.
Shavelson, R. J., Zlatkin-Troitschanskaia, O., & Marino, J. P. (2018). International performance assessment of learning in higher education (iPAL): Research and development. In O. Zlatkin-Troitschanskaia, M. Toepell, & R. J. Shavelson (Eds.), Assessment of learning outcomes in higher education: Cross-national comparisons and perspectives (pp. 193–214). Springer. https://doi.org/10.1007/978-3-319-74338-7_10
Stupple, E. J. N., Maratos, F. A., Elander, J., Hunt, T. E., Cheung, K. Y. F., & Aubeeluck, A. V. (2017). Development of the Critical Thinking Toolkit (CriTT): A measure of student attitudes and beliefs about critical thinking. Thinking Skills and Creativity, 23, 91–100. https://doi.org/10.1016/j.tsc.2016.11.007
Thomas, K., & Lok, B. (2015). Teaching critical thinking: An operational framework. In M. Davies & R. Barnett (Eds.), The Palgrave handbook of critical thinking in higher education (pp. 93–105). Palgrave Macmillan US. https://doi.org/10.1057/9781137378057_6
VERBI Software. (2023). MAXQDA 2024 [Computer software]. https://www.maxqda.com
Voogt, J., & Roblin, N. P. (2010). 21st century skills: Discussion paper. Kennisnet, 23(3), 1–40.
Wells, I., Burton, A., & Burton, E. (2006). The OCR AS level examination as a means of assessing the critical thinking skills of undergraduate students: A pilot study. Psychology Learning and Teaching, 5(1), 32–36. https://doi.org/10.2304/plat.2005.5.1.32
Willingham, D. T. (2007). Critical thinking: Why is it so hard to teach? American Educator, 31(2), 8–19.

Citation

Tiemann, R., Danial, L., & Koenen, J. (2026). Critical Thinking Performance Assessment in an Undergraduate Physical Chemistry Laboratory Course: Developing a Contextual Critical Thinking Coding Manual. American Journal of Qualitative Research, 10(1), 37-60. https://doi.org/10.29333/ajqr/17367

Tiemann, R., Danial, L., and Koenen, J. (2026). Critical Thinking Performance Assessment in an Undergraduate Physical Chemistry Laboratory Course: Developing a Contextual Critical Thinking Coding Manual. American Journal of Qualitative Research, 10(1), pp. 37-60. https://doi.org/10.29333/ajqr/17367

Tiemann R, Danial L, Koenen J. Critical Thinking Performance Assessment in an Undergraduate Physical Chemistry Laboratory Course: Developing a Contextual Critical Thinking Coding Manual. AM J QUALITATIVE RES. 2026;10(1), 37-60. https://doi.org/10.29333/ajqr/17367

Tiemann, Rüdiger, Lilian Danial, and Jenna Koenen. "Critical Thinking Performance Assessment in an Undergraduate Physical Chemistry Laboratory Course: Developing a Contextual Critical Thinking Coding Manual". American Journal of Qualitative Research 2026 10 no. 1 (2026): 37-60. https://doi.org/10.29333/ajqr/17367

Tiemann, Rüdiger et al. "Critical Thinking Performance Assessment in an Undergraduate Physical Chemistry Laboratory Course: Developing a Contextual Critical Thinking Coding Manual". American Journal of Qualitative Research, vol. 10, no. 1, 2026, pp. 37-60. https://doi.org/10.29333/ajqr/17367

Tiemann R, Danial L, Koenen J. Critical Thinking Performance Assessment in an Undergraduate Physical Chemistry Laboratory Course: Developing a Contextual Critical Thinking Coding Manual. AM J QUALITATIVE RES. 2026;10(1):37-60. https://doi.org/10.29333/ajqr/17367