Analysis of scribbles in the margins of diagnostic forms when solving the cognitive reflection test
DOI:
https://doi.org/10.54359/kcc2w057Abstract
One indirect piece of evidence supporting the existence of cognitive reflection as a switching mechanism between the first (intuitive) and second (analytical) thinking systems is the presence of characteristic margin notes or scribbles. The potential value of analyzing such marginalia has been noted by the author of the Cognitive Reflection Test (CRT), S. Frederick, as well as other researchers. This article presents both qualitative and quantitative analyses of marginal scribbles, conceptualized as by-products of in-person testing – spontaneous, unplanned visualizations of respondents’ reflective processes and task-solving mechanisms. The study addresses two empirical questions: (1) whether such scribbles serve as evidence supporting the hypothesis of cognitive reflection as a distinct facet of rationality, and (2) whether they instead reflect mathematical problem-solving skills.
The sample included 452 students (aged 17 – 30) from universities and vocational institutions in Perm, Russia. All participants completed two sets of CRT items adapted from prior studies: S. Frederick’s original CRT-3, the extended CRT-7 (M. Toplak, R. West, K. Stanovich), and a verbal CRT version (K. Thomson, D. Oppenheimer). Scribbles were observed in 14.8% of the paper-based CRT forms. The highest frequency occurred with the classic CRT-3 tasks. Five categories of scribbles were identified: (1) correction of an incorrect intuitive answer to a correct reflective one without additional notes; (2) correction with written explanation; (3) visualizations of correct mathematical solutions; (4) incorrect calculations; and (5) uncertain responses. In most cases (77%), classic corrections from incorrect intuitive to correct reflective answers were not accompanied by any written calculations – indirectly supporting the cognitive reflection hypothesis. By contrast, mathematical calculations were equally associated with both correct reflective and incorrect intuitive responses, suggesting that a tendency toward calculation does not ensure correct answers to tasks designed with intuitive lures. Moreover, engaging the second thinking system without adequate mathematical skills may lead to incorrect responses, which might be mistakenly interpreted as intuitive. In conclusion, current evidence is insufficient to decisively confirm or refute either hypothesis. Both the cognitive reflection and mathematical skills hypotheses receive partial support. Nonetheless, the analysis of marginal scribbles in CRT forms appears to be a promising methodological tool for future research into the interaction between the two systems of thinking.
Downloads
References
Attali Y., Bar-Hillel M. The false allure of fast lures. Judgment & Decision Making, 2020, No. 15(1), 93–111. DOI: 10.1017/S1930297500006938
Bago B., De Neys W. The smart System 1: Evidence for the intuitive nature of correct responding on the bat-and-ball problem. Thinking & Reasoning, 2019, No. 3, 257–299. DOI: 10.1080/13546783.2018.1507949
De Neys W., Pennycook G. Logic, Fast and Slow: Advances in Dual-Process Theorizing. Current Directions in Psychological Science, 2019, No. 28, 503–509. DOI: 10.1177/0963721419855658
Easton C. Women and ‘the philosophical personality’: evaluating whether gender differences in the Cognitive Reflection Test have significance for explaining the gender gap in Philosophy. Synthese, 2021, No. 198, 139–167. DOI: 10.1007/s11229-018-01986-w
Evans J.St.B.T. On the resolution of conflict in dual-process theories of reasoning. Thinking and Rea-soning, 2007, No. 13, 321–329.
Evans J.St.B.T. In Two Minds: Dual-Process Accounts of Reasoning. Trends in Cognitive Sciences, 2003, No. 7, 454–459. DOI: 10.1016/j.tics.2003.08.012
Frederick S. Cognitive reflection and decision making. The Journal of Economic Perspectives, 2005, No. 19, 25–42.
Haigh M. Has the standard cognitive reflection test become a victim of its own success? Advances in Cognitive Psychology, 2016, No. 12(3), 145–149. DOI: 10.5709/acp 0193-5
Handley S.J., Newstead S.E., Trippas, D. Logic, beliefs, and instruction: A test of the default interven-tionist account of belief bias. Journal of Experimental Psychology: Learning, Memory, and Cognition, 2011, No. 37(1), 28–43. DOI: 10.1037/a0021098
Livengood J., Sytsma J., Feltz A., Scheines R., Machery E. Philosophical temperament. Philosophical Psychology, 2010, No. 23(3), 313–330. DOI: 10.1080/09515089.2010.490941
Martín M., Valiña M. Heuristics, Biases and the Psychology of Reasoning: State of the Art. Psychol-ogy, 2023, No. 14, 264–294. DOI: 10.4236/psych.2023.142016
Otero I., Salgado J., Moscoso S. Cognitive reflection, cognitive intelligence, and cognitive abilities: A meta-analysis. Intelligence, 2020, No. 90, 1–13. DOI: 10.1016/j.intell.2021.101614
Pennycook G., Cheyne J.A., Koehler D.J., Fugelsang J.A. Is the cognitive reflection test a measure of both reflection and intuition? Behavior Research Methods, 2016, No. 48, 341–348. DOI: 10.3758/s13428-015-0576-1
Primi C., Morsanyi K., Chiesi F., Donati M., Hamilton J. The development and testing of a new ver-sion of the cognitive reflection test applying item response theory (IRT). Journal of Behavioral Deci-sion Making, 2015, No. 29(5), 453–469. DOI: 10.1002/bdm.188314
Reilly D. Gender, culture, and sex-typed cognitive abilities. PloS one Public Library of Science, 2012, No. 7(7), e39904, 1–16. DOI: 10.1371/journal.pone.0039904
Risko E., Gilbert S. Cognitive offloading. Trends in Cognitive Sciences, 2016, No. 20, 676–688. DOI: 10.1016/j.tics.2016.07.002
Robson S., Martire K., Drew M., Nicholls K., Faasse K. Thinking false and slow: Implausible beliefs and the Cognitive Refection Test Psychonomic. Bulletin & Review, 2023, No. 30(6), 2387–2396. DOI: 10.3758/s13423-023-02321-2
Rodina O.N., Prudkov P.N. Aprobatsiya russkoyazychnykh versii testa kognitivnoi refleksii. Voprosy psikhologii, 2019, No. 4, 155–162. (in Russian)
Sinayev A. Peters E. Cognitive reflection vs. calculation in decision making. Frontiers in Psychology, 2015, No. 6(532), 1–6. DOI: 10.3389/fpsyg.2015.00532
Sloman S. The empirical case for two systems of reasoning. Psychological Bulletin, 1996, No. 119(1), 3–22. DOI: 10.1037/0033-2909.119.1.3
Stanovich K., West R. On the relative independence of thinking biases and cognitive ability. Journal of personality and social psychology, 2008, No. 94(4), 672–695. DOI: 10.1037/0022-3514.94.4.672
Szaszi B., Szollosi A., Palfi B., Aczel D. The cognitive reflection test revisited: exploring the ways individuals solve the test. Thinking & Reasoning, 2017, No. 23(3), 207–234. DOI:10.1080/13546783.2017.1292954
Thomson K., Oppenheimer D. Investigating an alternate form of the cognitive reflection test. Judg-ment and Decision Making, 2016, No. 11(1), 99–113. DOI: https://doi.org/10.1017/S1930297500007622
Toplak M., West R., Stanovich K. The cognitive reflection test as a predictor of performance on heu-ristics-and-biases tasks. Memory & Cognition, 2011, No. 201439(7), 1275–1289. DOI: 10.3758/s13421-011-0104-1
Vikhman А.А. Diagnostika kognitivnoj refleksii: analiz testov. Vestnik Udmurtskogo universiteta. Seriya Filosofiya. Psihologiya. Pedagogika, 2025, No. 1, 13–27. DOI: 10.35634/2412-9550-2025-35-1-13-27 (in Russian)
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
