QALMRI & SPARK Tutorials

This page consists resources for writing QALMRI and SPARK summaries.

The QALMRI Method

You will write QALMRIs for weekly assignments as well as for your final project. This page is intended to help you understand the elements of a good and thorough QALMRI.

Acknowledgement: The QALMRI method was first introduced by:

Kosslyn, S. M., & Rosenberg, R. S. (2001). Psychology: The Brain, The Person, The World. Boston: Allyn & Bacon

The description of the QALMRI method below is adapted and modified from Dr. Matthew J. C. Crump’s textbook:

Matthew J. C. Crump. (2021). Instances of Cognition: Questions, Methods, Findings, Explanations, Applications, and Implications. https://crumplab.com/cognition/textbook

QALMRI is an acronym for critical parts of research articles. It stands for Question, Alternatives, Logic, Method, Results, and Inference. This section demonstrates using QALMRI as a guide for reading a primary research article. QALMRI can also be used as an activity or assignment, and an example of a QALMRI assignment is given at the end.

Finding a paper

The first step is to find a primary research article to read. To demonstrate QALMRI, I chose a paper from my own research. The article is titled, “Warning: this keyboard will deconstruct – The role of the keyboard in skilled typewriting”. This paper was published in the journal Psychonomic Bulletin and Review in 2010 and can be freely downloaded as a pdf here.

Before continuing, I suggest you download the paper and give it a quick glance. It it is only 5 pages long.

Anatomy of a primary research article

The paper you just downloaded is an example of a primary research article. The pieces of different research articles are very similar. For example, this one has a title, an abstract, an introduction, methods and results sections for two experiments, a general discussion, and references. Most research papers have similar components:

  • Title: The concise title captures the main focus of the study.

  • Abstract: The brief summary provides an overview of the research objectives, methods, key findings, and significance.

  • Introduction: This section presents background information, previous research, and identifies gaps in knowledge.

  • Methods: The section describes the study design, sample size, data collection procedures, and statistical analysis methods.

  • Results: The findings of the study are presented objectively, using data tables, figures, or graphs.

  • General Discussion: This section interprets the results, discusses implications, limitations, and future research directions.

  • References: A list of cited sources, following a specific citation format, allows readers to access the cited works.

By adhering to this structure, a primary research article ensures clarity, transparency, and replicability, enabling evaluation of the study’s validity and significance.

Research papers are often written for a technical audience, which refers to other researchers who are already familiar with the domain. As a result, students may find it challenging to extract and comprehend major points of research articles. We will use QALMRI as an aid to help you identify and comprehend the components of a research paper.

What does QALMRI stand for?

Q stands for Question

Researchers ask and answer research questions. So, your first task is to identify what questions are being asked in the research article. Research questions are usually stated in the abstract or the introduction, and sometimes they are restated near the beginning of the general discussion.

You may notice many different kinds of questions being stated in a research paper. It is helpful to distinguish between broad questions and specific questions. A single research paper is rarely capable of answering very broad questions, but it may have answered a specific question.

The paper you downloaded is about cognitive abilities mediating skilled typewriting on a computer keyboard. This topic is associated with many broad questions. For example, how do people control their own body movements? How do people learn to type without thinking about what their fingers are doing? How do people learn skills? How do people learn language? All of these questions are too big for a single research paper to answer.

The paper also asks specific questions. The overarching specific question is about how the feeling of the keyboard influences typing performance. People can type on keyboards with keys, or on flat surfaces without keys. How does tactile feedback from the keyboard influence typing performance?

A stands for Alternatives

Experimental research in cognition is often designed to test alternative explanations or hypotheses about a cognitive ability. When reading a paper you should attempt to identify the alternative explanations or hypotheses being discussed. Note also, sometimes a paper may only discuss one alternative hypothesis. More important, the alternative or hypothesis should contain an implication that can be tested later on.

The example paper has two major alternative ideas about cognitive processes in typing behavior. Both alternatives provide ideas about how people are able to move their fingers to individual keys very quickly and accurately, even without looking at the keyboard.

One possibility is that people have an internal cognitive map of the keyboard. The cognitive map represents the location of the keys on a computer, and typists may use this internal “mind” map to direct their fingers to appropriate locations while typing. An implication of the cognitive map idea is that typists may not need to rely on the feeling or touch of a keyboard in order to type quickly and accurately. Instead, their fingers can move to each key based entirely on directions from the internal map.

An alternative possibility is that people do not use an internal map of the keyboard. Instead, people may learn associations between the finger movements required for each keystroke and they way they feel (tactile, haptic, and proprioceptive feedback). An implication of this idea is that feeling the keyboard may be an important component of typing skill, especially for typists who learned to type on keyboards with keys.

L stands for Logic

The logic of the alternatives or hypotheses is nearly identical to the hypotheses themselves, but more formally stated in terms of if/then statements. Here are two example logic statements for each of the alternatives discussed above:

IF typists use an internal cognitive map that does not require feedback from the keyboard to guide their fingers, THEN typing performance should not be influenced by manipulations that remove tactile feedback, such as typing on keys vs a flat surface.

IF typists use feedback from the keyboard to guide their fingers, THEN typing performance should be influenced by manipulations that remove tactile feedback, such as typing on keys vs a flat surface.

M stands for Method

The method refers to the tools used to answer the research questions. More specifically, the method will usually be carefully designed to test the logical implications of the alternative explanations. In experimental research, the method will involve at least one independent variable or manipulation, and at least one dependent variable or measurement.

In our example, the method involved having skilled typists copy text while they were typing on keyboards with different surfaces, that provided less and less tactile feedback. The primary manipulation had four levels, and involved changing the feeling of the keyboard: there was a regular keyboard, a rubber button keyboard, a flat surface keyboard, and a laser projection keyboard. Measurements of typing performance (keystroke typing times and accuracy) were collected for each typist as they typed words on each keyboard.

R stands for Results

The results are a final product of the research cycle. A given research article may present many results depending on the complexity of the method. In general, the results refer to an analysis of whether the experimental manipulation influenced the measurements.

In our example, typing performance was measured for four different keyboards. Results were reported in a graph and showing different measures of typing performance, such as reaction time (time to start typing a word), inter-keystroke interval (time between keystrokes), and error rate. The major result was that typing performance WAS influenced by the keyboard manipulation. Typists were fastest and most accurate on a regular keyboard, and always slower and less accurate on the keyboards with less tactile feedback.

I stands for Inferences

Inferences may be the most important product of a research project. Inferences connect the results back to the original research questions. So, what inferences about the alternative explanations under investigation can we draw based on the results of the research? Inferences are possible if the methods do a good job of testing the logic of the alternatives.

In our example, the logic of the internal keyboard map idea suggested that manipulations to the feeling of the keyboard should not influence typing performance. However, the results of the study showed that reducing tactile feedback from the keyboard causes slower and more error-prone typing. So, one inference from the study could be that typists do not rely upon an internal map of the keyboard.

Writing a QALMRI summary

Writing a QALMRI for any research paper (one that you are writing, or one that you are reading) involves writing short answers to each of the QALMRI points using clear and concise language. It is a condensed, short-form, version of the research. Your task is to answer these questions:

  • Question: What was the broad question? What was the specific question?

  • Alternative hypotheses: What were the hypotheses?

  • Logic: If hypothesis 1 was true, what was the predicted outcome? What was the predicted outcome if hypothesis 2 was true?

  • Method: What was the experimental design?

  • Results: What was the pattern of data?

  • Inferences: What can be concluded about the hypotheses based on the data? What can be concluded about the specific and broad question? What are the next steps?

Example QALMRI

Even if you haven’t read the article, reading a QALMRI should provide you with enough information to get a basic idea of what the article was about. The following QALMRI summarizes our example article by Crump & Logan (2010).

Article citation

Article: Crump, M. J., & Logan, G. D. (2010). Warning: This keyboard will deconstruct—The role of the keyboard in skilled typewriting. Psychonomic bulletin & review, 17(3), 394-399.

  • Note: The APA citation is the first thing your QALMRI summary should contain. This informs the reader (usually me) which article you have chosen to summarize.

What was the broad and specific question?

The broad questions are about spatial cognition. How do people understand and represent the spatial relationships between objects in the environment? Do people have “spatial maps” in their head?

The specific question is how do typists know where the keys are on the QWERTY keyboard?

What are the alternatives?

  1. Typists have an internal cognitive spatial map of the keyboard that they use to guide their fingers during typing

  2. Typists do not have a map-like representation, instead they rely on learned associations between cues such as the feel of the keyboard to guide their fingers during typing

What is the logic?

  1. If typists have an internal map of the keyboard, then they should be able to guide their fingers to correct locations based on the map alone and no feedback from the environment. For example, if we could measure “air-typing” without a keyboard, then typists should still be able to put their fingers in the correct locations even when the keyboard is missing because they are relying on their internal map.

  2. If typists do not use an internal map of the keyboard, then their finger movements should become slow and inaccurate when they try to type without a keyboard, or in other conditions that change the normal feel of the keyboard, and thereby remove the cues that typists use to direct their fingers.

What is the Method?

  • Note: If a paper has multiple experiments, you are expected to report the methods and results of ALL experiments in the QALMRI, and also briefly provide reasoning about how the experiments are connected (see below).

  • In Experiment 1, typists copied paragraphs in four conditions that manipulated tactile (touching) feedback from the keyboard. They typed on a normal keyboard, a keyboard with the keys removed exposing the rubber buttons underneath, a flat circuit board without, and on a flat table with a laser projection keyboard.

  • In Experiment 2, typists performed the same task as Experiment 1, but only with the regular and laser projection keyboard. Additionally, the number of practice trials with the laser projection keyboard were increased, to evaluate whether the results of Experiment 1 could simply be explained by novetly/lack of practice with the laser projection keyboard.

Results

Experiment 1: Typists were fast and accurate in the normal keyboard condition. Typists were slow and inaccurate in all of the other keyboard conditions, where normal tactile feedback was removed.

Experiment 2: Typists were again fast and accurate in the normal keyboard condition compared to the laser keyboard condition, despite increased practice with the laser keyboard.

Inference

The results are not consistent with the internal map hypothesis. If typists had an internal map, and did not rely on tactile cues, then they should have typed normally even when the cues were removed (as in the case of the laser projection keyboard). The results are consistent with learned association hypothesis, that typists rely on cues, like the feel of the keyboard, that are associated with particular key locations.

Note: For this class, you should also reflect more broadly on the inferences in this section. You should discuss how the discussion in class elaborated on the inferences as well as any other broader themes that jumped out at you from the class session.

The SPARK Method

This guide aims to help you make the most of reading review articles in psychology or summarizing an academic podcast.

Acknowledgement: I used ChatGPT to build this section:

OpenAI. (2023). ChatGPT (May 24 version) [Large language model]. https://chat.openai.com/chat

Review articles provide a broader understanding of a topic by integrating and summarizing existing research. Whether you’re studying psychology in a seminar course, exploring the field in a research context, or out of personal interest, this guide will help you read and engage with review articles effectively.

Academic podcasts seek to provide an informative, thought-provoking, and comprehensive exploration of a chosen topic. They offer a platform for in-depth conversations, where academic expertise converges with real-world applications, fostering a deeper understanding and appreciation of the subject matter among the listeners.

Finding a paper or podcast

The first step is to find a review article to read or a podcast to listen to. To demonstrate SPARK, we will choose the following review article:

Carpenter, S. K., Pan, S. C., & Butler, A. C. (2022). The science of effective learning with spacing and retrieval practice. Nature Reviews Psychology, 1(9), 496-511.

This paper was published in the journal Nature Reviews Psychology in 2022 and can be freely downloaded as a pdf here.

Before continuing, please download the paper and peruse through it briefly. As you might have noticed, review articles tend to be longer than primary research articles, as they summarize several different strands of research.

We will also use the following podcast as an example:

DeepMind Podcast: AI and neuroscience: The virtuous circle

Anatomy of a review article

Review articles typically follow some kind of structure:

  • Introduction: Sets the stage, explains the relevance of the topic, and outlines the objectives of the review.
  • Historical context or background: Provides a historical perspective or background information on the topic to offer context.
  • Themes or subtopics covered: Organizes the content into themes or subtopics, giving you an overview of the field.
  • Summary of key findings: Presents the main conclusions and key findings from the reviewed literature. Often, these findings are presented for each theme or subtopic.
  • Conclusion and future directions: Interim conclusions/future directions summarize the main points for each theme/subtopic and suggest avenues for future research. Broader conclusions/future directions are often at the very end of the review article, that try to bring together common ideas across themes/subtopics.
  • References: Provides a list of sources for further exploration.

Anatomy of a Podcast

Podcasts come in several formats, but for an academic podcast, here’s a more detailed outline:

  • Host(s) and speaker(s): Engaging hosts and knowledgeable speakers guide the conversation, providing expertise and facilitating a dynamic discussion.

  • Broader questions and ideas: Each episode centers around a thought-provoking and overarching question or idea, fostering critical thinking and exploration of diverse perspectives.

  • Specific studies/findings: The podcast delves into specific studies, research findings, and scholarly work relevant to the topic, providing evidence-based insights and analysis.

  • Connections and implications: The discussions draw connections between the research and its broader implications, highlighting real-world applications, potential impacts, and avenues for further exploration.

What does SPARK stand for?

SPARK stands for Subject, Perspectives, Analysis, Reflection, and Knowledge. Let’s explore each subpoint in more detail:

S stands for Subject

The subject refers to the main topic being discussed in the review article or podcast. It represents the central theme or subject matter that the article or podcast aims to address. It is important to understand the significance of the chosen topic within the field of psychology. What makes it relevant, timely, or impactful? Consider how the subject contributes to existing knowledge, addresses gaps, or brings new insights to the field.

P stands for Perspectives

This subpoint involves identifying the main themes or subtopics covered in the review article or podcast. Look for the different perspectives, viewpoints, or angles explored by the author(s) regarding the chosen subject. These perspectives can provide a comprehensive understanding of the topic by considering diverse viewpoints, theoretical frameworks, or empirical evidence. Consider the range of perspectives presented and how they contribute to the overall understanding of the subject.

A stands for Analysis

In this stage, focus on the key findings summarized for each subtopic or theme discussed in the review article or podcast. Explore how the author(s) evaluated the evidence and drew conclusions. Consider the research studies, theories, or empirical data presented to support the key findings. Evaluate the strength of the evidence and the quality of the analysis provided. Reflect on what is currently known about the topic based on the presented evidence.

R stands for Reflection

This subpoint requires examining any limitations or gaps in the reviewed literature that the authors mentioned. Consider the authors’ reflections on the existing knowledge or potential biases in the research studies reviewed. Additionally, pay attention to any suggestions made by the authors for future research directions. Reflect on the implications of the identified limitations or gaps and how they contribute to the understanding of the subject. Consider the potential areas for further investigation or improvement in the field.

K stands for Knowledge

Think about how reading the review article or listening to the podcast has influenced your own perspectives or raised new questions about the topic. This is also a place to reflect on the class discussion we had in a given week. Identify the key insights or findings you gained from this week’s discussion. Consider how these insights contribute to your overall understanding of the subject. Reflect on any new knowledge acquired and its potential impact on your future thinking, research, or practice within the field of psychology.

By following the SPARK framework, you can engage in a comprehensive analysis of a review article or podcast, extracting key information, evaluating the evidence, and gaining a deeper understanding of the chosen subject matter in the field of psychology.

Writing a SPARK summary

Writing a SPARK summary involves answering a set of questions:

  • Subject: What is the main topic of the review article/podcast, and why is it significant in the field of psychology?

  • Perspectives: What ideas or subthemes have been considered or presented?

  • Analysis: What key findings were summarized for each subtopic/theme, and how did the author(s) evaluate the evidence? What do we know so far?

  • Reflection: What limitations or gaps in the reviewed literature do you see or the authors highlight? What potential areas of future research do they suggest?

  • Knowledge: What key insights or findings did you gain from reading the review article or listening to the podcast and the class discussion? How do these insights contribute to your understanding of the topic?

Example SPARK: Review Article

The review article we have chosen is titled “The science of effective learning with spacing and retrieval practice.”. Below is an example SPARK summary for the article:

Article: Carpenter, S. K., Pan, S. C., & Butler, A. C. (2022). The science of effective learning with spacing and retrieval practice. Nature Reviews Psychology, 1(9), 496-511.

  • Note: The APA citation is the first thing your SPARK summary should contain. This informs the reader (usually me) which article you have chosen to summarize.

Subject: In the review article, the subject is effective learning, i.e., how can we be effective learners? This topic is significant because humans learn new skills and knowledge throughout their lifetime, and this knowledge helps them navigate the world around them. Therefore, knowing how to learn effectively is important - especially early in life when humans are being exposed to a large amount of new information.

Perspectives: The review focuses on two main learning strategies: spacing and retrieval practice. Spacing is the strategy of spacing out the learning of material, and retrieval practice involves repeatedly engaging in active retrieval of the material. The review also discusses the research on metacognition and how it may contribute to the adoption and effectiveness of these strategies in real-world contexts. The research on metacognition in cognitive psychology and education is discussed.

Analysis:

  • Note: This piece is likely going to be the longest part of your SPARK summary.

  • The article first discusses the research on spacing. The authors summarize a large collection of research (Table 1) across many age groups (preschool to older adults), learning materials (words, pictures, courses, etc.), and spacing implementations (how spaced does learning need to be? 30 seconds to 1-2 weeks). The broader consensus is that spacing works. There is no clear sense of what the ideal spacing schedule looks like but 1 to 7 days is a generally good ballpark. The recommendation from the authors is that the material should feel “still familiar but not fresh in the mind”. They also briefly touch on why spacing works. There are many theories, such as spacing creates “distinct learning experiences” which serve as memory cues, or that it provides a “mental break”, or that it encourages retrieval practice by forcing people to retrieve from the earlier session.

  • The article then discusses the research on retrieval practice. As before, the authors summarize many years of research (table 2) across age groups, materials, and implementation (what kind of retrieval practice? multiple-choice, true-false, cued recall, short answer, etc.), with the consensus that it works. They also emphasize how feedback is important for retrieval practice as it can help identify gaps in knowledge. They also discuss why retrieval practice works: by providing additional cues/routes to the same information and likely some neural consolidation mechanisms. Finally, they discuss that while retrieval practice is very robust for near transfer (to very similar problems/tasks), the research on far transfer is less robust.

  • Finally, the authors discuss metacognition: the ability to think about one’s thinking and regulate one’s actions accordingly. They talk about how the cognitive psychology research has focused on micro-level constructs: awareness/monitoring and regulation/control. The research seems to suggest that learners are bad at both. The educational psychology research has focused on macro-level constructs: planning, goals, evaluation, motivation, and affect. This work seems to suggest that learners’ motivation is critical and may play into both awareness and regulation and thus complements the cognitive research.

Reflection: The authors discuss how the successful relearning could happen by combining the benefits of spacing and retrieval practice. However, they also emphasize that improving metacognition is key to successful relearning, but is also very difficult. Research suggests that simply making students aware, or giving them experience with the strategies is not sufficient. They discuss false beliefs about learning may be key here: students and teachers both may not know what strategies are effective and may believe that strategies that need more effort or time may not be as effective, contrary to what the evidence suggests. They suggest that a comprehensive framework that emphasizes knowledge, belief, commitment, and planning may be required to improve metacognition. They also discuss how it is important to truly understand learner’s perspectives in future research, as well how technology may change how stategies are implemented and adopted in the future.

Knowledge:

  • Note: This is where you would consolidate all the information you learned from the paper AND the class discussion.

  • Personally, I found the paper really interesting and succinct in describing the current research on effective learning strategies. When I was an undergraduate, I had very little information about what strategies worked and didn’t, and was often left to guess how I should apprach my learning. I was good in some courses and not so good at others, and it was very unclear if it was my learning style, my motivation, or the environment. Knowing what I know now would have been really helpful! I would have liked to know a bit more about interventions that target motivational or affective aspects of metacognition.

Example SPARK: Podcast

The podcast we have chosen is titled “AI and neuroscience: The virtuous circle”. Below is an example SPARK summary for the podcast:

Podcast: DeepMind Podcast: AI and neuroscience: The virtuous circle

  • Note: The citation is the first thing your SPARK summary should contain. This informs the reader (usually me) which article/podcast you have chosen to summarize.

Subject: The podcast discusses the goals of artificial intelligence, what it means for a machine to be intelligent, and how closely should it mimic human/animal minds. These ideas/questions are important as we are living in a world where artificial systems are all around us, and so understanding the motivations of the researchers behind these tools is important, just as important as figuring out exactly how AI learns from or builds on neuroscience.

Perspectives: Hannah Fry (the host) talks to Demis Hassabis the founder of DeepMind, as well as other scientists in the podcasts, such as Jess Hamrick, Matt Botvinick, Greg Wayne. They first discuss what the goals and potential of AI is: what can AI achieve and what should we want it to achieve? They next discuss how AI has learned from and can learn from human and animal minds. They talk about the idea of Moravec’s paradox: where the seemingly simple things humans do turn out to be the most complex things to program in an AI. Finally, they discuss how AI has also informed and advanced neuroscience.

Analysis:

  • Goals: Demis believes that AI could be used to solve humanity’s biggest problems, such as climate change, cancer, language, energy, etc. But to achieve all this, we need to decide what kind of AI do we want to build - should it be like a human or completely different? Hamrick weighs in on this and says that it is important for AI to be a little bit like us because if AI systems are deployed in the real world and they are not like us, or don’t think like us, then there will be no trust in them and this lack of understanding what it’s doing will make its use problematic. Fry discusses the case of an AI trained to diagnose skin cancer using images, where the AI was making judgments based on random information (a ruler in the photos). Botvinick discusses how AI needs to definitely “think like a human” in some ways but may need to go beyond that. They discuss the Wright brothers and how the idea originated from observing birds for years but then moving a step ahead to make planes.
  • Neuroscience inspiration: Fry discusses the idea of replay in human and animal brains, where past experiences are replayed over and over, which has been recast into reinforcement learning in AI, where the program remembers and learns from previous plays to inform future plays in games. Hamrick discusses the idea of mental simulation and how it helps us make predictions about the world, something that AI systems are also using now. Wayne discusses animal research on scrub jays and their ability to remember information about where they have stored their food on very long time scales.
  • Reward: Fry describes how AI systems learn in the first place: not through rules, but through keeping track of 0s and 1s and learning through prediction, i.e., by predicting when a reward might occur, the system learns a particular behavior.
  • Moravec’s paradox: Fry discusses the overarching theme that emerges from these discussions: that tasks that are seemingly simple for humans such as sitting, picking things up, etc. are the ones that are very hard to teach machines. But with advances in neuroscience and AI, Hassabis thinks this no longer needs to be a barrier as we have begun to understand how vision works through pattern recognition, etc.
  • AI back to neuroscience: Botvinick discusses the research on dopamine, where they were trying to understand why dopamine is released in the brain. The brain research came across computer scientists who were working on reinforcement learning, which led to a revolution in neuroscience: dopamine wasn’t the response to the reward, but it was the prediction error (what was the expectation vs. what was received).

Reflection: Overall, this podcast introduced a lot of exciting themes and questions. The host reflected on the history of the field in tandem with developments in neuroscience and where the current research is heading in AI. There appear to be many different motives and ideas in the field.

Knowledge:

  • Note: This is where you would consolidate all the information you learned from the paper AND the class discussion.

  • I found the podcast interesting and learned quite a bit! As a cognitive scientist, it was interesting for me to learn some of the early connections between neuroscience and AI, and how it truly is a circle. I am curious to know more about how different cognitive aspects of the human/animal mind may be essential vs. exciting to implement in artificial systems.