Think back to the last academic paper you read. Maybe it was for a coursework assignment or maybe just for fun. Which sections did you find easiest and most enjoyable to read? Which did you find most challenging? If you’re like many students (and academics!), the Method section was the most difficult to read (Hubbard & Dunbar, 2017). Yet this section is often the most important part of the paper. In this post, I’ll share some practical tips for reading and thinking critically about research methods. Then, I’ll show you what that looks like in practice by evaluating the methods used in one of my own studies as an example.

 

Why are methods so important?

Think of the Method section like the recipe for the study. It outlines the key ingredients and steps that ultimately produced the results: who took part, what materials were used, and how the data were analysed. Perhaps a common misconception is that understanding research methods is only relevant to those who plan to produce research. But while most of us won’t go on to produce research, we will almost certainly all consume research. Whether you’re evaluating the latest health claim on TikTok or reviewing evidence for a new type of therapy, being able to critically analyse research methods is what allows us to determine how much confidence we can have in a study’s findings.

 

How to tackle tricky methods sections.

While important, Method sections are notoriously difficult to understand, let alone evaluate. This is especially the case when we’re learning about particular technique for the first time. Below are some tips and tricks that I’ve gathered over the years that help me tackle difficult Method sections, and hopefully they’ll work for you too!

 

Step 1: Start with the basics

When first reading through the Methods section, don’t try to evaluate it right away. Start by describing what the researchers did. You might not understand every detail – focus on what you do understand. You’ll likely be able to identify some key features: Who participated? What materials were used? What did participants do? What were the outcome measures?

 

Step 2: Get curious about the details

Now that you understand the basics, it’s time to get curious about how the researchers implemented their chosen technique. For example, if you’re reading a systematic review, dig deeper into how the researchers identified papers and what criteria they used for exclusion.

 

You can also take a close look at the decisions the researchers made. Every study involves what are known as “researcher degrees of freedom” (Simmons et al., 2011), seemingly small decisions that can actually influence the results .

 

Example: A researcher studying whether musicians have an advantage for understanding language must first decide what constitutes a “musician”. Is it years of regular practice? Formal training?  How many years count?

 

These decisions are necessary but, in some cases, can dramatically alter the outcome. Returning to the musicians example, it makes a big difference if a musician means someone who learned the recorder for a few years in primary school or someone who actively plays in a professional orchestra! So when reading the Method section, look at whether and how researchers justify their choices and try to understand why the researchers made the decisions they did.

 

Step 3: Think critically about the methods

What do you think? Now it’s time to critically evaluate the methods. This can feel intimidating, especially if you don’t feel like an expert yet. Fortunately, critical evaluation is a transferable skill. Just use the same reasoning you’d use to evaluate whether you trust a claim you see on the news. Ask yourself:

 

• Do the methods allow the researchers to adequately test their research questions?

 

• Could any methodological decisions the researchers made influence the results?

 

• How might future studies overcome these limitations?

 

Now let’s look at how this evaluation process (Step 3) plays out in practice. Below is an annotated example from a blog post where I evaluate the methods used in one of my own studies. In this study, I was interested in whether musical training impacts how we understand speech. You can find the full piece here but for now, all you need to know is that the study was conducted online and involved musicians and non-musicians listening to speech. But as with all techniques, online testing has some key limitations that I have critically evaluated here:

 

 

Note how the paragraph starts by briefly reminding the reader what the method was and why it was chosen. Only then does it begin the critical evaluation, which starts out fairly surface-level explanation of what I think could be a problem before digging deeper into why a noisy environment would be an issue for this study specifically. Often, people get stuck at the surface-level. This not only impacts students’ assessment marks but has real-world consequences: changing someone’s mind about questionable research requires explaining not just what’s wrong with a study, but why it matters.

 

If this process seems like a lot of work, you might be wondering, can’t I just get AI to do this? While AI is good for many things, critical evaluation is not one of them. Generative AI models like ChatGPT have limited training data about highly technical subjects like research methods and are therefore more likely to make mistakes. In addition, their critiques are often surface-level.

 

Example: As a test, I asked ChatGPT for a critical evaluation of one of my own paper’s methods. As you can see below, it identified one of the study’s key limitations but does not explain why it matters.

 

“Also, the study was conducted entirely online. This opens doors to diverse participation, but it also means participants may have been using everything from studio headphones to laptop speakers in a noisy café.”

 

In addition, ChatGPT made some key factual errors, like praising the study for being preregistered when it wasn’t. While full access to stimuli and code were provided, this is not the same as preregistration.

 

“The authors preregistered their analyses, provided full access to stimuli and code, and published their data and materials on the Open Science Framework.”

 

So for understanding and evaluating research methods, your brain is still better than AI!

 

To sum up

If you find Method sections tricky, you’re not alone! Methods are often the most challenging parts of the paper to read. But they are still important! They allow us to determine how much confidence we can have in the research findings. Next time you read a paper, try applying some of these tips and see if they work for you. Method sections might still be challenging, but you’ll have some tools to tackle them.

 

References

Hubbard, K. E., & Dunbar, S. D. (2017). Perceptions of scientific research literature and strategies for reading papers depend on academic career stage. PloS one, 12(12), e0189753. https://doi.org/10.1371/journal.pone.0189753

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632