Online Science Experiments: The Paradigm Shift during COVID-19 and the Rise of Cognition

March 2020: the labs close

Before 2020, most behavioral and cognitive experiments were run the same way they had been for decades: a participant came to a university building, sat in front of a lab computer, and a research assistant pressed start. Online experiments existed, but for many labs they were a side project rather than the main method.

When universities closed their buildings in March 2020, that pipeline stopped overnight. Ethics boards suspended in-person testing, undergraduate participant pools went quiet, and doctoral students with data collection scheduled for that spring had to find another way. Within weeks the question in lab meetings changed from "should we run this online?" to "how quickly can we run this online?".

Where the experiments went

Running a study online requires two things: somewhere to recruit participants and somewhere to host the experiment. On the recruitment side, Amazon Mechanical Turk had been used by psychologists since the early 2010s, and Prolific, a platform built specifically for research, saw rapid growth during the pandemic. Universities also moved their own participant pools (often managed through SONA) to remote participation.

On the hosting side, researchers chose between graphical builders such as Gorilla, Pavlovia (the online counterpart of PsychoPy), Qualtrics for survey-based work, and code-first options such as jsPsych hosted on a lab server or on a dedicated service like Cognition. The choice often came down to how much control a lab wanted over timing and stimulus presentation versus how much time it could spend on programming.

jsPsych as the open standard

Much of the code-first migration converged on jsPsych, the open-source JavaScript library introduced by Joshua de Leeuw in 2015 (de Leeuw, 2015). Its timeline-and-plugin model maps naturally onto how experimenters already think about their designs, and because it is plain JavaScript running in the browser, an experiment written in jsPsych can be hosted anywhere. Version 7 introduced a modular architecture and npm packages, and version 8 refined the timeline system; both are documented in de Leeuw, Gilbert and Luchterhandt (2023).

A library is not a hosting service, though. A researcher still needs a server, HTTPS, a way to store incoming data safely, and a way to hand out links. Cognition grew out of exactly that gap: you paste jsPsych code into a web editor, upload stimuli, and receive a https://<token>.cognition.run link. Trial data is written to the server as it is produced rather than in one upload at the end, so a participant who closes the tab halfway through still leaves usable data.

Is online data any good?

Skepticism about online data predates the pandemic, and so does the evidence addressing it. Crump, McDonnell and Gureckis (2013) replicated a series of classic cognitive paradigms (Stroop, flanker, task switching, subliminal priming and others) on Mechanical Turk and found that most effects reproduced well, with the notable exception of tasks depending on very brief stimulus presentations. The message was that browser timing is good enough for most reaction-time work, but not for everything.

Attention is the other concern. Hauser and Schwarz (2016) found that MTurk workers actually passed instructional attention checks at higher rates than student samples in the lab. More recently, Peer, Rothschild, Gordon, Evernden and Damer (2022) compared several recruitment platforms on attention, comprehension, honesty and reliability, and reported that Prolific produced the highest-quality data in their tests, with MTurk showing more problems than it had a decade earlier.

What stayed after the pandemic

Buildings reopened, but many labs did not go back to running everything in person. Online collection is faster, cheaper per participant, and reaches people outside the undergraduate population. Institutional review boards now have templates for remote consent, and journals routinely accept online samples for paradigms that were once lab-only.

Some limitations are structural and have not gone away:

• Timing. Browser and display refresh variability add noise to reaction times and make sub-50 ms stimulus durations unreliable. Within-subject designs absorb much of this, but the precision of a lab setup is not available.
• Environment. You do not know the screen size, the viewing distance, whether the participant is on a phone, or whether a television is on in the room. Tools such as the jsPsych virtual chinrest plugin, full-screen mode and device checks mitigate but do not remove this.
• Bots and low-effort responding. Automated and careless participants exist on every platform. Attention checks, response-time filters, single-use links and post hoc exclusion criteria registered in advance are standard practice.

Where Cognition fits

Cognition is one of the hosting options that came out of this period, and its design reflects the problems above. Query parameters such as PROLIFIC_PID are stored as columns on every trial, which makes bonus payments and exclusions straightforward (see our guide on integrating with Prolific, SONA and MTurk). Single-use links limit repeat participation, a service worker buffers data locally if the connection drops, between-subject conditions are balanced on the server, and recruitment can be stopped on a date or after a set number of participants. jsPsych versions from 6.0.5 to 8.2.3 are supported, with a free migrator for older code.

If you want to try it, the free plan allows four tasks with sixty participants each. Create a free account, or look at one of the demos such as the Stroop test or the flanker task to see what a hosted experiment looks like from the participant's side.