Psychology Experiments: Traditional Approaches and the Shift Online

A practical guide

This article is a working checklist for designing and running a psychology experiment, with notes on where online data collection changes the decisions. It assumes you already have a hypothesis and covers design, recruitment, data quality, ethics and a concrete pipeline with jsPsych and Cognition.

1. Design

Start by naming your independent variable(s), the levels of each, and the dependent measure (reaction time, accuracy, a rating). Then decide how participants are allocated to levels.

• Within-subject designs expose every participant to every condition. They need fewer participants because each person is their own control, and they are robust to hardware differences online, since a slow keyboard slows all conditions equally. The cost is order effects (practice, fatigue, carry-over).
• Between-subject designs assign each participant to one condition. Necessary when exposure to one condition would contaminate another (e.g. a manipulation that reveals the hypothesis), but they require larger samples and balanced allocation.
• Counterbalancing handles order effects in within-subject designs. With two conditions, alternate AB and BA; with more, use a Latin square. Randomise trial order inside blocks with jsPsych's randomize_order: true.
• Sample size should come from a power analysis, not convention. For a within-subject comparison with a medium effect (Cohen's d = 0.5), 80 % power at α = 0.05 needs about 34 participants; the same effect between subjects needs about 64 per group. Use G*Power or the pwr package in R, and base the expected effect on prior studies or a pilot, remembering that published effects are usually inflated.

2. Recruitment

Traditional recruitment means a departmental participant pool, usually run through SONA Systems, where students take part for course credit. Online, the main options are Prolific, a panel built for research with pre-screening and a requirement to pay at least a minimum hourly rate, and Amazon Mechanical Turk, which is larger and cheaper but has a higher share of inattentive and automated responders. All three pass a participant identifier in the URL and expect the participant to be redirected back to a completion page at the end. The details, including how to capture PROLIFIC_PID and issue completion codes without losing data, are in our guide to integrating jsPsych with Prolific, SONA and MTurk.

3. Data quality online

You are not in the room, so quality control has to be built into the task and the analysis. Decide the rules before collecting data and state them in your preregistration.

• Attention checks. Insert occasional trials with an unambiguous correct answer ("Press J on this trial") or a questionnaire item with an instructed response. Exclude participants who fail more than one.
• Reaction-time exclusion. Remove trials faster than about 150–200 ms (anticipations) and slower than a task-appropriate ceiling, and exclude participants whose overall accuracy is near chance or whose median RT is far outside the group.
• Bots and duplicate submissions. Require fullscreen with jsPsychFullscreen, check that total duration is plausible, look for identical response sequences, and use single-use links so a participant cannot run the task twice.
• Environment. Ask participants to use a laptop or desktop with a physical keyboard, close other tabs, and (for audio) wear headphones. Record navigator.userAgent and window size so you can exclude phones afterwards.

4. Ethics

Online studies need the same ethics approval as lab studies. Consent has to be given on screen, with a clear option to decline and to withdraw at any point by closing the tab. Tell participants what is recorded. Under the GDPR, participant IDs from recruitment platforms are personal data, so decide where data are stored and for how long, and avoid collecting anything you do not need. Pay at least the recruitment platform's minimum rate, and provide a debriefing page, especially if any part of the task was not fully explained in advance.

Laboratory vs. online at a glance

5. The full pipeline with jsPsych and Cognition

1. Write the task. Build the experiment in jsPsych 8. A minimal structure is a jsPsychInstructions trial, a jsPsychFullscreen trial, a practice block, the main block built from timeline_variables, and a debriefing. Tag every trial with data: { task: "main", condition: … } so filtering is easy later.
2. Paste it into the editor. Cognition's web editor has a live preview and an error console. Select the jsPsych version (8.2.3 is the latest; 6.x and 7.x are also supported, and a free migration tool converts older code). Upload images or audio as stimuli.
3. Configure the study. Add the consent text in markdown with a redirect URL for those who decline. Set the number of between-subject conditions; the server balances allocation and your code reads window.CONDITION. Decide when recruitment stops: immediately, on a date, or after N participants.
4. Share the link. Every task gets a https://<token>.cognition.run URL. Paste it into Prolific or SONA with their ID placeholders; query parameters are added as columns to every trial. For tighter control, generate single-use links in batches of up to 500.
5. Collect data. Trials are sent to the server as they finish. If a participant loses their connection, a service worker stores the data locally and resends it; at the end, everything is uploaded before your on_finish runs, so the completion redirect never outruns the data.
6. Download and analyse. Export CSV or JSON (one file, or a ZIP with a file per participant) and apply your preregistered exclusions in R or Python. For a first look, the integrated analyzer runs t-tests and ANOVAs and plots condition means directly from the collected data.

A free account includes four tasks with sixty participants each, enough for a pilot or a class project. Create a free account, or see the documentation for the editor, consent and export options.