A lead researcher explaining the benefits of Cognition.run

Data Quality in Online Experiments: Attention Checks, Reaction Time Exclusions and Bot Detection in jsPsych

Online data collection is fast and cheap, and that is exactly why it attracts participants you do not want: bots, response farms, people running several studies at once, and lately, AI agents that can drive a browser and click through a survey. None of this makes online experiments unusable. It means that data quality has to be designed into the task, not hoped for afterwards.

This article collects practical recipes in jsPsych 8 (the syntax also works in jsPsych 7) and explains which metadata Cognition already records for every run. To be clear about scope: Cognition does not run attention checks, VPN detection or fingerprinting for you. What it does is host your task, store every trial as it happens, and attach per-run columns that make the checks below easy to apply when you analyze the data.

Why this matters more in 2025–26

The problem is not new. Chmielewski and Kucker (2020) documented a marked drop in data quality on Amazon Mechanical Turk during 2018, with more failed checks and weaker psychometric properties in the same scales that had behaved well before. Peer, Rothschild, Gordon, Evernden and Damer (2022, Behavior Research Methods) compared Prolific, MTurk, CloudResearch, Qualtrics and Dynata on attention, comprehension, honesty and reliability; platforms differed considerably, and in their data Prolific and CloudResearch came out ahead of the others on most quality measures.

What is new is automation. Agley and colleagues (2025) describe "computer-use" AI agents as an emerging threat: software that can read a screen, move a mouse and answer questions in a way that looks human in the resulting data. Recruitment platforms are reacting; Prolific, for example, introduced additional authenticity checks for participants in 2025. Those checks happen before the participant reaches your task, though. Once the link is opened, the quality of the data is your responsibility.

Principle: decide the exclusion rules before you collect data

Every check described below produces a number, and every number invites a threshold. If you choose the threshold after looking at the results, you are giving yourself researcher degrees of freedom (Simmons, Nelson and Simonsohn, 2011) and the exclusions become part of the effect. Write down the rules, ideally in a preregistration: which attention checks a participant must pass, which RT cutoffs apply, what accuracy counts as chance, and whether a flagged participant is excluded entirely or only from specific analyses. Then implement those rules in code so they are applied identically to everyone.

Attention and instruction checks in jsPsych 8

Three kinds of checks cover most needs.

  1. Instructional manipulation check (Oppenheimer, Meyvis and Davidenko, 2009): a question whose instructions tell the participant which option to pick. It detects people who are not reading. With jsPsychSurveyMultiChoice or jsPsychHtmlButtonResponse it is a single trial.
  2. Comprehension check with repetition: after the instructions, ask one or two questions about the task. If the answer is wrong, show the instructions again. A loop_function does this.
  3. Catch trials inside the task itself: trials with an obvious answer (in a flanker task, for instance, a congruent trial with a large stimulus or a longer presentation time) tagged with data: { task: "catch" }. They are harder to game than a one-off question because they are spread across the session.

Here is a comprehension loop. The instructions and the check are repeated until the participant answers correctly, and the number of attempts is stored so you can use it as an exclusion criterion.

let attempts = 0;

const instructions = {
  type: jsPsychInstructions,
  pages: ["<p>Press F if the central arrow points left, J if it points right.</p>"],
  show_clickable_nav: true
};

const check = {
  type: jsPsychHtmlButtonResponse,
  stimulus: "<p>Which key do you press when the central arrow points right?</p>",
  choices: ["F", "J"],
  data: { task: "comprehension_check" },
  on_finish: (data) => {
    attempts++;
    data.correct = data.response === 1; // index of "J"
  }
};

const comprehension_loop = {
  timeline: [instructions, check],
  loop_function: (data) => {
    const last = data.values().slice(-1)[0];
    return !last.correct; // repeat while wrong
  }
};

timeline.push(comprehension_loop);

const imc = {
  type: jsPsychHtmlButtonResponse,
  stimulus: "<p>To show that you are reading, select "Strongly agree" below.</p>",
  choices: ["Strongly disagree", "Disagree", "Neutral", "Agree", "Strongly agree"],
  data: { task: "imc" },
  on_finish: (data) => { data.correct = data.response === 4; }
};
timeline.push(imc);

Exclusion criteria based on reaction time and accuracy

For speeded tasks, the response distribution itself tells you who was not doing the task.

  • Very fast responses (below roughly 150–200 ms) cannot be based on the stimulus and are usually anticipations or key mashing.
  • Very slow responses (more than about 3 SD above the participant's mean, or an absolute cap such as 10 s) usually mean the participant was distracted.
  • Accuracy at chance: with two response keys and 100 trials, a binomial test tells you that fewer than about 59 correct responses is not distinguishable from guessing at p < .05. Compute the threshold for your own design rather than copying a number.
  • Identical responses: the same key on every trial, or a strict alternation, is a sign of a script or a disengaged participant.

Ratcliff (1993) showed that the choice of outlier treatment changes the power and sometimes the sign of RT effects, which is one more reason to fix the rule in advance. The snippet below computes a simple low_quality flag in the global on_finish and writes it to every row with addProperties, so it shows up as a column in the data you download from Cognition.

const jsPsych = initJsPsych({
  on_finish: () => {
    const trials = jsPsych.data.get().filter({ task: "flanker" });
    const n = trials.count();
    const rts = trials.select("rt").values.filter(rt => rt !== null);

    const too_fast = rts.filter(rt => rt < 200).length / n;
    const too_slow = rts.filter(rt => rt > 10000).length / n;
    const accuracy = trials.filter({ correct: true }).count() / n;
    const catch_accuracy = jsPsych.data.get().filter({ task: "catch", correct: true }).count()
      / jsPsych.data.get().filter({ task: "catch" }).count();

    const responses = trials.select("response").values;
    const same_key = responses.every(r => r === responses[0]);

    const low_quality = too_fast > 0.10 || too_slow > 0.10 ||
      accuracy < 0.59 || catch_accuracy < 0.80 || same_key || attempts > 2;

    jsPsych.data.addProperties({
      prop_too_fast: too_fast.toFixed(3),
      prop_too_slow: too_slow.toFixed(3),
      accuracy: accuracy.toFixed(3),
      catch_accuracy: catch_accuracy.toFixed(3),
      comprehension_attempts: attempts,
      low_quality: low_quality
    });
  }
});

On Cognition, the trial data is uploaded before your on_finish runs, and the properties added here are included with the final upload, so the flag is available as a column without any extra work.

Environment signals: tab switches, fullscreen, screen size, device

A participant who leaves the tab in the middle of a block is probably doing something else. jsPsych already records focus and blur events, plus fullscreen enter and exit, in its interaction data: call jsPsych.data.getInteractionData() at the end and store a summary. You can also count visibility changes yourself with the Page Visibility API, which works even outside jsPsych trials.

let hidden_count = 0;
document.addEventListener("visibilitychange", () => {
  if (document.hidden) {
    hidden_count++;
    jsPsych.data.addProperties({ tab_hidden_count: hidden_count });
  }
});

timeline.push({ type: jsPsychFullscreen, fullscreen_mode: true });

// at the end, in on_finish:
const interactions = jsPsych.data.getInteractionData();
jsPsych.data.addProperties({
  blur_events: interactions.filter({ event: "blur" }).count(),
  fullscreen_exits: interactions.filter({ event: "fullscreenexit" }).count(),
  window_width: window.innerWidth,
  window_height: window.innerHeight,
  touch_device: navigator.maxTouchPoints > 0
});

The touch_device flag matters if your task requires a keyboard: a participant on a phone will either fail or respond with a virtual keyboard, and neither gives you the RTs you expect. If the task needs a minimum window size, check it in the first trial and ask the participant to resize or to come back on a computer.

Duplicate participants and bots

The same person completing your task twice, or completing two of your related tasks, is a quieter problem than bots, but it inflates your sample and violates independence. Cognition adds every URL query parameter as a column on all trials, so if you recruit through Prolific, SONA or MTurk the PROLIFIC_PID, SONA ID or workerId travels with the data (see the integration guide). Cross-checking those IDs across tasks is a one-line join once you have the CSVs.

For the rest, a few facts about what Cognition records and filters:

  • Requests whose user agent identifies a robot or crawler (including link previews from Facebook or WhatsApp) do not create a run in production, and neither does a visit with ?demo=true. This keeps empty runs out of your table but does not catch a headless browser pretending to be Chrome.
  • Each run stores the user agent, the browser language (accept-language) and the referer, together with the query parameters. A batch of runs from an unusual browser language, or with no referer when you expect one from the recruitment platform, is worth a look.
  • Storing the participant's IP address is an opt-in setting per task, off by default. It can reveal clusters of runs from the same address, but an IP is personal data under the GDPR, so enable it only if your ethics approval and consent form cover it, and say so in your privacy notice.
  • Single-use links (up to 500 per batch) stop a link from being shared or reused, and "stop after N runs" closes recruitment once the planned sample is in, so a leaked link does not keep collecting.

Dropout is a data quality problem too

Participants who quit are not a random subset. Zhou and Fishbach (2016) showed that when dropout differs between conditions, for example because one condition is more tedious, the participants who remain are no longer comparable, and the resulting effect can be an artifact of who stayed. The remedy is to report attrition per condition and test whether it differs.

Cognition makes this possible because trials are uploaded while the task runs, not at the end. A run whose browser was closed is marked Dropped; one that was still open after an hour without finishing is marked Timeout. Both keep the partial data and the condition column. The download page has a "not finished" filter, so you can compute how far into the task each condition's participants got before leaving and include that in your report.

What to do with flagged data

Two reasonable options exist, and the preregistration should say which one you take. The first is to exclude flagged participants from all analyses, using the rules written before data collection. The second is to keep everyone in the main analysis and run a sensitivity analysis without the flagged participants; if the conclusions hold either way, the result is more convincing than one that depends on the exclusions. In both cases, report how many participants were flagged by each criterion. Exclusion rates are informative in themselves, and reviewers increasingly ask for them.

If you host your task on Cognition, all the flags above arrive as columns in the CSV or JSON, next to run_id, condition, the query parameters and the browser metadata. You can filter the download by finished or unfinished runs, and the built-in analyzer lets you compare conditions with t-tests and ANOVA once the data is cleaned. Create a free account to try it with your own task.