A protocol for checking AI answers about gambling legality

To check AI answers about gambling legality, ask a fixed set of neutral questions about named jurisdictions, run them on each assistant under recorded conditions, and score each answer against a dated source of truth such as the state regulator's list of licensed operators. The output is a rate, for example "correct on 14 of 20 US state questions", with every answer kept so the scoring can be audited.

Published studies so far, which we summarise in our evidence review of AI search and gambling, have focused on Europe and on risky prompts about bonuses and self-exclusion. This protocol fills a narrower gap: the everyday factual questions people ask about North American and UK markets, where the right answer depends entirely on location.

Why accuracy, rather than visibility

Most AI tracking in marketing asks whether a brand is mentioned. That is a commercial question, and it has its place. Our question comes first logically. If an assistant says online casino is legal "in the US", or lists a provincial lottery site as Ontario's only option, then every brand appearing in that answer sits inside a false frame. Accuracy also matters to regulators: the KVA study in the Netherlands found three of ten tools presenting unlicensed brands as legal.

Step 1: build the question bank

Questions are neutral, short and tied to one jurisdiction. They are written as people ask them, not as keywords. We use five types.

We never test prompts that seek to avoid age checks, geolocation or self-exclusion. Those prompts are covered by the European studies, and repeating them adds risk without new insight.

Step 2: fix the ground truth before testing

For each question we record the correct answer and its source before any assistant is asked. For US states that means the state regulator and our US legal states tracker; for Canada, provincial regulators and our guide to online gambling by province; for the UK, the Gambling Commission. Fixing the answer first stops the scorer from being nudged by a confident-sounding reply.

Step 3: run under recorded conditions

Answers vary with location, account history, model and time, so every run records the platform, the product or mode (Google's AI Overviews and AI Mode are different products), the model where shown, the date, the approximate location and whether the session was logged in. The protocol runs each question at least twice per round and keep the full text of every answer, including any cited links.

Step 4: score each answer

Each answer receives one of five labels: correct; correct but incomplete (for example, right on sports betting but silent on casino); outdated (true at some earlier date); wrong; or declined. Separately, we note whether the answer named any operator that is not licensed in that jurisdiction, and whether it cited a regulator. Where possible, a second reviewer should score a sample independently, and any disagreement is resolved against the source.

Step 5: report rates and examples

Results are reported per platform and per question type as rates over the full bank, with the round date. We quote individual answers only as examples and always with their full context. We do not produce an overall "AI accuracy score", because a single number hides whether the errors were about age limits or about naming unlicensed operators, which matter very differently.

Limits of the method

The question bank is a sample, not every question people ask. Location and personalisation cannot be fully controlled, and models change without notice, so a round describes a moment rather than a permanent state. Ground truth can also move: when a launch date slips, as Maine's online casino launch has, answers that were wrong can become right and vice versa. That is why each round is dated and why we re-check the trackers before every run.

Anyone can use this protocol. If you run it for your own markets, the most useful single habit is to write the correct answer down, with its source, before you ask the first question.