> For the complete documentation index, see [llms.txt](https://learning.contextqa.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learning.contextqa.com/planning-and-decision-guides/conference-demo-to-evaluation.md).

# Turn a Conference Demo into an Evaluation

Capture a conference or vendor claim, distinguish slides from observed evidence, and turn one relevant idea into a bounded post-event experiment.

> **Editorial owner:** ContextQA Documentation · **Last updated:** September 12, 2026

## Quick answer

Leave the event with one testable claim, its source and date, the artifact actually observed, and an experiment owner. Do not report a speaker’s claim as your team’s result or treat a polished demo as proof that the workflow fits your application.

## Ask five evidence questions

1. What exact problem, application type, and baseline does this claim describe?
2. Which parts were shown live, prerecorded, documented, or only asserted?
3. What human setup, review, repair, and infrastructure work remains?
4. What happens on a real product failure, timeout, missing permission, or unavailable dependency?
5. Which artifact would let our team reproduce or falsify the claim?

## Keep a claim log

| Source/session                  | Claim            | Evidence observed                        | Constraint or unknown            | Relevance       |
| ------------------------------- | ---------------- | ---------------------------------------- | -------------------------------- | --------------- |
| Title, speaker, organizer, date | Exact paraphrase | Live result, code, trace, slide, or none | Data, scale, environment, review | High/medium/low |

Use a short quotation only when necessary and permitted; otherwise paraphrase accurately and link the official session or approved material.

## Distinguish evidence strength

* **Observed artifact** — you inspected a live result, code, trace, or reproducible fixture.
* **Documented behavior** — current official documentation supports the claim.
* **Speaker/vendor claim** — useful hypothesis, not independently verified.
* **Your inference** — clearly labeled interpretation.
* **Unknown** — missing evidence or context.

## Design one bounded experiment

```
Claim to test:
Why it matters to our workflow:
Current baseline:
Representative fixture:
Normal and seeded-failure cases:
Required data, permissions, and integrations:
Evidence to retain:
Success, revise, and stop criteria:
Owner:
Deadline:
```

Prefer a test that can fail meaningfully. If the claim concerns test generation, seed an application defect or weaken a prerequisite and verify the resulting assertion. If it concerns triage, compare diagnosis time and evidence completeness on the same failure.

## Compare demos fairly

When several vendors address the same question, use the [AI Testing Vendor Demo Scorecard](/planning-and-decision-guides/ai-testing-vendor-demo-scorecard.md). Supply the same task and avoid awarding full credit to a presentation when another vendor ran the workflow live.

## Run the post-event debrief

Within a week:

1. remove claims unrelated to a current team decision;
2. confirm official sources and access to artifacts;
3. select no more than one owned experiment per decision owner;
4. schedule the experiment and review;
5. update the relevant evergreen guide only if verified behavior changes; and
6. close unowned ideas instead of keeping an indefinite trend backlog.

## Frequently asked questions

### Does a live conference demo count as proof for our environment?

No. It is stronger than a slide, but it still demonstrates the presenter’s prepared environment. Treat it as a useful hypothesis until the workflow runs with your application, permissions, data, failure modes, and evidence requirements.

### How many post-event experiments should we run?

Select no more than one active experiment per decision owner. Prioritize the claim with the clearest connection to a current operating problem and close the rest unless capacity and a defined decision exist.

### What if we cannot reproduce the conference claim?

Record the setup, versions, fixture, evidence, and unresolved difference. Classify the claim as not reproduced rather than false unless the test actually contradicts it, then ask the source or vendor one bounded clarification question.

## Related documentation

* [Choose a QA Conference](/planning-and-decision-guides/choose-a-qa-conference.md)
* [AI Testing Pilot Scorecard](/evaluation-and-governance/ai-testing-pilot-scorecard.md)
* [Prove an AI-Generated Test Can Fail](/ai-features/ai-test-generation.md#prove-a-generated-test-can-fail)
* [Planning and Decision Guides](/planning-and-decision-guides/decision-guides.md)

{% hint style="info" %}
[**Test one conference claim with ContextQA →**](https://contextqa.com/book-a-demo/?utm_source=learning.contextqa.com\&utm_medium=referral\&utm_campaign=docs_demand_gen\&utm_content=conference_demo_evaluation) using the claim log, representative workflow, and acceptance criteria from this guide.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://learning.contextqa.com/planning-and-decision-guides/conference-demo-to-evaluation.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.