📊 Status Dashboard ↑ all runs

util-corpus-a-324f4ab3-c63-20260802

util-corpus-a · preserved partial run · iOS sim
This is the preserved partial result for this run. Its original utterance evidence is shown below.
Rows
16
Pass
12 (75%)
Fail
3 (19%)
Unverified
1 (6%)
Pass rate
80%
Avg difficulty
—
"Unverified" = the grader couldn't confidently call it PASS or FAIL from the trace — needs a human look (that's you 👍/👎-ing it). "Pass rate" = pass ÷ (pass + fail) — it excludes Unverified rows, so it differs slightly from the Pass % above (which is share of ALL rows). Unverified breaks down as: 1 unclassified — the "grader coverage gap" ones aren't ambiguous, the grader just doesn't fully score that intent type yet.

Why the fails happened — comprehension vs execution vs cosmetic

Comprehension — picked the wrong action/target (the hard problem)
3 (100%)
Of 3 fails: if most are comprehension, that's the hard problem (the app didn't figure out the right action/target); execution/cosmetic means the decision was right but something downstream broke. Click a bar for the sub-split + example utterances.

Handled correctly? — by expected action

Click a row to see the ones it got wrong; click a wrong utterance to jump to its full detail below.
Supposed toNCorrectWrongUnverified
▸ LOG — log the entry 77 (100%) 0 (0%) 0 (0%)
▸ UPDATE — update the entry 53 (60%) 2 (40%) 0 (0%)
▸ INTERNAL_QUERY — answer from app data 11 (100%) 0 (0%) 0 (0%)
▸ OTHER — handle appropriately 10 (0%) 0 (0%) 1 (100%)
▸ CLARIFY — ask a clarifying question 10 (0%) 1 (100%) 0 (0%)
▸ CLARIFY_ANSWER — complete the pending action 11 (100%) 0 (0%) 0 (0%)
Total1612 (80%)31

Accuracy by difficulty

Pending A1's per-utterance difficulty score (requested 2026-07-05) — this bar chart lights up once that lands.

Clarification follow-ups — scored separately

Second turn: app asked, we replied — did it resolve correctly?
Follow-ups
1
Resolved
1 (100%)
Mishandled
0 (0%)
Unverified
0

Cosmetic only

Not yet classified — pending A1 adding cosmetic-issue detection (double response, wording variance, rounding) to the grader. Nothing fabricated here.

System / infra

Not yet classified — pending confirmation from A1 whether trace data already carries infra-error signal (TTS/sync/DB-write errors) or needs new detection.

Latency

Avg (time to ready)
0.3s
p90
0.4s
Max
0.9s
Went async
0 (0%)
Each dot is one utterance (green=pass, red=fail, amber=unverified), positioned by how long it took to be ready for review. Bigger dots are the slow outliers — click any dot to jump to its detail.
0s
1s
2s
5s
Response path — quick (single response) vs async (an ack like "Working on it…" before the real answer).
Quick response
15
Sync clarification
1
Slowest 8 utterances (click to jump to detail):
"Append also grab oat milk to my latest note."0.9s
"I want to take a new note in the category Travel."0.7s
"Remember to bring the charger next time."0.4s
"Add a to-do to call the dentist."0.3s
"Make sure I text Sarah back and also add a to-do to buy milk."0.3s
"Take a note in the category Recipes that says add cardamom to the coffee cake."0.3s
"Add a task to renew my passport in the category Admin."0.3s
"Add to the note about the wifi password: it changes every month."0.2s

Filter — controls the list below

Pass / Fail / Unverified
PASS 12 FAIL 3 UNVERIFIED 1
Module (intended for)
todos (9)notes (7)
Utterance sub-type (within module)
16 shown — 12 pass, 3 fail, 1 unverified

Per-utterance detail

PASSshould log the entry"Add a to-do to call the dentist." (difficulty —)0.3s
Verdict Expected LOG — should log the entry. PASS: Notes write-truth verified: call the dentist.
Why verdict Notes write-truth verified: call the dentist.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.3s
1 · TTS said Added to-do: call the dentist. Priority 3.
2 · Card shown Added to-do: call the dentist. Priority 3.
3 · App data rows written created todo_item: call the dentist
4 · UI did navigated to "todos" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:15:26.381Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould log the entry"Remind me to pick up dry cleaning tomorrow." (difficulty —)0.2s
Verdict Expected LOG — should log the entry. PASS: Notes write-truth verified: pick up dry cleaning.
Why verdict Notes write-truth verified: pick up dry cleaning.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.2s
1 · TTS said Added to-do: pick up dry cleaning. Priority 3, due Aug 3. Tell me if the title or due date should be different.
2 · Card shown Added to-do: pick up dry cleaning. Priority 3, due Aug 3. Tell me if the title or due date should be different.
3 · App data rows written created todo_item: pick up dry cleaning
4 · UI did navigated to "todos" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:15:37.714Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould log the entry"Add a task to renew my passport in the category Admin." (difficulty —)0.3s
Verdict Expected LOG — should log the entry. PASS: Notes write-truth verified: renew my passport.
Why verdict Notes write-truth verified: renew my passport.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.3s
1 · TTS said Added to-do: renew my passport. Priority 3.
2 · Card shown Added to-do: renew my passport. Priority 3.
3 · App data rows written created todo_item: renew my passport
4 · UI did navigated to "todos" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:15:49.181Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould log the entry"Make sure I text Sarah back and also add a to-do to buy milk." (difficulty —)0.3s
Verdict Expected LOG — should log the entry. PASS: Notes write-truth verified: buy milk, text Sarah back.
Why verdict Notes write-truth verified: buy milk, text Sarah back.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.3s
1 · TTS said Added 2 to-dos: text Sarah back; buy milk.
2 · Card shown Added 2 to-dos: text Sarah back; buy milk.
3 · App data rows written created todo_item: text Sarah back
created todo_item: buy milk
4 · UI did navigated to "todos" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:16:00.655Z
6 · Why this food
  • The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • Matched the common-food list to "Milk" at 90% confidence.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould update the entry"Mark call the dentist as done." (difficulty —)0.1s
Verdict Expected UPDATE — should update the entry. PASS: Notes UPDATE write-truth verified (changed:todo_items×1).
Why verdict Notes UPDATE write-truth verified (changed:todo_items×1).
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.1s
1 · TTS said Marked done: call the dentist.
2 · Card shown Marked done: call the dentist.
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "todos" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:16:11.901Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould update the entry"I finished the buy milk task." (difficulty —)0.2s
Verdict Expected UPDATE — should update the entry. PASS: Notes UPDATE write-truth verified (changed:todo_items×1).
Why verdict Notes UPDATE write-truth verified (changed:todo_items×1).
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.2s
1 · TTS said Marked done: buy milk.
2 · Card shown Marked done: buy milk.
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "todos" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:16:23.285Z
6 · Why this food
  • The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • Matched the common-food list to "Milk" at 90% confidence.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
FAILshould update the entry"Reopen the dry cleaning to-do." (difficulty —)0.0s
Verdict Expected UPDATE — should update the entry. FAIL: OVER-ASK — asked instead of update (no notes-domain mutation).
Why verdict OVER-ASK — asked instead of update (no notes-domain mutation).
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.0s
1 · TTS said I could not find a to-do matching dry cleaning. Did you mean: buy milk; call the dentist?
2 · Card shown I could not find a to-do matching dry cleaning. Did you mean: buy milk; call the dentist?
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "todos" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:16:34.390Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould answer from app data"What are my most important to-dos?" (difficulty —)0.0s
Verdict Expected INTERNAL_QUERY — should answer from app data. PASS: Answered.
Why verdict Answered.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.0s
1 · TTS said Most important to-dos: pick up dry cleaning (P3, due Aug 3); renew my passport (P3); text Sarah back (P3).
2 · Card shown Most important to-dos: pick up dry cleaning (P3, due Aug 3); renew my passport (P3); text Sarah back (P3).
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "todos" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:16:45.594Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
UNVERIFIEDunclassifiedshould handle appropriately"Delete the dentist to-do." (difficulty —)0.1s
Verdict Expected OTHER — should handle appropriately. UNVERIFIED: OTHER unknown subtype — no mechanical checks defined (key: )
Why verdict OTHER unknown subtype — no mechanical checks defined (key: )
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.1s
1 · TTS said I didn't find any food entries logged yet today to correct. Could you tell me which entry you meant?
2 · Card shown I didn't find any food entries logged yet today to correct. Could you tell me which entry you meant?
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did no screen events captured
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:16:56.737Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould log the entry"Create a new note that says the wifi password is sunflower22." (difficulty —)0.2s
Verdict Expected LOG — should log the entry. PASS: Notes write-truth verified: says the wifi password is sunflower22.
Why verdict Notes write-truth verified: says the wifi password is sunflower22.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.2s
1 · TTS said Logged your note in General about says the wifi password is sunflower22.
2 · Card shown Logged your note in General about says the wifi password is sunflower22.
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "notes" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:17:08.137Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould log the entry"Take a note in the category Recipes that says add cardamom to the coffee cake." (difficulty —)0.3s
Verdict Expected LOG — should log the entry. PASS: Notes write-truth verified: says add cardamom to the coffee cake, Recipes.
Why verdict Notes write-truth verified: says add cardamom to the coffee cake, Recipes.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.3s
1 · TTS said Logged your note in Recipes about says add cardamom to the coffee cake.
2 · Card shown Logged your note in Recipes about says add cardamom to the coffee cake.
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "notes" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:17:19.581Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould log the entry"Remember to bring the charger next time." (difficulty —)0.4s
Verdict Expected LOG — should log the entry. PASS: Notes write-truth verified: bring the charger next time.
Why verdict Notes write-truth verified: bring the charger next time.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.4s
1 · TTS said Logged your note in General about bring the charger next time.
2 · Card shown Logged your note in General about bring the charger next time.
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "notes" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:17:31.166Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
FAILshould update the entry"Append also grab oat milk to my latest note." (difficulty —)0.9s
Verdict Expected UPDATE — should update the entry. FAIL: WRITE-TRUTH FAIL — UPDATE touched notes-domain rows but none matched expected content (Append also grab oat milk to my latest note.).
Why verdict WRITE-TRUTH FAIL — UPDATE touched notes-domain rows but none matched expected content (Append also grab oat milk to my latest note.).
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.9s
1 · TTS said Updated your note in General about bring the charger next time.
2 · Card shown Updated your note in General about bring the charger next time.
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "notes" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:17:43.497Z
6 · Why this food
  • The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • Matched the common-food list to "Milk" at 90% confidence.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould update the entry"Add to the note about the wifi password: it changes every month." (difficulty —)0.2s
Verdict Expected UPDATE — should update the entry. PASS: Notes UPDATE write-truth verified (changed:personal_notes×1).
Why verdict Notes UPDATE write-truth verified (changed:personal_notes×1).
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.2s
1 · TTS said Updated your note in General about says the wifi password is sunflower22.
2 · Card shown Updated your note in General about says the wifi password is sunflower22.
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "notes" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:17:54.546Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
FAILshould ask a clarifying question"I want to take a new note in the category Travel." (difficulty —)0.7s
Verdict Expected CLARIFY — should ask a clarifying question. FAIL: Logged a BLIND guess — no stated assumption, no correction invited.
Why verdict Logged a BLIND guess — no stated assumption, no correction invited.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.7s
1 · TTS said Logged your note in Travel. about I want to take a new note in the category....
2 · Card shown Logged your note in Travel. about I want to take a new note in the category....
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "notes" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:18:13.002Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)
PASSshould complete the pending action"The note should say pack the passport and adapter." (difficulty —)0.2s
Verdict Expected CLARIFY_ANSWER — should complete the pending action. PASS: Completed the pending action.
Why verdict Completed the pending action.
My override
Things I think are wrong
Use this when the report passed something that should have failed.
Comprehension
Execution
Cosmetic
Things the report marked wrong but I think are right
Use this when the report failed something that should have passed.
Actually right
Report/grader issue
Pick every reason you think applies. Multiple boxes are allowed because one utterance can be wrong in more than one way. Use the note below for "Other" or extra detail.
Timing 0.2s
1 · TTS said Logged your note in General about pack the passport and adapter.
2 · Card shown Logged your note in General about pack the passport and adapter.
3 · App data rows written No food/workout/list/note data row captured for this utterance.This does not mean "nothing anywhere was stored"; spoken/chat evidence is shown above, and some chat/session records may live outside this write-capture field. It means the report did not capture an app data-row write such as food_log_entries or workout_sets.
4 · UI did navigated to "notes" screen
5 · Seed/context data
Seeded starting state: none listed in the corpus row.
  • No DB snapshot captured for this utterance; the report cannot prove whether yesterday/personal data existed.
snapshot captured 2026-08-02T21:18:24.343Z
6 · Why this food The lookup compared against planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
7 · Lookup method
  • The report captured the app food lookup that ran before or alongside the final decision.
  • It searched planned foods, your saved foods and aliases, your past food logs, the shared nutrition database, the common-food list, the exercise list, and the workout equipment list.
  • The lookup evidence took 0.0 seconds to collect.
  • This lookup is supporting evidence in the report; by itself it does not prove that food was saved.
8 · Candidates → decider no candidate list captured (this path did not run DB resolution — e.g. provisional branded log)