HSA flight opsReplay · no live inferenceLocal modelsn = 200 hand-labelled

A support agent that knows when to call no-go.

An AI pipeline for Hulu’s public Twitter support. Before any reply leaves, every escalation rule is polled. One no-go hands the case to a person, with the reason on record.

Built and measured as an engineering experiment: it classifies the issue, retrieves how Hulu resolved similar ones, drafts a grounded reply, and reports its own failures. Not a Hulu product.

Explore the system Inspect the evaluation Source on GitHub

MON 1 · golden #197

msg: "Hi I forgot to put my account on hold and was charged for this month. If I cancel my account can I possibly be refunded?"
intent: billing_charge · 1.00
evidence: 0.94 · 0.92 · 0.91
draft: "Hi! We're here to help. Please contact us here: https://t.co/6YdK7bvQN7 so we can look into your billing issue. 😊"
check: no fabricated claim
rule: sensitive_intent → hand to a person

A recorded run from the evaluation, replayed. No model is running on this page.

From customer message to accountable decision.

Six stages, each returning its inputs and signals alongside its output, so any reply can be audited after the fact. Select a stage to see what goes in, what comes out, and the code that does it.

Classify

In: Clean message plus the 10 written intent definitions.
Does: Qwen2.5-7B-Instruct (local, temperature 0) picks one intent and states a confidence, as JSON.
Out: Intent label and a self-reported confidence.

Why it matters

Intent decides whether the case is sensitive. “other” is a deliberate escape hatch: forcing a substantive label turns “I don't know” into a confident wrong answer.

Taxonomy came from TF-IDF + KMeans over 14,080 conversation openers, then collapsed by hand. One raw cluster held 51% of messages; another was just “Rick and Morty”.
If the JSON fails to parse or names an unknown intent, the label falls back to “other”.
Self-reported confidence is not a probability. On the golden set the model only ever said 0.8, 0.9 or 1.0.

Read the code

Two signals in this pipeline look like safety mechanisms and mostly aren’t. Self-reported confidence predicts a correct intent barely better than chance (AUC 0.586), and evidence similarity barely does either (0.556). The policy uses them only after the sensitive-intent rule, never alone.

Follow one message through the machine.

Every case below is a real run from the evaluation, replayed from reports/predictions.csv: the recorded intent, the retrieved exchanges, the draft and the decision, set against the human label. Step through it one stage at a time.

A disputed monthly charge golden #197

Recorded run · replayed · no live model inference

A · Incoming message

Hi I forgot to put my account on hold and was charged for this month. If I cancel my account can I possibly be refunded?

Sampled from the escalation_sensitive stratum · conversation opener

Policy lab

Deterministic rules only · runs in your browser · no live model inference

Type a message and the escalation policy runs on it, ported line for line from src/taxonomy.py and src/agent.py. The LLM classifier and embedding search can’t run here, so intent comes from the keyword-rules baseline (which you can override), and you set the similarity and confidence yourself. Nothing you type leaves this page.

Start from a scenario

Scenarios are illustrative, not recorded runs.

Customer messageIntent (keyword rules matched “game”)

Best evidence similarity: 0.88 (threshold 0.80)

Self-reported confidence: 0.90 (threshold 0.75)

Draft parsed from the model outputDraft reply (type one to test the fabricated-claim regex)Regex matched “I've refunded”

Decision

Sensitive intentintent ∈ {billing_charge, account_access}GO (did not fire)
Unclassifiable messageintent == otherGO (did not fire)
Insufficient evidencetop-1 similarity < 0.80GO (did not fire)
Low self-reported confidenceself-reported confidence < 0.75GO (did not fire)
No usable draftdraft failed to parseGO (did not fire)
Fabricated claim in draftdraft matches FORBIDDEN_CLAIMSNO-GO (fired)

Hand to a person

Draft contained an unsupported claim: 'I've refunded'

A result is only as good as the evidence behind it.

200 messages, each labelled by hand by one person who never saw a model suggestion while labelling. Three baselines. Bootstrap intervals from 2,000 resamples. Every number here is from the evaluation report.

ADoes it classify better than something simple?

Intent macro-F1, with bootstrap 95% intervals

Agent (Qwen2.5-7B)

Prompted with the 10 intent definitions and asked to choose “other” when nothing fits. Not fine-tuned.

macro-F1: 0.606
95% CI: 0.513–0.673
accuracy: 63.0%

BWhen does it hand off, and when should it have?

Two ways to be wrong, on 72 true escalations among 200

Missed escalation: should have gone to a person, was auto-handledNeedless escalation: a person was pulled in without need

Always escalate

100%

Sensitive intents only

65%

Agent policy

46%

25%

Escalation metrics by system
System	Recall	Precision	Missed	Needless
Always escalate	1	0.36	0	1
Sensitive intents only	0.35	0.93	0.65	0.02
Agent policy	0.54	0.55	0.46	0.25

Agent policy: recall 0.54, precision 0.55. Always escalating is safe and useless; escalating only on sensitive intents is precise and misses 65%. The agent sits between them and still misses nearly half.

Where it matters most: billing disputes

16 / 18 escalated, recall 0.89. Two billing_charge messages were missed (outlined).

Promising for the category the policy was built around. It does not carry over to the aggregate: most of the 33 misses are content, live-TV and device cases the rules don’t treat as sensitive.

Which rule decided, across all 200

Auto-handle129
Sensitive intent39
Insufficient evidence24
Unclassifiable message8
Low self-reported confidence0
No usable draft0
Fabricated claim in draft0

Low self-report never fired: the model only ever said 0.8 or higher, and the threshold is 0.75.

CDoes generating a reply add anything?

Generated reply minus copied reply, paired over 200 messages

Copy the nearest reply4.89

Grounded generation4.87

Constant apology3.58

Mean LLM-judge overall score, 1–5.

Generation − copy

The interval [−0.12, +0.09] straddles zero. On this evaluation, the LLM writing a new reply is not shown to beat pasting the most similar reply Hulu already wrote.

Scores come from Llama 3.1 8B judging Qwen’s output. That judge agrees with a human only moderately (ρ = 0.54) and is generous by +1.48 points on average, more so on weak replies, which squeezes gaps together. A “no difference” result survives that bias; a reported win would not.

DDoes retrieval earn its place?

Retrieval ablation: each setting minus the shipped k=3

60-message subsample, same messages and judge. k=3 scores 4.883.

k=0 · No evidence (4.017)

k=1 · One exchange (4.983)

No evidence costs 0.87 points (CI [+0.67, +1.08] in k=3’s favour). The grounding is where the quality comes from.
One exchange beat three by 0.10 (CI [0.02, 0.18]). Not acted on: it was measured on the same examples as every other number, so switching to k=1 now would be tuning on the test set. It needs a full-sample rerun and a held-out check first.

EIntent by intent

Per intent. Select a row to see where it goes wrong.

Several classes have fewer than 20 examples; one or two messages move these numbers a lot.


17	0.87	0.76	0.81
18	0.67	0.89	0.76
42	0.71	0.81	0.76
7	1.00	0.57	0.73
30	0.81	0.43	0.57
25	0.49	0.68	0.57
25	0.63	0.48	0.55
16	0.47	0.50	0.48
13	0.43	0.46	0.44
7	0.38	0.43	0.40

Where live_tv_sports_issue went

Agent predictions for the 30 messages a human labelled live_tv_sports_issue. Counted from the recorded predictions.

live_tv_sports_issue13
playback_error9
content_availability6
device_app_issue1
other1

Blue: correct. Pink: misrouted.

Full confusion matrix (rows: human label)

Human label	playback	device	live TV	content	billing	account	feature	complaint	praise	other
playback	17	2	1	1			3			1
device	4	12		1	2	2		3		1
live TV	9	1	13	6						1
content			1	34	1		4	1		1
billing	1			1	16
account	1	1			2	13
feature		1		4	1		8	2
complaint	1	1		1	2		1	6		1
praise	1		1					1	4
other	1	1					1	1		3

Where the system breaks is part of the result.

74 of 200 intents are still wrong after correction. These are the patterns behind them, each with the recorded evidence and the next thing worth testing.

Live TV read as a playback error

What happened

Half of the true live-TV and sports messages were routed elsewhere: 15 of 30, nine to playback_error and six to content_availability. It is the largest single error cluster.

Observed evidence

live is so poor right now. Chopped playback, not holding video quality.

Human label live_tv_sports_issue · agent playback_error · auto-handled

live_tv_sports_issue: precision 0.81, recall 0.43. When the agent says live TV it’s usually right; it just rarely says it.

Why

The message carries a playback symptom (“chopped”, “quality”) and a live-context cue (“live”). The two classes overlap by construction, and the model weighs symptom vocabulary more heavily.

What to investigate next

State a precedence rule in the taxonomy (“if it's live, live wins”) or merge the classes, then re-measure. Fixing the boundary is a taxonomy decision before it is a model one.

The evaluation was wrong before the audit caught it.

A low score can mean a weak model, a broken evaluation, or both. Here it was the evaluation, and the only reason the current number can be trusted at all is that someone went back and read the labels.

Step 1 · first draw
0.487
Macro-F1 on the first 60 hand-labelled messages. Only one of them was a billing dispute.
Step 2 · stratified to 200
0.222
Invalid
The expanded set appeared to collapse the result. It looked like sobering news about the model.
Step 3 · audit
101 / 140 labels from a rushed second session didn’t match their message. One billing_charge label was on Are the new Christmas Movies going to be on?
Step 4 · corrected
0.606
Each label re-read against the written definitions, the same standard as the first 60. Both versions stay in git history.

These are not three comparable results. The first is a smaller sample, the second measured labelling noise, and only the fourth is a measurement of the system. Without the audit, 0.222 would have shipped as a false finding. Full diagnosis in the build log and decision log.

The other checks, and what they don’t cover

One annotator, no model in the loop

All 200 labels, and every correction, were made by one person reading the message against the written intent definitions. No model prediction was shown during labelling. That blinds the labels to the model; it doesn't measure how far a second annotator would disagree.

A retrieval leak, found and closed

The index first contained the evaluation messages themselves, so a test message retrieved its own historical reply at similarity 1.0000, and the copy-nearest baseline was quietly emitting ground truth. Every golden example and its whole conversation thread (233 rows) is now excluded, and two tests fail if it regresses.

The judge was tested before it was trusted

Llama 3.1 judges Qwen's replies, a different model family on purpose. On constructed degradations it ranks a real reply above a stripped one, a generic apology and an invented refund (ρ -0.829 with the known order; 75% pairwise). Against 21 blind human ratings it agrees only moderately, and its relevance dimension barely discriminates: it gave a reply with an invented refund 4.68 out of 5.

Thresholds come from reference data, not the test set

The 0.80 evidence threshold is the 10th percentile of top-1 similarity on 1,500 reference-corpus probes. An earlier version read it off the golden set, which would have tuned the policy on the examples used to report it.

Zero detected fabrications is not zero fabrications

The fabricated-claim check is a regex over specific phrasings. It fired on none of the 200 drafts. A phrasing it doesn't list, such as “a credit will appear on your next bill”, would pass straight through. Try one in the policy lab above.

Judge score vs. a blind human rating, 21 replies

Dashed line: perfect agreement. 18 of 21 replies sit above it. Spearman ρ = 0.54, exact agreement 14%, mean bias +1.48. Numbers in circles count overlapping replies.

Small enough to inspect. Serious enough to evaluate.

Plain Python, about twenty files, no agent or RAG framework. All three models run locally through Ollama, so a full run has no marginal cost, and every call is cached on disk by a hash of its exact input.

Generator

Qwen2.5-7B-Instruct

qwen2.5:7b-instruct-q4_K_M

Classifies and drafts.

Judge

Llama 3.1 8B Instruct

llama3.1:8b-instruct-q4_K_M

A different model family, so it isn't grading its own style.

Embeddings

nomic-embed-text

For retrieval.

Build the ground

Offline, once. Turns 2.8M tweets into an index, a taxonomy and a test set.

Answer a message

Online path. Every stage returns its signals for audit.

Measure it

The evaluation harness. This is where most of the work is.

Determinism

temperature=0, a fixed seed, and a disk cache keyed by SHA-256 of model, system prompt and prompt. Re-running replays the same answers.

Cost of a full run

About 1,000 LLM calls for the 200-example evaluation: one classify, one draft and three judge calls per message, plus a one-off ~21.5k-message embedding pass.

What was deliberately not built

Fine-tuning, multi-turn dialogue, an agent framework, and any production safety layer beyond the policy and the regex.

Inspect the evidence. Reproduce the result.

The numbers on this page come from the repository, not from this website. This site runs no Python and no model; to check the results, run them yourself.

Replay the reported results

git clone https://github.com/hey-shiv/hulu-support-agent.git
cd hulu-support-agent
python3.12 -m venv .venv
./.venv/bin/pip install -r requirements.txt
make reproduce

Rebuild from scratch (optional)

ollama pull qwen2.5:7b-instruct-q4_K_M
ollama pull llama3.1:8b-instruct-q4_K_M
ollama pull nomic-embed-text
# put twcs.csv from Kaggle in data/raw/
make all

make reproduceabout 14 s here; under 20 s per the README: Replays every cached model call and prints the metrics table, calibration and judge validity. It should print agent macro-F1 0.606; anything else is a bug.; Needs: Python 3.12. No GPU, no API key, no network, no Ollama.
make allabout 30 min: Rebuilds everything from raw tweets: conversations, brand choice, clusters, index, silver labels, evaluation, metrics.; Needs: Ollama with the three models, and the raw Kaggle CSV (493 MB).
make testseconds: Runs the test suite (34 tests), including the regression tests that fail if golden examples leak back into the retrieval index.; Needs: The same environment.

Repository Evaluation report Kaggle dataset

What remains unsolved.

A research prototype with known gaps. It is not fit to answer customers unsupervised.

Single messages only: No conversation history; mid-thread fragments are mostly misread.
One annotator, 200 examples: No measured inter-annotator agreement, and the annotator also built the system.
2017 public Twitter data: Hulu's product and support playbook have changed since; only public conversations are visible.
Historical replies aren't proven fixes: Retrieval assumes past replies were good resolutions. Nobody checked that they resolved anything.
Weak uncertainty signals: Self-report AUC 0.586; similarity barely better. Escalation recall is 54%.
A generous, moderately agreeing judge: ρ 0.54 against a human on 21 ratings, +1.48 bias, and a relevance score that barely discriminates.
An unconfirmed ablation: k=1 beat k=3 on a 60-message subsample, measured on the test set.
A regex guardrail: It catches the phrasings it lists and nothing else.

Proposed next experiments

Not yet done.

1Re-run the retrieval ablation on all 200, then test k=1 vs. k=3 on a genuinely held-out set before changing anything.
2Double-label 50 examples blind, a day apart, for an agreement ceiling.
3Widen the blind human ratings past 21 to tighten the judge-bias estimate.
4Add a precedence rule for live TV vs. playback, the largest error cluster.
5Replace the judge's relevance dimension.
6Pass prior turns for mid-thread messages.
7Sweep the evidence threshold against an explicitly stated cost ratio.

Already fixed during the project

Implemented and reflected in every number on this page.

Golden examples and their threads removed from the retrieval index, with regression tests.
101 mislabelled examples audited and corrected; both versions kept in git.
Evidence threshold re-derived from reference data instead of the golden set.
Unparseable drafts now escalate instead of auto-sending an empty reply.
21 blind human ratings collected to test the judge.