Evaluation before claims.

What it does

VidWise is retrieval over video transcripts with a point of view: measure first, claim later. It chunks long transcripts, retrieves the relevant slices for a question, and answers with citations that have to point back to a real span. If the video does not contain the answer, it is supposed to refuse, not guess.

The part I cared most about is the eval harness. Not RAGAS, not a hosted suite. A small Python harness that defines three metrics, computes confidence intervals, and refuses to commit a number unless a versioned run reproduces it.

• Chunked transcript retrieval instead of stuffing the whole video into a prompt
• Recall at K with Wilson confidence intervals
• Citation must ground to a transcript span or the answer fails
• Negative queries that test abstention, not just accuracy
• Versioned run artifacts so every metric is reproducible

Limitation: chunking is heuristic. Long, noisy transcripts with overlapping speakers still hurt recall.

How it works

The pipeline is plain: transcribe, chunk, embed, retrieve, generate with citations. The interesting code lives in evaluation, where a single run is never treated as a claim.

# one run is not a claim, report interval and sample size
import math

def wilson_ci(hits, total, z=1.96):
    if total == 0:
        return (0.0, 0.0)
    p = hits / total
    denom = 1 + z*z/total
    centre = p + z*z/(2*total)
    delta = z * math.sqrt(p*(1-p)/total + z*z/(4*total*total))
    return ((centre - delta)/denom, (centre + delta)/denom)

recall = hits / total
lo, hi = wilson_ci(hits, total)
# only log as committed if hi - lo is tight and run is versioned

# citation is valid only if it matches a span in the transcript
def citation_grounds(answer, transcript):
    for cite in answer.citations:
        span = transcript[cite.start:cite.end]
        if cite.text.strip() not in span:
            return False  # fail the example
    return True

# if the video does not contain the answer, abstain
def should_abstain(query, video_id, index):
    if query.label == "negative":
        hits = index.retrieve(query.text, k=8)
        if not hits:
            return True
        # even with hits, model must say it does not know
        return answer.text.lower().startswith("i do not")
    return False

What I learned

Writing the harness before chasing scores kept the work honest. When every number needs a Wilson interval and a run artifact, you stop polishing anecdotes and start fixing retrieval.