Skip to content
Baumbart logo Baumbart Documentation

chatbot/__benchmarks__/types

This page is auto-generated from TSDoc output. Edit the source file, not this Markdown.

chatbot/__benchmarks__/types

Section titled “chatbot/__benchmarks__/types”
  • Package: @leipzigtreechat/chatbot
  • Source file: packages/chatbot/src/__benchmarks__/types.ts

Summary

Section titled “Summary”

No summary is available for this file.

Types

Section titled “Types”

BenchmarkCaseKind

Section titled “BenchmarkCaseKind”
type BenchmarkCaseKind = "question" | "dialogue"

Defined at: line 3

BenchmarkAxis

Section titled “BenchmarkAxis”
type BenchmarkAxis = "task_completion" | "grounding" | "flow_correctness" | "graceful_failure" | "presentation"

Defined at: line 4

BenchmarkStatus

Section titled “BenchmarkStatus”
type BenchmarkStatus = "pass" | "soft-fail" | "fail"

Defined at: line 5

BenchmarkSignalTarget

Section titled “BenchmarkSignalTarget”
type BenchmarkSignalTarget = "assistant_last" | "assistant_any" | "transcript" | "metadata"

Defined at: line 6

BenchmarkMatchMode

Section titled “BenchmarkMatchMode”
type BenchmarkMatchMode = "includes" | "regex"

Defined at: line 7

ClarificationExpectation

Section titled “ClarificationExpectation”
type ClarificationExpectation = "required" | "forbidden" | "allowed"

Defined at: line 8

BenchmarkSignal

Section titled “BenchmarkSignal”
type BenchmarkSignal = {
  label: string;
  value: string;
  axis: BenchmarkAxis;
  critical: boolean;
  target?: BenchmarkSignalTarget;
  mode?: BenchmarkMatchMode;
  flags?: string;
  caseSensitive?: boolean;
}

Defined at: line 10

BenchmarkExpectedFlow

Section titled “BenchmarkExpectedFlow”
type BenchmarkExpectedFlow = {
  clarification: ClarificationExpectation;
  requiredMetadataStatuses?: ChatBotMetadataStatus[];
  forbiddenMetadataStatuses?: ChatBotMetadataStatus[];
  minAssistantMessages?: number;
}

Defined at: line 21

BenchmarkFixture

Section titled “BenchmarkFixture”
type BenchmarkFixture = {
  id: string;
  title: string;
  kind: BenchmarkCaseKind;
  userTurns: string[];
  expectedFlow: BenchmarkExpectedFlow;
  requiredSignals: BenchmarkSignal[];
  forbiddenSignals: BenchmarkSignal[];
  notes: string;
}

Defined at: line 28

BenchmarkTranscriptEntry

Section titled “BenchmarkTranscriptEntry”
type BenchmarkTranscriptEntry = {
  role: "user" | "assistant";
  content: string;
  timestamp: string;
  turnIndex: number | null;
}

Defined at: line 39

BenchmarkMetadataEntry

Section titled “BenchmarkMetadataEntry”
type BenchmarkMetadataEntry = {
  status: ChatBotMetadataStatus;
  message: string;
  timestamp: string;
}

Defined at: line 46

BenchmarkTurnTiming

Section titled “BenchmarkTurnTiming”
type BenchmarkTurnTiming = {
  turnIndex: number;
  userPrompt: string;
  startedAt: string;
  firstAssistantResponseAt?: string;
  latencyMs?: number;
}

Defined at: line 52

BenchmarkCheckResult

Section titled “BenchmarkCheckResult”
type BenchmarkCheckResult = {
  label: string;
  axis: BenchmarkAxis;
  critical: boolean;
  passed: boolean;
  details: string;
}

Defined at: line 60

BenchmarkAxisResult

Section titled “BenchmarkAxisResult”
type BenchmarkAxisResult = {
  axis: BenchmarkAxis;
  passed: boolean;
  applicable: boolean;
  summary: string;
  checks: BenchmarkCheckResult[];
}

Defined at: line 68

BenchmarkEvaluation

Section titled “BenchmarkEvaluation”
type BenchmarkEvaluation = {
  status: BenchmarkStatus;
  score: number;
  failureReason?: string;
  axes: BenchmarkAxisResult[];
}

Defined at: line 76

BenchmarkCaseResult

Section titled “BenchmarkCaseResult”
type BenchmarkCaseResult = {
  fixture: BenchmarkFixture;
  transcript: BenchmarkTranscriptEntry[];
  metadata: BenchmarkMetadataEntry[];
  turnTimings: BenchmarkTurnTiming[];
  totalDurationMs: number;
  evaluation: BenchmarkEvaluation;
  runtimeError?: string;
}

Defined at: line 83

BenchmarkRunResult

Section titled “BenchmarkRunResult”
type BenchmarkRunResult = {
  startedAt: string;
  finishedAt: string;
  totalDurationMs: number;
  cases: BenchmarkCaseResult[];
}

Defined at: line 93

In chatbot

Section titled “In chatbot”