> ## Documentation Index > Fetch the complete documentation index at: https://docs.keywordsai.co/llms.txt > Use this file to discover all available pages before exploring further. # Observability data model > Understanding how Keywords AI organizes and presents your LLM data ## Overview Keywords AI's observability is built around one core concept: **different views of the same underlying data**. All views present the same log data, just organized differently for different use cases: **Plain view** - Individual LLM requests as they happen **Hierarchical view** - Multi-step workflows and AI agent operations **Conversational view** - Linear chat interface for dialogue systems **Evaluation view** - Quality assessments and performance metrics All four views show the **same underlying data** - they just organize and present it differently. The core data structure remains consistent across all views. ## Logs A log represents a single LLM request and contains all the information about that interaction. This is the foundational data that powers all other views. ### Core fields * start\_time — Request start time (RFC3339) * timestamp — Span end time (RFC3339) * prompt\_tokens — Tokens in the prompt/input * completion\_tokens — Tokens in the model output * prompt\_cache\_hit\_tokens — Tokens served from cache * prompt\_cache\_creation\_tokens — Tokens added to cache * total\_request\_tokens — Sum of prompt and completion tokens * latency — Total request latency in seconds * time\_to\_first\_token — Time to first token (seconds) * tokens\_per\_second — Output token throughput (TPS) * routing\_time — Deprecated: time spent deciding the model/route * cost — Total request cost (USD) * unique\_id — Request unique identifier * UNIQUE\_ORGANIZATION\_ID\_KEY — Organization unique identifier * organization\_key\_id — API key identifier * environment — Runtime environment (e.g., test, prod) * customer\_identifier — User/customer-level identifier * evaluation\_identifier — Evaluator run identifier * prompt\_id — Prompt identifier * prompt\_version\_number — Prompt version number * custom\_identifier — Custom identifier provided by client * thread\_identifier — Logical thread identifier * thread\_unique\_id — Unique thread identifier * trace\_unique\_id — Trace identifier * span\_unique\_id — Span identifier * span\_name — Span name * span\_parent\_id — Parent span identifier * span\_workflow\_name — Workflow name associated with span * trace\_group\_identifier — Group identifier for related traces * deployment\_name — Deployment name * provider\_id — Provider identifier * model — Model name * status\_code — HTTP-like status code * status — Status string * tool\_calls — Tool calls recorded * stream * stream\_options * temperature * max\_tokens * logit\_bias * logprobs * top\_logprobs * frequency\_penalty * presence\_penalty * stop * n * response\_format * verbosity * tools * input — Universal input field containing JSON-serialized request data * output — Universal output field containing JSON-serialized response data * prompt\_messages — (Legacy) Messages sent to the model (extracted from input for chat/completion types) * completion\_message — (Legacy) Final assistant message from the model (extracted from output for chat/completion types) ### Universal input/output design **Keywords AI uses universal `input` and `output` fields across all log types.** These fields contain JSON-serialized data that represents the complete request and response for any span type. The system automatically: * Serializes your data into `input` and `output` fields * Extracts type-specific fields (like `prompt_messages`, `completion_message`) for backward compatibility * Calculates metrics (cost, tokens, latency) * Validates and sanitizes all fields This design allows you to log any data structure while maintaining compatibility with legacy fields. ### Log types All logs are categorized by type to help with filtering and organization: * **`chat`**: Conversational chat completions (most common) * Input: Array of messages `[{"role": "user", "content": "..."}]` * Output: Message object `{"role": "assistant", "content": "..."}` * **`completion`**: Legacy completion format * **`response`**: Response API calls * **`embedding`**: Vector embedding generation * Input: Text string or array of texts * Output: Vector embeddings array * **`transcription`**: Speech-to-text conversion * Input: JSON with audio metadata `{"audio_url": "...", "filename": "..."}` * Output: Transcribed text string * **`speech`**: Text-to-speech generation * Input: Text string to be spoken * Output: JSON with audio metadata `{"audio_url": "...", "filename": "..."}` * **`text`**: (Deprecated) Use `chat` instead * **`workflow`**: Root-level workflow orchestration * **`task`**: Individual tasks within workflows * **`tool`**: Tool or function calls * **`agent`**: AI agent operations * **`handoff`**: Agent-to-agent task transfers * **`guardrail`**: Safety and validation checks * **`function`**: Standalone function executions * **`custom`**: User-defined custom operations * **`generation`**: Detailed generation process tracking * **`unknown`**: Fallback for unrecognized types ### Example log structure ```json theme={"system"} { "id": "log_123", "log_type": "chat", "model": "gpt-4", // Universal fields (always present) "input": "[{\"role\":\"user\",\"content\":\"What is the capital of France?\"}]", "output": "{\"role\":\"assistant\",\"content\":\"The capital of France is Paris.\"}", // Extracted legacy fields (for backward compatibility) "prompt_messages": [ {"role": "user", "content": "What is the capital of France?"} ], "completion_message": { "role": "assistant", "content": "The capital of France is Paris." }, // Metrics "prompt_tokens": 8, "completion_tokens": 7, "total_tokens": 15, "cost": 0.0003, "latency": 1.2, "timestamp": "2024-01-15T10:30:00Z", "status": "success" } ``` ## Traces Traces organize the same log data into hierarchical workflows, perfect for complex AI agent operations and multi-step processes. Traces group related logs using **spans** to show the execution flow of multi-step processes. Each span corresponds to a log entry but adds hierarchical context. ### Trace structure ```mermaid theme={"system"} graph TD T[Trace: trace_unique_id] --> RS[Root Span: WORKFLOW
span_workflow_name] RS --> S1[Span 1: CHAT
span_parent_id → Root] RS --> S2[Span 2: TASK
span_parent_id → Root] RS --> S3[Span 3: EMBEDDING
span_parent_id → Root] S1 --> S1A[Span 1A: GENERATION
span_parent_id → Span 1] S1 --> S1B[Span 1B: FUNCTION
span_parent_id → Span 1] S2 --> S2A[Span 2A: GUARDRAIL
span_parent_id → Span 2] ``` ### Key trace fields * **`trace_unique_id`**: Groups all spans in the same workflow * **`span_unique_id`**: Individual span identifier (maps to log ID) * **`span_parent_id`**: Creates the hierarchical structure * **`span_name`**: Descriptive name for the operation * **`span_workflow_name`**: The nearest workflow this span belongs to ### Multi-trace grouping Complex workflows can span multiple traces using `trace_group_identifier`: ```mermaid theme={"system"} graph TD TG[trace_group_identifier: workflow-group-456] --> T1[Trace A: trace-123] TG --> T2[Trace B: trace-124] T1 --> T1RS[Root: WORKFLOW
Parallel Process 1] T1RS --> T1S1[CHAT Span] T1RS --> T1S2[TASK Span] T2 --> T2RS[Root: WORKFLOW
Parallel Process 2] T2RS --> T2S1[EMBEDDING Span] T2RS --> T2S2[SPEECH Span] ``` ## Threads Threads organize the same log data in a conversational format, ideal for chat applications and dialogue systems. ### Thread structure * **Thread ID**: Unique identifier for the conversation * **Messages**: Ordered sequence of user and assistant messages (each message maps to log entries) Notice how each message in the thread references a `log_id` - this shows how threads are just a different presentation of the same underlying log data. ## Scores Scores organize the same log data with evaluation metrics and quality assessments, perfect for monitoring LLM performance and conducting evaluations. Scores are evaluation results that are attached to specific logs. Each score represents a quality assessment or performance metric for a particular LLM request. ### Score structure Scores are linked to logs through the `log_id` field and can be created by two types of evaluators: * **Platform evaluators**: Use `evaluator_id` (UUID from Keywords AI platform) * **Custom evaluators**: Use `evaluator_slug` (your custom string identifier) ### Key score fields * **`id`**: Unique score identifier * **`log_id`**: Links the score to its corresponding log entry * **`evaluator_id`**: UUID of Keywords AI platform evaluator (optional) * **`evaluator_slug`**: Custom evaluator identifier (optional) * **`is_passed`**: Whether the evaluation passed defined criteria * **`cost`**: Cost of running the evaluation * **`created_at`**: When the score was created Each evaluator can only have one score per log, ensuring data integrity and preventing duplicate evaluations. ### Score value types **Critical: Use the correct value field based on evaluator's `score_value_type`** Scores support four different value types based on the evaluator's `score_value_type`: | Evaluator's `score_value_type` | Use This Field | Data Type | Example | | ------------------------------ | ------------------- | ---------------- | --------------------------- | | `numerical` | `numerical_value` | number | `4.5` | | `boolean` | `boolean_value` | boolean | `true` | | `categorical` | `categorical_value` | array of strings | `["excellent", "coherent"]` | | `comment` | `string_value` | string | `"Good response quality"` | #### Detailed descriptions **Numerical Scores** * **Use case**: Ratings, confidence scores, quality metrics * **Range**: Defined by evaluator's `min_score` and `max_score` * **Example**: Rating response quality from 1-5 **Boolean Scores** * **Use case**: Pass/fail evaluations, binary classifications * **Values**: `true` or `false` * **Example**: Content safety check **Categorical Scores** * **Use case**: Multi-choice classifications * **Values**: Array of predefined choices from evaluator's `categorical_choices` * **Example**: `["relevant", "accurate", "helpful"]` **Comment Scores** * **Use case**: Qualitative feedback, explanations * **Values**: Free-form text * **Example**: Detailed evaluation reasoning ### Evaluator types Scores are created by evaluators, which come in three types: **LLM Evaluators (`type: "llm"`)** * AI-powered evaluation using language models * Requires `evaluator_definition` prompt * Supports all score value types **Human Evaluators (`type: "human"`)** * Manual evaluation by human reviewers * Often used with categorical or comment scores * Requires predefined choices for categorical **Code Evaluators (`type: "code"`)** * Programmatic evaluation using custom code * Requires `eval_code_snippet` * Most flexible for complex logic Legacy fields (`llm_input`, `llm_output`) are normalized to `input`/`output` when reading the `inputs` field.