Update evaluator

PATCH /api/evaluators/{evaluator_id}/

Update evaluator

cURL

curl --request PATCH \
  --url https://api.keywordsai.co/api/evaluators/{evaluator_id}/ \
  --header 'Authorization: Bearer <token>'

Updates specific fields of an evaluator. Supports partial updates of configuration fields.

Authentication

All endpoints require API key authentication:

Authorization: Bearer YOUR_API_KEY

Path Parameters

Parameter	Type	Description
evaluator_id	string	The unique ID of the evaluator to update

Request Body

New Format: You can now update score_config, passing_conditions, llm_config, and code_config fields to add or modify automation for any evaluator type.

You can update any of the following fields. Only provide the fields you want to update:

Field	Type	Description
name	string	Display name for the evaluator
description	string	Description of what the evaluator does
score_config	object	New: Score type configuration (min/max, choices, etc.)
passing_conditions	object	New: Passing conditions using universal filter format
llm_config	object	New: LLM automation config
code_config	object	New: Code automation config
configurations	object	Legacy type-specific configuration settings
categorical_choices	array	Legacy choices (use score_config.choices in new format)
custom_required_fields	array	Additional required fields
starred	boolean	Whether the evaluator is starred
tags	array	Tags for organization

Note: Configuration fields are merged with existing values. Non-null values take precedence over existing null values.

Examples

New Format: Add LLM Config to Existing Evaluator

This example shows adding LLM automation to any evaluator type, demonstrating the decoupling of annotation method from evaluator type.

Python

import requests

evaluator_id = "0f4325f9-55ef-4c20-8abe-376694419947"
url = f"https://api.keywordsai.co/api/evaluators/{evaluator_id}/"
headers = {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
}

# Add LLM automation to an existing evaluator
data = {
    "llm_config": {
        "model": "gpt-4o-mini",
        "evaluator_definition": "Rate the quality:\n<input>{{input}}</input>\n<output>{{output}}</output>",
        "scoring_rubric": "1=Poor, 5=Excellent",
        "temperature": 0.1,
        "max_tokens": 200
    },
    "passing_conditions": {
        "primary_score": {
            "operator": "gte",
            "value": 3
        }
    }
}

response = requests.patch(url, headers=headers, json=data)
print(response.json())

New Format: Add Code Config to Existing Evaluator

Python

# Add code automation to an existing evaluator
data = {
    "code_config": {
        "eval_code_snippet": "def main(eval_inputs):\n    output = eval_inputs.get('output', '')\n    return len(str(output)) > 10"
    }
}

response = requests.patch(url, headers=headers, json=data)
print(response.json())

New Format: Update Score Config

Python

# Update score configuration
data = {
    "score_config": {
        "min_score": 0,
        "max_score": 10,
        "choices": [
            {"name": "Poor", "value": 0},
            {"name": "Average", "value": 5},
            {"name": "Excellent", "value": 10}
        ]
    }
}

response = requests.patch(url, headers=headers, json=data)
print(response.json())

Legacy Format: Update LLM Evaluator Configuration

Python

import requests

evaluator_id = "0f4325f9-55ef-4c20-8abe-376694419947"
url = f"https://api.keywordsai.co/api/evaluators/{evaluator_id}/"
headers = {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
}

# Update scoring rubric and passing score
data = {
    "configurations": {
        "scoring_rubric": "Updated: 1=Very Poor, 2=Poor, 3=Fair, 4=Good, 5=Excellent",
        "passing_score": 4.0
    }
}

response = requests.patch(url, headers=headers, json=data)
print(response.json())

Update Name and Description

Python

data = {
    "name": "Enhanced Response Quality Evaluator",
    "description": "Advanced evaluator for response quality assessment with updated criteria"
}

response = requests.patch(url, headers=headers, json=data)
print(response.json())

Update Categorical Choices

Python

# For categorical evaluators
categorical_evaluator_id = "cat-eval-123"
url = f"https://api.keywordsai.co/api/evaluators/{categorical_evaluator_id}/"

data = {
    "categorical_choices": [
        { "name": "Outstanding", "value": 5 },
        { "name": "Very Good", "value": 4 },
        { "name": "Good", "value": 3 },
        { "name": "Fair", "value": 2 },
        { "name": "Poor", "value": 1 }
    ]
}

response = requests.patch(url, headers=headers, json=data)
print(response.json())

Update Code Evaluator

Python

# For code evaluators
code_evaluator_id = "bool-eval-456"
url = f"https://api.keywordsai.co/api/evaluators/{code_evaluator_id}/"

data = {
    "name": "Enhanced Length Checker",
    "configurations": {
        "eval_code_snippet": "def evaluate(llm_input, llm_output, **kwargs):\n    '''\n    Enhanced length checker with word count\n    Returns True if response has >= 10 words, False otherwise\n    '''\n    if not llm_output:\n        return False\n    \n    word_count = len(llm_output.strip().split())\n    return word_count >= 10"
    }
}

response = requests.patch(url, headers=headers, json=data)
print(response.json())

Update Tags and Starred Status

Python

data = {
    "starred": True,
    "tags": ["quality", "assessment", "production"]
}

response = requests.patch(url, headers=headers, json=data)
print(response.json())

Update LLM Engine and Model Options

Python

data = {
    "configurations": {
        "llm_engine": "gpt-4o",
        "model_options": {
            "temperature": 0.2,
            "max_tokens": 300
        }
    }
}

response = requests.patch(url, headers=headers, json=data)
print(response.json())

Response

Status: 200 OK

Returns the updated evaluator object with all current field values:

{
  "id": "0f4325f9-55ef-4c20-8abe-376694419947",
  "name": "Enhanced Response Quality Evaluator",
  "evaluator_slug": "response_quality_v1",
  "type": "llm",
  "score_value_type": "numerical",
  "eval_class": "",
  "description": "Advanced evaluator for response quality assessment with updated criteria",
  "configurations": {
    "evaluator_definition": "Rate the response quality based on accuracy, relevance, and completeness.\n<llm_input>{{llm_input}}</llm_input>\n<llm_output>{{llm_output}}</llm_output>",
    "scoring_rubric": "Updated: 1=Very Poor, 2=Poor, 3=Fair, 4=Good, 5=Excellent",
    "llm_engine": "gpt-4o",
    "model_options": {
      "temperature": 0.2,
      "max_tokens": 300
    },
    "min_score": 1.0,
    "max_score": 5.0,
    "passing_score": 4.0
  },
  "created_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[e-mail]"
  },
  "updated_by": {
    "first_name": "Keywords AI",
    "last_name": "Team",
    "email": "[e-mail]"
  },
  "created_at": "2025-09-11T09:43:55.858321Z",
  "updated_at": "2025-09-11T10:15:22.123456Z",
  "custom_required_fields": [],
  "categorical_choices": null,
  "starred": true,
  "organization": 2,
  "tags": ["quality", "assessment", "production"]
}

Configuration Update Rules

Partial Configuration Updates

When updating configurations, the system performs a merge operation:

• Existing configuration fields are preserved unless explicitly overridden
• New fields are added to the configuration
• Setting a field to null removes it from the configuration
• Nested objects (like model_options) are completely replaced, not merged

Example Configuration Merge

Original Configuration:

{
  "evaluator_definition": "Original prompt",
  "scoring_rubric": "Original rubric",
  "llm_engine": "gpt-4o-mini",
  "min_score": 1.0,
  "max_score": 5.0
}

Update Request:

{
  "configurations": {
    "scoring_rubric": "Updated rubric",
    "passing_score": 3.0
  }
}

Resulting Configuration:

{
  "evaluator_definition": "Original prompt",
  "scoring_rubric": "Updated rubric",
  "llm_engine": "gpt-4o-mini",
  "min_score": 1.0,
  "max_score": 5.0,
  "passing_score": 3.0
}

Error Responses

400 Bad Request

{
  "configurations": [
    "Configuration validation failed: llm_engine 'invalid-model' is not supported"
  ]
}

401 Unauthorized

{
  "detail": "Your API key is invalid or expired, please check your API key at https://platform.keywordsai.co/platform/api/api-keys"
}

403 Forbidden

{
  "detail": "You do not have permission to update this evaluator."
}

404 Not Found

{
  "detail": "Not found."
}

422 Unprocessable Entity

{
  "categorical_choices": [
    "This field is required when score_value_type is 'categorical'."
  ]
}