Datasets

Transform your LLM requests into curated datasets for model fine-tuning, evaluation, and analysis. Helicone Datasets let you select, organize, and export your best examples with just a few clicks.

Why Use Datasets#

Creating Datasets#

From the Requests Page#

The easiest way to create datasets is by selecting requests from your logs:

1

Filter your requests

Use custom properties and filters to find the requests you want

2

Select requests

Check the boxes next to requests you want to include in your dataset

3

Add to dataset

Click "Add to Dataset" and choose to create a new dataset or add to an existing one

Via API#

Create datasets programmatically for automated workflows:

// Create a new datasetconst response = await fetch('https://api.helicone.ai/v1/helicone-dataset', {  method: 'POST',  headers: {    'Authorization': `Bearer ${HELICONE_API_KEY}`,    'Content-Type': 'application/json'  },  body: JSON.stringify({    name: 'Customer Support Examples',    description: 'High-quality support interactions for fine-tuning'  })});const dataset = await response.json();// Add requests to the datasetawait fetch(`https://api.helicone.ai/v1/helicone-dataset/${dataset.id}/request/${requestId}`, {  method: 'POST',  headers: {    'Authorization': `Bearer ${HELICONE_API_KEY}`  }});

Building Quality Datasets#

The Curation Process#

Transform raw requests into high-quality training data through careful curation:

1

Collect broadly, then filter

Start by adding many potential examples, then narrow down to the best ones. It's easier to remove than to find examples later.

2

Review each example

Examine each request/response pair for:

• Accuracy - Is the response correct and helpful?
• Consistency - Does it match the style and format you want?
• Completeness - Does it fully address the user's request?

3

Remove poor examples

Delete any examples that are:

• Incorrect or misleading responses
• Off-topic or irrelevant
• Inconsistent with your desired behavior
• Edge cases that might confuse the model

4

Balance your dataset

Ensure you have:

• Examples covering all common use cases
• Both simple and complex queries
• Appropriate distribution matching real usage

Dataset Dashboard#

Access all your datasets at helicone.ai/datasets:

From the dashboard you can:

• Track progress - Monitor dataset size and last updated time
• Access datasets - Click to view and curate contents
• Export data - Download datasets when ready for fine-tuning
• Maintain quality - Regularly review and improve your collections

Exporting Data#

Export Formats#

Download your datasets in various formats:

{"messages": [{"role": "user", "content": "Hello"}, {"role": "assistant", "content": "Hi there!"}]}{"messages": [{"role": "user", "content": "Help me"}, {"role": "assistant", "content": "I'd be happy to help!"}]}

request_id,created_at,model,prompt_tokens,completion_tokens,cost,user_message,assistant_responsereq_123,2024-01-15,gpt-4o,50,100,0.002,"Hello","Hi there!"req_124,2024-01-15,gpt-4o,45,95,0.0019,"Help me","I'd be happy to help!"

API Export#

Retrieve dataset contents programmatically:

// Query dataset contentsconst response = await fetch(`https://api.helicone.ai/v1/helicone-dataset/${datasetId}/query`, {  method: 'POST',  headers: {    'Authorization': `Bearer ${HELICONE_API_KEY}`,    'Content-Type': 'application/json'  },  body: JSON.stringify({    limit: 100,    offset: 0  })});const data = await response.json();

Use Cases#

Replace Expensive Models with Fine-Tuned Alternatives#

The most common use case - using your expensive model logs to train cheaper, faster models:

Task-Specific Evaluation Sets#

Build evaluation datasets to test model performance:

// Create eval sets for different capabilitiesconst datasets = {  reasoning: 'Complex multi-step problems with verified solutions',  extraction: 'Structured data extraction with known correct outputs',  creativity: 'Creative writing with human-rated quality scores',  edge_cases: 'Unusual inputs that often cause failures'};

Use these to:

• Compare model versions before deploying
• Test prompt changes against consistent examples
• Identify model weaknesses and blind spots

Continuous Improvement Pipeline#

Build a data flywheel for model improvement:

1. Tag requests with custom properties for easy filtering
2. Score outputs based on user feedback or automated metrics
3. Auto-collect winners into datasets when they meet quality thresholds
4. Regular retraining with newly curated examples
5. A/B test new models against production traffic

Best Practices#

Related Features#

Datasets turn your production LLM logs into valuable training and evaluation resources. Start small with a focused use case, then expand as you see the benefits of curated, high-quality data.