User Feedback - Helicone OSS LLM Observability

When building AI applications, you need real-world signals about response quality to improve prompts, catch regressions, and understand what users find helpful. User Feedback lets you collect positive/negative ratings on LLM responses, enabling data-driven improvements to your AI systems based on actual user satisfaction.

Why use User Feedback#

Improve response quality: Identify patterns in poorly-rated responses to refine prompts and model selection
Catch regressions early: Monitor feedback trends to detect when changes negatively impact user experience
Build training datasets: Use highly-rated responses as examples for fine-tuning or few-shot prompting

Quick Start#

Make a request and capture the ID

Capture the Helicone request ID from your LLM response:

import OpenAI from "openai";const openai = new OpenAI({  apiKey: process.env.OPENAI_API_KEY,  baseURL: "https://oai.helicone.ai/v1",  defaultHeaders: {    "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,  },});// Use a custom request ID for feedback trackingconst customId = crypto.randomUUID();const response = await openai.chat.completions.create({  model: "gpt-4o-mini",  messages: [{ role: "user", content: "Explain quantum computing" }]}, {  headers: {    "Helicone-Request-Id": customId  }});// Use your custom ID for feedbackconst heliconeId = customId;

Alternative: Getting request ID from response

You can also try to get the Helicone ID from response headers, though this may not always be available:

const response = await openai.chat.completions.create({  model: "gpt-4o-mini",  messages: [{ role: "user", content: "Explain quantum computing" }]});// Try to get the Helicone request ID from response headersconst heliconeId = response.response?.headers?.get("helicone-id");// If not available, you'll need to use a custom ID approachif (!heliconeId) {  console.log("Helicone ID not found in headers, use custom ID approach instead");}

Submit feedback rating

Send a positive or negative rating for the response:

const feedback = await fetch(  `https://api.helicone.ai/v1/request/${heliconeId}/feedback`,  {    method: "POST",    headers: {      "Authorization": `Bearer ${process.env.HELICONE_API_KEY}`,      "Content-Type": "application/json",    },    body: JSON.stringify({      rating: true  // true = positive, false = negative    }),  });

View feedback analytics

Access feedback metrics in your Helicone dashboard to analyze response quality trends and identify areas for improvement.

Configuration Options#

Feedback collection requires minimal configuration:

Parameter	Type	Description	Default	Example
`rating`	`boolean`	User's feedback on the response	N/A	`true` (positive) or `false` (negative)
`helicone-id`	`string`	Request ID to attach feedback to	N/A	UUID

Processing multiple feedback ratings

When you need to submit feedback for multiple requests, use parallel API calls:

// Note: There is no bulk feedback endpoint - each rating requires a separate API callconst feedbackBatch = [  { requestId: "f47ac10b-58cc-4372-a567-0e02b2c3d479", rating: true },  { requestId: "6ba7b810-9dad-11d1-80b4-00c04fd430c8", rating: false },  { requestId: "6ba7b811-9dad-11d1-80b4-00c04fd430c8", rating: true }];// Submit feedback in parallel for better performanceconst feedbackPromises = feedbackBatch.map(({ requestId, rating }) =>  fetch(`https://api.helicone.ai/v1/request/${requestId}/feedback`, {    method: "POST",    headers: {      "Authorization": `Bearer ${process.env.HELICONE_API_KEY}`,      "Content-Type": "application/json",    },    body: JSON.stringify({ rating }),  }));// Wait for all feedback submissions to completeconst results = await Promise.all(feedbackPromises);// Check for any failed submissionsresults.forEach((result, index) => {  if (!result.ok) {    console.error(`Failed to submit feedback for request ${feedbackBatch[index].requestId}`);  }});

Use Cases#

Track user satisfaction with AI assistant responses:

import OpenAI from "openai";const openai = new OpenAI({  apiKey: process.env.OPENAI_API_KEY,  baseURL: "https://oai.helicone.ai/v1",  defaultHeaders: {    "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,  },});// In your chat handlerasync function handleChatMessage(userId: string, message: string) {  const requestId = crypto.randomUUID();    const response = await openai.chat.completions.create(    {      model: "gpt-4o-mini",      messages: [        { role: "system", content: "You are a helpful assistant." },        { role: "user", content: message }      ]    },    {      headers: {        "Helicone-Request-Id": requestId,        "Helicone-User-Id": userId,        "Helicone-Property-Feature": "chat"      }    }  );  // Store request ID for later feedback  await storeRequestMapping(userId, requestId, response.id);    return response;}// When user clicks thumbs up/downasync function handleUserFeedback(userId: string, responseId: string, isPositive: boolean) {  const requestId = await getRequestId(userId, responseId);    await fetch(    `https://api.helicone.ai/v1/request/${requestId}/feedback`,    {      method: "POST",      headers: {        "Authorization": `Bearer ${process.env.HELICONE_API_KEY}`,        "Content-Type": "application/json",      },      body: JSON.stringify({ rating: isPositive }),    }  );}

Collect feedback on generated code quality:

// After generating code for the userconst codeGenResponse = await openai.chat.completions.create(  {    model: "gpt-4o-mini",    messages: [      { role: "system", content: "You are an expert programmer." },      { role: "user", content: `Generate a ${language} function to ${task}` }    ]  },  {    headers: {      "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,      "Helicone-Property-Feature": "code-generation",      "Helicone-Property-Language": language    }  });// Track if the generated code workedconst codeWorked = await userTestedCode(); // Your logic here// Auto-submit feedback based on code executionconst heliconeId = codeGenResponse.headers?.get("helicone-id");if (heliconeId) {  await fetch(    `https://api.helicone.ai/v1/request/${heliconeId}/feedback`,    {      method: "POST",      headers: {        "Authorization": `Bearer ${process.env.HELICONE_API_KEY}`,        "Content-Type": "application/json",      },      body: JSON.stringify({ rating: codeWorked }),    }  );}// Analyze which languages/tasks have highest success rates

Measure effectiveness of automated support responses:

// Support ticket handlerasync function handleSupportQuery(ticketId: string, query: string) {  const requestId = `ticket-${ticketId}-${Date.now()}`;    const response = await openai.chat.completions.create(    {      model: "gpt-4o-mini",      messages: [        {           role: "system",           content: "You are a technical support specialist. Provide clear, helpful solutions."         },        { role: "user", content: query }      ],      temperature: 0.3 // Lower temperature for consistent support answers    },    {      headers: {        "Helicone-Auth": `Bearer ${process.env.HELICONE_API_KEY}`,        "Helicone-Request-Id": requestId,        "Helicone-Property-Type": "support",        "Helicone-Property-TicketId": ticketId      }    }  );  // Send response to user  await sendSupportResponse(ticketId, response.choices[0].message.content);    // Follow up after resolution  setTimeout(async () => {    const wasHelpful = await checkIfTicketResolved(ticketId);        await fetch(      `https://api.helicone.ai/v1/request/${requestId}/feedback`,      {        method: "POST",        headers: {          "Authorization": `Bearer ${process.env.HELICONE_API_KEY}`,          "Content-Type": "application/json",        },        body: JSON.stringify({ rating: wasHelpful }),      }    );  }, 24 * 60 * 60 * 1000); // Check after 24 hours}

Understanding User Feedback#

How it works#

User feedback creates a continuous improvement loop for your AI application:

Each LLM request gets a unique Helicone ID
Users rate responses as positive (helpful) or negative (not helpful)
Feedback is linked to the original request for analysis
Dashboard aggregates feedback to show quality trends

Explicit vs Implicit Feedback#

Explicit feedback is when users directly rate responses (thumbs up/down, star ratings). While valuable, it has low response rates since users must take deliberate action.

Implicit feedback is derived from user behavior and is much more valuable since it reflects actual usage patterns:

Track user actions that indicate response quality:

// Code completion acceptance (like Cursor)async function trackCodeCompletion(requestId: string, suggestion: string) {  // Monitor if user accepts the completion  const accepted = await waitForUserAction(suggestion);    await fetch(`https://api.helicone.ai/v1/request/${requestId}/feedback`, {    method: "POST",    headers: {      "Authorization": `Bearer ${process.env.HELICONE_API_KEY}`,      "Content-Type": "application/json",    },    body: JSON.stringify({       rating: accepted // true if accepted, false if rejected/ignored    }),  });}// Chat engagement patternsasync function trackChatEngagement(requestId: string, response: string) {  // Track user behavior after response  const userActions = await monitorUserBehavior(60000); // 1 minute    const implicitRating =     userActions.continuedConversation || // User asked follow-up    userActions.copiedResponse ||        // User copied the answer    userActions.sharedResponse ||        // User shared/saved    userActions.timeSpent > 30;          // User read for >30 seconds    await submitFeedback(requestId, implicitRating);}// Search/recommendation clicksasync function trackSearchResult(requestId: string, results: string[]) {  // Monitor if user clicks on suggested results  const clicked = await trackClicks(results, 300000); // 5 minutes    // High click-through rate = good recommendations  const rating = clicked.length > 0;  await submitFeedback(requestId, rating);}