Skip to main content
Helicone OSS LLM Observability

Building and Monitoring AI Agents with Helicone

Learn how to build autonomous AI agents, monitor and optimize their performance using Helicone's Sessions.
4 min read

AI agents are transforming how we interact with software, moving beyond simple question-answer systems to tools that can actually do things for us. But as agents become more autonomous and complex, monitoring their behavior becomes critical.

This guide shows you how to build a true AI agent—one that can think, decide, and act autonomously—while using Helicone's Sessions to track every decision, tool usage, and interaction.

What Makes a True AI Agent?#

The key distinction between a true agent and an automation (also known as a "workflow") lies in autonomy and dynamic decision-making:

  • Workflows are like a GPS with a fixed route—if there's a roadblock, it can't adapt
  • Agents are like having a local guide who knows all the shortcuts and can change plans on the fly

What We'll Build#

We'll create a stock information agent that can:

  1. Fetch real-time stock prices using the Yahoo Finance API
  2. Find company CEOs from stock data
  3. Identify ticker symbols from company names
  4. Chain tool calls to answer complex queries

What makes this a true agent is that it autonomously decides:

  • Which tool to use for each query
  • When to chain multiple tools together
  • When to ask the user for more information
  • How to handle errors and retry with different approaches

And with Helicone's Sessions, we can monitor every decision and tool execution the agent makes to pinpoint issues and optimize performance.

Prerequisites#

You'll need:

  • Python 3.7 or higher
  • A Helicone API key (get one free at helicone.ai)
  • An OpenAI API key (get one free at openai.com)

Create a project directory and install packages:

Create a .env file:

Building the AI Agent#

Set up the Agent with Helicone

First, let's create our agent class an initialize an OpenAI client with Helicone integration. We'll also initialize the Helicone Manual Logger to log tool usage:

Initialize Session Tracking

Sessions help you track complete agent conversations and see how tools chain together:

Create and Monitor Tools with Sessions & Manual Logging

Each tool execution is logged separately with detailed results:

Implement the Agent's Decision-Making Loop

Implement the main processing loop, which calls tools as needed until it has a complete answer:

Add the Chat Interface

Finally, create the interactive chat loop, which serves as the entry point for the agent and kicks off the session:

Run Your Agent

Running the agent is simple, navigate to the project directory and run the following command:

Real-World Example#

Here's how the monitored agent handles a complex query:

The agent autonomously:

  1. Recognized "EV company from China" was ambiguous
  2. Asked which specific company
  3. Found the ticker symbol for NIO
  4. Retrieved the CEO information
  5. Fetched the current stock price
  6. Composed a complete answer

In your Helicone dashboard, you'll see each operation tracked in detail as part of the session flows as shown in the image below.

Viewing Agent Operations in Helicone#

With Sessions integration, your agent's operations appear beautifully organized in your Helicone dashboard:

Helicone Sessions view showing agent operations with timeline and detailed request tracking

The session view shows:

  • Timeline visualization of agent operations flowing from reasoning to tool execution
  • Hierarchical session paths showing the flow from /stock-chat to specific operations like /price/tsla
  • Individual request details with status, timing, and model information
  • Complete conversation context across multiple tool calls

Each operation is logged with rich metadata:

  • Tool executions show success/failure status and detailed results
  • LLM reasoning calls include full conversation context
  • Session paths create a logical hierarchy of operations
  • Timing information helps identify performance bottlenecks

Debugging Complex Agent Interactions#

Using Helicone Sessions provides several debugging advantages:

Separate Tool Tracking#

Each tool execution is logged individually, making it easy to identify which tools fail or succeed.

Rich Metadata#

Tool calls include detailed input/output information and error states for comprehensive debugging.

Session Flow Visualization#

See exactly how your agent chains tools together and where decision points occur.

Performance Monitoring#

Track timing for both LLM reasoning and tool execution to optimize agent performance.

Complete Implementation#

Click to expand the full agent code

Next Steps#

With Helicone's Manual Logger, you have complete visibility into your agent's decision-making process. From here, you can:

  • Extend the agent with more tools like news retrieval or financial analysis
  • Optimize performance based on the data available in the sessions dashboard
  • Debug complex interactions using session flow visualization
  • Monitor production usage with detailed request tracking