wangtao
/
autogen
Not watched
1
0
Fork 0
Code
Releases 127 Wiki evaluate Activity
Issues 0 Pull Requests 0 Datasets Model Cloudbrain HPC
You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
Tree: 04be2ee930
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '04be2ee930'
${ noResults }
autogen/website/docs/ecosystem/agentops.md

4.2 kB
Raw Blame History

Agent Monitoring and Debugging with AgentOps

AgentOps logo

AgentOps provides session replays, metrics, and monitoring for AI agents.

At a high level, AgentOps gives you the ability to monitor LLM calls, costs, latency, agent failures, multi-agent interactions, tool usage, session-wide statistics, and more. For more info, check out the AgentOps Repo.
📊 Replay Analytics and Debugging Step-by-step agent execution graphs
💸 LLM Cost Management Track spend with LLM foundation model providers
🧪 Agent Benchmarking Test your agents against 1,000+ evals
🔐 Compliance and Security Detect common prompt injection and data exfiltration exploits
🤝 Framework Integrations Native Integrations with CrewAI, AutoGen, & LangChain
Agent Dashboard Agent Dashboard
Session Analytics Session Analytics
Session Replays Session Replays

Installation

AgentOps works seamlessly with applications built using Autogen.

  1. Install AgentOps
pip install agentops

  1. Create an API Key:
    Create a user API key here: Create API Key

  2. Configure Your Environment:
    Add your API key to your environment variables

AGENTOPS_API_KEY=<YOUR_AGENTOPS_API_KEY>
  1. Initialize AgentOps

To start tracking all available data on Autogen runs, simply add two lines of code before implementing Autogen.

import agentops
agentops.init() # Or: agentops.init(api_key="your-api-key-here")

After initializing AgentOps, Autogen will now start automatically tracking your agent runs.

Features

  • LLM Costs: Track spend with foundation model providers
  • Replay Analytics: Watch step-by-step agent execution graphs
  • Recursive Thought Detection: Identify when agents fall into infinite loops
  • Custom Reporting: Create custom analytics on agent performance
  • Analytics Dashboard: Monitor high level statistics about agents in development and production
  • Public Model Testing: Test your agents against benchmarks and leaderboards
  • Custom Tests: Run your agents against domain specific tests
  • Time Travel Debugging: Save snapshots of session states to rewind and replay agent runs from chosen checkpoints.
  • Compliance and Security: Create audit logs and detect potential threats such as profanity and PII leaks
  • Prompt Injection Detection: Identify potential code injection and secret leaks

Autogen + AgentOps examples

Extra links