You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.

connection.py 17 kB

Enable LLM Call Observability in AGS (#5457) <!-- Thank you for your contribution! Please review https://microsoft.github.io/autogen/docs/Contribute before opening a pull request. --> <!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. --> It is often helpful to inspect the raw request and response to/from an LLM as agents act. This PR, does the following: - Updates TeamManager to yield LLMCallEvents from core library - Run in an async background task to listen for LLMCallEvent JIT style when a team is run - Add events to an async queue and - yield those events in addition to whatever actual agentchat team.run_stream yields. - Update the AGS UI to show those LLMCallEvents in the messages section as a team runs - Add settings panel to show/hide llm call events in messages. - Minor updates to default team <img width="1539" alt="image" src="https://github.com/user-attachments/assets/bfbb19fe-3560-4faa-b600-7dd244e0e974" /> <img width="1554" alt="image" src="https://github.com/user-attachments/assets/775624f5-ba83-46e8-81ff-512bfeed6bab" /> <img width="1538" alt="image" src="https://github.com/user-attachments/assets/3becbf85-b75a-4506-adb7-e5575ebbaec4" /> ## Why are these changes needed? <!-- Please give a short summary of the change and the problem this solves. --> ## Related issue number <!-- For example: "Closes #1234" --> Closes #5440 ## Checks - [ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally. - [ ] I've added tests (if relevant) corresponding to the changes introduced in this PR. - [ ] I've made sure all auto checks have passed.
1 year ago
Enable LLM Call Observability in AGS (#5457) <!-- Thank you for your contribution! Please review https://microsoft.github.io/autogen/docs/Contribute before opening a pull request. --> <!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. --> It is often helpful to inspect the raw request and response to/from an LLM as agents act. This PR, does the following: - Updates TeamManager to yield LLMCallEvents from core library - Run in an async background task to listen for LLMCallEvent JIT style when a team is run - Add events to an async queue and - yield those events in addition to whatever actual agentchat team.run_stream yields. - Update the AGS UI to show those LLMCallEvents in the messages section as a team runs - Add settings panel to show/hide llm call events in messages. - Minor updates to default team <img width="1539" alt="image" src="https://github.com/user-attachments/assets/bfbb19fe-3560-4faa-b600-7dd244e0e974" /> <img width="1554" alt="image" src="https://github.com/user-attachments/assets/775624f5-ba83-46e8-81ff-512bfeed6bab" /> <img width="1538" alt="image" src="https://github.com/user-attachments/assets/3becbf85-b75a-4506-adb7-e5575ebbaec4" /> ## Why are these changes needed? <!-- Please give a short summary of the change and the problem this solves. --> ## Related issue number <!-- For example: "Closes #1234" --> Closes #5440 ## Checks - [ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally. - [ ] I've added tests (if relevant) corresponding to the changes introduced in this PR. - [ ] I've made sure all auto checks have passed.
1 year ago
Enable LLM Call Observability in AGS (#5457) <!-- Thank you for your contribution! Please review https://microsoft.github.io/autogen/docs/Contribute before opening a pull request. --> <!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. --> It is often helpful to inspect the raw request and response to/from an LLM as agents act. This PR, does the following: - Updates TeamManager to yield LLMCallEvents from core library - Run in an async background task to listen for LLMCallEvent JIT style when a team is run - Add events to an async queue and - yield those events in addition to whatever actual agentchat team.run_stream yields. - Update the AGS UI to show those LLMCallEvents in the messages section as a team runs - Add settings panel to show/hide llm call events in messages. - Minor updates to default team <img width="1539" alt="image" src="https://github.com/user-attachments/assets/bfbb19fe-3560-4faa-b600-7dd244e0e974" /> <img width="1554" alt="image" src="https://github.com/user-attachments/assets/775624f5-ba83-46e8-81ff-512bfeed6bab" /> <img width="1538" alt="image" src="https://github.com/user-attachments/assets/3becbf85-b75a-4506-adb7-e5575ebbaec4" /> ## Why are these changes needed? <!-- Please give a short summary of the change and the problem this solves. --> ## Related issue number <!-- For example: "Closes #1234" --> Closes #5440 ## Checks - [ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally. - [ ] I've added tests (if relevant) corresponding to the changes introduced in this PR. - [ ] I've made sure all auto checks have passed.
1 year ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432
  1. import asyncio
  2. import logging
  3. import traceback
  4. from datetime import datetime, timezone
  5. from typing import Any, Callable, Dict, Optional, Union
  6. from uuid import UUID
  7. from autogen_agentchat.base._task import TaskResult
  8. from autogen_agentchat.messages import (
  9. AgentEvent,
  10. ChatMessage,
  11. HandoffMessage,
  12. MultiModalMessage,
  13. StopMessage,
  14. TextMessage,
  15. ToolCallExecutionEvent,
  16. ToolCallRequestEvent,
  17. )
  18. from autogen_core import CancellationToken
  19. from autogen_core import Image as AGImage
  20. from fastapi import WebSocket, WebSocketDisconnect
  21. from ...database import DatabaseManager
  22. from ...datamodel import LLMCallEventMessage, Message, MessageConfig, Run, RunStatus, TeamResult
  23. from ...teammanager import TeamManager
  24. logger = logging.getLogger(__name__)
  25. class WebSocketManager:
  26. """Manages WebSocket connections and message streaming for team task execution"""
  27. def __init__(self, db_manager: DatabaseManager):
  28. self.db_manager = db_manager
  29. self._connections: Dict[UUID, WebSocket] = {}
  30. self._cancellation_tokens: Dict[UUID, CancellationToken] = {}
  31. # Track explicitly closed connections
  32. self._closed_connections: set[UUID] = set()
  33. self._input_responses: Dict[UUID, asyncio.Queue] = {}
  34. self._cancel_message = TeamResult(
  35. task_result=TaskResult(
  36. messages=[TextMessage(source="user", content="Run cancelled by user")], stop_reason="cancelled by user"
  37. ),
  38. usage="",
  39. duration=0,
  40. ).model_dump()
  41. def _get_stop_message(self, reason: str) -> dict:
  42. return TeamResult(
  43. task_result=TaskResult(messages=[TextMessage(source="user", content=reason)], stop_reason=reason),
  44. usage="",
  45. duration=0,
  46. ).model_dump()
  47. async def connect(self, websocket: WebSocket, run_id: UUID) -> bool:
  48. try:
  49. await websocket.accept()
  50. self._connections[run_id] = websocket
  51. self._closed_connections.discard(run_id)
  52. # Initialize input queue for this connection
  53. self._input_responses[run_id] = asyncio.Queue()
  54. await self._send_message(
  55. run_id, {"type": "system", "status": "connected", "timestamp": datetime.now(timezone.utc).isoformat()}
  56. )
  57. return True
  58. except Exception as e:
  59. logger.error(f"Connection error for run {run_id}: {e}")
  60. return False
  61. async def start_stream(self, run_id: UUID, task: str, team_config: dict) -> None:
  62. """Start streaming task execution with proper run management"""
  63. if run_id not in self._connections or run_id in self._closed_connections:
  64. raise ValueError(f"No active connection for run {run_id}")
  65. team_manager = TeamManager()
  66. cancellation_token = CancellationToken()
  67. self._cancellation_tokens[run_id] = cancellation_token
  68. final_result = None
  69. try:
  70. # Update run with task and status
  71. run = await self._get_run(run_id)
  72. if run:
  73. run.task = MessageConfig(content=task, source="user").model_dump()
  74. run.status = RunStatus.ACTIVE
  75. self.db_manager.upsert(run)
  76. input_func = self.create_input_func(run_id)
  77. async for message in team_manager.run_stream(
  78. task=task, team_config=team_config, input_func=input_func, cancellation_token=cancellation_token
  79. ):
  80. if cancellation_token.is_cancelled() or run_id in self._closed_connections:
  81. logger.info(f"Stream cancelled or connection closed for run {run_id}")
  82. break
  83. formatted_message = self._format_message(message)
  84. if formatted_message:
  85. await self._send_message(run_id, formatted_message)
  86. # Save messages by concrete type
  87. if isinstance(
  88. message,
  89. (
  90. TextMessage,
  91. MultiModalMessage,
  92. StopMessage,
  93. HandoffMessage,
  94. ToolCallRequestEvent,
  95. ToolCallExecutionEvent,
  96. LLMCallEventMessage,
  97. ),
  98. ):
  99. await self._save_message(run_id, message)
  100. # Capture final result if it's a TeamResult
  101. elif isinstance(message, TeamResult):
  102. final_result = message.model_dump()
  103. if not cancellation_token.is_cancelled() and run_id not in self._closed_connections:
  104. if final_result:
  105. await self._update_run(run_id, RunStatus.COMPLETE, team_result=final_result)
  106. else:
  107. logger.warning(f"No final result captured for completed run {run_id}")
  108. await self._update_run_status(run_id, RunStatus.COMPLETE)
  109. else:
  110. await self._send_message(
  111. run_id,
  112. {
  113. "type": "completion",
  114. "status": "cancelled",
  115. "data": self._cancel_message,
  116. "timestamp": datetime.now(timezone.utc).isoformat(),
  117. },
  118. )
  119. # Update run with cancellation result
  120. await self._update_run(run_id, RunStatus.STOPPED, team_result=self._cancel_message)
  121. except Exception as e:
  122. logger.error(f"Stream error for run {run_id}: {e}")
  123. traceback.print_exc()
  124. await self._handle_stream_error(run_id, e)
  125. finally:
  126. self._cancellation_tokens.pop(run_id, None)
  127. async def _save_message(self, run_id: UUID, message: Union[AgentEvent | ChatMessage, ChatMessage]) -> None:
  128. """Save a message to the database"""
  129. run = await self._get_run(run_id)
  130. if run:
  131. db_message = Message(
  132. session_id=run.session_id,
  133. run_id=run_id,
  134. config=message.model_dump(),
  135. user_id=None, # You might want to pass this from somewhere
  136. )
  137. self.db_manager.upsert(db_message)
  138. async def _update_run(
  139. self, run_id: UUID, status: RunStatus, team_result: Optional[dict] = None, error: Optional[str] = None
  140. ) -> None:
  141. """Update run status and result"""
  142. run = await self._get_run(run_id)
  143. if run:
  144. run.status = status
  145. if team_result:
  146. run.team_result = team_result
  147. if error:
  148. run.error_message = error
  149. self.db_manager.upsert(run)
  150. def create_input_func(self, run_id: UUID) -> Callable:
  151. """Creates an input function for a specific run"""
  152. async def input_handler(prompt: str = "", cancellation_token: Optional[CancellationToken] = None) -> str:
  153. try:
  154. # Send input request to client
  155. await self._send_message(
  156. run_id,
  157. {
  158. "type": "input_request",
  159. "prompt": prompt,
  160. "data": {"source": "system", "content": prompt},
  161. "timestamp": datetime.now(timezone.utc).isoformat(),
  162. },
  163. )
  164. # Wait for response
  165. if run_id in self._input_responses:
  166. response = await self._input_responses[run_id].get()
  167. return response
  168. else:
  169. raise ValueError(f"No input queue for run {run_id}")
  170. except Exception as e:
  171. logger.error(f"Error handling input for run {run_id}: {e}")
  172. raise
  173. return input_handler
  174. async def handle_input_response(self, run_id: UUID, response: str) -> None:
  175. """Handle input response from client"""
  176. if run_id in self._input_responses:
  177. await self._input_responses[run_id].put(response)
  178. else:
  179. logger.warning(f"Received input response for inactive run {run_id}")
  180. async def stop_run(self, run_id: UUID, reason: str) -> None:
  181. if run_id in self._cancellation_tokens:
  182. logger.info(f"Stopping run {run_id}")
  183. stop_message = self._get_stop_message(reason)
  184. try:
  185. # Update run record first
  186. await self._update_run(run_id, status=RunStatus.STOPPED, team_result=stop_message)
  187. # Then handle websocket communication if connection is active
  188. if run_id in self._connections and run_id not in self._closed_connections:
  189. await self._send_message(
  190. run_id,
  191. {
  192. "type": "completion",
  193. "status": "cancelled",
  194. "data": stop_message,
  195. "timestamp": datetime.now(timezone.utc).isoformat(),
  196. },
  197. )
  198. # Finally cancel the token
  199. self._cancellation_tokens[run_id].cancel()
  200. except Exception as e:
  201. logger.error(f"Error stopping run {run_id}: {e}")
  202. # We might want to force disconnect here if db update failed
  203. # await self.disconnect(run_id) # Optional
  204. async def disconnect(self, run_id: UUID) -> None:
  205. """Clean up connection and associated resources"""
  206. logger.info(f"Disconnecting run {run_id}")
  207. # Mark as closed before cleanup to prevent any new messages
  208. self._closed_connections.add(run_id)
  209. # Cancel any running tasks
  210. await self.stop_run(run_id, "Connection closed")
  211. # Clean up resources
  212. self._connections.pop(run_id, None)
  213. self._cancellation_tokens.pop(run_id, None)
  214. self._input_responses.pop(run_id, None)
  215. async def _send_message(self, run_id: UUID, message: dict) -> None:
  216. """Send a message through the WebSocket with connection state checking
  217. Args:
  218. run_id: UUID of the run
  219. message: Message dictionary to send
  220. """
  221. if run_id in self._closed_connections:
  222. logger.warning(f"Attempted to send message to closed connection for run {run_id}")
  223. return
  224. try:
  225. if run_id in self._connections:
  226. websocket = self._connections[run_id]
  227. await websocket.send_json(message)
  228. except WebSocketDisconnect:
  229. logger.warning(f"WebSocket disconnected while sending message for run {run_id}")
  230. await self.disconnect(run_id)
  231. except Exception as e:
  232. logger.error(f"Error sending message for run {run_id}: {e}, {message}")
  233. # Don't try to send error message here to avoid potential recursive loop
  234. await self._update_run_status(run_id, RunStatus.ERROR, str(e))
  235. await self.disconnect(run_id)
  236. async def _handle_stream_error(self, run_id: UUID, error: Exception) -> None:
  237. """Handle stream errors with proper run updates"""
  238. if run_id not in self._closed_connections:
  239. error_result = TeamResult(
  240. task_result=TaskResult(
  241. messages=[TextMessage(source="system", content=str(error))],
  242. stop_reason="An error occurred while processing this run",
  243. ),
  244. usage="",
  245. duration=0,
  246. ).model_dump()
  247. await self._send_message(
  248. run_id,
  249. {
  250. "type": "completion",
  251. "status": "error",
  252. "data": error_result,
  253. "timestamp": datetime.now(timezone.utc).isoformat(),
  254. },
  255. )
  256. await self._update_run(run_id, RunStatus.ERROR, team_result=error_result, error=str(error))
  257. def _format_message(self, message: Any) -> Optional[dict]:
  258. """Format message for WebSocket transmission
  259. Args:
  260. message: Message to format
  261. Returns:
  262. Optional[dict]: Formatted message or None if formatting fails
  263. """
  264. try:
  265. if isinstance(message, MultiModalMessage):
  266. message_dump = message.model_dump()
  267. message_dump["content"] = [
  268. message_dump["content"][0],
  269. {
  270. "url": f"data:image/png;base64,{message_dump['content'][1]['data']}",
  271. "alt": "WebSurfer Screenshot",
  272. },
  273. ]
  274. return {"type": "message", "data": message_dump}
  275. elif isinstance(message, TeamResult):
  276. return {
  277. "type": "result",
  278. "data": message.model_dump(),
  279. "status": "complete",
  280. }
  281. elif isinstance(
  282. message,
  283. (
  284. TextMessage,
  285. StopMessage,
  286. HandoffMessage,
  287. ToolCallRequestEvent,
  288. ToolCallExecutionEvent,
  289. LLMCallEventMessage,
  290. ),
  291. ):
  292. return {"type": "message", "data": message.model_dump()}
  293. return None
  294. except Exception as e:
  295. logger.error(f"Message formatting error: {e}")
  296. return None
  297. async def _get_run(self, run_id: UUID) -> Optional[Run]:
  298. """Get run from database
  299. Args:
  300. run_id: UUID of the run to retrieve
  301. Returns:
  302. Optional[Run]: Run object if found, None otherwise
  303. """
  304. response = self.db_manager.get(Run, filters={"id": run_id}, return_json=False)
  305. return response.data[0] if response.status and response.data else None
  306. async def _update_run_status(self, run_id: UUID, status: RunStatus, error: Optional[str] = None) -> None:
  307. """Update run status in database
  308. Args:
  309. run_id: UUID of the run to update
  310. status: New status to set
  311. error: Optional error message
  312. """
  313. run = await self._get_run(run_id)
  314. if run:
  315. run.status = status
  316. run.error_message = error
  317. self.db_manager.upsert(run)
  318. async def cleanup(self) -> None:
  319. """Clean up all active connections and resources when server is shutting down"""
  320. logger.info(f"Cleaning up {len(self.active_connections)} active connections")
  321. try:
  322. # First cancel all running tasks
  323. for run_id in self.active_runs.copy():
  324. if run_id in self._cancellation_tokens:
  325. self._cancellation_tokens[run_id].cancel()
  326. run = await self._get_run(run_id)
  327. if run and run.status == RunStatus.ACTIVE:
  328. interrupted_result = TeamResult(
  329. task_result=TaskResult(
  330. messages=[TextMessage(source="system", content="Run interrupted by server shutdown")],
  331. stop_reason="server_shutdown",
  332. ),
  333. usage="",
  334. duration=0,
  335. ).model_dump()
  336. run.status = RunStatus.STOPPED
  337. run.team_result = interrupted_result
  338. self.db_manager.upsert(run)
  339. # Then disconnect all websockets with timeout
  340. # 10 second timeout for entire cleanup
  341. async with asyncio.timeout(10):
  342. for run_id in self.active_connections.copy():
  343. try:
  344. # Give each disconnect operation 2 seconds
  345. async with asyncio.timeout(2):
  346. await self.disconnect(run_id)
  347. except asyncio.TimeoutError:
  348. logger.warning(f"Timeout disconnecting run {run_id}")
  349. except Exception as e:
  350. logger.error(f"Error disconnecting run {run_id}: {e}")
  351. except asyncio.TimeoutError:
  352. logger.warning("WebSocketManager cleanup timed out")
  353. except Exception as e:
  354. logger.error(f"Error during WebSocketManager cleanup: {e}")
  355. finally:
  356. # Always clear internal state, even if cleanup had errors
  357. self._connections.clear()
  358. self._cancellation_tokens.clear()
  359. self._closed_connections.clear()
  360. self._input_responses.clear()
  361. @property
  362. def active_connections(self) -> set[UUID]:
  363. """Get set of active run IDs"""
  364. return set(self._connections.keys()) - self._closed_connections
  365. @property
  366. def active_runs(self) -> set[UUID]:
  367. """Get set of runs with active cancellation tokens"""
  368. return set(self._cancellation_tokens.keys())