Resolves #3247

This PR introduces `WorkBench`.

A workbench provides a group of tools that share the same resource and
state. For example, `McpWorkbench` provides the underlying tools on the
MCP server. A workbench allows tools to be managed together and abstract
away the lifecycle of individual tools under a single entity. This makes
it possible to create agents with stateful tools from serializable
configuration (component configs), and it also supports dynamic tools:
tools change after each execution.

Here is how a workbench may be used with AssistantAgent (not included in
this PR):

```python
workbench = McpWorkbench(server_params)
agent = AssistantAgent("assistant", tools=workbench)
result = await agent.run(task="do task...")
```


TODOs:
1. In a subsequent PR, update `AssistantAgent` to use workbench as an
alternative in the `tools` parameter. Use `StaticWorkbench` to manage
individual tools.
2. In another PR, add documentation on workbench.

---------

Co-authored-by: EeS <chiyoung.song@motov.co.kr>
Co-authored-by: Minh Đăng <74671798+perfogic@users.noreply.github.com>

- Added the support Azure AI Agent. The new agent is named AzureAIAgent.
- The agent supports Bing search, file search, and Azure search tools.
- Added a Jupiter notebook to demonstrate the usage of the AzureAIAgent.

## What's missing?
- AzureAIAgent support only text message responses
-  Parallel execution for the custom functions. 



## Related issue number


[5545](https://github.com/microsoft/autogen/issues/5545#event-16626859772)

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

The DockerCommandLineCodeExecutor doesn't currently offer GPU support.
By simply using DeviceRequest from the docker python API, these changes
expose GPUs to the docker container and provide the ability to execute
CUDA-accelerated code within autogen.

## Related issue number

Closes: #6302 

## Checks

- [x] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

## Why are these changes needed?
`convert_tools` failed if Optional args were used in tools (the `type`
field doesn't exist in that case and `anyOf` must be used).

This uses the `anyOf` field to pick the first non-null type to use.  

## Related issue number

Fixes #6323

## Checks

- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

---------

Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

## Why are these changes needed?


I was getting the following exception when doing tool calls with
anthropic - the exception was coming form the `__str__` in
`LLMStreamStartEvent`.

```
('Object of type ToolUseBlock is not JSON serializable',)
```

The issue is that when creating the LLMStreamStartevent in the
`create_stream`, the messages weren't being serialized first.
## Related issue number

Signed-off-by: Peter Jausovec <peter.jausovec@solo.io>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

The current implementation of consecutive `SystemMessage` merging
applies only to models where `model_info.family` starts with
`"gemini-"`.

Since PR #6327 introduced the `multiple_system_messages` field in
`model_info`, we can now generalize this logic by checking whether the
field is explicitly set to `False`.

This change replaces the hardcoded family check with a conditional that
merges consecutive `SystemMessage` blocks whenever
`multiple_system_messages` is set to `False`.

Test cases that previously depended on the `"gemini"` model family have
been updated to reflect this configuration flag, and renamed accordingly
for clarity.

In addition, for consistency across conditional logic, a follow-up PR is
planned to refactor the Claude-specific transformation condition
(currently implemented via `create_args.get("model",
"unknown").startswith("claude-")`)
to instead use the existing `is_claude()`.

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This is an initial exploration of what could be a solution for #6214 .
It implements a simple text canvas using difflib and also a memory
component and a tool component for interacting with the canvas. Still in
early testing but would love feedback on the design.

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

---------

Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

## Why are these changes needed?
`SocietyOfMindAgent` has multiple system message, however many
client/model does not support it.

## Related issue number
Related #6290

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

This PR introduces a safer and more controllable execution environment
for LLM code execution in version 0.4 by enabling the use of Jupyter
inside a container. This enhancement addresses security concerns and
provides a more robust execution context. In particular, it allows:

Isolation of code execution via containerized Jupyter environments.

Persistent memory of variables and their values throughout the
conversation.

Memory of code execution results to support more advanced reasoning and
follow-up tasks.

These improvements help build a more interactive and stateful LLM-agent
programming experience, especially for iterative code generation and
debugging scenarios.

## Related issue number

Open #6153

## Checks

- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

## Why are these changes needed?
This PR fixes a bug where the underlying azure `SearchClient` was being
closed prematurely due to use of `async with client` : inside the tool's
run method. this caused the users to encounter errors "HTTP transport
has already been closed"

## Related issue number

Closes #6308 "

## Checks

- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [X] I've made sure all auto checks have passed.

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

Resolves #6232, #6198

This PR introduces an optional parameter `session` to `mcp_server_tools`
to support reuse of the same session.

```python
import asyncio

from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.conditions import TextMentionTermination
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.ui import Console
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_ext.tools.mcp import StdioServerParams, create_mcp_server_session, mcp_server_tools


async def main() -> None:
    model_client = OpenAIChatCompletionClient(model="gpt-4o", parallel_tool_calls=False)  # type: ignore
    params = StdioServerParams(
        command="npx",
        args=["@playwright/mcp@latest"],
        read_timeout_seconds=60,
    )
    async with create_mcp_server_session(params) as session:
        await session.initialize()
        tools = await mcp_server_tools(server_params=params, session=session)
        print(f"Tools: {[tool.name for tool in tools]}")

        agent = AssistantAgent(
            name="Assistant",
            model_client=model_client,
            tools=tools,  # type: ignore
        )

        termination = TextMentionTermination("TERMINATE")
        team = RoundRobinGroupChat([agent], termination_condition=termination)
        await Console(
            team.run_stream(
                task="Go to https://ekzhu.com/, visit the first link in the page, then tell me about the linked page."
            )
        )


asyncio.run(main())
``` 

Based on discussion in this thread: #6284, we will consider
serialization and deserialization of MCP server tools when used in this
manner in a separate issue.

This PR also replaces the `json_schema_to_pydantic` dependency with
built-in utils.

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

An app can pass untyped dicts to set configuration options of various
Task-Centric Memory classes. But tools like pyright can complain about
the loose typing. This PR exposes 4 TypedDict classes that apps can
optionally use.

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [x] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

`IncludeEnum` was removed in ChromaDB when it was updated to `1.0.0`.
This caused issues when using `ChromaDBVectorMemory`. This PR fixes
those issues

## Related issue number

Closes #6241

## Checks

- [x] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

---------

Co-authored-by: Victor Dibia <victordibia@microsoft.com>

## Why are these changes needed?

bug fix : add get_embedding() implementation

## Related issue number

"Closes #6240 " -->

## Checks

- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [X] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [X] I've made sure all auto checks have passed.

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

- Exposes a few optional memory controller parameters for more detailed
control and evaluation.
- Fixes a couple formatting issues in the documentation.

## Related issue number

None

## Checks

- [x] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

Resolves #6152

This pull request introduces a new feature to the
`DockerCommandLineCodeExecutor` class, which allows temporary files
generated by code execution to be deleted after code execution. The most
important changes include adding a new configuration option, updating
the class to handle this option, and adding tests to verify the new
functionality.

### New Feature: Temporary File Deletion

*
[`python/packages/autogen-ext/src/autogen_ext/code_executors/docker/_docker_code_executor.py`](diffhunk://#diff-8ef47c21141ed8b0a757b0e6f9d1491561fc31684756d22ed0253edbfcfcdf91R81):
Added `delete_tmp_files` attribute to the
`DockerCommandLineCodeExecutorConfig` class and updated the
`DockerCommandLineCodeExecutor` class to handle this attribute. This
includes initializing the attribute, adding it to the configuration
methods, and implementing the file deletion logic in the
`_execute_code_dont_check_setup` method.
[[1]](diffhunk://#diff-8ef47c21141ed8b0a757b0e6f9d1491561fc31684756d22ed0253edbfcfcdf91R81)
[[2]](diffhunk://#diff-8ef47c21141ed8b0a757b0e6f9d1491561fc31684756d22ed0253edbfcfcdf91R128)
[[3]](diffhunk://#diff-8ef47c21141ed8b0a757b0e6f9d1491561fc31684756d22ed0253edbfcfcdf91R177)
[[4]](diffhunk://#diff-8ef47c21141ed8b0a757b0e6f9d1491561fc31684756d22ed0253edbfcfcdf91R231)
[[5]](diffhunk://#diff-8ef47c21141ed8b0a757b0e6f9d1491561fc31684756d22ed0253edbfcfcdf91R318)
[[6]](diffhunk://#diff-8ef47c21141ed8b0a757b0e6f9d1491561fc31684756d22ed0253edbfcfcdf91R346-R352)
[[7]](diffhunk://#diff-8ef47c21141ed8b0a757b0e6f9d1491561fc31684756d22ed0253edbfcfcdf91R527)
[[8]](diffhunk://#diff-8ef47c21141ed8b0a757b0e6f9d1491561fc31684756d22ed0253edbfcfcdf91R547)

### Testing

*
[`python/packages/autogen-ext/tests/code_executors/test_docker_commandline_code_executor.py`](diffhunk://#diff-635dbdcdeca161e620283399d5cd43ca756ec0f88d4429f059ee4f6b346874e4R318-R363):
Added a new test `test_delete_tmp_files` to verify the behavior of the
`delete_tmp_files` attribute. This test checks that temporary files are
correctly deleted or retained based on the configuration.<!-- Thank you
for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

This PR improves fallback safety when an invalid `model_family` is
supplied to `get_transformer()`. Previously, if a user passed an
arbitrary or incorrect `family` string in `model_info`, the lookup could
fail without falling back to `ModelFamily.UNKNOWN`.

Now, we explicitly check whether `model_family` is a valid value in
`ModelFamily.ANY`. If not, we fallback to `_find_model_family()` as
intended.


## Related issue number

Related #6011#issuecomment-2779957730

## Checks

- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

Fix a bug that caused tool calls to be truncated in
OllamaChatCompletionClient when streaming is on.

Messages sent as part of `LLMCallEvent` for Anthropic were not fully serializable
The example below shows TextBlock and ToolUseBlocks inside the content of messages - these throw downsteam errors in apps like AGS (or event sinks) that expect serializable dicts inside the LLMCallEvent.
```
[
{'role': 'user', 'content': 'What is the weather in New York?'}, 
{'role': 'assistant', 'content': [TextBlock(citations=None, text='I can help you find the weather in New York. Let me check that for you.', type='text'), ToolUseBlock(id='toolu_016W8g55GejYGBzRRrcsnt7M', input={'city': 'New York'}, name='get_weather', type='tool_use')]}, 
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_016W8g55GejYGBzRRrcsnt7M', 'content': 'The weather in New York is 73 degrees and Sunny.'}]}
]


```
This PR attempts to first serialize content of anthropic messages before they are passed to `LLMCallEvent`

```
[
{'role': 'user', 'content': 'What is the weather in New York?'}, 
{'role': 'assistant', 'content': [{'citations': None, 'text': 'I can help you find the weather in New York. Let me check that for you.', 'type': 'text'}, {'id': 'toolu_016W8g55GejYGBzRRrcsnt7M', 'input': {'city': 'New York'}, 'name': 'get_weather', 'type': 'tool_use'}]}, 
{'role': 'user', 'content': [{'type': 'tool_result', 'tool_use_id': 'toolu_016W8g55GejYGBzRRrcsnt7M', 'content': 'The weather in New York is 73 degrees and Sunny.'}]}
]

```

# Azure AI Search Tool Implementation

This PR adds a new tool for Azure AI Search integration to autogen-ext,
enabling agents to search and retrieve information from Azure AI Search
indexes.

## Why Are These Changes Needed?
AutoGen currently lacks native integration with Azure AI Search, which
is a powerful enterprise search service that supports semantic, vector,
and hybrid search capabilities. This integration enables agents to:
1. Retrieve relevant information from large document collections
2. Perform semantic search with AI-powered ranking
3. Execute vector similarity search using embeddings
4. Combine text and vector approaches for optimal results

This tool complements existing retrieval capabilities and provides a
seamless way to integrate with Azure's search infrastructure.

## Features
- **Multiple Search Types**: Support for text, semantic, vector, and
hybrid search
- **Flexible Configuration**: Customizable search parameters and fields
- **Robust Error Handling**: User-friendly error messages with
actionable guidance
- **Performance Optimizations**: Configurable caching and retry
mechanisms
- **Vector Search Support**: Built-in embedding generation with
extensibility

## Usage Example
```python
from autogen_ext.tools.azure import AzureAISearchTool
from azure.core.credentials import AzureKeyCredential
from autogen import AssistantAgent, UserProxyAgent
# Create the search tool
search_tool = AzureAISearchTool.load_component({
   "provider": "autogen_ext.tools.azure.AzureAISearchTool",
   "config": {
       "name": "DocumentSearch",
       "description": "Search for information in the knowledge base",
       "endpoint": "https://your-service.search.windows.net",
       "index_name": "your-index",
       "credential": {"api_key": "your-api-key"},
       "query_type": "semantic",
       "semantic_config_name": "default"
   }
})
# Create an agent with the search tool
assistant = AssistantAgent(
   "assistant",
   llm_config={"tools": [search_tool]}
)
# Create a user proxy agent
user_proxy = UserProxyAgent(
   "user_proxy",
   human_input_mode="TERMINATE",
   max_consecutive_auto_reply=10,
   code_execution_config={"work_dir": "coding"}
)
# Start the conversation
user_proxy.initiate_chat(
   assistant,
   message="What information do we have about quantum computing in our knowledge base?"
)
```

## Testing
- Added unit tests for all search types (text, semantic, vector, hybrid)
- Added tests for error handling and cancellation
- All tests pass locally

## Documentation
- Added comprehensive docstrings with examples
- Included warnings about placeholder embedding implementation
- Added links to Azure AI Search documentation

## Related issue number

Closes #5419 

## Checks

- [x] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

This PR improves how model_family is resolved when selecting a
transformer from the registry.
Previously, model families were inferred using a simple prefix-based
match like:
```
if model.startswith(family): ...
```
This works for cleanly prefixed models (e.g., `gpt-4o`, `claude-3`) but
fails for models like `mistral-large-latest`, `codestral-latest`, etc.,
where prefix-based matching is ambiguous or misleading.

To address this:
	•	model_family can now be passed explicitly (e.g., via ModelInfo)
• _find_model_family() is only used as a fallback when the value is
"unknown"
	•	Transformer lookup is now more robust and predictable
• Example integration in to_oai_type() demonstrates this pattern using
self._model_info["family"]

This change is required for safe support of models like Mistral and
other future models that do not follow standard naming conventions.

Linked to discussion in
[#6151](https://github.com/microsoft/autogen/issues/6151)
Related : #6011

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

The initializer for ACADynamicSessionsCodeExecutor creates a new GUID to
use as the session ID for dynamic sessions.

In some scenarios it is desirable to be able to re-create the agent
group chat from saved state. In this case, the
ACADynamicSessionsCodeExecutor needs to be associated with a previous
instance (so that any execution state is still valid)

This PR adds a new argument to the initializer to allow a session ID to
be passed in (defaulting to the current behaviour of creating a GUID if
absent).

Closes #6119

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

## Why are these changes needed?

This PR fixes a `400 - invalid_request_error` that occurs when using
Anthropic models and the **final message is from the assistant and ends
with trailing whitespace**.

Example error:

```
Error code: 400 - {'error': {'code': 'invalid_request_error', 'message': 'messages: final assistant content cannot end with trailing whitespace', ...}}
```

To unblock ongoing internal usage, this patch introduces an **ad-hoc
fix** that strips trailing whitespace if the model is Anthropic and the
last message is from the assistant.

## Related issue number

Ad-hoc fix for issue discussed here:  
https://github.com/microsoft/autogen/issues/6167

Follow-up structural proposal here:  
https://github.com/microsoft/autogen/issues/6167
https://github.com/microsoft/autogen/issues/6167#issuecomment-2768592840

## Why are these changes needed?

Changed default working directory of code executors, from the current
directory `"."` to Python's
[`tempfile`](https://docs.python.org/3/library/tempfile.html#tempfile.TemporaryDirectory).
These changes simplify file cleanup and prevent the model from accessing
code files or other sensitive data that should not be accessible.
These changes simplify file cleanup and prevent the model from accessing
code files or other sensitive data that should not be accessible.

Changes made:
- The default `work_dir` parameter in code executors is changed to
`None`; when invoking the `start()` method, if not `work_dir` was
specified (`None`) a temporary directory is created.
- The `start()` and `stop()` methods of code executors handle the
creation and cleanup of the working directory, for the default temporary
directory.
- For maintaining backward compatibility:
- A `DeprecationWarning` is emitted when the current dir, `"."`, is used
as `work_dir` as it is in the current code executor implementation. The
deprecation warning is tested in `test_deprecated_warning()`.
- For existing implementation that do not call the `start()` method and
do not specify a `work_dir`, the executors will continue using the
current directory `"."` as the working directory, mantaining backward
compatibility.
- Updated test suites:
- Added tests to confirm that by default code executors use a temporary
directory as their working directory: `test_default_work_dir_is_temp()`;
- Implemented test to ensure that a `DeprecationWarning` is raised when
the current directory is used as the default directory:
`test_deprecated_warning()`;
- Added tests to ensure that errors arise when invalid paths (doesn't
exist or user has not the right permissions) are provided:
`test_error_wrong_path()`.

Feel free to suggest any additions or improvements!

## Related issue number

Close #6041 

## Checks

- [x] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

This PR adds a module-level docstring to `_message_transform.py`, as
requested in the review for [PR
#6063](https://github.com/microsoft/autogen/pull/6063).

The documentation includes:
- Background and motivation behind the modular transformer design
- Key concepts such as transformer functions, pipelines, and maps
- Examples of how to define, register, and use transformers
- Design principles to guide future contributions and extensions

By embedding this explanation directly into the module, contributors and
maintainers can more easily understand the structure, purpose, and usage
of the transformer pipeline without needing to refer to external
documents.

## Related issue number

Follow-up to [PR #6063](https://github.com/microsoft/autogen/pull/6063)

## Why are these changes needed?
This change addresses a compatibility issue when using Google Gemini
models with AutoGen. Specifically, Gemini returns a 400 INVALID_ARGUMENT
error when receiving a response with an empty "text" parameter.

The root cause is that Gemini does not accept empty string values (e.g.,
"") as valid inputs in the history of the conversation.

To fix this, if the content field is falsy (e.g., None, "", etc.), it is
explicitly replaced with a single whitespace (" "), which prevents the
Gemini model from rejecting the request.

- **Gemini API compatibility:** Gemini models reject empty assistant
messages (e.g., `""`), causing runtime errors. This PR ensures such
messages are safely replaced with whitespace where appropriate.
- **Avoiding regressions:** Applying the empty content workaround **only
to Gemini**, and **only to valid message types**, avoids breaking OpenAI
or other models.
- **Reducing duplication:** Previously, message transformation logic was
scattered and repeated across different message types and models.
Modularizing this pipeline removes that redundancy.
- **Improved maintainability:** With future model variants likely to
introduce more constraints, this modular structure makes it easier to
adapt transformations without writing ad-hoc code each time.
- **Testing for correctness:** The new structure is verified with tests,
ensuring the bug fix is effective and non-intrusive.

## Summary

This PR introduces a **modular transformer pipeline** for message
conversion and **fixes a Gemini-specific bug** related to empty
assistant message content.

### Key Changes

- **[Refactor]** Extracted message transformation logic into a unified
pipeline to:
  - Reduce code duplication
  - Improve maintainability
  - Simplify debugging and extension for future model-specific logic

- **[BugFix]** Gemini models do not accept empty assistant message
content.
- Introduced `_set_empty_to_whitespace` transformer to replace empty
strings with `" "` only where needed
- Applied it **only** to `"text"` and `"thought"` message types, not to
`"tools"` to avoid serialization errors

- **Improved structure for model-specific handling**
- Transformer functions are now grouped and conditionally applied based
on message type and model family
- This design makes it easier to support future models or combinations
(e.g., Gemini + R1)

- **Test coverage added**
- Added dedicated tests to verify that empty assistant content causes
errors for Gemini
  - Ensured the fix resolves the issue without affecting OpenAI models

---

## Motivation

Originally, Gemini-compatible endpoints would fail when receiving
assistant messages with empty content (`""`).
This issue required special handling without introducing brittle, ad-hoc
patches.

In addressing this, I also saw an opportunity to **modularize** the
message transformation logic across models.
This improves clarity, avoids duplication, and simplifies future
adaptations (e.g., different constraints across model families).

---


## 📘 AutoGen Modular Message Transformer: Design & Usage Guide

This document introduces the **new modular transformer system** used in
AutoGen for converting `LLMMessage` instances to SDK-specific message
formats (e.g., OpenAI-style `ChatCompletionMessageParam`).
The design improves **reusability, extensibility**, and
**maintainability** across different model families.

---

### 🚀 Overview

Instead of scattering model-specific message conversion logic across the
codebase, the new design introduces:

- Modular transformer **functions** for each message type
- Per-model **transformer maps** (e.g., for OpenAI-compatible models)
- Optional **conditional transformers** for multimodal/text hybrid
models
- Clear separation between **message adaptation logic** and
**SDK-specific builder** (e.g., `ChatCompletionUserMessageParam`)

---

### 🧱 1. Define Transform Functions

Each transformer function takes:
- `LLMMessage`: a structured AutoGen message
- `context: dict`: metadata passed through the builder pipeline

And returns:
- A dictionary of keyword arguments for the target message constructor
(e.g., `{"content": ..., "name": ..., "role": ...}`)

```python
def _set_thought_as_content_gemini(message: LLMMessage, context: Dict[str, Any]) -> Dict[str, str | None]:
    assert isinstance(message, AssistantMessage)
    return {"content": message.thought or " "}
```

---

### 🪢 2. Compose Transformer Pipelines

Multiple transformer functions are composed into a pipeline using
`build_transformer_func()`:

```python
base_user_transformer_funcs: List[Callable[[LLMMessage, Dict[str, Any]], Dict[str, Any]]] = [
    _assert_valid_name,
    _set_name,
    _set_role("user"),
]

user_transformer = build_transformer_func(
    funcs=base_user_transformer_funcs,
    message_param_func=ChatCompletionUserMessageParam
)
```

- The `message_param_func` is the actual constructor for the target
message class (usually from the SDK).
- The pipeline is **ordered** — each function adds or overrides keys in
the builder kwargs.

---

### 🗂️ 3. Register Transformer Map

Each model family maintains a `TransformerMap`, which maps `LLMMessage`
types to transformers:

```python
__BASE_TRANSFORMER_MAP: TransformerMap = {
    SystemMessage: system_transformer,
    UserMessage: user_transformer,
    AssistantMessage: assistant_transformer,
}

register_transformer("openai", model_name_or_family, __BASE_TRANSFORMER_MAP)
```

- `"openai"` is currently required (as only OpenAI-compatible format is
supported now).
- Registration ensures AutoGen knows how to transform each message type
for that model.

---

### 🔁 4. Conditional Transformers (Optional)

When message construction depends on runtime conditions (e.g., `"text"`
vs. `"multimodal"`), use:

```python
conditional_transformer = build_conditional_transformer_func(
    funcs_map=user_transformer_funcs_claude,
    message_param_func_map=user_transformer_constructors,
    condition_func=user_condition,
)
```

Where:

- `funcs_map`: maps condition label → list of transformer functions
```python
user_transformer_funcs_claude = {
    "text": text_transformers + [_set_empty_to_whitespace],
    "multimodal": multimodal_transformers + [_set_empty_to_whitespace],
}
```

- `message_param_func_map`: maps condition label → message builder
```python
user_transformer_constructors = {
    "text": ChatCompletionUserMessageParam,
    "multimodal": ChatCompletionUserMessageParam,
}
```

- `condition_func`: determines which transformer to apply at runtime
```python
def user_condition(message: LLMMessage, context: Dict[str, Any]) -> str:
    if isinstance(message.content, str):
        return "text"
    return "multimodal"
```

---

### 🧪 Example Flow

```python
llm_message = AssistantMessage(name="a", thought="let’s go")
model_family = "openai"
model_name = "claude-3-opus"

transformer = get_transformer(model_family, model_name, type(llm_message))
sdk_message = transformer(llm_message, context={})
```

---

### 🎯 Design Benefits

| Feature | Benefit |
|--------|---------|
| 🧱 Function-based modular design | Easy to compose and test |
| 🧩 Per-model registry | Clean separation across model families |
| ⚖️ Conditional support | Allows multimodal / dynamic adaptation |
| 🔄 Reuse-friendly | Shared logic (e.g., `_set_name`) is DRY |
| 📦 SDK-specific | Keeps message adaptation aligned to builder interface
|

---

### 🔮 Future Direction

- Support more SDKs and formats by introducing new message_param_func
- Global registry integration (currently `"openai"`-scoped)
- Class-based transformer variant if complexity grows



---

## Related issue number
Closes #5762

## Checks

- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ v ] I've made sure all auto checks have passed.

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

Rename the `ChatMessage` and `AgentEvent` base classes to `BaseChatMessage` and `BaseAgentEvent`. 

Bring back the `ChatMessage` and `AgentEvent` as union of built-in concrete types to avoid breaking existing applications that depends on Pydantic serialization. 

Why?

Many existing code uses containers like this:

```python
class AppMessage(BaseModel):
   name: str
   message: ChatMessage 

# Serialization is this:
m = AppMessage(...)
m.model_dump_json()

# Fields like HandoffMessage.target will be lost because it is now treated as a base class without content or target fields.
```

The assumption on `ChatMessage` or `AgentEvent` to be a union of concrete types could be in many existing code bases. So this PR brings back the union types, while keep method type hints such as those on `on_messages` to use the `BaseChatMessage` and `BaseAgentEvent` base classes for flexibility.

Anthropic SDK could not takes multiple system messages.
However some autogen Agent(e.g. SocietyOfMindAgent) makes multiple
system messages.

And... Gemini with OpenaiSDK do not take error. However is not working
mulitple system messages.
(Just last one is working)

So, I simple change of, "merge multiple system message" at these cases.

## Related issue number
Closes #6116
Closes #6117


---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

When using the `ACADynamicSessionsCodeExecutor` it includes the stdout
from the execution but also the `results` property from the call to
dynamic sessions. In some situations, when the executed code results in
a file being saved this is included in the result:

```console
Plot saved as 'results_by_date.png'
{'type': 'image', 'format': 'png', 'base64_data': 'iVBORw0KGgoAAAANSUhEUgAAA90AAAJOCAYAAACqS2TfAAAAOXRFWHRTb2Z0d2FyZQ...
```

In some situations, this additional output is not desirable:
- when displaying the code output to a user - in this case, the stdout
content is dwarfed by the base64 encoded file content
- when an LLM agent is going to evaluate the code output to determine
next steps - in this case, the base64 content will be included in the
message history sent to the LLM increasing the prompt token cost

To handle these cases, this PR adds a new (optional) argument to the
`ACADynamicSessionsCodeExecutor` constructor that would allow
suppressing the result content (but default to False to preserve the
current behaviour in the default case)

(from #6042)
Closes #6042 


Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

This PR adds missing model entries for OpenAI-compatible endpoints,
including gpt-4.5-turbo, gpt-4.5-turbo-preview, and claude-3.5-sonnet.
This improves coverage and avoids potential fallback or mismatch issues
when initializing clients.

Resolves #6096

Additionally: make sure MCP errors are formatted correctly, added unit
tests for mcp servers and upgrade mcp version.

This PR refactored `AgentEvent` and `ChatMessage` union types to
abstract base classes. This allows for user-defined message types that
subclass one of the base classes to be used in AgentChat.

To support a unified interface for working with the messages, the base
classes added abstract methods for:
- Convert content to string
- Convert content to a `UserMessage` for model client
- Convert content for rendering in console.
- Dump into a dictionary
- Load and create a new instance from a dictionary

This way, all agents such as `AssistantAgent` and `SocietyOfMindAgent`
can utilize the unified interface to work with any built-in and
user-defined message type.

This PR also introduces a new message type, `StructuredMessage` for
AgentChat (Resolves #5131), which is a generic type that requires a
user-specified content type.

You can create a `StructuredMessage` as follow:

```python

class MessageType(BaseModel):
  data: str
  references: List[str]

message = StructuredMessage[MessageType](content=MessageType(data="data", references=["a", "b"]), source="user")

# message.content is of type `MessageType`. 
```

This PR addresses the receving side of this message type. To produce
this message type from `AssistantAgent`, the work continue in #5934.

Added unit tests to verify this message type works with agents and
teams.

Closes #6031 

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

So the behavior of hosted R1 model is the same as locally hosted R1 model.
Addresses: #5989
---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->



<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Add utf encoding to file reading. 
Without this, a default system encoding will be used. On Windows
machines this can default to any local encoding causing errors.

```python
with open(
            os.path.join(os.path.abspath(os.path.dirname(__file__)), "page_script.js"), "rt", encoding="utf-8"
        ) as fh:
```

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

Closes #6093


## Checks

- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

added support for the thought process in tool calls for
`OpenAIChatCompletionClient`, allowing additional text produced by a
model alongside tool calls to be preserved in the thought field of
`CreateResult`. This PR extends the same functionality to
`AzureAIChatCompletionClient` for consistency across model clients.

#5650
Co-authored-by: Jay Prakash Thakur <jathakur@microsoft.com>

## Why are these changes needed?
This PR fixes a `TypeError: Cannot instantiate typing.Union` that occurs
when using the `MultimodalWebSurfer_agent` with Anthropic models. The
error was caused by the incorrect usage of `typing.Union` as a class
constructor instead of a type hint within the `_anthropic_client.py`
file. The code was attempting to instantiate `typing.Union`, which is
not allowed. The fix correctly uses `typing.Union` within type hints,
and uses the correct `Base64ImageSourceParam` type. It also updates the
`pyproject.toml` dependency.

## Related issue number
Closes #6035 

## Checks

- [ ] I've included any doc changes needed for
<https://microsoft.github.io/autogen/>. See
<https://github.com/microsoft/autogen/blob/main/CONTRIBUTING.md> to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [v] I've made sure all auto checks have passed.

---------

Co-authored-by: Victor Dibia <victordibia@microsoft.com>

Fix error:

TypeError: Input stream must be opened in bytes mode, not in text mode.

Markdown converter takes binary stream

Resolves #6015

Optionally limit what files and folders FileSurfer can access
(constraining it to a subtree of the FS).

This is not a replacement for Docker sandboxing, but can be used in
conjunction with sandboxing to help prevent FileSurfer from accessing
sensitive files.