<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Currently the way to accomplish RAG behavior with agent chat,
specifically assistant agents is with the memory interface, however
there is no way to configure it via the declarative API.

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

---------

Co-authored-by: Victor Dibia <chuvidi2003@gmail.com>

Just fix typo

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [x] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

<!-- Please give a short summary of the change and the problem this
solves. -->

To allow serialization of OAI Assistant Agent.

## Related issue number

<!-- For example: "Closes #1234" -->

Closes #5130 

## Checks

- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

Closes #5302

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Fix permission issue in ags windows (env files now stored in a user
directory)

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->
Closes #5355
## Checks

- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

Modify M1 agents to support text-only settings.
This allows M1 to be used with models like o3-mini and Llama3.1+

Allow AssistantAgent to drop images when not equipped with a multi-modal model.

Adds a corresponding utility function, which can be used in autogen-ext and teams, to accomplish the same.

This PR does the following: 

- Fix warning messages in AGS on launch.
- Improve Cli message to include app URL on startup from command line
- Minor improvements default gallery generator. (add more default tools)
- Improve new session behaviour.



## Related issue number

Closes #5097

## Checks

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Fix bug where the `model_info` field is not serialized for the
`OpenAIChatCompletionClient` class. This was because the `_raw_config`
field was based on a version of the args that had been sanitized
(model_info removed). We need the full model info field for non-openai
models

```python
from autogen_ext.agents.web_surfer import MultimodalWebSurfer
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_core.models import ModelInfo
mistral_vllm_model = OpenAIChatCompletionClient(
    model="TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
    base_url="http://localhost:1234/v1",
    api_key="empty",
    model_info=ModelInfo(vision=False, function_calling=True, json_output=False, family="unkown"),
)
(mistral_vllm_model.dump_component().model_dump_json())
```

Before
```
{
  "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
  "component_type": "model",
  "version": 1,
  "component_version": 1,
  "description": "Chat completion client for OpenAI hosted models.",
  "label": "OpenAIChatCompletionClient",
  "config": {
    "model": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
    "api_key": "empty",
    "base_url": "http://localhost:1234/v1"
  }
}

```

After
```
{
  "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
  "component_type": "model",
  "version": 1,
  "component_version": 1,
  "description": "Chat completion client for OpenAI hosted models.",
  "label": "OpenAIChatCompletionClient",
  "config": {
    "model": "TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
    "api_key": "empty",
    "model_info": {
      "vision": false,
      "function_calling": true,
      "json_output": false,
      "family": "unkown"
    },
    "base_url": "http://localhost:1234/v1"
  }
}


```
<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

After #5431 some command-line arguments became string rather than lists.
This PR fixes the issue by checking the type.

Allow m1 cli to read a configuration from a yaml file.

## Why are these changes needed?
This pull request introduces the 'o3' model family and adds support for
the 'o3-mini' model.

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

<!-- Please give a short summary of the change and the problem this
solves. -->

We are seeing this issue more often now, probably related to the load on
the API servers. Hence this PR:
1. Demotes the the `max_consecutive_empty_chunk_tolerance` parameter
from function to inline threshold
2. Change exception to a one time warning

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [x] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

---------

Signed-off-by: Mohammad Mazraeh <mazraeh.mohammad@gmail.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

Resolves #5322

This PR adds a method that approximately extracts the text visible in
the viewport of the web browser (as opposed to always printing the first
50 lines, or relying entirely on OCR).

Resolves #5179

Closes #5163
---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

This PR removes the older `autogen_magentic_one` package, and directs
people to use the new AgentChat implementation.

Hopefully this eases confusion.

---------

Co-authored-by: Jack Gerrits <jack@jackgerrits.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

Because.

Resolves #5281

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

This PR updates AGS to use the declarative config serialization native
to AgentChat.
The effect? You can build your teams/artifacts directly in python, run
`team.dump_component()` and immediately run it in AGS.

Some change details:

- Removes ComponentFactory. Instead TeamManager just loads team specs
directly using `Team.load_component`.
- Some fixes to the UI to simplify drag and drop experience.  
- Improve layout of nodes...


<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->
Closes #4439 
Closes #5172

## Checks

- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.


cc @EItanya @nour-bouzid

Resolves #5255 

---------

Co-authored-by: afourney <adamfo@microsoft.com>

Removed the TERMINATE clause from the system prompt since M1 handles
termination via the Orchestrator, and it is just ignored.

Removed the clause about saving to a particular file name, since tmp_
files are created in the current CodeExecutors.

This PR adds a `sources` optional parameter to CodeExecutorAgent
(similar to the termination conditions), that allows finer-grained
control on which agents can provide code for execution.

It also moves the `_extract_markdown_code_blocks` subroutine to a member
method, so that it can be overridden by subclasses. I've found this to
be very important to support benchmarks like HumanEval, where we need to
add a test harness around the implementation.

Partially address #5205 and #5226

Resolves #5248

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

FunctionTool supports passing in a partial

## Related issue number

Closes #5151 

## Checks

- [x] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

Closes #5074

Signed-off-by: Mohammad Mazraeh <mazraeh.mohammad@gmail.com>

Resolves #3983

* introduce `model_client_stream` parameter in `AssistantAgent` to
enable token-level streaming output.
* introduce `ModelClientStreamingChunkEvent` as a type of `AgentEvent`
to pass the streaming chunks to the application via `run_stream` and
`on_messages_stream`. Although this will not affect the inner messages
list in the final `Response` or `TaskResult`.
* handle this new message type in `Console`.

Resolves #5046

Update a minor typo and updated the `response_format` documentation to
the new value

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

Resolves #5043

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

It is currently hard to add a description to a component (defaults to
None also) .. you have to call super.dump() modify and return. This PR
makes the experience better.

- allows you specify `component_description` and `component_label` as an
optional class var. label is an optional human readable name for the the
component.
- will use component_description if provided int he description field
when dumped if there is no description, will use the first line of class
docstring. Takes advantage of all the good practices we have in writing
good docstrings. label defaults to component type.
 

For example 

```python
model_client=OpenAIChatCompletionClient( model="gpt-4o-2024-08-06" )
config = model_client.dump_component()
print(config.model_dump_json())
```
Note the description field below is no longer None and there is a label
```python
{
  "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
  "component_type": "model",
  "version": 1,
  "component_version": 1,
  "description": "Chat completion client for OpenAI hosted models.",
  "label": "OpenAIChatCompletionClient",
  "config": { "model": "gpt-4o-2024-08-06" }
}


```

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->
None, felt faster to fix.

## Checks

- [x] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [x] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [x] I've made sure all auto checks have passed.

Resolves #5222

Typo in Report Agent

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

Partially resolves: #5118

Once the extension page is ready, update the tutorial pages to reduce
duplication.

---------

Co-authored-by: Victor Dibia <victordibia@microsoft.com>

<!-- Thank you for your contribution! Please review
https://microsoft.github.io/autogen/docs/Contribute before opening a
pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?

Make AssistantAgent and Handoff use BaseTool.  
This ensures that they can be made declarative/serialized

<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've included any doc changes needed for
https://microsoft.github.io/autogen/. See
https://microsoft.github.io/autogen/docs/Contribute#documentation to
build and test documentation locally.
- [ ] I've added tests (if relevant) corresponding to the changes
introduced in this PR.
- [ ] I've made sure all auto checks have passed.

* Refactor client id retrieval

* WIP

* fixes

* future annotations

* Fix tests

* remove import

* Rebase to latest main branch

* Moved _azure module to azure

* Validate extra_create_args in and json response

* Added Support for Github Models

* Added normalize_name and assert_valid name

* Added Tests for AzureAIChatCompletionClient

* WIP: Azure AI Client

* Added: object-level usage data
* Added: doc string
* Added: check existing response_format value
* Added: _validate_config and _create_client

* lint

* merge dependencies

* add tests for img and function calling

* support actual tests through env vars

* address mypy errors

* doc example fix

* fmt

* fix doc fmt

* Update python/packages/autogen-ext/src/autogen_ext/models/azure/_azure_ai_client.py

---------

Co-authored-by: Rohan Thacker <thackerrohan4@gmail.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>

Communicate client id via metadata