* Re-commit of code from PR (#2167) addressing #1861, due to wrong basing

* Update website/docs/topics/non-openai-models/best-tips-for-nonopenai-models.md

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Removed unnecessary notebook images

* Update conversation-patterns.ipynb

Updated to include note about being applicable when auto.

* Updated to include checks that the role is not blank/None. Added tests.

* Changed try-except to use pytest

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* multimodal carryover

* adds mm carryover tests

* more tests + cleanup code

* check content instead

* beibin suggestion

* cleanup

* fix async

* use deepcopy

* handle carryover method

* remove content copy

* sonichi suggestions

---------

Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* wip

* Adds docstrings

* fixed spellings

* wip

* fixed errors

* better class names

* adds tests

* added tests to workflow

* improved token counting

* improved notebook

* improved token counting in test

* improved docstrings

* fix inconsistencies

* changed by mistake

* fixed docstring

* fixed details

* improves tests + adds openai contrib test

* fix spelling oai contrib test

* clearer docstrings

* remove repeated docstr

* improved notebook

* adds metadata to notebook

* Improve outline and description (#2125)

* better dir structure

* clip max tokens to allowed tokens

* more accurate comments/docstrs

* add deperecation warning

* fix front matter desc

* add deperecation warning notebook

* undo local notebook settings changes

* format notebook

* format workflow

---------

Co-authored-by: gagb <gagb@users.noreply.github.com>

* Mark cache as a protocl and update type hints to reflect

* int

* undo init change

* Introducing IOStream

* bug fixing

* polishing

* refactoring

* refactoring

* refactoring

* wip: async tests

* websockets added

* wip

* merge with main

* notebook added

* FastAPI example added

* wip

* merge

* getter/setter to iostream added

* website/blog/2024-03-03-AutoGen-Update/img/dalle_gpt4v.png: convert to Git LFS

* website/blog/2024-03-03-AutoGen-Update/img/gaia.png: convert to Git LFS

* website/blog/2024-03-03-AutoGen-Update/img/teach.png: convert to Git LFS

* add SSL support

* wip

* wip

* exception handling added to on_connect()

* refactoring: default iostream is being set in a context manager

* test fix

* polishing

* polishing

* polishing

* fixed bug with new thread

* polishing

* a bit of refactoring and docs added

* notebook added to docs

* type checking added to CI

* CI fix

* CI fix

* CI fix

* polishing

* obsolete todo comment removed

* fixed precommit error

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* draft agent optimizer

* refactor

* remove

* change openai config interface

* notebook

* update blog

* add test

* clean up

* redir

* update

* update interface

* change model name

* move to contrib

* Update autogen/agentchat/contrib/agent_optimizer.py

Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com>

---------

Co-authored-by: “skzhang1” <“shaokunzhang529@gmail.com”>
Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com>
Co-authored-by: Jieyu Zhang <jieyuz2@cs.washington.edu>
Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com>

* tried implementing my own regex

* improves tests

* finally works

* removes prints

* fixed test

* adds start and end

* delete unused imports

* refactored to use new tool

* significantly improved algo

* tag content -> tag attr

* fix tests + adds new field

* return full match

* return remove start and end

* update docstrings

* update docstrings

* update docstrings

---------

Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* context to kwargs

* add tag

* add test

* text to kwargs

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* Add vision capability

* Configurate: description_prompt

* Print warning instead of raising issues for type

* Skip vision capability test if dependencies not installed

* Append "vision" to agent's system message when enabled VisionCapability

* GPT-4V notebook update with ConversableAgent

* Clean GPT-4V notebook

* Add vision capability test to workflow

* Lint import

* Update system message for vision capability

* Add a `custom_caption_func` to VisionCapability

* Add custom function example for vision capability

* Skip test Vision capability custom func

* GPT-4V notebook metadata to website

* Remove redundant files

* The custom caption function takes more inputs now

* Add a more complex example of custom caption func

* Remove trailing space

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* Encapsulate colored into a module

* lint fix

* add missing file

* undo change

* conform with original colored func

* change import strategy

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* fix test_human_input

* shorten tests

* terminate chat early

* limit max turns

* limit max_rounds

* check skip_openai

* update pre-commit

* update pre-commit.ci

* lint fix

* adjust order of side bars

* doc improvement

* limit max turns

* improve tests

* improve doc

* improve docstr

* fix test

* Update test/agentchat/test_agent_logging.py

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* add assistant config

* add test

* change notebook to use assistant config

* use assistant config in testing

* code refinement

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* allow sender in nested chat

* description

* update example list

* doc format

* meta data

* add test

* adds image generation capability

* add todo

* readded cache

* wip

* fix content str bugs

* removed todo: delete imshow

* wip

* fix circular imports

* add notebook

* improve prompt

* improved text analyzer + notebook

* notebook update

* improve notebook

* smaller notebook size

* made changes to the wrong branch :(

* resolve comments + 1

* adds doc strings

* adds cache doc string

* adds doc string to add_to_agent

* adds doc string to ImageGeneration

* instructions are not configurable

* removed unnecessary imports

* changed doc string location

* more doc strings

* improves testability

* adds tests

* adds cache test

* added test to github workflow

* compatible llm config format

* configurable reply function position

* skip_openai + better comments

* fix test

* fix test?

* please fix test?

* last fix test?

* remove type hint

* skip cache test

* adds mock api key

* dalle-2 test

* fix dalle config

* use apu key function

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* fix type and default value of the code_execution_config of UserProxAgent

* fix type and default value of the code_execution_config of UserProxAgent

* set default value of llm_config in UserProxyAgent to None

* fixed tests

* revert llm_config to False

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* improve validation of llm_config

* fixed test_register_for_llm_without_LLM

* docstr about llm_config=None

* Make None a sentinel

* pop tools

---------

Co-authored-by: Davor Runje <davor@airt.ai>

* add message field

* send

* message func doc str

* test dict message

* retiring soon

* generate_init_message docstr

* remove todo

* update notebook

* CompressibleAgent

* update notebook

* add test

* retrieve agent

* update test

* summary_method args

* summary

* carryover

* dict message

* update nested doc

* generate_init_message

* fix typo

* update docs for mathchat

* Fix missing message

* Add docstrings

* model

* notebook

* default naming

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>
Co-authored-by: kevin666aa <yrwu000627@gmail.com>
Co-authored-by: Li Jiang <bnujli@gmail.com>
Co-authored-by: Li Jiang <lijiang1@microsoft.com>

* work in progress

* wip

* groupchat type hints

* clean up

* formatting

* formatting

* clean up

* address comments

* better comment

* updates docstring a_send

* resolve comments

* agent.py back to original format

* resolve more comments

* rename carryover type exception

* revert next_agent changes + keeping UndefinedNextagent

* fixed ciruclar dependencies?

* fix cache tests

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* init PR

* update

* update code check

* update

* update

* update

* update

* Test the ability to have agents a,u,t,o,g,e,n speak in turn.

* update

* update

* update

* Evidence that groupchat not terminating because of the TERMINATE substring.

* Raising NoEligibleSpeakerException allows graceful exit before max turns

* update

* To confirm with author that custom function is meant to override graph constraints

* Confirmed the expected test behaviour with author

* Update autogen/agentchat/groupchat.py

* update

* update

---------

Co-authored-by: Joshua Kim <Joshua@spectdata.com>
Co-authored-by: Qingyun Wu <qingyun0327@gmail.com>

* add doc about effects for capabilities

* remove unnecessary imports

* improve doc

* test

* test location

* polish

* improve import of colored

* termcolor

* termcolor

* use pull request for openai test

* resolved errors happening when using function calling and clear history

* checking in nr_of_messages_to_preserve were provided

* code formatting

* test added, dict signature improved

* test added, dict signature improved

* test added, dict signature improved

* test added, dict signature improved

* test added, dict signature improved

* test added, dict signature improved

* test added, dict signature improved

* test added, dict signature improved

* Test updated

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* test improved

* test improved

* comment about preserving additional message added

* commentary about clear history called in tool response improved

* created test for clear hisotry called from tool response

* code formatting

* added 'USER INTERRUPTED' as internal content of tool response

* added separate vatiable 'nr_messages_to_preserve_internal'

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* Add headless browser to the WebSurferAgent, closes #1481

* replace soup.get_text() with markdownify.MarkdownConverter().convert_soup(soup)

* import HeadlessChromeBrowser

* implicitly wait for 10s

* inicrease max. wait time to 99s

* fix: trim trailing whitespace

* test: fix headless tests

* better bing query search

* docs: add example 3 for headless option

---------

Co-authored-by: Vijay Ramesh <vijay@regrello.com>

* remove default trigger value for register_nested_chats

* update notebook and test

Tests that contain `if __name__ == "__main__":` now have a shebang line
and execute permission.

* Change defualt model for `lmm`

* Try to use PIL image for LMM's _oai_messages

* Update test cases and llava

* Remove redundant files

* Update the imports for lmm tests

* Test case fix

* Docstring update

* LMM notebook lint

* Typo correction for img_utils and its test

* Fix self.client bug for Dalle Agent
The self.client member variable is reserved for OpenAI's LLM-compatible
client, which requires usage summary support.
So, we rename self.client to self._dalle_client to resolve this issue

* hook-list are updated to accept a sender argument

* Update the hook function Signature & add assertions for Sender

* fix

* update

* reformat

---------

Co-authored-by: “skzhang1” <“shaokunzhang529@gmail.com”>

* add util

* add notebook

* SoM notebook

* doc

* update notebook

* SoM

* optiguide

* rename

* add implementation

* update notebook

* update notebook

* update notebook

* summary method

* initiate_chats enhancements

* callable summary_method

* summary method

* summary method default

* docstr

* add timeout to slient pip install test

* consolidate_chat_info

* a_initiate_chat

* AssertionError tests

* AssertionError test

* update tests

* update test

* remove redudant file

* kwargs

* update notebook

* update notebook

* nested

* update SoM

* update notebook

* max_turns

* add notebook

* updatenotebook

* add notebooks

* update notebook and test

* add links to the example page

* annotations

* documentation

* default values

* Expanded details about the purpose of each cell

* update default value

* default value for reply_func_from_nested_chats

* default reply func name

* update notebook

* update optiGuide

* upload fig

* optiGuide notebook

* doc

* update notebook

* update notebook

* notebook format

* update optiguide notebook

* raise value error

* update notebook

* Improve language

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Co-authored-by: signalprime <15487280+signalprime@users.noreply.github.com>
Co-authored-by: gagb <gagb@users.noreply.github.com>

* process message before send

* rename

* Allow the GroupChatManager to send introductions.

* Fixed function name.

* Added test cases for sending introductions.

* Trying to sort out why remote pytest is failing.

* Fixed broken plugin behavior.

* Update autogen/agentchat/groupchat.py

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* Updated as per Chi's suggestions.

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* support getting model from both llm config and config list

* address comments

* address commentsd

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* try to fix blog

* modify blog

* fix test error in #717; fix blog typo in installation; update blogs with output examples.

* pre-commit

* pre-commit

* Update website/blog/2023-11-26-Agent-AutoBuild/index.mdx

Co-authored-by: Qingyun Wu <qingyun.wu@psu.edu>

* add future work

* fix grammar

* update agent_builder

* solve #941; add detailed debug info; support json string config

* pre-commit

* solve #954

* pre-commit

* [new feature] build group chat agents from library.

* pre-commit

* add authors' info in notebook; add a new notebook for build_from_library; reduce prompt effort

* update test and example for build_from_library

* pre-commit

* add notebook; update docs

* change notebook name

* change description for notebook and doc

* remove default value for default_llm_config

* add embedding similarity agent selection

* pre-commit

* update test

* add dependency installation in github workflow

* update test

* pre-commit

* update notebook

* support directly json as library; support customize embedding model

* update test

* pre-commit

* update github test workflow

* Update autobuild_agent_library.ipynb

* add agent description

* refine prompt; update notebook

* pre-commit

* update test example

* update test

* update test

* update test

* change `config_path` to `config_path_or_env`; update test

* pre-commit

* update test

* update test

* update test: add config_file_location

* change `config_path_or_env` to `config_file_or_env`

* update test

* solve noqa

* fix import error for conftest

* fix test error

* pre-commit

* * update error message in `_create_agent`.
* replace `gpt-4-1106-preview` to `gpt-4` in test file.

* add comment on local server creation; modify notebook; update contrib-openai.yml for test; add autobuild option in setup.py; add autotest model name statement

* move import huggingface_hub to _create_agent

* pre-commit

* add uncover comment in the endpoint creation code block

* recover contrib-openai.yml for merge

* remove TERMINATE assertion for sys msg

* remove TERMINATE assertion

---------

Co-authored-by: Jieyu Zhang <jieyuz2@cs.washington.edu>
Co-authored-by: Qingyun Wu <qingyun.wu@psu.edu>

* Allow None for sender field

* Apply formatting fixes

* Add test for ConversableAgent generate_reply with message is provided and sender is None

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* async_initiate_chats init commit

* Fix a_get_human_input bug

* Add agentchat_multi_task_async_chats.ipynb with concurrent exampls.

* Addess the comments, Update unit test

* Add agentchat_multi_task_async_chats.ipynb to Examples.md

* Fix type for Python 3.8

---------

Co-authored-by: Qingyun Wu <qingyun0327@gmail.com>

* refactored code to simplify

* optimize function. Instead of iterating over each character, guess at size and then iterate by token.

* adding tests

* Add missing tests

* minor test fix

* simplified token truncation by using tiktoken to encode and decode

* updated truncated notification message

* Fix llm_config spec to use os.environ

* Add test case and fix bug in loop

---------

Co-authored-by: gagb <gagb@users.noreply.github.com>

* Reproduce #1440

* Updated code with latest APIs

* Reran notebook

* Fix usage of cache

* Change defualt model for `lmm`

* Try to use PIL image for LMM's _oai_messages

* Update test cases and llava

* Remove redundant files

* Update the imports for lmm tests

* Test case fix

* Docstring update

* LMM notebook lint

* Typo correction for img_utils and its test

* Update test_llava.py

debug, reformat

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>
Co-authored-by: Shaokun Zhang <shaokunzhang529@gmail.com>
Co-authored-by: Shaokun Zhang <shaokun.zhang@psu.edu>

* use str for hook key

* bump version to 0.2.15

* remove github

* disable assertion

* max_turns

* add notebook

* simplify code

* groupchat _prepare_chat

* doc

* add doc

* Update autogen/agentchat/conversable_agent.py

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>