* Start Gemini integration: works ok with Text now

* Gemini notebook lint

* try catch "import" for Gemini

* Debug: id issue for chat completion in Gemini

* Add RAG example

* Update docs for RAG

* Fix missing pydash

* Remove temp folder

* Fix test error in runs/7206014032/job/19630042864

* Fix tqdm warning

* Fix notebook output

* Gemini's vision model is supported now

* Install instructions for the Gemini branch

* Catch and retry when see Interval Server Error 500

* Allow gemini to take more flexible messages
i.e., it can take messages where "user" is not the last role.

* Use int time for Gemini client

* Handle other exceptions in gemini call

* rename to "create" function for gemini

* GeminiClient compatible with ModelClient now

* Lint

* Update instructions in Gemini notebook

* Lint

* Remove empty blocks from Gemini notebook

* Add gemini into example page

* self.create instead of call

* Add py and Py into python execution

* Remove error code from merging

* Remove pydash dependency for gemini

* Add cloud-gemini doc

* Remove temp file

* cache import update

* Add test case for summary with mm input

* Lint: warnings instead of print

* Add test cases for gemini

* Gemini test config

* Disable default model for gemini

* Typo fix in gemini workflow

* Correct grammar in example notebook

* Raise if "model" is not provided in create(...)

* Move TODOs into a roadmap

* Update .github/workflows/contrib-tests.yml

Co-authored-by: Davor Runje <davor@airt.ai>

* Gemini test config update

* Update setup.py

Co-authored-by: Davor Runje <davor@airt.ai>

* Update test/oai/test_gemini.py

Co-authored-by: Davor Runje <davor@airt.ai>

* Update test/oai/test_gemini.py

Co-authored-by: Davor Runje <davor@airt.ai>

* Remove python 3.8 from gemini
No google's generativeai for Windows with Python 3.8

* Update import error handling for gemini

* Count tokens and cost for gemini

---------

Co-authored-by: Li Jiang <bnujli@gmail.com>
Co-authored-by: Davor Runje <davor@airt.ai>

* add test

* fix test

* fix

---------

Co-authored-by: AnonymousRepoSub <“shaokunzhang529@outlook.com” >

* fix local test failures

* set skip

* gather_usage_summary has been updated

* updated cost info to 'usage_including_cached_inference' and 'usage_excluding_cached_inference'

* fix: pre-commit formatting for cost_info

* improved cost explanation and doc

* improved cost info doc

* include - exclude

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* Standardize printing of MessageTransforms

* Fix pre-commit black failure

* Add test for transform_messages printing

* Return str instead of printing

* Rename to_print_stats to verbose

* Cleanup

* t i# This is a combination of 3 commits.

Update requirements

* Remove lazy-fixture

* Avoid calling apply_transform in two code paths

* Format

* Replace stats with logs

* Handle no content messages in TokenLimiter get_logs()

* Move tests from test_transform_messages to test_transforms

---------

Co-authored-by: Wael Karkoub <wael.karkoub96@gmail.com>

* Add "py" as lang in conversable agent (#1062)

* Add conditions to allow for python executable variants (#1062)

* reverted import (#1062)

* Parameterized tests, moved Python variants to a constant (#1062)

* Moved Python variants to a constant (#1062)

* Update autogen/code_utils.py (#1062)

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Update autogen/coding/local_commandline_code_executor.py (#1062)

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Added PYTHON_VARIANTS as imported constant (#1062)

* ran pre-commit-check  (#1062)

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* support llm_config in agentoptimizer

* fix doc

* restore seed timeout

---------

Co-authored-by: “skzhang1” <“shaokunzhang529@gmail.com”>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* Added vectordb base and chromadb

* Remove timer and unused functions

* Added filter by distance

* Added test utils

* Fix format

* Fix type hint of dict

* Rename test

* Add test chromadb

* Fix test no chromadb

* Add coverage

* Don't skip test vectordb utils

* Add types

* Fix tests

* Fix docs build error

* Add types to base

* Update base

* Update utils

* Update chromadb

* Add get_docs_by_ids

* Improve docstring

* Add get all docs

* Move chroma_results_to_query_results to utils

* Improve type hints

* Update logger

* Update init, add embedding func

* Improve docstring of vectordb, add two attributes

* Improve test workflow

* Add source to the answer for default prompt

* Fix qdrant

* Fix tests

* Update docstring

* Fix check files

* Fix qdrant test error

* Use assistant tag to  filter config

* bump version to 0.2.23

* fixed types related to function calling

* polishing

* fixed types in tests

* bump version to 0.2.22

* doc update

* link update

* consider the failure case

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* Add isort

* Apply isort on py files

* Fix circular import

* Fix format for notebooks

* Fix format

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

Co-authored-by: Yiran Wu <32823396+kevin666aa@users.noreply.github.com>

* Add in memory cache class

* formatting

* Update mm test: create dummy image in case file corrput

* Update vision capability test for MacOS

* Lint

* Upload image for testing

* Allow custom name for functions module

* update tests

* reorder

* Re-commit of code from PR (#2167) addressing #1861, due to wrong basing

* Update website/docs/topics/non-openai-models/best-tips-for-nonopenai-models.md

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Removed unnecessary notebook images

* Update conversation-patterns.ipynb

Updated to include note about being applicable when auto.

* Updated to include checks that the role is not blank/None. Added tests.

* Changed try-except to use pytest

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* multimodal carryover

* adds mm carryover tests

* more tests + cleanup code

* check content instead

* beibin suggestion

* cleanup

* fix async

* use deepcopy

* handle carryover method

* remove content copy

* sonichi suggestions

---------

Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* wip

* Adds docstrings

* fixed spellings

* wip

* fixed errors

* better class names

* adds tests

* added tests to workflow

* improved token counting

* improved notebook

* improved token counting in test

* improved docstrings

* fix inconsistencies

* changed by mistake

* fixed docstring

* fixed details

* improves tests + adds openai contrib test

* fix spelling oai contrib test

* clearer docstrings

* remove repeated docstr

* improved notebook

* adds metadata to notebook

* Improve outline and description (#2125)

* better dir structure

* clip max tokens to allowed tokens

* more accurate comments/docstrs

* add deperecation warning

* fix front matter desc

* add deperecation warning notebook

* undo local notebook settings changes

* format notebook

* format workflow

---------

Co-authored-by: gagb <gagb@users.noreply.github.com>

* fixed get_stream in new thread by introducing a global default

* fixed get_stream in new thread by introducing a global default

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* simplify getting-started; update news

* bug fix

* Implement user defined functions feature for local cli exec, add docs

* add tests, update docs

* fixes

* fix test

* add pandas test dep

* install test

* provide template as func

* formatting

* undo change

* address comments

* add test deps

* formatting

* test only in 1 env

* formatting

* remove test for local only

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Mark cache as a protocl and update type hints to reflect

* int

* undo init change

* Introducing IOStream

* bug fixing

* polishing

* refactoring

* refactoring

* refactoring

* wip: async tests

* websockets added

* wip

* merge with main

* notebook added

* FastAPI example added

* wip

* merge

* getter/setter to iostream added

* website/blog/2024-03-03-AutoGen-Update/img/dalle_gpt4v.png: convert to Git LFS

* website/blog/2024-03-03-AutoGen-Update/img/gaia.png: convert to Git LFS

* website/blog/2024-03-03-AutoGen-Update/img/teach.png: convert to Git LFS

* add SSL support

* wip

* wip

* exception handling added to on_connect()

* refactoring: default iostream is being set in a context manager

* test fix

* polishing

* polishing

* polishing

* fixed bug with new thread

* polishing

* a bit of refactoring and docs added

* notebook added to docs

* type checking added to CI

* CI fix

* CI fix

* CI fix

* polishing

* obsolete todo comment removed

* fixed precommit error

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* draft agent optimizer

* refactor

* remove

* change openai config interface

* notebook

* update blog

* add test

* clean up

* redir

* update

* update interface

* change model name

* move to contrib

* Update autogen/agentchat/contrib/agent_optimizer.py

Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com>

---------

Co-authored-by: “skzhang1” <“shaokunzhang529@gmail.com”>
Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com>
Co-authored-by: Jieyu Zhang <jieyuz2@cs.washington.edu>
Co-authored-by: Jack Gerrits <jackgerrits@users.noreply.github.com>

* tried implementing my own regex

* improves tests

* finally works

* removes prints

* fixed test

* adds start and end

* delete unused imports

* refactored to use new tool

* significantly improved algo

* tag content -> tag attr

* fix tests + adds new field

* return full match

* return remove start and end

* update docstrings

* update docstrings

* update docstrings

---------

Co-authored-by: Beibin Li <BeibinLi@users.noreply.github.com>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* context to kwargs

* add tag

* add test

* text to kwargs

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* Add vision capability

* Configurate: description_prompt

* Print warning instead of raising issues for type

* Skip vision capability test if dependencies not installed

* Append "vision" to agent's system message when enabled VisionCapability

* GPT-4V notebook update with ConversableAgent

* Clean GPT-4V notebook

* Add vision capability test to workflow

* Lint import

* Update system message for vision capability

* Add a `custom_caption_func` to VisionCapability

* Add custom function example for vision capability

* Skip test Vision capability custom func

* GPT-4V notebook metadata to website

* Remove redundant files

* The custom caption function takes more inputs now

* Add a more complex example of custom caption func

* Remove trailing space

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* fix

* adds tests

* check if windows

* adds windows shells

* modifies exit code

* fix powershell

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Encapsulate colored into a module

* lint fix

* add missing file

* undo change

* conform with original colored func

* change import strategy

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* fix test_human_input

* shorten tests

* terminate chat early

* limit max turns

* limit max_rounds

* check skip_openai

* update pre-commit

* update pre-commit.ci

* lint fix

* macos bugfix

* logs permissionerror warning only if lang is powershell command

* condensed the if statements down

* Update code_utils.py

* fixed formatting

* handled powershell_command = None case

* bugfix

* raising exceptions instead of logging warnings

* code formatting fixed

* removed return sh statement

* fixed code formatting

* update get_powershell_command

* Update code_utils.py

fixed code format

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* adjust order of side bars

* doc improvement

* limit max turns

* improve tests

* improve doc

* improve docstr

* Request for longer response to make cache test more robust.

* 1000 words --> 100 words

* fix test

* Update test/agentchat/test_agent_logging.py

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* add assistant config

* add test

* change notebook to use assistant config

* use assistant config in testing

* code refinement

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* allow sender in nested chat

* description

* update example list

* doc format

* meta data

* add test

* adds image generation capability

* add todo

* readded cache

* wip

* fix content str bugs

* removed todo: delete imshow

* wip

* fix circular imports

* add notebook

* improve prompt

* improved text analyzer + notebook

* notebook update

* improve notebook

* smaller notebook size

* made changes to the wrong branch :(

* resolve comments + 1

* adds doc strings

* adds cache doc string

* adds doc string to add_to_agent

* adds doc string to ImageGeneration

* instructions are not configurable

* removed unnecessary imports

* changed doc string location

* more doc strings

* improves testability

* adds tests

* adds cache test

* added test to github workflow

* compatible llm config format

* configurable reply function position

* skip_openai + better comments

* fix test

* fix test?

* please fix test?

* last fix test?

* remove type hint

* skip cache test

* adds mock api key

* dalle-2 test

* fix dalle config

* use apu key function

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* init PR

* update

* update code check

* update

* update

* update

* update

* update

* Test the ability to have agents a,u,t,o,g,e,n speak in turn.

* update

* update

* update

* Evidence that groupchat not terminating because of the TERMINATE substring.

* Raising NoEligibleSpeakerException allows graceful exit before max turns

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* udpate to comments

* update

* Update website/blog/2024-02-29-StateFlow/index.mdx

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* update links

* remove unnecessary changes

* Update website/blog/2024-02-29-StateFlow/index.mdx

Co-authored-by: Joshua Kim <joshkyh@users.noreply.github.com>

* Update website/blog/2024-02-29-StateFlow/index.mdx

Co-authored-by: Joshua Kim <joshkyh@users.noreply.github.com>

* remove test for custom notebook

* update

* update

---------

Co-authored-by: Joshua Kim <Joshua@spectdata.com>
Co-authored-by: Chi Wang <wang.chi@microsoft.com>
Co-authored-by: Joshua Kim <joshkyh@users.noreply.github.com>

* fix type and default value of the code_execution_config of UserProxAgent

* fix type and default value of the code_execution_config of UserProxAgent

* set default value of llm_config in UserProxyAgent to None

* fixed tests

* revert llm_config to False

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* consistent file saving across cli executors

* test fixes

* feedback

* make path

* formatting

* run timeout test on windows

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>