* Re-added missing notebook

* Test installing postgres

* Error handle the connection.

* Fixed import.

* Fixed import.

* Fixed creation of collection without client.

* PGVector portion working. OpenAI untested.

* Fixed prints.

* Added output.

* Fixed pre-commits.

* Run pgvector notebook

* Improve efficiency of get_collection

* Fix delete_collection

* Fixed issues with pytests and validated functions.

* Validated pytests.

* Fixed pre-commits

* Separated extra_requires to allow more logic. Retrieve_chat base dependencies included on pgvector and qdrant.

* Fixed extra newline.

* Added username and password fields.

* URL Encode the connection string parameters to support symbols like %

* Fixed pre-commits.

* Added pgvector service

* pgvector doesn't have health intervals.

* Switched to colon based key values.

* Run on Ubuntu only. Linux is only option with container service support.

* Using default credentials instead.

* Fix postgres setup

* Fix postgres setup

* Don't skip tests on win and mac

* Fix command error

* Try apt install postgresql

* Assert table does not exist when deleted.

* Raise value error on a empty list or None value provided for IDs

* pre-commit

* Add install pgvector

* Add install pgvector

* Reorg test files, create a separate job for test pgvector

* Fix format

* Fix env format

* Simplify job name, enable test_retrieve_config

* Fix test_retrieve_config

* Corrected behavior for get_docs_by_ids with no ids returning all docs.

* Corrected behavior for get_docs_by_ids with no ids returning all docs.

* Fixed pre-commits.

* Added return values for all functions.

* Validated distance search is implemented correctly.

* Validated all pytests

* Removed print.

* Added default clause.

* Make ids optional

* Fix test, make it more robust

* Bump version of openai for the vector_store support

* Added support for choosing the sentence transformer model.

* Added error handling for model name entered.

* Updated model info.

* Added model_name db_config param.

* pre-commit fixes and last link fix.

* Use secrets password.

* fix: link fixed

* updated tests

* Updated config_list.

* pre-commit fix.

* Added chat_result to all output.
Unable to re-run notebooks.

* Pre-commit fix detected this requirement.

* Fix python 3.8 and 3.9 not supported for macos

* Fix python 3.8 and 3.9 not supported for macos

* Fix format

* Reran notebook with MetaLlama3Instruct7BQ4_k_M

* added gpt model.

* Reran notebook

---------

Co-authored-by: Li Jiang <bnujli@gmail.com>
Co-authored-by: Hk669 <hrushi669@gmail.com>

* Added vectordb base and chromadb

* Remove timer and unused functions

* Added filter by distance

* Added test utils

* Fix format

* Fix type hint of dict

* Rename test

* Add test chromadb

* Fix test no chromadb

* Add coverage

* Don't skip test vectordb utils

* Add types

* Fix tests

* Fix docs build error

* Add types to base

* Update base

* Update utils

* Update chromadb

* Add get_docs_by_ids

* Improve docstring

* Update init params

* Update init vector db

* Add get all docs

* Move chroma_results_to_query_results to utils

* Add init vectordb

* Convert format of results for old version

* Improve type hints

* Update get_context for new query results format

* Fix typo

* Improve init db

* Update default folder

* Update logger

* Update init, add embedding func

* Update distance_threshold

* Fix logger name

* Update qdrant

* Fix init db

* Update notebooks

* Use kwargs to improve readability

* Improve docstring of vectordb, add two attributes

* Add db_config

* Update gitignore

* Update comments

* Add source

* Fix file downloaded from urls have the same name

* Remove files added by mistake

* Improve docstring

* Update docstring

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* Update docstring

* Update docstring

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* add RAG

* demo

* correct notebook

* Update quarto installation

* Update gitignore

* Update format

* RAG doc

---------

Co-authored-by: Li Jiang <bnujli@gmail.com>

* add message field

* send

* message func doc str

* test dict message

* retiring soon

* generate_init_message docstr

* remove todo

* update notebook

* CompressibleAgent

* update notebook

* add test

* retrieve agent

* update test

* summary_method args

* summary

* carryover

* dict message

* update nested doc

* generate_init_message

* fix typo

* update docs for mathchat

* Fix missing message

* Add docstrings

* model

* notebook

* default naming

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>
Co-authored-by: kevin666aa <yrwu000627@gmail.com>
Co-authored-by: Li Jiang <bnujli@gmail.com>
Co-authored-by: Li Jiang <lijiang1@microsoft.com>

* create topics dir and move llm config

* fix redirect

* fix link

* Update process_notebook to use metadata instead of a yaml comment

* upgrade quarto and version check in tool

* formatting

* address comments

* Extend process_notebooks for testing

* add command

* spelling and lint

* update docs

* Update contributing.md

* add shebang

* Update contributing.md

* lint

* update some notebooks

* Update contributing.md

* remove os

---------

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Rewrite and consolidate configuration docs

* add link to filtering

* updates based on comments

* update lockfile

* update redirect

* spelling mistake

* update the noqa

* update links

* fix broken links

* lint fixes

* address comments

* update wording about model selection

* update cache_seed and json pattern

* add periods

* Update the config json func and other small changes

* update wording of json string

* update intro

* update description of caching

* fix link

* spelling mistake

* fix llm config

* Update website/docs/llm_configuration.ipynb

Co-authored-by: gagb <gagb@users.noreply.github.com>

* Add examples back as notebook

---------

Co-authored-by: gagb <gagb@users.noreply.github.com>
Co-authored-by: Aaron <aaronlaptop12@hotmail.com>

* Initial infrasctructure for notebooks page

* migrate two notebooks

* add readme notification for notebook dir

* override 'text' prism language to add basic syntactical structure to autogens output

* Rework to retain existing directory and not expose front matter to consumers of the notebook

* improve error handling of process notebooks

* format, ruff and type fixes

* undo changes to navbar

* update readme, CI

* whitespace

* spelling mistakes

* spelling

* Add contributing guide for notebooks

* update notebook

* formatting

* nbqa adedd to pre-commit, added black and ruff for notebooks

* polishing

* polishing

* polishing

* fixed spelling, minor errors and reformatted using black

* polishing

* added codespell to pre-commit hooks, fixed a number of spelling errors and a few minor bugs in the code

* update autogen library version in notebooks

* update autogen library version in notebooks

* update autogen library version in notebooks

* update autogen library version in notebooks

* update autogen library version in notebooks

* wording

* grammar and wording

* readability suggested by shruti222patel

* period

* Add custom text types and recursive

* Add custom text types and recursive

* Fix format

* Update qdrant, Add pdf to unstructured

* Use unstructed as the default text extractor if installed

* Add tests for unstructured

* Update tests env for unstructured

* Fix error if last message is a function call, issue #569

* Remove csv, md and tsv from UNSTRUCTURED_FORMATS

* Update docstring of docs_path

* Update test for get_files_from_dir

* Update docstring of custom_text_types

* Fix missing search_string in update_context

* Add custom_text_types to notebook example

* Add speaker selection methods

* Update groupchat RAG

* Update seed to cache_seed

* Update RetrieveChat notebook

* Update parameter name

* Add test

* Add more tests

* Add mock to test

* Add mock to test

* Fix typo speaking

* Add gracefully exit manual input

* Update round_robin docstring

* Add method checking

* Remove participant roles

* Fix versions in notebooks

* Minimize installation overhead

* Fix missing lower()

* Add comments for try_count 3

* Update warning for n_agents < 3

* Update warning for n_agents < 3

* Add test_n_agents_less_than_3

* Add a function for manual select

* Update version in notebooks

* Fixed bugs that allow speakers to go twice in a row even when allow_repeat_speaker = False

---------

Co-authored-by: Adam Fourney <adamfo@microsoft.com>

* Fix docstring of get_or_create

* Improve docstring

* api_base -> base_url (#383)

* InvalidRequestError -> BadRequestError (#389)

* remove api_key_path; close #388

* close #402 (#403)

* openai client (#419)

* openai client

* client test

* _client -> client

* _client -> client

* extra kwargs

* Completion -> client (#426)

* Completion -> client

* Completion -> client

* Completion -> client

* Completion -> client

* support aoai

* fix test error

* remove commented code

* support aoai

* annotations

* import

* reduce test

* skip test

* skip test

* skip test

* debug test

* rename test

* update workflow

* update workflow

* env

* py version

* doc improvement

* docstr update

* openai<1

* add tiktoken to dependency

* filter_func

* async test

* dependency

* migration guide (#477)

* migration guide

* change in kwargs

* simplify header

* update optigude description

* deal with azure gpt-3.5

* add back test_eval_math_responses

* timeout

* Add back tests for RetrieveChat (#480)

* Add back tests for RetrieveChat

* Fix format

* Update dependencies order

* Fix path

* Fix path

* Fix path

* Fix tests

* Add not run openai on MacOS or Win

* Update skip openai tests

* Remove unnecessary dependencies, improve format

* Add py3.8 for testing qdrant

* Fix multiline error of windows

* Add openai tests

* Add dependency mathchat, remove unused envs

* retrieve chat is tested

* bump version to 0.2.0b1

---------

Co-authored-by: Li Jiang <bnujli@gmail.com>

* Update comments to make it more clear

* Update Installation

* fix typo

* wording

* typo

* UPDATE - Updated retrieve_utils.py to have the ability to parse text from pdf files

* UNDO - change to recursive condition

* UPDATE - updated agentchat_RetrieveChat.ipynb to clarify which file types are accepted to be in the docs path

* ADD - missing import

* UPDATE - setup.py to have PyPDF2 in retrievechat

* RE-ADD - urls

* ADD - tests for retrieve utils, and removed deprecated PyPdf2

* Update agentchat_RetrieveChat.ipynb

* Update retrieve_utils.py

Fix format

* Update retrieve_utils.py

Replace print with logger

* UPDATE - added more specific exception to PDF decryption try/catch

* FIX - typo, return statement at wrong indentation in extract_text_from_pdf

---------

Co-authored-by: Ward <award40@LAMU0CLP74YXVX6.uhc.com>
Co-authored-by: Li Jiang <bnujli@gmail.com>

* fix bug for windows

* fix bug for windows

* more clear example

* link to example

* add test

* format

* comment

* fix assertion error

* fix test error and links

---------

Co-authored-by: Chi Wang (MSR) <chiw@microsoft.com>

* Upsert in batch

* Improve update context, support customized answer prefix

* Update tests

* Update intermediate answer

* Fix duplicate intermediate answer, add example 6 to notebook

* Add notebook results

* Works better without intermediate answers in the context

* Bump version to 0.1.2

* Remove commented code and add descriptions to _generate_retrieve_user_reply

---------

Co-authored-by: Qingyun Wu <qingyun.wu@psu.edu>