afourney
e5ab7d55cf
Some pandas series were not being handled correctly ( #5972 )
1 year ago
afourney
22b68b96b6
Added a flag to agbench to enable Azure identity. ( #5977 )
1 year ago
Eric Zhu
483532180a
Improvements to agbench ( #5776 )
1. Add host network support in Docker and remove unused requirements
from argument check.
2. Use Pandas to simplify summary statistic calculations.
3. Add running time to summary statistics
```
Using tabulation method defined in '/home/ekzhu/autogen/python/packages/agbench/benchmarks/HumanEval/Scripts/custom_tabulate.py'
Task Id Trial 0 Success Trial 0 Time
-- ------------ ----------------- --------------
0 HumanEval_0 True 3
1 HumanEval_1 False 15
2 HumanEval_2 True 2
3 HumanEval_3 True 11
4 HumanEval_4 True 4
5 HumanEval_5 True 2
6 HumanEval_6 False 18
7 HumanEval_7 True 2
8 HumanEval_8 True 2
9 HumanEval_9 True 12
10 HumanEval_10 False 11
11 HumanEval_11 True 2
12 HumanEval_12 True 3
13 HumanEval_13 True 1
14 HumanEval_14 True 4
15 HumanEval_15 True 1
16 HumanEval_16 True 2
17 HumanEval_17 False 76
18 HumanEval_18 True 4
19 HumanEval_19 True 3
20 HumanEval_20 True 5
21 HumanEval_21 True 3
22 HumanEval_22 True 1
23 HumanEval_23 True 2
24 HumanEval_24 nan
Summary Statistics
Successes Failures Missing Total Average Success Rate Average Time Total Time
------- ----------- ---------- --------- ------- ---------------------- -------------- ------------
Trial 0 20 4 1 25 0.8 7.875 189
CAUTION: 'autogenbench tabulate' is in early preview and is not thoroughly tested.
Please do not cite values from these calculations in academic work without first inspecting and verifying the results in the run logs yourself.
```
Now the default tabulate output looks like this
---------
Co-authored-by: Ryan Sweet <rysweet@microsoft.com>
1 year ago
afourney
af5dcc7fdf
Significant updates to agbench. ( #5313 )
- Updated HumanEval template to use AgentChat
- Update templates to use config.yaml for model and other configuration
- Read environment from ENV.yaml (ENV.json still supported but
deprecated)
- Temporarily removed WebArena and AssistantBench. Neither had viable
Templates after `autogen_magentic_one` was removed. Templates need to be
update to AgentChat (in a future PR, but this PR is getting big enough
already)
1 year ago
afourney
088a50faa5
Remove old autogen_magentic_one package. ( #5305 )
This PR removes the older `autogen_magentic_one` package, and directs
people to use the new AgentChat implementation.
Hopefully this eases confusion.
---------
Co-authored-by: Jack Gerrits <jack@jackgerrits.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
1 year ago
Jack Gerrits
538f39497b
Replace create_completion_client_from_env with component config ( #4928 )
* Replace create_completion_client_from_env with component config
* json load
1 year ago
Jack Gerrits
fadff4aece
Fix definition of workspace package, remove uv pin ( #4830 )
* Fix definition of workspace package, remove uv pin
* add --all-packages
* pin docs uv versions for older project structure
* try old version to verify CI
* Use workflow target
* change syntax
* change check
* try with var in matrix
* add all packages to workspace
* remove project table
1 year ago
Jack Gerrits
87011ae01b
Migrate model context and models modules out of components ( #4613 )
* Move model context out of components
* move models out of components
* rename docs file
1 year ago
Eric Zhu
8dac072658
Update references in docs ( #4590 )
* Update agent doc
* Remove outdated doc
* Update references
* Update readme
* Update readme
1 year ago
Eric Zhu
fa550c2c36
fix docs ( #4589 )
* fix doc on distributed runtime
* Fix references
* Update references
* Fix import paths in user guide notebooks for code executor components
1 year ago
Jack Gerrits
2b878763f8
Move grpc runtimes to ext, flatten application ( #4553 )
* Move grpc runtimes to ext, flatten application
* rename to grpc
* fmt
1 year ago
Victor Dibia
777f2abbd7
Load and Save state in AgentChat ( #4436 )
1. convert dataclass types to pydantic basemodel
2. add save_state and load_state for ChatAgent
3. state types for AgentChat
---------
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
1 year ago
Jack Gerrits
3022369eeb
Flatten core base and components ( #4513 )
* Flatten core base and components
* remove extra files
* dont export from deprecated locations
* format
* fmt
1 year ago
Griffin Bassman
e037596228
typo: agbench readme ( #4302 )
1 year ago
Leonardo Pinheiro
38f62e1609
migrate models ( #3848 )
* migrate models
* Update python/packages/autogen-agentchat/src/autogen_agentchat/agents/_tool_use_assistant_agent.py
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
* refactor missing imports
* ignore type check errors
* Update python/packages/autogen-ext/src/autogen_ext/models/_openai/_model_info.py
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
* update packages index page
---------
Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
1 year ago
Hussein Mozannar
e11d84b996
Adding Benchmarks to agbench ( #3803 )
* Move from tomllib to tomli
* added example code for magentic-one + code comments
* adding benchmarks temporarily
* add license for datasets
* revert changes to magentic-one
* change license location
---------
Co-authored-by: Ryan Sweet <rysweet@microsoft.com>
1 year ago
Max Golovanov
636591a149
Use MCR registry instead of Docker's registry ( #3814 )
* Update FunctionCallGenerator.cs to address race condition
Update FunctionCallGenerator.cs to address race condition
* Update Dockerfile
Use MCR registry
* Update Dockerfile
Use MCR registry
1 year ago
Hussein Mozannar
373adc9a34
Adding Benchmarks back into agbench and updates to agbench ( #3711 )
1 year ago
Jack Gerrits
2526c69ce9
Include license file in package ( #3703 )
1 year ago
Jack Gerrits
1174fcd92e
Merge branch 'main' into staging
1 year ago
afourney
d7190cbe9e
Removes easyocr from mdconvert ( #653 )
* Removes easyocr from mdconvert
* Updated uv lock
* Remove unused variable.
1 year ago
Jack Gerrits
dc02719f7c
Check for prints ( #616 )
* Check for prints
* format
1 year ago
Jack Gerrits
93e7127f1f
Change references from agenext to autogen ( #610 )
1 year ago
Jack Gerrits
6bf62262a4
fix config for pyright for most ( #561 )
1 year ago
Ryan Sweet
7d7fc8a912
.NET cleanup and refactor ( #558 )
Moves some shared code from samples into core.
complete/cleanup the rename to Microsoft.AutoGen
adds new projects in AutoGen.Extensions
1 year ago
Jack Gerrits
306541e247
Fixup ruff config and inclusions ( #495 )
* add tests to ruff for core
* fmt
* lint
* lint fixes
* fixup more dirs
* dont include non python
* lint fixes
* lint fixes
* fix dir name
* dont relative include
1 year ago
afourney
243c095796
Updated the root path discovery in agbench to reflect latest folder structure. ( #433 )
1 year ago
Jack Gerrits
4ff5610853
Migrate to uv and poe for workspace management and task running ( #424 )
* Migrate to uv and poe for workspace management and task running
* install python
* try fix
* ensure workspace venv in used
* package dir
* move nbqa to mypy task
* separate sync, clarify docs
1 year ago
Jack Gerrits
5e8840d13c
Python: organize packages in package directory ( #420 )
* Move packages to packages directory
* remove screenshot
* update some paths
1 year ago