autogen/samples/tools/webarena

• WebArena Benchmark
• • Installing WebArena
  • Running with AutoGen agents
  • References

WebArena Benchmark

This directory helps run AutoGen agents on the WebArena benchmark.

Installing WebArena

WebArena can be installed by following the instructions from WebArena's GitHub repository

If using WebArena with AutoGen there is a clash on the versions of OpenAI and some code changes are needed in WebArena to be compatible with AutoGen's OpenAI version:

• webarena's openai version is openai==0.27.0
• autogen's openai version is: openai>=1.3

Prior to installation, in the WebArena codebase, any file containing openai.error needs to be replaced with openai.

Running with AutoGen agents

You can use the run.py file in the webarena directory to run WebArena with AutoGen. The OpenAI (or AzureOpenAI or other model) configuration can be setup via OAI_CONFIG_LIST. The config list will be filtered by whatever model is passed in the --model argument.

e.g. of running run.py:

mkdir myresultdir
python run.py --instruction_path agent/prompts/jsons/p_cot_id_actree_2s.json --test_start_idx 27 --test_end_idx 28 --model gpt-4 --result_dir myresultdir

The original run.py file has been modified to use AutoGen agents which are defined in the webarena_agents.py file.

References

WebArena: A Realistic Web Environment for Building Autonomous Agents

Zhou, Shuyan and Xu, Frank F and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Bisk, Yonatan and Fried, Daniel and Alon, Uri and others

https://arxiv.org/pdf/2307.13854.pdf