Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
|
|
6 months ago | |
|---|---|---|
| .. | ||
| Readme.md | 6 months ago | |
| dataset_record.yml | 6 months ago | |
| dataset_record_avif.yml | 6 months ago | |
| policy_inference.yml | 6 months ago | |
Guide for recording imitation learning datasets with the SO101 arm using Dora and train policies with LeRobot.
To begin recording, edit the dataflow.yml file. You must update camera settings, robot ports, and dataset configuration based on your setup.
Update the camera device id, you can also add or remove cameras based on your setup:
# Laptop camera
CAPTURE_PATH: "0" # Laptop camera is defaults to 0
# External camera
CAPTURE_PATH: "1" # Change this to match your external camera device
Identify and set the correct USB ports for both the leader and follower SO101 arms
PORT: "/dev/ttyACM0" # change this
Edit the following fields:
REPO_ID: "your_username/so101_dataset" # HuggingFace dataset path
SINGLE_TASK: "Pick up the red cube and place it in the blue box" # Your task description
CAMERA_NAMES: "laptop, front" # Name your camera sources depending on your setup
CAMERA_LAPTOP_RESOLUTION: "480,640,3"
CAMERA_FRONT_RESOLUTION: "480,640,3"
You can adjust the following parameters as needed:
TOTAL_EPISODES: "2" # Number of episodes
#you may want to try with 2-3 episodes to test, then atleast 50 episodes for training is recommended
EPISODE_DURATION_S: "60" # Duration of each episode (in seconds) - depends on complexity of task
RESET_DURATION_S: "15" # Time to reset the environment between episodes
FPS: "30" # Should match camera fps
PUSH_TO_HUB: "false" # Set to "true" to auto-upload dataset to HuggingFace
ROOT_PATH: "full path where you want to save the dataset"
# if not defined then it will be stored at ~/.cache/huggingface/lerobot/repo_id
Once everything is updated in dataflow.yml, you are ready to record your dataset.
Build and Run
uv venv
dora build dataflow.yml --uv
dora run dataflow.yml --uv
In the rerun window you can see the the info regarding Start of episodes, start of reset phase and saving of episodes.
Episode Active:
Reset Phase:
Repeat until all episodes are complete
Recording Tips
Your dataset will be saved locally. Check the recording was successful:
It will be stored in ~/.cache/huggingface/lerobot/repo_id
After successfully recording your dataset, we will be training imitation learning policies and deploying them on your SO101 robot arm.
ACT (Recommended for SO101)
Diffusion Policy
uv run dora-dataset-lerobot-train
--dataset.repo_id=${HF_USER}/your_repo_id \ # provide full path of your dataset
--policy.type=act \
--output_dir=outputs/train/act_so101_test \
--job_name=act_so101_test \
--policy.device=cuda \
--wandb.enable=true \
--policy.repo_id=${HF_USER}/my_policy
You can monitor your training progress on wandb
For more details regarding training check lerobot guide on Imitation learning for SO101
After training your policy, you can test it on your SO101 arm using the policy_inference.yml.
Edit the policy_inference.yml file and update the following settings based on your hardware setup:
Update the camera device IDs to match your setup (same as recording):
# Laptop camera
CAPTURE_PATH: "0" # Usually 0 for built-in laptop camera
# External camera
CAPTURE_PATH: "1" # Change this to your external camera device ID
Set the correct USB port for your follower SO101 arm:
PORT: "/dev/ttyACM1" # Update this to match your follower robot port
Update the path to your trained model and task description:
MODEL_PATH: "./outputs/train/act_so101_test/checkpoints/last/pretrained_model" # Path to your trained model
TASK_DESCRIPTION: "Pick up the red cube and place it in the blue box"
Ensure camera settings match your recording configuration:
CAMERA_NAMES: "laptop, front" # Must match training setup
CAMERA_LAPTOP_RESOLUTION: "480,640,3" # Must match training
CAMERA_FRONT_RESOLUTION: "480,640,3" # Must match training
Once you've updated the configuration:
dora build policy_inference.yml
dora run policy_inference.yml
DORA (Dataflow-Oriented Robotic Architecture) is middleware designed to streamline and simplify the creation of AI-based robotic applications. It offers low latency, composable, and distributed datafl
Rust Python TOML Markdown C other