eval

Websocket evaluation runner for the BEHAVIOR-1K challenge.

Drives the OmniGibson Evaluator against a policy served over a websocket (e.g. the openpi or GR00T scripts/b1k/serve_b1k.py server). For each test instance of a task it runs a rollout and writes a per-rollout result JSON compatible with omnigibson/eval/utils/score_utils.py (q_score, time, agent_distance / normalized_agent_distance).

Example

python -m omnigibson.eval.eval --task-name turning_on_radio --robot-config omnigibson/eval/r1pro.yaml --mode public_test --host 127.0.0.1 --port 8000 --instance-indices 0 --max-steps 500 --output-dir outputs/b1k_eval --write-video