Skip to main content
Harbor is a framework for evaluating and optimizing agents and language models in sandboxed environments, from the creators of Terminal-Bench. Harbor runs each trial in an isolated container, so you can parallelize evaluations and rollouts across many environments at once. The langsmith Harbor environment runs those trials on LangSmith sandboxes. Select it with -e langsmith to execute Harbor jobs on LangSmith infrastructure, alongside providers such as AgentCore, Daytona, E2B, and Modal.

​
Prerequisites

​
Install

Install Harbor with the langsmith extra:

​
Authenticate

Harbor authenticates with your LangSmith credentials. Set an API key:
LANGCHAIN_API_KEY works as well. Alternatively, select a LangSmith SDK profile instead of exporting a key:

​
Run an evaluation

Run a Harbor job and select the LangSmith environment with -e langsmith:
Harbor creates one LangSmith sandbox per trial, runs the agent and verifier inside it, then tears the sandbox down when the trial finishes.

​
Configure the sandbox environment

The LangSmith environment boots each sandbox from a filesystem snapshot. Provide one of the following in your Harbor task:
  • Prebuilt image: set [environment].docker_image in task.toml. Harbor reuses or creates a snapshot from that image.
  • Existing snapshot: pass environment.kwargs.snapshot_name to boot from a snapshot you already created.
  • Dockerfile: include an environment/Dockerfile. Harbor builds a snapshot from it with the build-from-Dockerfile flow, using the task environment/ directory as the build context.
Tune the sandbox lifecycle with environment kwargs, passed on the command line with --ek:
  • idle_ttl_seconds: stops an idle sandbox after this many seconds. Set 0 to disable the idle timeout.
  • delete_after_stop_seconds: deletes a stopped sandbox after this many seconds.

​
Run Deep Agents on LangSmith

Deep Agents run on LangSmith sandboxes through Harbor’s built-in langgraph agent and the -e langsmith environment. The full setup: staging the Deep Agents packages, the LangGraph project (langgraph.json with the deepagent graph), and the exact harbor run command, is maintained in the Deep Agents eval guide: → Running Deep Agents on Harbor / Terminal Bench 2.0 The LangSmith-specific options (authentication, -e langsmith, and the --ek sandbox-lifecycle kwargs) apply to those runs as well.

​
Multi-container tasks

The LangSmith environment supports multi-container tasks. Include an environment/docker-compose.yaml file in your task definition to run several containers per trial. See the Harbor sandbox documentation for details.
Connect these docs to Claude, VSCode, and more via MCP for real-time answers.
Edit this page on GitHub or file an issue.