StepFun is an AI company best known for its Step series of LLMs (such as Step 3.7), along with AI products for multimodal models, voice, image generation, and agent development. As StepFun scaled its agent applications, the team needed a way to capture, search, replay, and analyze agent trace data without waiting for batch pipelines.
That need produced StepTrace, StepFun’s PB-scale agent observability platform, built on Apache Doris. StepTrace handles real-time trace ingestion, full-text search, cost and latency analysis, session-level replay, and evaluation workflows in a single system.
Why agent observability is hard
Agent systems are harder to debug than traditional applications. The systems under observation are probabilistic: they change over time, fail in unexpected ways, and need continuous evaluation feedback to improve.
In a traditional microservice, engineers can usually trace a failure through logs, code paths, and system architecture. The troubleshooting path is embedded in the code and the system design. In an agent system, a single response may involve multiple rounds of reasoning, tool calls, model decisions, sandbox execution, and infrastructure dependencies, and the execution path changes from run to run.
To observe how agents actually run, StepFun set out to build a trace system with six capabilities:
-
Agent-native data model: Agent traces carry fields such as prompts, reasoning, and tool calls. The data model had to support these fields from the start.
-
Session-level analysis: Agent tasks often span multiple rounds of conversation, so the system had to restore and analyze a full session end to end.
-
Cost analysis: Each agent step consumes tokens, so the system had to track token usage and support cost governance.
-
Trace search: Teams search traces by user, tag, environment, and other metadata, and run full-text search across inputs and outputs.
-
Evaluation loop: Traces feed analysis and replay, so teams can turn production behavior into test data and evaluation samples.
-
Infrastructure correlation: Agent quality depends on the model, the harness, and the infrastructure underneath, so a single agent call may need to connect trace data with sandbox execution, KV cache behavior, and scheduling.
What agent traces demand from the data layer
These six capabilities translate into heavy requirements for the data infrastructure. Trace data arrives as semi-structured JSON full of high-cardinality fields, and writes must become visible in real time. The same system then has to serve metadata search, full-text search, point lookups, and analytical queries, because AgentOps teams evaluate traces by quality, latency, cost, and system behavior.
StepFun summarized four requirements for the data foundation:
-
Wide-table modeling: Trace attributes are modeled and analyzed in wide tables, and dynamic fields such as prompts, tool calls, and retrieval results keep growing.
-
Flexible search: In addition to point lookups by trace ID, the system needs keyword full-text search and search over nested JSON metadata.
-
Multi-dimensional aggregation: The base trace table needs flexible rollups for success rate, quality metrics, and token cost.
-
Mixed workload governance: Online queries and batch-style analytics run on the same data, so the platform needs workload isolation and resource management across query patterns.
Why StepFun chose Apache Doris
Apache Doris matched these requirements point by point:
-
VARIANT type for semi-structured JSON: During ingestion, Apache Doris automatically identifies field names and types in JSON data, extracts them as subcolumns, and stores them in columnar format. This improves compression and query performance, and wide trace tables absorb new dynamic fields without schema changes.
-
Inverted indexes: Apache Doris supports efficient point lookups and keyword filtering through inverted indexes. Doris added inverted index support in 2023, and the feature has run in many PB-scale production environments since.
-
Asynchronous materialized views: Async materialized views give StepTrace a lightweight rollup mechanism with transparent query rewrite, which covers the multi-dimensional aggregation requirement.
-
Stream Load and Workload Groups: Stream Load supports high-throughput real-time ingestion. In StepFun’s practice, Stream Load reached p99 latency under 1 second, and a single request could carry a 500MB batch. Workload Groups isolate online and offline workloads on the same cluster.
-
FE/BE separation: Apache Doris separates metadata management (FE) from data storage and execution (BE), so StepFun can expand resources and run upgrades without manually migrating data. This operational model was one of the main reasons the team chose Apache Doris over ClickHouse.
StepTrace in production
StepTrace is compatible with the Langfuse and OpenTelemetry protocols. For trace context, it follows the W3C traceparent standard and reuses the OpenTelemetry gRPC and collector stack.
Collected trace data flows into Apache Doris through Stream Load. Under GB/s-level write throughput, data becomes visible within seconds.
After ingestion, materialized views take over pre-aggregation and precomputation. Each span or agent step generates token consumption, for example, and StepTrace precomputes token cost per trace and per session so cost queries hit the rollup directly.
Use case: SWE-Agent
SWE-Agent is one of StepTrace’s main use cases: StepFun runs it against SWE-bench, an important benchmark for coding ability. By instrumenting SWE-Agent with SDK-based tracing, StepFun observes each part of the workflow: image pulling, sandbox creation, sandbox execution, multi-turn model calls, and tool calls.
The traces help StepFun monitor SWE-Agent stability and show how each step behaves across rollout and evaluation flows.
Before StepTrace, algorithm engineers relied on logs and trajectory files, and analyses such as horizontal comparison across agent versions, time-series performance analysis, and success-rate analysis were difficult to run. StepTrace put these analyses in one system.
Use case: smart vehicle agent
With stronger on-device computing, vehicle OTA updates, and cloud-edge collaboration, more smart vehicle products are introducing agents to improve the driving and passenger experience. Agents in a vehicle also bring new challenges around safety and cost.
Safety comes first. Vehicle agents ship with many safety constraints, and teams need evidence that those constraints work and that the agent completes tasks in test scenarios.
User input makes evaluation harder: drivers speak in fragmented voice snippets, and intent often becomes clear only after several turns of conversation, which makes the final outcome uncertain.
To improve agent quality, StepFun observes every reasoning step in the execution path, analyzes why the agent missed the expected result, and makes targeted improvements. The team also maintains a reliable evaluation set. When StepFun switches to a new model or updates the architecture, quantitative evaluation confirms whether users will notice the improvement.
What’s next
StepFun’s long-term goal is a unified agent data analytics platform on Apache Doris. Three areas are in progress:
-
Infrastructure-level observability: Production agent execution crosses model decisions, sandbox execution, inference scheduling, and engine-level optimization. StepFun wants unified observability across these layers to locate where an unexpected result came from. The work may involve vLLM prefill and decode, non-intrusive eBPF observability in the sandbox, and security interception.
-
Data feedback loop: StepFun plans to connect trace data with ATIF, a format for agent execution trajectories, and through ATIF with frameworks and benchmarks such as OpenHands, SWE-bench, and Gemini CLI. Trace data then becomes the data foundation for SFT, RL, and evaluation.
-
Data integration: StepFun also plans to connect the trace platform with enterprise data platforms and internal big data components, building a data warehouse around trace data.
If you enjoy reading about this story, join the Apache Doris community on Slack and connect with other Doris experts and users. If you're looking for a fully managed Apache Doris cloud service, contact the VeloDB team.
