Apache Doris + Lance: Vector Search and SQL Analytics on Multimodal Data for Autonomous Driving and Embodied AI

TL;DR. Apache Doris 4.2, planned for late September 2026, reads Lance datasets through a Lance Catalog and runs approximate nearest neighbor search on existing Lance vector indexes with a vector_search() table function. Autonomous driving and robotics teams can pre-filter by scenario, search by embedding, and aggregate disengagement or task failure rates by model version in one SQL statement. Lance access is read-only in 4.2.

What does Apache Doris 4.2 add for Lance?

Apache Doris 4.2 adds read access to Lance, the open-source columnar format for multimodal AI data, through two paths: a Lance Catalog that maps Lance namespaces to Doris databases and tables, and the s3() and local() table functions with "format" = "lance". On top of that, a vector_search() table function reuses the vector indexes that Lance ecosystem tools have already built, so a similarity search and the OLAP aggregation that follows it run in the same query. Upstream tools keep writing the data; Doris queries it in place.

The rest of this post explains why this matters for Physical AI teams, walks through an autonomous driving pipeline and a robotics pipeline, and shows the SQL.

What is Lance, and why does Physical AI data end up in it?

Lance is an open source columnar format for multimodal AI, maintained by the lance-format project and used by LanceDB. It stores column data in pages optimized for random access on object storage, and it treats vector, scalar and full-text indexes as first-class objects that live alongside the data. The Lance Lakehouse Format is defined as a stack of specifications: a file format, a table format with fragments and ACID commits, index formats, and two catalog specs, a Directory Catalog for zero-infrastructure deployments on object stores and a REST Catalog for governed enterprise access. When this post says Lance Catalog, it means the catalog layer of the Lance format as Apache Doris reads it.

An autonomous vehicle generates video, point clouds, vehicle state, and model outputs all day. Every robot task leaves behind multiple camera streams, joint states, action trajectories, language instructions and execution results. Teams store this in Lance because training loaders need random access to individual samples and because the embeddings and indexes can sit next to the raw data. What decides how fast a model iterates is whether a team can answer these questions quickly, from the full corpus:

• Which clips contain scenes similar to this incident?

• Did the new model’s disengagement rate rise at night, in construction zones, or around unusual obstacles?

• Which robot tasks fail most often, and what visual or motion patterns appeared before the failure?

• How do the selected high value episodes get handed to labeling, training and regression evaluation?

A traditional data warehouse aggregates structured facts well. A vector database finds similar images well. Object storage keeps petabytes of files cheaply. Physical AI needs semantic search, structured filtering, business metrics and the raw evidence in the same analytical chain.

Why does Physical AI need semantic search and SQL analytics together?

Autonomous driving and embodied AI data is organized around episodes. An episode is a reproducible physical process: sensors observe the environment, a model makes a decision, the control system acts, and the real world responds.

In autonomous driving, “find the frames that look like the 5 seconds before the incident” is only the first step. The engineering team then narrows by vehicle model, sensor version, road type, weather, model version and time range, and compares disengagement rate, false positive rate or planning deviation across the candidates. Embodied AI works the same way: a failed grasp involves visual semantics plus task type, robot arm model, policy version, joint state, action trajectory and final result.

High value data discovery therefore runs through at least three layers of computation: structured conditions that narrow the search space to the right business context; vector similarity search that surfaces related scenes the label taxonomy has not covered yet; and aggregation, joins, ranking and window analysis that turn candidate clips into model evaluation conclusions and executable datasets.

Lance and Doris complement each other at exactly this point. Lance manages the AI datasets, random access and vector indexes. Doris returns the search results to an explainable, composable SQL analysis system. One autonomous driving company already runs multimodal search on Apache Doris with data loaded into Doris tables; Lance Catalog extends the same query model to data that stays in the lake.

How do Apache Doris and Lance divide the work?

Figure 1. Where Apache Doris + Lance sit in a Physical AI data architecture: upstream tools write Lance datasets and indexes to storage, Doris reads them through the Lance Catalog, and people and agents query through SQL.

Upstream, teams keep using the Lance SDK, Ray, Spark and similar tools to produce and maintain Lance datasets: features, embeddings, labels, model outputs and existing Lance vector indexes. The data can live on a local file system, S3-compatible object storage or Alibaba Cloud OSS. Lance access in Doris is read-only: upstream tools write and maintain the datasets and indexes, and Doris queries them.

Apache Doris discovers datasets and schemas through the Lance Catalog. Within a single query it reads one consistent dataset version, splits the work by fragment or index segment, runs scans and vector search in parallel, and finishes with global top-k and OLAP computation. Data stays where it is: no copy into a dedicated analytics replica, and no AI data asset locked into a single compute system.

Doris covers the layer that is easiest to neglect before and after training: giving engineers, algorithm teams, business analysts and agents one consistent way to discover data, validate models, locate problems and assemble the next training set. This is the same lakehouse query model Doris already applies to Iceberg and Paimon, extended to a format built for embeddings and media.

How does corner case mining work for autonomous driving with Doris + Lance?

Finding high value scenes in fleet data is the bottleneck

In autonomous driving R&D, corner case mining, also called edge case or hard case mining, means finding the scenes where a model misclassifies, misses a detection, triggers abnormal braking or forces a disengagement. These clips are a tiny share of all fleet data and have an outsized effect on model safety and long tail coverage.

Finding the right data in a huge driving log is the hard part. The main bottlenecks, and how the architecture addresses each:

• Anomalies are rare and their context is scattered. A vehicle continuously produces multiple camera streams, point clouds, localization, vehicle state and model outputs. One anomaly may exist in a few seconds of footage, under a particular vehicle model, road, weather and sensor version. Doris narrows the search space first by time, vehicle model, road, weather, model version and other business conditions.

• Existing labels do not cover every meaning. Querying by time, vehicle or label alone misses clips that are semantically similar and still unlabeled. Lance stores the episode metadata, embeddings and vector index; Doris reuses the index to retrieve similar scenes.

• Similar results are not yet an engineering conclusion. Vector similarity on its own can return results that are irrelevant to the business context, and it says nothing about problem frequency, regression across model versions or sample value. Once candidates are in Doris, distributed statistics and version comparison decide which episodes are worth labeling, training and evaluation.

A corner case mining pipeline that works in practice

Suppose the team wants to find clips that are visually similar to a specific hard braking event caused by an unusual obstacle, restricted to urban roads, and check whether the problem concentrates in one perception model version.

1. The Lance dataset stores episode_id, timestamp, road and weather labels, vehicle model, model version, disengagement or hard braking outcome, image or clip embeddings, and pointers to the raw media.

2. Doris pushes down the time range, urban road, vehicle model and model version conditions, narrowing the search space before vector candidates are generated.

3. Doris reuses the existing Lance vector index and searches in parallel. Fragments the index does not cover are completed automatically with flat (brute-force) search.

4. Each parallel task returns local candidates, and Doris merges them into a global top-k.

5. SQL aggregates disengagement rate, hard braking rate and sample count by model version, weather and road type, producing the corner case distribution.

6. Candidate episode_ids move on to human review, labeling, the training set or the regression evaluation set, with the raw media read precisely through the Lance SDK.

Figure 2. Corner case mining for autonomous driving: pre-filter, vector search on the Lance index, global top-k merge, SQL aggregation, then review, labeling, training and regression evaluation.

The point of this path is to keep “similar scenes” attached to their business context. The algorithm engineer can answer which versions the problem appears in, whether it is regressing, how many vehicles and road conditions it covers, and which episodes deserve a place in the next training round.

Finding similar scenes with one SQL statement

Assume an upstream tool has already built a vector index on the scene_embedding column of the Lance dataset. The intervention column records whether the drive ended in a disengagement. Doris runs the search through vector_search():

SELECT
    episode_id,
    event_time,
    road_type,
    weather,
    model_version,
    intervention,
    _distance
FROM vector_search(
    "table" = "lance_ai.autodrive.scene_episodes",
    "column" = "scene_embedding",
    "query_vector" = "[0.12, -0.08, 0.31, ...]",
    "top_k" = "200",
    "metric" = "cosine",
    "nprobes" = "20",
    "refine_factor" = "10",
    "filter" = "road_type = 'urban' AND event_time >= '2026-08-01'",
    "use_index" = "true"
)
ORDER BY _distance ASC, episode_id;

The filter here is a pre-filter: it fixes the scene scope first and generates vector candidates within it, which matches the semantics of corner case mining better than post-filtering with a WHERE clause outside the table function. A smaller _distance means a closer match. Adding a unique key as the second sort field keeps the return order stable when distances tie.

Vector candidates can flow straight into standard SQL aggregation. For example, the disengagement distribution across similar scenes by model version:

WITH similar_scenes AS (
    SELECT model_version, weather, intervention, _distance
    FROM vector_search(
        "table" = "lance_ai.autodrive.scene_episodes",
        "column" = "scene_embedding",
        "query_vector" = "[0.12, -0.08, 0.31, ...]",
        "top_k" = "1000",
        "metric" = "cosine",
        "filter" = "road_type = 'urban'",
        "use_index" = "true"
    )
)
SELECT
    model_version,
    weather,
    COUNT(*) AS scene_count,
    SUM(CASE WHEN intervention THEN 1 ELSE 0 END) AS intervention_count
FROM similar_scenes
GROUP BY model_version, weather
ORDER BY intervention_count DESC;

How does embodied AI episode analysis work with Doris + Lance?

Robot data is hard because of multi-rate alignment, random access and cross-episode analysis

A single embodied AI task produces data at several rates and in several shapes at once. Arm state and control signals may be sampled at hundreds of hertz. Multiple cameras record the environment at tens of frames per second. On top of that come language instructions, per-frame labels, intermediate model results and the success, failure and retry record for the task.

When this data is split across video files, columnar files and JSON, the team maintains timestamp alignment and version mapping by hand and rewrites the same stitching logic for training, replay and problem analysis. A training framework cares about reading a batch of samples at high throughput. R&D analysis cares about locating similar failures across millions of episodes, comparing policy and hardware versions, and deciding whether a problem belongs to perception, planning, control or execution. Random access alone cannot do that cross-episode aggregation, and a warehouse style full table scan is the wrong tool for every multimodal sample access.

Lance organizes state, actions, features, embeddings and media pointers by episode and timeline, and keeps the random access path that training and review need. Doris performs conditional filtering, vector search and cross-episode aggregation on the fields it supports. The split lets robot data serve model training and unified analysis and review at the same time.

From a failed task to a training data loop

For a problem like “the arm fails to grasp transparent or reflective objects”, the team can run this flow:

1. Multi-camera streams, joint states, action trajectories, instructions, policy version and task result land in Lance per episode.

2. An upstream model generates visual or trajectory embeddings and builds a vector index in Lance.

3. Doris pre-filters by robot model, task type, policy version and success = false.

4. Vector search finds frames or motion segments similar to the target failure pattern.

5. Doris aggregates failure reason, retry count, task duration, hardware and policy version to decide whether the problem is perception, planning, control or hardware related.

6. High value episodes go to review, relabeling and training, and the repaired model is regression tested on the same dataset.

Figure 3. Embodied AI episode analysis: multi-rate sensor data aligned per episode in Lance, cross-episode analysis in Doris, and a training data loop closed on the same dataset version.

This loop fits agentic data workflows well. An agent can scope the task with SQL, call vector search to find similar failures, and then produce distribution statistics and a candidate episode list. Every step keeps explicit filter conditions, SQL and a data version, which is far easier to explain and review than letting a model search blindly through a file directory or a vector store. The same pattern underpins context engineering on VeloDB, where agents retrieve fresh, filtered context through SQL.

A robot failure pattern search

The query_vector is generated upstream with the same embedding model used for frame_embedding. A reference image, a representative frame from a video, or a motion trajectory goes into the model, and the resulting float array is passed to vector_search(). The query vector’s dimension, numeric type and normalization must match the dataset’s vector column and the settings used when the index was built. The [-0.04, 0.27, 0.18, …] below only illustrates the parameter format.

SELECT
    episode_id,
    robot_model,
    policy_version,
    task_type,
    failure_reason,
    retry_count,
    _distance
FROM vector_search(
    "table" = "lance_ai.robotics.task_episodes",
    "column" = "frame_embedding",
    "query_vector" = "[-0.04, 0.27, 0.18, ...]",
    "top_k" = "300",
    "metric" = "cosine",
    "filter" = "task_type = 'pick' AND success = false",
    "use_index" = "true"
)
ORDER BY _distance ASC, episode_id;

The results can be aggregated by policy_version, robot_model or failure_reason, or returned as episode_id plus media pointer fields so the application opens the matching video frames, point clouds or other blobs through the Lance SDK for review.

How do you use Lance Catalog in Apache Doris?

Connect to a Lance dataset on object storage

The example below uses S3-compatible object storage. In production, supply access keys through a secure credential manager and keep real keys out of shared scripts.

CREATE CATALOG lance_ai PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "filesystem",
    "warehouse" = "s3://<bucket>/physical-ai",
    "s3.endpoint" = "https://<object-storage-endpoint>",
    "s3.region" = "<region>",
    "s3.access_key" = "<access-key>",
    "s3.secret_key" = "<secret-key>"
);

Doris also supports the local file system, Alibaba Cloud OSS, and Lance REST Catalogs authenticated with a bearer token, API key or custom header. A filesystem catalog (the Lance specification calls it a directory catalog) maps namespaces by directory: the first level of directories under the warehouse root maps to Doris databases, and each .lance directory maps to a table.

Discover datasets, schemas and existing indexes

SHOW DATABASES FROM lance_ai;
SHOW TABLES FROM lance_ai.autodrive;
DESC lance_ai.autodrive.scene_episodes;
SHOW INDEX FROM lance_ai.autodrive.scene_episodes;
SHOW INDEX lists the indexes that Lance ecosystem tools have already created.

Run a vector query through the catalog

SELECT episode_id, event_time, model_version, _distance
FROM vector_search(
    "table" = "lance_ai.autodrive.scene_episodes",
    "column" = "scene_embedding",
    "query_vector" = "[0.12, -0.08, 0.31, ...]",
    "top_k" = "200",
    "metric" = "cosine",
    "filter" = "road_type = 'urban'",
    "use_index" = "true"
)
ORDER BY _distance ASC, episode_id;

Creating the catalog, discovering datasets and running vector search all happen inside the same catalog namespace. The filter restricts the search to urban road scenes before candidates are generated, and _distance returns the distance under the chosen metric. The query vector’s dimension and the distance metric must match the ones used upstream to generate the embeddings and build the Lance index.

Pre-filtering or post-filtering: which should you use?

Pre-filtering and post-filtering differ in when the condition participates in candidate generation.

• A condition in the filter parameter of vector_search() is a pre-filter. Lance selects the rows that satisfy the condition first and runs the vector search inside that subset, so top_k means top-k within the filtered data. Use it for road type, time range, vehicle model, tenant, data permissions and any other condition that must constrain the search space.

• A condition in the WHERE clause outside the table function is a post-filter. Vector search produces candidates first and Doris filters them afterward. Use it for expressions Lance cannot push down yet, or for a “search by meaning first, then apply business filters” flow. The final result may contain fewer than top_k rows, and the system does not widen the search to make up the difference.

Real queries combine both. Put the strong constraints that define the search scope in filter, put complex computation and remaining conditions in the outer WHERE, and use EXPLAIN to confirm which conditions were pushed down.

Confirm predicate pushdown and index hits with EXPLAIN

Run EXPLAIN on the same vector_search() query before executing it. In the plan, lancePushdownPredicate shows that the pre-filter condition was pushed down, and lanceSearchIndexSegments shows that Lance index segments were planned. Fragments added after the index was built are completed with flat (brute-force) search, so the result is complete for the current dataset version.

EXPLAIN
SELECT episode_id, _distance
FROM vector_search(
    "table" = "lance_ai.autodrive.scene_episodes",
    "column" = "scene_embedding",
    "query_vector" = "[0.12, -0.08, 0.31, ...]",
    "top_k" = "200",
    "metric" = "cosine",
    "filter" = "road_type = 'urban'",
    "use_index" = "true"
)
ORDER BY _distance ASC, episode_id;

Read a dataset ad hoc with a table function, without a catalog

For a one-off analysis of a specific Lance dataset that you do not want to register in a catalog, the s3() table function reads it by path. The uri must point to the root directory of one Lance dataset. The table function suits path-based reads and analysis on scalar columns. When you need to reuse the datasets, schemas and vector indexes in a catalog, use the Lance Catalog and vector_search() described above.

SELECT episode_id, event_time, model_version
FROM s3(
    "uri" = "s3://<bucket>/physical-ai/autodrive/scene_episodes.lance",
    "s3.endpoint" = "https://<object-storage-endpoint>",
    "s3.region" = "<region>",
    "s3.access_key" = "<access-key>",
    "s3.secret_key" = "<secret-key>",
    "format" = "lance"
)
WHERE event_time >= '2026-08-01';

What Doris + Lance does not do yet

Lance access in Doris 4.2 is read-only. Creating, updating or deleting Lance tables, and exporting query results as Lance files, stay with the Lance SDK and upstream tools; Lance export is on the roadmap below. Queries always read the current dataset version, so there is no time travel to an earlier snapshot. Full-text and hybrid search over Lance columns are out of scope in this release, even though Doris supports both on its own tables. Index management from Doris (create, rebuild, drop) is tracked in the 4.2 roadmap issue and should be confirmed against the release notes.

Performance numbers are also still to come. The benchmarks listed in the roadmap have not been published, so treat the query patterns here as a design guide and validate latency and recall on your own data scale, vector dimension and filter selectivity before committing to production SLAs. Recall depends on nprobes and refine_factor for IVF indexes and on ef for HNSW indexes, and the right values differ by dataset.

Finally, if your workload is a single application serving low latency k-nearest-neighbor lookups with no filtering, joins or aggregation behind them, a vector database or the LanceDB query path alone may be the simpler fit. Doris + Lance earns its place when the similarity search is the first step of an analysis, and the analysis is the deliverable.

What is on the Doris + Lance roadmap?

The Iceberg, Paimon and Lance multimodal lakehouse capabilities have been merged into the Doris 4.2 release branch. Doris 4.2 is planned for release at the end of September 2026, with 5.0 planned for November 2026 per the roadmap issue.

Follow-on work on Doris + Lance concentrates on performance, stability and usability:

• reproducible benchmarks across data scale, vector dimension, top-k, filter selectivity, index coverage, concurrency and cold versus warm cache;

• measurement of P50, P95 and P99 latency, throughput, recall, bytes scanned, object storage requests, network traffic, peak CPU and peak memory;

• continued optimization of split balancing, large top-k merges, high dimensional vectors, wide result column fetches, and object storage read and cache paths;

• scenario-based guidance for nprobes and refine_factor on IVF indexes, ef on HNSW indexes, use_index and related parameters, plus capacity planning and troubleshooting guides;

• fuller Lance table management;

• export of data as Lance files.

The Apache Doris x Lance series will continue with deeper articles on the distributed execution design, along with measured performance results and parameter tuning experience. Beyond Lance, the multimodal lakehouse series will cover Iceberg, Paimon and other open formats.

Frequently asked questions

Does Apache Doris support the Lance format? Yes, starting with Apache Doris 4.2. Doris reads Lance datasets through a Lance Catalog (filesystem or REST catalog types) or through the s3() and local() table functions with "format" = "lance", and it runs vector search on existing Lance vector indexes with the vector_search() table function. The Doris documentation lists Lance support as available from 4.2, which is planned for late September 2026.

Can Apache Doris write to Lance tables? No. In Doris 4.2, Lance access is read-only: creating, writing, updating and deleting Lance tables is unsupported, and there is no time travel. Datasets and indexes are written and maintained upstream with the Lance SDK, Ray, Spark or similar tools. Export of query results as Lance files is on the Doris + Lance roadmap.

What is the difference between pre-filtering and post-filtering in vector search? Pre-filtering applies the condition before candidates are generated, so the top-k results come from the filtered subset only. In Doris + Lance, the filter parameter of vector_search() is a pre-filter. Post-filtering runs the vector search first and applies the condition to the candidates afterward, so the final result can contain fewer than top-k rows. In Doris + Lance, a WHERE clause outside the table function is a post-filter.

Which vector index types does Doris support on Lance datasets? The Doris 4.2 roadmap lists IVF_FLAT, IVF_SQ, IVF_PQ and the IVF_HNSW variants, with L2, cosine, dot product and hamming distance metrics. The indexes are built by Lance ecosystem tools, and Doris reuses them; fragments that the index does not cover are searched with flat (brute-force) search so the result is complete for the current dataset version. Query-time parameters are nprobes and refine_factor for IVF indexes and ef for HNSW indexes.

When should you use Lance instead of Parquet for multimodal AI data? Lance is the better fit when training loaders, review tools or search need random access to individual rows and blobs, and when vector, scalar or full-text indexes should live next to the data as versioned table objects. Parquet remains the default for scan-heavy analytics where whole column chunks are read in bulk. With Lance Catalog support, Doris can query both formats from the same SQL interface, so the choice can be made per dataset.

Closing

In autonomous driving and embodied AI, the scarce resource is the high-value episode: the one that explains a model problem, covers a long-tail scenario and drives the next iteration. Lance makes AI data organized, indexed and randomly accessible. Doris makes that data filterable, searchable and aggregatable by people and agents through one SQL interface. When “find the similar scenes” flows into “which version did this affect, how often, is it regressing, and which training set should it enter”, multimodal data has entered the loop of model iteration.

A commercial preview build with Lance Catalog support is available. Teams evaluating Iceberg, Paimon or Lance, or with real-time lakehouse and multimodal analysis requirements, can contact VeloDB for technical discussion, solution assessment and testing support.