Semi-Structured Analytics on AWS: Iceberg V3 Variant, S3 Tables, and VeloDB

AI agents constantly change data and context. And those agent interactions produce semi-structured data, often stored as JSON, with fields that vary across models, tools, searches, and workflows.

Getting real-time analytics on constantly updating semi-structured data is no easy feat. Two common approaches come with painful trade-offs: storing JSON as raw strings adds parsing work to every query, flattening it into predefined columns creates ongoing schema and pipeline maintenance.

The VARIANT data type resolves this. VARIANT preserves JSON flexibility while making the data efficient to query.

Join engineers from AWS and VeloDB to explore a cost-efficient way to store and query semi-structured data with S3 tables and VeloDB.

We'll cover:

• How Iceberg V3's VARIANT type and S3 handle evolving data in open tables.
• How VeloDB combines its own VARIANT type with columnar storage and inverted indexes for fast SQL aggregation and full-text search.
• Reference architecture for semi-structured analytics on AWS, cut query-time JSON parsing and no need for schema flattening.

Speakers:

Saunak Chandra, Principal Solutions Architect, AWS

Kevin Shen, Principal Product Manager, VeloDB