Storage and Lifecycle Management

Course README · Open Lab 13 · Open Quiz 13

Module goal

Remove expired months by partition, record what each maintenance step changed, and know before the change how to bring back a month that was removed by mistake.

Learning objectives

After completing this module, you will be able to:

1. Choose partition boundaries that match a retention rule and the time that queries filter on.
2. Explain how a row belongs to a partition, a tablet, a replica, a rowset, and a segment.
3. Record before-and-after evidence of a maintenance change with SHOW PARTITIONS, SHOW TABLETS, and SHOW CREATE TABLE.
4. Tell query visibility, metadata changes, and reclaimed space apart, and say where compaction fits.
5. Recover a dropped or truncated partition from the recycle bin, and say when only a backup or an archive can help.
6. Write a maintenance runbook with its scope, checks, evidence, and a recovery path tested beforehand.

Module structure

13.1 Partition by the Retention Rule

An order table holds two years of data, and the retention rule keeps the current calendar month and the 12 months before it. Maintenance in February 2026 keeps February 2025 to February 2026 and removes the months before. A colleague runs DELETE FROM orders WHERE order_date < '2025-02-01': the statement returns quickly, the old orders disappear from queries, and the disk barely changes, because the rows stay in their files until compaction; and if the condition was wrong, there is no way back. Before cleanup SQL, answer four questions:

The third answer is the partition key. Partitioned by load time, a January order that arrives in February would sit in February's partition, out of reach of January's cleanup, and a filter on order_date would not prune. A rule in calendar months also differs from a rolling window of 365 days: whole months can be removed by partition, a window ending on some day cannot.

On a Duplicate Key table, a conditional DELETE writes a version with a delete predicate instead of rewriting files; SHOW DELETE lists it, and queries filter by it until compaction merges the rowsets. Many predicates slow queries down.

Monthly partitions offer two statements:

Dynamic partitioning can create future partitions and remove historical ones on a schedule; it removes old ones only when dynamic_partition.start says how many to keep. Whether the cleanup is manual or scheduled, check the archive of a month before you remove it.

13.2 Where a Row Lives

The business speaks of January's orders; Doris stores them in tablets and rowsets. Follow order 130001 of Lab 13, of 2025-01-15:

A cleanup of p202501 touches January's tablets and leaves February's and March's alone. The layout decides how many tablets there are:

Each row of SHOW TABLETS is one replica, which is why a partition clause helps on a real table. Too few buckets make large tablets and little parallelism; too many make small tablets and more metadata to schedule. One BE and one replica illustrate the relations; they are not a size for production.

13.3 Evidence Before and After Maintenance

A wrong grant can be revoked; a wrong cleanup has no undo statement that always works. Record the state before the change, and run the same checks afterward:

VisibleVersion changes as soon as a load commits; RowCount reads -1 or 0 until the BE's next report, about a minute later. SHOW TABLETS has no partition column, so name the partition. The sizes mean different things in integrated and in decoupled storage and compute; this course's sandbox is integrated.

13.4 What TRUNCATE Changes

SQL reading example: the step of Lab 13 that clears January. Do not rerun it after the Lab.

TRUNCATE TABLE ops_orders_l3 PARTITION (p202501);

TRUNCATE replaces the partition with a new, empty one of the same name and range; it neither deletes rows one by one nor records a delete predicate, so it does not wait for compaction. The recycle bin keeps the old partition for catalog_trash_expire_second, 86400 seconds by default: one day. TRUNCATE ... FORCE skips the recycle bin.

Compaction merges a tablet's rowsets into fewer, larger ones in the background, and removes the rows of delete predicates as it goes; that is why the colleague's DELETE changed the queries before the disk. It has nothing to do with whether a truncation succeeded. Watch its pressure with VersionCount and the load and query rates, not with one small sample.

13.5 Ways Back

The recycle bin keeps truncated and dropped partitions, and RECOVER PARTITION brings one back while it is there. Lab 13 recovers the truncated January:

1. SHOW CATALOG RECYCLE BIN WHERE NAME = 'p202501' lists every dropped p202501, of every table and every run; keep the entry whose PartitionId you recorded before the truncation.
2. RECOVER PARTITION p202501 FROM ops_orders_l3 is refused while the empty replacement is in the table: partition[p202501] already exist in table[ops_orders_l3]. Check that the replacement is still empty: rows written to it since would be lost.
3. Drop the replacement, which goes to the recycle bin too, and recover by the ID.

SQL reading example: the recovery of Lab 13. Do not rerun it after the Lab.

ALTER TABLE ops_orders_l3 DROP PARTITION p202501;
RECOVER PARTITION p202501 <the recorded PartitionId> FROM ops_orders_l3;

January comes back with its original ID and its order: 3 orders and 600.00. Without the ID, measured on Doris 4.1.3, RECOVER PARTITION p202501 FROM ops_orders_l3 recovers the newest p202501 in the recycle bin, the empty replacement, and January stays empty. A DROP PARTITION without FORCE, as on Lab 13's second table, comes back with a RECOVER by name, because only one p202501 of that table is there.

The recycle bin is a short way back. The durable ones are prepared before the change:

Lab 13's optional sections run a repository backup: they start a small local object store, the Silo of Module 4, register it as a repository, back up a table, drop it with FORCE, and restore its snapshot under another name in another database, where every row is compared. Doris and the object store share one computer there, so losing the computer would lose both copies: a backup for disaster recovery needs another failure domain, a tested recovery point and recovery time, and managed credentials and retention. The drill covers an ordinary table of an integrated storage and compute deployment, not decoupled storage and compute, the cluster's configuration, or external lake tables.

13.6 A Maintenance Runbook

The next operator needs more than "clear January". Write the runbook before the change:

Test the recovery before the change: read the archive back and compare its rows with the partition, instead of finding out after a mistake that it cannot be read. Lab 13's practice checks a retention plan in SQL before it removes anything: from the maintenance date 2026-02-15 it computes the first month to keep, 2025-02-01, lists the partitions that end on or before it, drops them without FORCE, and checks that 13 months remain.

Lab 13: Partition Maintenance and Recovery

Open Lab 13: Partition Maintenance and Recovery. You:

1. Create ops_orders_l3 with three monthly partitions, one order each.
2. Record its partitions, tablets, and definition.
3. Clear January with TRUNCATE, and compare the IDs before and after.
4. Recover the truncated January by its partition ID.
5. Drop and recover January on a second table.
6. Optionally, back up a table to a repository, drop it with FORCE, and restore it in another database.
7. Practice: check a retention plan for 13 calendar months, then apply it.

You need only the course sandbox. The optional drill starts the object store and keeps its snapshot; the Lab says how to remove it.

Summary

• Partition by the time that decides expiry, the order date, in the rule's units, calendar months; drop an expired month, truncate one that you will reload.
• A row lives in a partition, a tablet of it, a replica of the tablet, a rowset, and its segments; a partition's cleanup touches only its tablets.
• Record each partition's ID and version, its tablets, and the table's definition before a change, and count rows per partition: RowCount lags.
• Queries change at once, metadata gets new IDs, and space comes back after the recycle bin's retention; compaction merges rowsets and removes deleted rows, apart from truncation.
• Recover a truncated partition by its recorded ID, after removing its empty replacement; only a backup, an archive, or the source can help after FORCE or the retention.
• A runbook names the scope, the checks, the change, the evidence, and a recovery that was tested before the change.

Quiz 13

Open Quiz 13: Storage and Lifecycle Management. It has six single-choice questions and needs no running environment.

Next module: Module 14: Resource Isolation and Query Protection.

References

• Apache Doris: Partitioning and bucketing
• Apache Doris: Dynamic partitioning
• Apache Doris: Delete
• Apache Doris: Truncate
• Apache Doris: Recycle bin
• Apache Doris: Backup
• Apache Doris: Restore
• Apache Doris: Data compaction