Apache Doris Python UDF: Calling the Python Ecosystem from SQL for Agent-Era

A payments team keeps its fraud rules in Python. The transaction data lives in Apache Doris. Every night, a job exports the day's payments to object storage, an Airflow task runs the scoring script, and the results load back into Doris for the morning dashboard.

The loop is slow, and it is slower to change. Scores trail the data by hours. When a rule changes, someone edits the Python repo, redeploys the job, and backfills the table. When a score looks wrong, the debugging session spans SQL, scheduler logs, and a Python codebase, each with its own permissions and its own owner.

The workarounds carry their own costs. Calling a scoring service row by row keeps the logic in one place, and query latency then swings with every network hiccup. Rewriting the rules in pure SQL keeps everything in the database, and the team gives up Pandas, NumPy, and the model libraries the logic was built on.

Python UDF, new in Apache Doris 4.1.3, ends the round trip. The scoring function runs inside the SQL query: data stays in Doris, logic stays in Python, and the score comes back in the same SELECT that reads the table.

What is a Python UDF?

A Python UDF is a Python function registered in Doris through CREATE FUNCTION and called from SQL like any built-in. Doris manages the execution end to end: it batches input columns, hands them to a managed Python runtime, and merges the returned values back into the query plan.

The call site is ordinary SQL:

SELECT
    user_id,
    amount,
    py_risk_level(amount) AS risk_level
FROM payment_events
WHERE dt = '2026-06-17'
ORDER BY user_id;

py_risk_level is Python. Everything around it is Doris.

That scalar shape is 1 of 3 function forms the extension supports, one per input-to-output relationship:

A cleaning function, a custom aggregate metric, and a JSON exploder all register the same way and run through the same machinery.

Batches, vectors, and process pools

Calling Python from a C++ engine costs time at the boundary. Doris keeps that cost low with 3 design choices: columnar batching, vectorized execution inside the batch, and a pooled, isolated runtime around it.

Column batches over Arrow Flight

During execution, the Doris BE packs input data into Arrow RecordBatches and sends them over Arrow Flight to a separate Python Server. The function computes on the whole batch, and results return to the query path in columnar form.

One boundary crossing covers thousands of rows, so process switching and serialization stop dominating the cost. The Python path stays consistent with Doris's columnar execution engine and keeps most of its query speed while running complex business logic.

The batch execution path (English rebuild of the original diagram)

Vectorized execution with Pandas Series

Inside each batch, the function can go faster still. Declare the input as pd.Series, and Doris hands your function whole columns, so the work runs through Pandas's underlying implementation instead of a Python loop. For example, bucketing payment amounts:

CREATE FUNCTION py_amount_bucket(DOUBLE)
RETURNS INT
PROPERTIES (
    "type" = "PYTHON_UDF",
    "symbol" = "evaluate",
    "runtime_version" = "3.10.12",
    "always_nullable" = "true",
    "volatility" = "immutable"
)
AS $$
import pandas as pd

# Declare pd.Series types explicitly to run vectorized
def evaluate(amount: pd.Series) -> pd.Series:
    return pd.cut(
        amount,
        bins=[-float("inf"), 100, 1000, 10000, float("inf")],
        labels=[0, 1, 2, 3]
    ).astype("Int64")
$$;

Vectorized execution cuts interpreter overhead, which pays off on the workloads UDFs attract most: string processing, feature computation, field conversion, and bucket mapping over large scans.

A pooled, isolated, self-healing runtime

The same design has to protect the engine from the code it runs. User functions execute in separate Python Server processes, pooled per runtime_version and managed by Doris.

Process pool management (English rebuild of the original diagram)

The key mechanisms and what each one buys:

To the developer, a Python UDF is still an ordinary SQL call. Doris handles process management, resource reuse, and failure recovery underneath.

Set up and call your first function

Getting to a working function takes 3 steps: configure the BE nodes, register the function, call it.

Python UDF ships in Apache Doris 4.1.3 and later. Enable the Python UDF settings on every BE node, and install pandas and pyarrow in the target Python environment. When you need to debug, the Python UDF Server writes its logs to output/be/log/python_udf_output.log.

Python UDF usage overview (English rebuild of the original diagram)

The payment-risk rule from the opening becomes:

DROP FUNCTION IF EXISTS py_risk_level(DOUBLE);

CREATE FUNCTION py_risk_level(DOUBLE)
RETURNS STRING
PROPERTIES (
    "type" = "PYTHON_UDF",
    "symbol" = "evaluate",
    "runtime_version" = "3.12.11",
    "always_nullable" = "true",
    "volatility" = "immutable"
)
AS $$
def evaluate(amount):
    if amount is None:
        return None
    if amount >= 10000:
        return "high"
    if amount >= 1000:
        return "medium"
    return "low"
$$;

With the function registered, the SELECT from the top of this post runs as written. The nightly export, the scheduler task, and the backfill all drop out; the score computes at query time.

Code organization scales with the function. For quick validation, write the Python inline in the CREATE FUNCTION statement, as above. For production, package the code as a ZIP and point file and symbol at the module entry:

CREATE FUNCTION py_add_one(INT)
RETURNS INT
PROPERTIES (
    "type" = "PYTHON_UDF",
    "file" = "file:///opt/doris/udf/math_ops.zip",
    "symbol" = "math_ops.add_one",
    "runtime_version" = "3.10.12",
    "volatility" = "immutable"
);

The ZIP path gives you code review, dependency management, and versioned releases, the same way the rest of your Python ships.

Where Python UDF pays off

The payment-risk score is one function. The pattern behind it, logic that used to need an export now running where the data lives, covers 3 kinds of work:

• Data processing. Text parsing, JSON splitting, field standardization, outlier handling.

• AI analysis. Tag extraction, feature engineering, model scoring, embedding generation.

• Business rules. Risk assessment, tiering and bucketing, compliance validation, complex mapping.

Where Python UDF applies (English rebuild of the original diagram)

The first column cleans what arrives, the second feeds models, and the third encodes judgment. All 3 used to sit in external scripts for the same reason: they needed Python. With the UDF path, Doris covers the aggregation work of classic real-time analytics and the scoring, tagging, and embedding work that AI applications add on top.

Try it on one function

Python UDF connects Doris's SQL analytics to Pandas, PyArrow, and the rest of the Python ecosystem. Rules, features, and model scores run inside the query path, pipelines get shorter, and data stays where it is governed.

The fastest way to evaluate it is small. Upgrade to Apache Doris 4.1.3, enable the BE settings, and port one function: a cleaning step, a risk rule, or a bucketing map. Then measure the pipeline stages it retires.

For production deployments, VeloDB, the commercial distribution of Apache Doris, supports Python UDF, UDAF, and UDTF with enterprise operations, stable releases, security compliance, and technical support. VeloDB Cloud has a free trial if you want the managed path.

Special thanks to @WencongLiu from ByteDance and @sjyango from Tencent for their collaboration and contributions to the community's work on Python UDF. Everyone interested in Doris is welcome to join the community and help the project grow.