What you describe here are so-called scalar UDFs. They are called "Scalar" because they - at least conceptionally - apply to a single row of data in a single call and also return a single Scalar value as a result.

We have thought about Scalar Python UDFs at length already. Implementing those not immediately straightforward due to two main issues:

1. Function Call Overhead: Calling a function in Python is rather expensive. It will be slow to loop over rows of data calling the function for every single row/value.
2. Global Interpreter Lock: We cannot call into the Python interpreter from multiple threads and get a good speedup because of Python's GIL. Hence we cannot directly use parallelism to alleviate issue 1 a bit.

There are some ideas that we could implement Python UDFs 1 and 2, they each come with their own drawbacks.

• UDF Servers: This is what APACHE Spark does for example. Start a couple of separate Python processes, ship the input data to those processes, run the UDFs there, and ship the results back. This is tricky, DuckDB would need to use sys calls or fork to create more Python processes, manage them somewhat (what happens if one dies etc), reload any Python packages in them, deal with serialisation of data, contention etc. The mechanics of an in-process database doing this are questionable to say the least.
• JIT-Vectorized UDFs: We could use something like Numba to generate a native function (Just-In-Time compilation) that wraps the user-supplied function to run on a larger chunk of data at the same time, much like DuckDB's internal execution engine works. The advantage here is that we do not need a separate process, we would need to study if the Numba semantics exactly match the Python ones, and for which kind of functions the JIT would work for. Not so sure either.

There are some good news, we have already implemented table-valued UDFs for Python. These functions get a chunk of the input table passed as a Pandas data frame, can manipulate it in whatever way they want and are required to return a data frame again. Again, the UDF only sees a chunk of the input at a time. Here is an example

import duckdb
conn = duckdb.connect()
query_result = conn.query("SELECT * FROM generate_series(10)")
print(query_result)
map_result = query_result.map(lambda df: df['generate_series'].add(42).to_frame())
print(map_result)

Here, the table-valued UDF is the lambda in the call to .map, it just adds 42 to the series in the query result.

@hannes I think you're overthinking scalar UDF's and trying to make them performant. I would also really like scalar UDF's in duckdb and I don't care about performance, I just need them to work. So I hope it wouldn't be difficult to implement something simple to use and register.

My use case is mainly using Duckdb for CI / Unit testing / Compatibility. I think scalar UDF's would be valuable regardless of performance. Having them available but slow would be a great value add imo.

hey @hannes is there any update on this feature request?

I've got a branch here that can do this:

import duckdb

from duckdb.typing import *

con = duckdb.connect()

def table_func(df):
	print(df)
	return df

con.register_table_function('no_op_tablefunc', table_func)
rel2 = con.sql('select * from no_op_tablefunc((from range(10)))')
print(rel2)

Empty DataFrame
Columns: [range]
Index: []
Empty DataFrame
Columns: [range]
Index: []
   range
0      0
1      1
2      2
3      3
4      4
5      5
6      6
7      7
8      8
9      9
┌───────┐
│ range │
│ int64 │
├───────┤
│     0 │
│     1 │
│     2 │
│     3 │
│     4 │
│     5 │
│     6 │
│     7 │
│     8 │
│     9 │
└───────┘

This just utilizes the same code as map, only instead of using it on a relation you can register as a function to be used in SQL queries.
Note that like Hannes already mentioned, this is not optimized in any way yet