I know it is eaarly (Python 3.11 has just been released yesterday) but we are hoping in Apache Airflow to a much faster cycle of adding new Python releases - especially that Pyhon 3.11 introduces huge performance improvements (25% is the average number claimed) due to a very focused effort to increase single-threaded Python performance (Specialized interpreter being the core of it but also many other improvements) without actually changing any of the Python code.

The databricks-sql-python is one of the dependencies of Airlfow that need to support newer Python version so this issue is here to make you aware that we are eaager (also to help if needed) to make it faster, possibly by talking to dependencies of Beam as well (who are oftten also direct dependencies of Airflow) and help them as well. I perfectly understand we need to bubble up support so that your dependencies support them first.

Nice summary of Py3.11 support is here: https://pyreadiness.org/3.11/ - it's not very green obviously, but I hope it gets greener soon.

I'd appreciate if someone in the Databricks team attempted to migrate and have a PR running (and failing possibly until all prerequisite are met so that we can also track the progress and possibly help each other to solve any problems. I know PyArrow is blocking for now, but this is going to be solved soon lilely with apache/arrow#14499 and just opening a failing PR with it might be a good start.

I just opened such PR in Apache Airflow yesterday and plan to keep it open until it gets green :)

apache/airflow#27264

I think it would be fantastic if we could as the open source community migrate to the new Python much faster.

Looking forward to cooperation on that one :)