Description

As part of the migration of the Databricks API from 2.1 to 2.2, the get_run method of the hook did not support pagination for specific array properties.
We can create Databricks Workflows with more than 100 tasks, but the monitoring of tasks in Airflow fails in random tasks because the get_run retrieves only 100 tasks. Therefore when we try to check the status of a task we get an KeyError

Solution

Added a new hook method get_run_tasks to support pagination on the tasks array property and solve the problem mentioned above.

Links

Databricks API Get a single job run

related: #59189
related: #59217

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.