I was able to verify the fix

Before:

After:

Setup Spark Connect Tutorial: https://medium.com/@yssmelo/spark-connect-launch-spark-applications-anywhere-with-the-client-server-architecture-dbt-f99399c566fe

Sample App:

import streamlit as st

from pyspark.sql import SparkSession

# spark = SparkSession.builder.getOrCreate()

spark = SparkSession.builder.remote("sc://0.0.0.0:15002").getOrCreate()

from datetime import datetime, date
import pandas as pd
from pyspark.sql import Row

df = spark.createDataFrame([
    Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),
    Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),
    Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))
])
st.write(type(df))
st.table(df)

@raethlein I noticed earlier that this PR failed on one of the linters. Would you like me to apply the changes to fix or are you in the middle of making more changes here? I don't have a preference either way

@OSalama It looks like I cannot push to your branch with some formatting to get the Enforce Pre-Commit Hooks gate to pass. Could you try to run the precommit hooks locally (or at least try to run ruff format) which seems to be the issue here?

Update: Oh I have just seen your comment 🙂 I wanted to help out but I don't seem to be able to push to your branch. Would you mind trying to run the formatter?

@OSalama It looks like I cannot push to your branch with some formatting to get the Enforce Pre-Commit Hooks gate to pass. Could you try to run the precommit hooks locally (or at least try to run ruff format) which seems to be the issue here?

Update: Oh I have just seen your comment 🙂 I wanted to help out but I don't seem to be able to push to your branch. Would you mind trying to run the formatter?

Done!