Why are you still debugging Spark on your host machine?
Last October, a "minor" dependency bump in a PySpark job cost us four hours of downtime and roughly $12,000 in cloud compute credits. A developer had tested a new UDF locally using an older version of delta-spark than what we ran on our EMR cluster. Locally, the serialization worked fine. In production, the different Scala/Java versions in the underlying runtime caused a java.io.NotSerializableException that only manifested when the data volume hit a specific shuffle threshold. We spent hours t…