Thanks for going through the post in detail. I appreciate your time and feedback. The code snippet is minimal to showcase the AQE and not the end-to end case.

you can setup spark session using -

```

# Import SparkSession

from pyspark.sql import SparkSession

# Create SparkSession

spark = SparkSession.builder \

.master("local[1]") \

.appName("AQE Example") \

.getOrCreate()

```

and data.csv is logical data, you can use Titanic dataset available.