1. 导入相应的包：

from pyspark.sql import SparkSession
from pyspark.sql.functions import col
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

2. 创建Spark会话并将S3存储桶作为Output路径：

spark = SparkSession.builder.appName("WritePartitionedParquetToS3").getOrCreate()
output_path = "s3://my-bucket/path/to/output"

3. 创建模拟数据集：

data = [("apple", 2), ("orange", 1), ("banana", 3), ("pineapple", 1)]
df = spark.createDataFrame(data, ["fruit", "quantity"])

4. 定义分区列：

partitionedBy = ["quantity"]

5. 将模拟数据集写入Parquet文件：

df.write.partitionBy(partitionedBy).mode("overwrite").parquet(output_path)

6. 配置S3 Committers：

spark.conf.set("spark.sql.parquet.output.committer.class", "org.apache.spark.internal.io.cloud.PathOutputCommitProtocol")
spark.conf.set("spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a", "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory")

7. 创建新的模拟数据集：

data2 = [("pear", 2), ("kiwi", 1), ("grape", 4), ("watermelon", 2)]
df2 = spark.createDataFrame(data2, ["fruit", "quantity"])

8. 将新的数据集写入Parquet文件，并使用动态分区覆盖：

df2.write.partitionBy(partitionedBy).mode("overwrite").option("overwriteSchema", "true").parquet(output_path)

9. 关闭Spark会话：

spark.stop()