Jobs failing with error Amazon.S3.AmazonS3Exception Part number must be an integer between 1 and 10000, inclusive

Jobs failing with error Amazon.S3.AmazonS3Exception: Part number must be an integer between 1 and 10000, inclusive

Root Cause

When writing data to S3 during a multi-part upload , the maximum number of parts can be 10000

Due to this threshold the jobs fail during execution

Solution

To overcome this limitation we need to increase the size of the ‘part’ with the below advanced configuration.

The part size can be up to 5 GB

Set the below property as an additional configuration in table properties as shown in below image. This will increase the capacity for each 'part'

key: ingestion_spark_configs

value: spark.hadoop.fs.s3a.multipart.size=104857600

Please rerun the job after setting the above advanced configuration.

Affects Version:

IWX 3.2,4.2 and above