LavenderDataLoader - Fault Handling

Lavender Data is designed to handle failures gracefully, ensuring your training pipeline remains robust.

Handling Failed Samples

By default, Lavender Data will skip failed samples. You can control this behavior with the skip_on_failure and max_retry_count parameters.

Parameter	Description	Default
skip_on_failure	Skip the sample if it fails (after max_retry_count attempts), instead of stopping the iteration	False
max_retry_count	Retry each failed sample up to this number of times	0

dataloader = LavenderDataLoader(
    dataset_id=dataset.id,
    shardsets=[shardset.id],
    skip_on_failure=True, # Skip the sample if it fails (after max_retry_count attempts)
    max_retry_count=3, # Retry each failed sample up to 3 times
)