LavenderDataLoader - Resume

One of Lavender Data’s powerful features is the ability to pause and resume iterations. Here’s how to resume an iteration.

Save the Iteration ID

You need to save the iteration id to resume it later.

dataloader = LavenderDataLoader(
    dataset_id=dataset.id,
    shardsets=[shardset.id],
)
print(dataloader.id)

Put back the Samples In Progress

Before resuming the iteration, you might want to put back the samples still in progress. For example, they can still in preprocessing, or they are prefetched but not used yet. You can put back them into the iteration queue by calling pushback method, or through the web UI.

CLI

lavender-data client iterations pushback it-...

Python

dataloader.pushback()
# or
lavender.pushback(dataloader.id)

Web UI

Click the “Pushback” button in the iteration detail page.

Resume an Iteration

Later, you can resume the iteration from where it left off:

for batch in LavenderDataLoader(iteration_id="it-..."):
    # Continue processing from where you left off
    process_batch(batch)