LavenderDataLoader - Progress

You can check the progress of the data loader, including how many samples are processed and left.

The progress object has the following fields:

Field	Description
Total	The total number of samples to process in the iteration.
Current	The latest index of the sample that is being processed.
Inprogress	The samples that are being processed. Includes the index of the sample, the timestamp when it started, and the rank of the worker that is processing it.
Completed	The number of samples that have been completed.
Filtered	The number of samples that have been filtered out.
Failed	The number of samples that have failed to process.

# List all iterations for a dataset
lavender-data client iterations list --dataset-id ds-...

{"dataset_id": "ds-...", "id": "it-..."}

# Get the progress of an iteration
lavender-data client iterations get-progress it-...

{
  "total": 100,
  "current": 3,
  "inprogress": [
    {
      "index": 2,
      "rank": 0,
      "started_at": 1744688747.979773
    }
  ],
  "completed": 2,
  "filtered": 0,
  "failed": 0
}

# List all iterations for a dataset
iterations = lavender.api.get_iterations(
    dataset_id=dataset.id,
)

# Get the progress of an iteration
progress = lavender.api.get_progress(
    iteration_id=iterations[0].id,
)