Managing datasets
List your stored datasets, share them, export several at once, save queries, and delete what you don't need.
Every dataset has one id, the datasetId. You get it from a job or run, or from the list below. The endpoints on this page all take that id.
List your datasets
/Everything stored for you, plus datasets your company shares with you. The list is paged by page number.
| Query parameter | Default | Description |
|---|---|---|
q | — | Search. |
status | all | Available, Failed, Stale, or Expired. |
scope | all | Job or Run — whether the dataset holds a whole job's data or one run's. |
section | production | demo lists datasets from demo agents instead. |
page, pageSize | 1, 25 | pageSize up to 100. |
| Field | Meaning |
|---|---|
id | The datasetId. |
agentName, scraperAgentId | The agent that collected it. |
agentJobHistoryId, agentRunHistoryId, groupRunId | The job, run, and group run it came from. |
status, sizeBytes, fileName | Whether it's downloadable, how big it is, and its file name. |
expiresAt, isExpired | When retention removes the file, and whether it already has. The record stays after the file is gone. |
accessLevel, canManage | Private or Company, and whether you can change or delete it. |
The page also reports availableCount and totalBytes across the whole filtered list.
Where a dataset came from
/The dataset's origin as a small graph: nodes (the agent, the job or run, and the dataset itself) and the edges between them.
Share with your company
A dataset is Private to the user who collected it until it's shared. Sharing makes it visible to everyone in your company.
//{ "accessLevel": "Company" }accessLevel is Private or Company. Only the dataset's owner can change it, and sharing needs your account to belong to a company — otherwise you get 400 INVALID_ACCESS_POLICY.
Export several at once
/{ "datasetIds": ["3fa85f64-5717-4562-b3fc-2c963f66afa6", "3fa85f64-5717-4562-b3fc-2c963f66afa7"] }Up to 50 datasets, streamed back as one zip. The X-Export-File-Count header says how many files it holds. Datasets whose files aren't available are left out; if none are, you get 404 DATASETS_NOT_FOUND.
One bulk export runs at a time per account and company. A second one returns 429 BULK_EXPORT_CAPACITY_EXCEEDED with Retry-After: 5.
Saved queries
Save a query to run again later. Saved queries are private to you.
| Endpoint | Does |
|---|---|
GET /api/data-explorer/saved-queries | Your saved queries, newest first (up to 100). |
POST /api/data-explorer/saved-queries | Save one: { "name": "...", "datasetId": "...", "definition": { ... } }. definition is a query request body. |
DELETE /api/data-explorer/saved-queries/{id} | Delete one. |
Delete datasets
//The bulk form takes { "datasetIds": [...] } (up to 50) and returns processed and skipped counts. You can delete only datasets you own; others are skipped.
Deleting removes the stored copy
The stored file and any saved queries on it are deleted. The run itself stays in your history.