curl "https://api.ideardev.com/api/v1/scraping_groups" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Scraping groups
Bundles several scrapings and runs them at once, substituting a value: a national ID, a tax ID, a license plate. The backend infers the variables from the templates.
apihttps://api.ideardev.com/api/v1api3https://api3.ideardev.com/api/v1Fallbacks are only accepted if they are on the same domain and ask for the same values.
A group runs its templates in parallel, without passing data from one to another. If what one template extracts has to feed the next, use a Flow, which also accepts whole groups as a step.
Details with items, variables_detected, schedules and share.
idpathcurl "https://api.ideardev.com/api/v1/scraping_groups/44" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Creates the group. scraping_ids goes outside the envelope.
scraping_groupbodyscraping_idsbodycurl -X POST "https://api.ideardev.com/api/v1/scraping_groups" \
-H "Authorization: Bearer $IDEARDEV_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"scraping_group": {
"name": "Background checks",
"project_id": 7
},
"scraping_ids": [
101,
102,
103
]
}'Updates the group. Without scraping_ids the templates are left untouched.
idpathscraping_groupbodycurl -X PUT "https://api.ideardev.com/api/v1/scraping_groups/44" \
-H "Authorization: Bearer $IDEARDEV_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"scraping_group": {
"name": "Background checks v2"
}
}'Copies the group (same templates, with their order and fallbacks) under another name. Runs, schedules and the share link are not copied. 422 if the name is missing or already taken.
idpathnamebodycurl -X POST "https://api.ideardev.com/api/v1/scraping_groups/44/duplicate" \
-H "Authorization: Bearer $IDEARDEV_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "Background checks (copy)"
}'curl -X DELETE "https://api.ideardev.com/api/v1/scraping_groups/44" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Values to send when running, and which templates each one lands in.
idpathcurl "https://api.ideardev.com/api/v1/scraping_groups/44/variables" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Adds a template. With fallback_of it becomes the fallback of another one.
idpathscraping_idbodyfallback_ofbodycurl -X POST "https://api.ideardev.com/api/v1/scraping_groups/44/items" \
-H "Authorization: Bearer $IDEARDEV_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"scraping_id": 118,
"fallback_of": 101
}'Enables, disables or moves a template within its fallback chain.
idpathscraping_idpathactivebodycurl -X PATCH "https://api.ideardev.com/api/v1/scraping_groups/44/items/101" \
-H "Authorization: Bearer $IDEARDEV_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"active": false
}'Removes the template from the group.
idpathscraping_idpathcurl -X DELETE "https://api.ideardev.com/api/v1/scraping_groups/44/items/101" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Checks each site before running. One request per template.
idpathcurl "https://api.ideardev.com/api/v1/scraping_groups/44/availability" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Estimates the duration from history, without running.
idpathrowsquerycurl "https://api.ideardev.com/api/v1/scraping_groups/44/estimate?rows=10" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Queues the run. Accepts the short form or a list of rows in values.
idpathcedulabodyciudadbodycurl -X POST "https://api3.ideardev.com/api/v1/scraping_groups/44/run" \
-H "Authorization: Bearer $IDEARDEV_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"cedula": [
"1086726464",
"52331890"
],
"ciudad": "Bogota"
}'{
"scraping_group_run": {
"id": 61,
"status": "created",
"total": 6
}
}Status, counters, results and timings of the run.
run_idpathcurl "https://api.ideardev.com/api/v1/scraping_group_runs/61" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
curl "https://api.ideardev.com/api/v1/scraping_groups/44/runs" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Relaunches only the failed extractions on the same run.
run_idpathcurl -X POST "https://api3.ideardev.com/api/v1/scraping_group_runs/61/retry_failed" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Retries a single result.
run_idpathresult_idpathcurl -X POST "https://api3.ideardev.com/api/v1/scraping_group_runs/61/results/66f1c0a2/retry" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Deletes the run. Responds 409 if it is still running.
run_idpathcurl -X DELETE "https://api.ideardev.com/api/v1/scraping_group_runs/61" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Deletes a result and returns the recalculated run.
run_idpathresult_idpathcurl -X DELETE "https://api.ideardev.com/api/v1/scraping_group_runs/61/results/66f1c0a2" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Every extracted record with the value that produced it. Filter by run or by attribute.
idpathattributequeryvaluequerymatchquerycurl "https://api.ideardev.com/api/v1/scraping_groups/44/data?attribute=cedula&value=1086726464&match=contains" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
{
"runs": [
{
"id": 61,
"records": [
{
"cedula": "1086726464"
}
]
}
]
}Downloads the run as csv or json.
run_idpathtypepathcurl "https://api.ideardev.com/api/v1/scraping_group_runs/61/export/json" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
curl -X POST "https://api.ideardev.com/api/v1/scraping_groups/44/schedules" \
-H "Authorization: Bearer $IDEARDEV_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"schedule": {
"start_date": "2026-10-02T07:00:00-05:00",
"is_daily": true,
"schedule_values": [
{
"cedula": "1086726464"
}
]
}
}'Edits a scheduled run.
idpathschedule_idpathcurl -X PATCH "https://api.ideardev.com/api/v1/scraping_groups/44/schedules/3" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Removes a scheduled run.
idpathschedule_idpathcurl -X DELETE "https://api.ideardev.com/api/v1/scraping_groups/44/schedules/3" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Generates or rotates the link to run without an account, capped by share_max_rows.
idpathcurl -X POST "https://api.ideardev.com/api/v1/scraping_groups/44/share" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
{
"shared": true,
"share_token": "…",
"share_max_rows": 20,
"execute_url": "https://…"
}curl -X DELETE "https://api.ideardev.com/api/v1/scraping_groups/44/share" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"