Extraction

Scraping groups

Bundles several scrapings and runs them at once, substituting a value: a national ID, a tax ID, a license plate. The backend infers the variables from the templates.

apihttps://api.ideardev.com/api/v1api3https://api3.ideardev.com/api/v1

Fallbacks are only accepted if they are on the same domain and ask for the same values.

GET/scraping_groups

List with templates_count, data_count and run_health.

api.ideardev.com

curl "https://api.ideardev.com/api/v1/scraping_groups" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

GET/scraping_groups/{id}

Details with items, variables_detected, schedules and share.

api.ideardev.com

idpath

curl "https://api.ideardev.com/api/v1/scraping_groups/44" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

POST/scraping_groups

Creates the group. scraping_ids goes outside the envelope.

api.ideardev.com

scraping_groupbody

scraping_idsbody

curl -X POST "https://api.ideardev.com/api/v1/scraping_groups" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "scraping_group": {
    "name": "Background checks",
    "project_id": 7
  },
  "scraping_ids": [
    101,
    102,
    103
  ]
}'

PUT/scraping_groups/{id}

Updates the group. Without scraping_ids the templates are left untouched.

api.ideardev.com

idpath

scraping_groupbody

curl -X PUT "https://api.ideardev.com/api/v1/scraping_groups/44" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "scraping_group": {
    "name": "Background checks v2"
  }
}'

DELETE/scraping_groups/{id}

Deletes the group.

api.ideardev.com

idpath

curl -X DELETE "https://api.ideardev.com/api/v1/scraping_groups/44" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

GET/scraping_groups/{id}/variables

Values to send when running, and which templates each one lands in.

api.ideardev.com

idpath

curl "https://api.ideardev.com/api/v1/scraping_groups/44/variables" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

POST/scraping_groups/{id}/items

Adds a template. With fallback_of it becomes the fallback of another one.

api.ideardev.com

idpath

scraping_idbody

fallback_ofbody

curl -X POST "https://api.ideardev.com/api/v1/scraping_groups/44/items" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "scraping_id": 118,
  "fallback_of": 101
}'

PATCH/scraping_groups/{id}/items/{scraping_id}

Enables, disables or moves a template within its fallback chain.

api.ideardev.com

idpath

scraping_idpath

activebody

curl -X PATCH "https://api.ideardev.com/api/v1/scraping_groups/44/items/101" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "active": false
}'

DELETE/scraping_groups/{id}/items/{scraping_id}

Removes the template from the group.

api.ideardev.com

idpath

scraping_idpath

curl -X DELETE "https://api.ideardev.com/api/v1/scraping_groups/44/items/101" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

GET/scraping_groups/{id}/availability

Checks each site before running. One request per template.

api.ideardev.com

idpath

curl "https://api.ideardev.com/api/v1/scraping_groups/44/availability" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

GET/scraping_groups/{id}/estimate

Estimates the duration from history, without running.

api.ideardev.com

idpath

rowsquery

curl "https://api.ideardev.com/api/v1/scraping_groups/44/estimate?rows=10" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

POST/scraping_groups/{id}/run

Queues the run. Accepts the short form or a list of rows in values.

api3.ideardev.com · runs→ 202

idpath

cedulabody

ciudadbody

curl -X POST "https://api3.ideardev.com/api/v1/scraping_groups/44/run" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "cedula": [
    "1086726464",
    "52331890"
  ],
  "ciudad": "Bogota"
}'

Response

{
  "scraping_group_run": {
    "id": 61,
    "status": "created",
    "total": 6
  }
}

GET/scraping_group_runs/{run_id}

Status, counters, results and timings of the run.

api.ideardev.com

run_idpath

curl "https://api.ideardev.com/api/v1/scraping_group_runs/61" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

GET/scraping_groups/{id}/runs

Run history.

api.ideardev.com

idpath

curl "https://api.ideardev.com/api/v1/scraping_groups/44/runs" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

POST/scraping_group_runs/{run_id}/retry_failed

Relaunches only the failed extractions on the same run.

api3.ideardev.com · runs

run_idpath

curl -X POST "https://api3.ideardev.com/api/v1/scraping_group_runs/61/retry_failed" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

POST/scraping_group_runs/{run_id}/results/{result_id}/retry

Retries a single result.

api3.ideardev.com · runs

run_idpath

result_idpath

curl -X POST "https://api3.ideardev.com/api/v1/scraping_group_runs/61/results/66f1c0a2/retry" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

DELETE/scraping_group_runs/{run_id}

Deletes the run. Responds 409 if it is still running.

api.ideardev.com

run_idpath

curl -X DELETE "https://api.ideardev.com/api/v1/scraping_group_runs/61" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

DELETE/scraping_group_runs/{run_id}/results/{result_id}

Deletes a result and returns the recalculated run.

api.ideardev.com

run_idpath

result_idpath

curl -X DELETE "https://api.ideardev.com/api/v1/scraping_group_runs/61/results/66f1c0a2" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

GET/scraping_groups/{id}/data

Every extracted record with the value that produced it. Filter by run or by attribute.

api.ideardev.com

idpath

attributequery

valuequery

matchquery

curl "https://api.ideardev.com/api/v1/scraping_groups/44/data?attribute=cedula&value=1086726464&match=contains" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

Response

{
  "runs": [
    {
      "id": 61,
      "records": [
        {
          "cedula": "1086726464"
        }
      ]
    }
  ]
}

GET/scraping_group_runs/{run_id}/export/{type}

Downloads the run as csv or json.

api.ideardev.com

run_idpath

typepath

curl "https://api.ideardev.com/api/v1/scraping_group_runs/61/export/json" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

POST/scraping_groups/{id}/schedules

Adds a scheduled run.

api.ideardev.com

idpath

schedulebody

curl -X POST "https://api.ideardev.com/api/v1/scraping_groups/44/schedules" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "schedule": {
    "start_date": "2026-10-02T07:00:00-05:00",
    "is_daily": true,
    "schedule_values": [
      {
        "cedula": "1086726464"
      }
    ]
  }
}'

PATCH/scraping_groups/{id}/schedules/{schedule_id}

Edits a scheduled run.

api.ideardev.com

idpath

schedule_idpath

curl -X PATCH "https://api.ideardev.com/api/v1/scraping_groups/44/schedules/3" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

DELETE/scraping_groups/{id}/schedules/{schedule_id}

Removes a scheduled run.

api.ideardev.com

idpath

schedule_idpath

curl -X DELETE "https://api.ideardev.com/api/v1/scraping_groups/44/schedules/3" \
  -H "Authorization: Bearer $IDEARDEV_TOKEN"

curl -X POST "https://api.ideardev.com/api/v1/scraping_groups" \ -H "Authorization: Bearer $IDEARDEV_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "scraping_group": { "name": "Background checks", "project_id": 7 }, "scraping_ids": [ 101, 102, 103 ] }'

curl -X PUT "https://api.ideardev.com/api/v1/scraping_groups/44" \ -H "Authorization: Bearer $IDEARDEV_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "scraping_group": { "name": "Background checks v2" } }'

curl -X POST "https://api.ideardev.com/api/v1/scraping_groups/44/items" \ -H "Authorization: Bearer $IDEARDEV_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "scraping_id": 118, "fallback_of": 101 }'

curl -X POST "https://api3.ideardev.com/api/v1/scraping_groups/44/run" \ -H "Authorization: Bearer $IDEARDEV_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "cedula": [ "1086726464", "52331890" ], "ciudad": "Bogota" }'

curl -X POST "https://api.ideardev.com/api/v1/scraping_groups/44/schedules" \ -H "Authorization: Bearer $IDEARDEV_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "schedule": { "start_date": "2026-10-02T07:00:00-05:00", "is_daily": true, "schedule_values": [ { "cedula": "1086726464" } ] } }'