curl "https://api.ideardev.com/api/v1/crawlings?page=1" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
Crawling
Walks a site from a start URL, follows its links and extracts data from each page.
apihttps://api.ideardev.com/api/v1api3https://api3.ideardev.com/api/v1curl "https://api.ideardev.com/api/v1/crawlings/44" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
curl -X POST "https://api.ideardev.com/api/v1/crawlings" \
-H "Authorization: Bearer $IDEARDEV_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"url": "https://store.com",
"max_pages": 50
}'curl -X PUT "https://api.ideardev.com/api/v1/crawlings/44" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
curl -X DELETE "https://api.ideardev.com/api/v1/crawlings/44" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
POST/crawling_run
Runs the crawl within the request. Use a long timeout and do not retry.
api3.ideardev.com · runs
idbodycurl -X POST "https://api3.ideardev.com/api/v1/crawling_run" \
-H "Authorization: Bearer $IDEARDEV_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"id": 12
}'Response
{
"message": "ok",
"crawling": {
"id": 12,
"status": "completed"
},
"data": []
}curl "https://api.ideardev.com/api/v1/data_crawlings/by_crawling_id/44" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
curl "https://api.ideardev.com/api/v1/crawling_export/44/json" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"
curl "https://api.ideardev.com/api/v1/crawlings/44/availability" \ -H "Authorization: Bearer $IDEARDEV_TOKEN"