Crawl API
Start a bounded website crawl, poll status, paginate retained results and cancel jobs with the AIHackr Crawl API. Includes practical request examples.
On this page
Start a small crawl
POST /v2/crawl starts an asynchronous managed job. Save the returned id in your application. This example requests at most three same-origin pages, one link level from the starting page, without executing JavaScript.
curl --fail-with-body https://aihackr.com/v2/crawl \
-H "Authorization: Bearer $AIHACKR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"limit": 3,
"depth": 1,
"render": false,
"formats": [
"markdown"
]
}'Parameters and bounds
| Field | Default | Allowed |
|---|---|---|
| url | required | One public HTTP(S) starting URL. |
| limit | 20 | 1–100 requested pages, subject to your remaining account allowance. |
| depth | 2 | 0–5. maxDiscoveryDepth is a supported alias. |
| render | true | Boolean. false retrieves static HTML without executing JavaScript. |
| formats | ["markdown"] | markdown and/or html, at most two. scrapeOptions.formats is an alias. |
| source | all | all, links or sitemaps. |
| maxAge | 0 | 0–604800 seconds; provider cache preference. |
| modifiedSince | unset | A timestamp string passed to the managed provider. |
| options | {} | includePatterns and excludePatterns only; each at most 20 patterns. |
Poll status
curl --fail-with-body https://aihackr.com/v2/crawl/YOUR_JOB_ID \
-H "Authorization: Bearer $AIHACKR_API_KEY"The workspace response normalizes the status to scraping, completed, failed or cancelled. providerStatus preserves the provider’s more detailed value. Poll with a delay, such as five seconds; stop when terminal. Inspect individual page outcomes instead of assuming every requested page succeeded.
Read every retained page
Each response returns at most 10 records. Follow the relative URL in next with the same account credentials until it is null. Do not construct provider job IDs or assume the first response contains every page.
const url = new URL(result.next, 'https://aihackr.com');
if (url.origin !== 'https://aihackr.com') throw new Error('Unexpected result origin');
const nextResponse = await fetch(url, {
headers: {Authorization: 'Bearer ' + process.env.AIHACKR_API_KEY}
});Cancel and retain results
curl --fail-with-body -X DELETE https://aihackr.com/v2/crawl/YOUR_JOB_ID \
-H "Authorization: Bearer $AIHACKR_API_KEY"Cancellation stops further work when the provider processes it; already running work may finish. Completed results are retained in your workspace for seven days. Save content you need beyond that window in your own storage. Crawl limits reserve the full requested maximum under current trial accounting; unused capacity is not refunded.
Scope and unsupported features
Crawling stays on the submitted origin. Subdomains and external origins are excluded. Two crawls can be active per workspace. Custom headers, cookies, scripts, actions and webhooks are rejected. Use limits and troubleshooting to plan workloads.