Scrape vs. crawl vs. map: choose the right request

The practical difference

These are the behaviors of AIHackr’s current API. Other products may use the same words for different functionality. In particular, AIHackr Map is page-level discovery, not a prebuilt index of every URL on a domain.

Scrape when you already know the page

Use Scrape for a setup guide, a public product description or another specific source. You can request Markdown for text workflows, HTML for structure or links for a direct link result. A browser can render JavaScript-produced content, subject to the target’s behavior and access restrictions.

curl --fail-with-body https://aihackr.com/v2/scrape \
  -H "Authorization: Bearer $AIHACKR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "formats": [
    "markdown"
  ]
}'

Do not crawl just because a page belongs to a larger site. Fetch the known source first and check whether it answers the task. One page is often easier to cite and review than a broad collection.

Map when you need to choose the next source

Use Map when you have a section overview but need to inspect the available paths. It reads one page and returns matching same-origin links. You can review link text, remove repetitive navigation and select a narrower starting point for a crawl.

curl --fail-with-body https://aihackr.com/v2/map \
  -H "Authorization: Bearer $AIHACKR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "limit": 10
}'

A limit of 10 means up to 10 returned links. It does not fetch 10 pages. An empty list can be a valid result if no same-origin links are exposed. It is not proof that the site has no other pages.

Crawl when coverage requires several pages

Use Crawl for a bounded documentation section or a small public content collection. Specify a page limit, depth and rendering preference. Start small enough to review the results manually.

curl --fail-with-body https://aihackr.com/v2/crawl \
  -H "Authorization: Bearer $AIHACKR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "limit": 3,
  "depth": 1,
  "render": false,
  "formats": [
    "markdown"
  ]
}'

Save the returned job ID and poll its status with a delay. At completion, inspect individual page outcomes and follow the next URL until all retained records have been read. A completed job does not mean every requested page was accessible or useful.

Set render: false when static HTML is sufficient; this avoids executing JavaScript for that crawl. Keep rendering enabled when the content depends on client-side scripts. Compare the actual target result rather than assuming either mode is always better.

Page bounds are part of the request

Current AIHackr trial accounting reserves a crawl’s requested maximum, including unused capacity. A limit of 20 can consume a new account’s full trial even if the site yields fewer pages. A single-page Scrape, Map or Extract reserves one page.

That makes the sequence important: inspect one source, choose the operation and set an explicit bound. The trial is not a recurring monthly allowance, and retries do not replenish it. Read limits and retention before building an unattended workflow.

A useful decision rule

If you know the URL and need its content, scrape it. If you need to choose URLs from a page, map it. If you need content from several connected pages, submit a bounded crawl and handle its lifecycle. If you need particular fields from one page, consider Extract instead.

The endpoint references for Scrape, Map and Crawl show the exact request shapes and supported options.