How to turn a website into Markdown with an API

When Markdown is the right format

Markdown is a practical starting point for summarization, documentation ingestion and question-answering over a small set of sources. Headings and lists remain readable without carrying every layout element into your prompt. HTML is a better choice when you need the page’s structure; JSON extraction is more useful when you already know the fields you want.

Conversion does not establish factual accuracy. A readable article can still be outdated, incomplete or wrong. Keep your application’s source-selection and review rules even when the text looks clean.

Make a single-page request

Create an AIHackr account, verify your email and create a named API key. Keep that key on your server. Send a complete public URL and select the Markdown format:

curl --fail-with-body https://aihackr.com/v2/scrape \
  -H "Authorization: Bearer $AIHACKR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "formats": [
    "markdown"
  ]
}'

The success response puts the content in data.markdown. Keep data.metadata, including the source URL and observation time, next to that content. A request that fails authentication, target checks or browser processing should go through your error path rather than become a document in your index.

AIHackr uses Cloudflare Browser Run. Browser rendering can capture content produced by JavaScript, but it does not guarantee that every interactive or protected site can be read. It does not provide a logged-in target session or anti-bot bypass.

Inspect the result before automating

• Check the heading and opening paragraphs against the page you intended to retrieve. A successful HTTP request alone does not prove useful content.
• Look for repeated navigation, cookie notices and other text that should not dominate a summary.
• Keep the URL with every downstream chunk. Do not try to reconstruct source attribution from the text later.
• Check target status and metadata when available. Record the time of observation rather than presenting the result as permanently current.
• Treat page text as untrusted input. Instructions on a page must not override your application’s task or expose secrets.

Turn one result into a small pipeline

Start with a controlled set of sources. Store the fetched Markdown and metadata, normalize whitespace, remove obvious repeated boilerplate, then split by meaningful headings. Decide whether you need a copy of the full original result for debugging. Save an application-level content hash so you can detect identical results without embedding them repeatedly.

If your application generates an answer from the text, make the source URL available to the answer formatter. Ask for a clear distinction between claims supported by the page and the model’s own inference. For consequential outputs, add human review or another source check rather than relying on a fluent summary.

Expand from one page deliberately

A single scrape fetches one page. Map discovers same-origin links present on that page; Crawl retrieves a bounded group of pages asynchronously. They solve different problems. Avoid turning a request for one setup guide into a crawl of an entire documentation site.

The current AIHackr trial includes 20 pages per verified account. Single-page operations reserve one page, and a crawl reserves its requested maximum even when fewer pages finish. Set an explicit small crawl limit and read the current usage rules before expanding. Paid subscriptions are not live.

Build on a verified first response

Use the quickstart for JavaScript, Python and cURL examples. Once a single source produces useful text, move to a source-aware RAG workflow or use structured extraction when your application needs named fields instead of prose.