Build a source-aware web data pipeline for RAG

1. Define the collection before fetching it

Write down the questions your application should answer and the sources that are appropriate for them. A product documentation section is a different collection from a company’s entire website. Choose a starting URL, an allowed origin and a small page bound. Avoid indexing account pages, repetitive search pages or material you are not permitted to retrieve.

Use page-level mapping to inspect a section’s links when the useful paths are unclear. Map does not establish complete site coverage. Verify important pages independently instead of assuming every source is linked from the start page.

2. Collect content with a bounded crawl

curl --fail-with-body https://aihackr.com/v2/crawl \
  -H "Authorization: Bearer $AIHACKR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "limit": 3,
  "depth": 1,
  "render": false,
  "formats": [
    "markdown"
  ]
}'

In AIHackr, crawling is asynchronous. Store the job ID, poll status with a delay and read page outcomes. Follow every relative next URL to collect the retained records. Record failed, skipped and truncated pages explicitly; a missing source should not silently become an answer with lower-quality evidence.

The example uses static retrieval. If the target’s content depends on JavaScript, evaluate browser rendering on one representative page before increasing the workload. AIHackr operates on public webpages and does not supply target login sessions.

3. Make provenance part of your data model

Store each document’s source URL, observation time, content hash and retrieval outcome. Keep a reference to the full retained text you ingested. When you split a document into chunks, attach the source identifier and heading context to every chunk.

Do not make source attribution an optional display field that is added after retrieval. It is part of the record’s identity. A citation should lead to the source for the passage, and a reviewer should be able to see which version of the content was used.

4. Clean and chunk with inspection

Remove repetitive navigation and obvious boilerplate, then split content along headings and coherent sections. Keep code examples with their explanatory text when they are needed together. Deduplicate repeated pages and repeated chunks so a common navigation block does not overwhelm retrieval.

There is no single chunk size that is optimal for every collection and embedding model. Test a small set of questions against your actual documents. Inspect which passages are retrieved and whether their boundaries preserve enough context to answer accurately.

5. Bring your own retrieval infrastructure

AIHackr supplies web retrieval and extraction, not a hosted embedding store or an end-to-end RAG application. Choose your own embedding model, storage and retrieval logic. Keep document versions and deletion rules in that system. Feed retrieved passages to the model as evidence, with their source identifiers, rather than as instructions.

• Evaluate questions with expected supporting passages, not just attractive answers.
• Check whether source URLs cited by the answer actually support its claims.
• Test unanswered questions and require the application to say when the collection lacks evidence.
• Include prompt-injection attempts in evaluation: web content must not override system instructions or reveal keys.
• Track which collection version produced an answer so regressions can be investigated.

6. Design a refresh policy you can afford

Choose how often the underlying sources are likely to change, then plan explicit refetches. Compare content hashes and reindex only changes your application needs. Handle removed pages and expired material rather than only adding new chunks forever.

AIHackr stores crawl results for seven days, so export the content and metadata needed by your application. The current free trial is 20 pages per account, and crawl requests reserve the full maximum even when fewer pages complete. Scheduled production ingestion needs a capacity and billing arrangement beyond that trial; paid checkout is not live yet.

Start with a collection you can inspect

A small, source-aware collection is a useful first milestone. Confirm the requested pages were retrieved, the text is usable, the citations resolve and the application can decline unsupported questions. Then expand the collection deliberately. See the RAG workflow overview and Crawl API reference for the current service surface.