Web search for RAG applications

Ground your LLM in the live web — not a stale training cutoff.

For

Engineers building retrieval-augmented generation pipelines, where the model needs fresh, verifiable context to produce trustworthy answers.

The pain

Without a real retrieval layer.

Out-of-the-box LLMs hallucinate because their training data has a cutoff and they have no notion of "I do not know." Pre-built vector indexes go stale within weeks. Building your own crawler is six months of pain you do not have to take on.

With SiftQ

One call, the right shape.

SiftQ is the retrieval primitive in a RAG stack. Hand it a query, get back ranked passages with source URLs in under 180ms. Feed those directly into your context window. Your LLM cites real, fresh sources and stops making things up.

Recommended config

Start with this shape.

scopewebpage

size8-12

optionsconciseSnippet=true (reduce token bloat) · includeRawContent=true only when LLM needs full page

Working code

Paste, run, ship.

Python — RAG with SiftQ + Claude

import os, anthropic, requests

claude = anthropic.Anthropic()

def retrieve(q: str, n: int = 8) -> list[dict]:
    r = requests.post(
        "https://api.siftq.com/v1/search",
        headers={"Authorization": f"Bearer {os.environ['SIFTQ_API_KEY']}"},
        json={"q": q, "scope": "webpage", "size": str(n), "conciseSnippet": True},
        timeout=15,
    )
    r.raise_for_status()
    return r.json().get("webpages", [])

def ground(question: str) -> str:
    hits = retrieve(question)
    context = "\n\n".join(
        f"[{i+1}] {h['title']}\n{h['snippet']}\nSource: {h['link']}"
        for i, h in enumerate(hits)
    )
    msg = claude.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": (
                f"Use only the sources below. Cite as [N].\n\n"
                f"SOURCES:\n{context}\n\n"
                f"QUESTION: {question}"
            ),
        }],
    )
    return msg.content[0].text

print(ground("What did Anthropic ship in the past 30 days?"))

Why SiftQ for this

✓Sub-180ms p50 — keeps your end-to-end response under a second
✓conciseSnippet mode cuts ~3× the token cost of pasting full pages
✓Per-result score (low/medium/high) for filtering before reranking
✓No vector DB to provision — SiftQ already ranks

FAQ

Likely questions.

Do I still need a vector database?

For pure web RAG, no — SiftQ already returns ranked, scored results from a live index. You add a vector DB when you have your own private documents (internal wiki, support tickets, etc.) to combine with web search.

How do I cite sources back to the user?

Each result includes a `link` field with the source URL. The standard pattern is to interleave inline numeric citations in the LLM prompt and ask the model to use the same numbering in its answer.

How much context should I send to the LLM?

Start with 8 results and conciseSnippet=true (typically ~2-3k tokens). Add includeRawContent only when the LLM needs the full page (rare; most answers come from snippets).

What if the user query is a long natural-language paragraph?

SiftQ handles natural language directly — no need to extract keywords first. For very long queries (>1k chars), pre-summarize the query with a small model before sending.

Keep reading

Use case

Search for autonomous research agents

Give your agent a primitive it can call dozens of times per task.

Use case

Grounding chatbots with live data

Stop your assistant from confidently making things up.

Glossary

RAG (Retrieval-Augmented Generation)

A pattern where an LLM is given relevant retrieved context at inference time rather than relying solely on its training data.

Glossary

Web grounding