Multi-modal retrieval for visual agents

One auth scheme, web + image + video + podcast under the same call.

For

Visual agent products — image-based discovery, video summarization, podcast research, e-commerce product matching — where retrieval needs to span more than web pages.

The pain

Without a real retrieval layer.

Most search APIs are text-only. To do images you bolt on Google Custom Search; for video another API; for podcasts a third. Three different auth schemes, three pricing models, three response shapes. Your agent code becomes a mess of adapters.

With SiftQ

One call, the right shape.

One endpoint, one auth token, one billing relationship — but ten different scopes including `image`, `video`, `podcast`. The response shape adapts (images have `imageUrl/imageWidth/imageHeight`; videos add `duration/coverImage`) so your client code stays clean.

Recommended config

Start with this shape.

scopeimage | video | podcast (per task)

size8-15

optionsUse scope-specific fields (imageUrl for images, duration for video)

Working code

Paste, run, ship.

Python — Visual product-match agent

import os, base64, anthropic, requests

claude = anthropic.Anthropic()
HEADERS = {"Authorization": f"Bearer {os.environ['SIFTQ_API_KEY']}"}

def find_visual_matches(description: str, k: int = 10) -> list[dict]:
    """Find images matching a textual description."""
    r = requests.post(
        "https://api.siftq.com/v1/search",
        headers=HEADERS,
        json={"q": description, "scope": "image", "size": str(k)},
        timeout=15,
    )
    return r.json().get("images", [])

def describe_with_vision(image_url: str) -> str:
    """Use Claude vision to describe a fetched image."""
    img_bytes = requests.get(image_url, timeout=10).content
    msg = claude.messages.create(
        model="claude-sonnet-4-6", max_tokens=200,
        messages=[{
            "role": "user",
            "content": [
                {"type": "image", "source": {
                    "type": "base64", "media_type": "image/jpeg",
                    "data": base64.b64encode(img_bytes).decode(),
                }},
                {"type": "text", "text": "What is this in 1 sentence?"},
            ],
        }],
    )
    return msg.content[0].text

q = "vintage 1970s minimalist serif typography poster"
for hit in find_visual_matches(q)[:3]:
    print(f"{hit['imageWidth']}×{hit['imageHeight']}: {hit['imageUrl']}")
    print("  →", describe_with_vision(hit['imageUrl']))

Why SiftQ for this

✓One scope flag swaps between text, image, video, podcast — no SDK switch
✓image scope returns dimensions for layout planning before download
✓video scope returns duration + coverImage for grid UIs
✓podcast scope returns duration for time-budgeted listening agents

FAQ

Likely questions.

Can I search images by uploading an image (reverse search)?

Not currently. Today the image scope is text-to-image (describe what you want). Reverse-image search is on the roadmap; subscribe to changelog for updates.

Are the images downloadable, or only thumbnail URLs?

Returns the canonical image URL from the source site. Fetch responsibly — respect rate limits and `robots.txt` on the origin if you cache.

What's the resolution / quality range?

Whatever the source publisher uses. You get `imageWidth` and `imageHeight` per result so you can filter by quality before fetching the bytes.

Keep reading

Use case

Search for autonomous research agents

Give your agent a primitive it can call dozens of times per task.

Use case

Web search for RAG applications

Ground your LLM in the live web — not a stale training cutoff.

Glossary