Two calls — search the web, and pull any page back as clean markdown. Available as a plain HTTP API and as an MCP server, behind one API key per consuming project.
Why this exists
Search and page reading kept being rebuilt per project, each with its own engine accounts, its own scraping code and its own way of failing. This service does it once: the engine accounts, the JavaScript rendering, the caching and the safety checks live in one place, and every project calls it the same way.
Four independent search indexes are asked at once and their rankings are fused, so a page several of them agree on comes back first. Consumers never hold an engine key.
HTML, plain text and PDF come back as markdown ready for a prompt. Script-heavy pages are re-fetched through a headless Chromium when static extraction returns too little.
Each project gets its own key with its own rate budget, so a runaway loop in one project cannot spend another's allowance — and a leaked key is revoked alone.
How it works
Only api is reachable. It authenticates the caller, enforces the rate budget, checks
every destination address before a page is fetched, and answers from its SQLite cache when it can.
The renderer sits on the internal network with no published port at all.
One published port — everything that touches untrusted content runs behind it.
API
Base URL https://websearch.softcreator.com. Everything except /v1/health
carries an X-API-Key header. Errors are always JSON, never an HTML error page, in the
shape {"error": {"code": …, "message": …}}.
| Field | Default | Meaning |
|---|---|---|
| query | — | required, 1–500 characters |
| max_results | 10 | 1–20 |
| time_range | null | day, week, month, year |
| language | "en" | result language |
| categories | null | news, science, it |
| allowed_domains | null | keep only these hosts, subdomains included |
| blocked_domains | null | drop these hosts — cannot be combined with the row above |
| safesearch | 0 | 0, 1 or 2 |
Response
{"results": [{"title": …, "url": …, "snippet": …, "engine": …,
"published": …, "found_by": 3}],
"provider": "brave_api",
"providers": ["brave_api", "tavily", "serper", "exa"],
"cached": false}found_by
How many independent indexes returned that page. On the deployed service 90% of results come back above 1 — weigh it, a page four indexes agree on is not the same as a page one returned.
| Field | Default | Meaning |
|---|---|---|
| url | — | required, http or https only |
| max_length | 30000 | characters returned by this call |
| start_index | 0 | where to continue from on the next call |
| headings_only | false | return just the outline |
| section | null | return only the section under this heading |
metadata.truncated says there is more;
call again with start_index = the previous start_index plus the length
you received. headings_only and section are applied before
that split, so total_length always describes the text you are being served, not the
whole document.
Response
{"content": "# Background Tasks\n\nYou can define …",
"metadata": {"url": …, "final_url": …, "content_type": "text/html",
"title": …, "truncated": true, "total_length": 7707,
"fetched_at": …, "rendered": false}}Reports ok (two or more keyed providers configured), degraded (one) or
down (none, leaving only the free engine), together with the commit the running
container was built from. No provider is probed — a probe is a billed query. The badge at the
top of this page is that endpoint.
web_search and web_fetch
The same two operations exposed as MCP tools, authenticated with the same X-API-Key
sent as a transport header. See Connect.
Getting started
Ask for the key that belongs to your project — there is one per consumer — then pick the way in. If
your code already runs a tool-calling loop, use HTTP: two POSTs and no protocol layer. If your client
speaks MCP, point it at /mcp and the two tools appear on their own.
Search
curl -sS https://websearch.softcreator.com/v1/search \
-H "X-API-Key: $WEBSEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"query": "fastapi background tasks", "max_results": 5}'Fetch a page
curl -sS https://websearch.softcreator.com/v1/fetch \
-H "X-API-Key: $WEBSEARCH_KEY" \
-H 'Content-Type: application/json' \
-d '{"url": "https://fastapi.tiangolo.com/tutorial/background-tasks/"}'import httpx
client = httpx.Client(
base_url="https://websearch.softcreator.com",
headers={"X-API-Key": WEBSEARCH_KEY},
timeout=60,
)
hits = client.post("/v1/search", json={"query": "pydantic v2 validators"}).json()
page = client.post("/v1/fetch", json={"url": hits["results"][0]["url"]}).json()
print(page["content"][:2000])One command registers the server. web_search and web_fetch then appear in
the MCP panel with no integration code at all.
claude mcp add --transport http websearch https://websearch.softcreator.com/mcp \
--header "X-API-Key: $WEBSEARCH_KEY"For clients configured through a file — Cursor, Claude Desktop, IDE extensions:
{
"mcpServers": {
"websearch": {
"type": "http",
"url": "https://websearch.softcreator.com/mcp",
"headers": { "X-API-Key": "<your key>" }
}
}
}Try it
Paste your key, then search and fetch against the live service. The requests go from this browser to
the same host that served the page — the same two endpoints your code will call, with the same
errors. The key stays in this browser's local storage and is never sent anywhere except as the
X-API-Key header of these requests.
No key set — both calls will come back 401.
Limits
Exceeding a per-key limit returns 429 with a Retry-After header. A separate
pacer, shared by every consumer, caps how hard the upstream engines are hit, so no single key can burn
them for everyone else.
| Limit | Default | Notes |
|---|---|---|
| Search requests | 30 / min per key | counted per consumer, not per project instance |
| Fetch requests | 20 / min per key | a cache hit still counts |
| Download size | 20 MB | larger responses are refused, not truncated |
| Search cache | 15 min | cached: true marks a cached answer |
| Fetch cache | 1 hour | keyed on the URL and the extraction options |
| Content types | html · text · pdf | anything else returns 415 |
Security
/v1/fetch resolves the hostname and refuses private, loopback, link-local,
reserved, multicast and cloud-metadata addresses — and re-checks after every redirect.
Chromium executes untrusted page scripts, so it has no published port, no volumes and no access to the environment file. The api validates a destination before handing it over.
A key is checked against every configured key with compare_digest, so timing never
reveals how much of a guess was correct.
The stack
FastAPI. Authentication, rate limiting, provider selection, the extraction chain, the SQLite
cache and the MCP transport at /mcp. Runs a single uvicorn worker — the rate
limiter's buckets live in process memory.
Headless Chromium side-car, used only when static extraction returns too little. If it fails, the request still returns whatever the static extraction found.