Web Research

The Web Resists Being Read

Most websites were built for humans, not agents. They serve JavaScript-heavy pages, block programmatic access with 403s, and bury useful content inside navbars, footers, and cookie banners. Standard HTTP fetch fails on a surprising fraction of real URLs.

Firecrawl is the service Amber uses to work around this. It converts any URL — static sites, JavaScript-rendered pages, council portals — into clean markdown that a model can actually reason over.

Firecrawl vs Just Searching the Web

A natural question: why not just ask the agent to search, or fetch the URL itself?

A web search returns snippets and links. The agent can see that a page exists and what it might contain, but not the actual content. If the answer lives in the body of a specific page, a search result isn’t enough.

A direct fetch (what Amber does by default with a URL) sends a plain HTTP request. No JavaScript execution, no browser headers. Many sites block it outright — a 403 is a common result. Even when it works, you get raw HTML: the content is buried in markup, and you need to parse it.

Playwright (a headless browser) goes further. It renders JavaScript and presents as a real browser, so it gets through most bot protection. But what you receive is still a rendered visual page — you’d need to extract the content from the DOM yourself.

Firecrawl sits on top of a headless browser but adds the extraction step. What comes back is clean markdown — no nav, no footer, no ads — structured for a model to read directly. The difference isn’t just access, it’s usability.

Method	Gets past bot protection	Returns usable content
Direct HTTP fetch	Sometimes	No — raw HTML
Web search	Yes	No — snippets only
Playwright headless	Yes	No — requires DOM parsing
Firecrawl	Yes	Yes — clean markdown

How Amber Uses It

Firecrawl is wired into Amber as a research skill. When you give Amber a URL and ask it to do something useful with the content, it routes the fetch through Firecrawl’s scrape endpoint rather than fetching directly.

The skill activates automatically on trigger words like “research”, “web research”, or “deep research” — or when the agent decides it needs to read a URL to answer your question. Your API key is never exposed in chat history; all requests go through a credential proxy that handles authentication transparently.

Technical Detail — Endpoints

The research skill supports four Firecrawl endpoints:

• /v1/scrape — single page to markdown
• /v1/search — web search returning scraped results
• /v1/crawl — async multi-page crawl with polling
• /v2/agent — Firecrawl’s own research agent mode

For most tasks Amber uses /v1/scrape. For deep research across many sources it can crawl a whole site or invoke /v2/agent. The skill runs Python in a sandboxed container and writes results to a file — a pattern that handles large responses without blowing the context window.

A Live Example — Swinburne FC

A Swinburne FC member asked Amber to write a social media post about a Boroondara Council grant that could subsidise members’ registration fees. They pasted in the URL for the online application form.

Amber scraped the form, extracted the eligibility rules (Boroondara resident, current government means-tested card), and produced two draft posts — short and punchy for Instagram, longer with suburb callouts for Facebook. All in one turn.

Then they asked it to pull further detail from the council’s main info page. Amber’s standard web fetch returned a 403. The council site had bot protection in place.

Here’s what that page looks like to a browser — what a human or Playwright sees:

And here’s what Firecrawl returned for the same URL — clean markdown, straight to the point:

# Individual Participation Grants

Individual participation grants help disadvantaged community members
participate in sport and recreation.

You can apply for up to 75% of annual sports club fees (capped at $200),
subject to availability. Funding is paid directly to your sports club,
which will then reduce your registration fee.

## Eligibility

To be eligible for individual participation grant funding, you must:

- live in the City of Boroondara
- provide evidence of a current government means-tested card
- choose an eligible Boroondara sporting club (not for profit)

## Applications

Applications are accepted all year round, until the allocated funding
has been used.

That’s the detail the post needed — the $200 cap, the direct-to-club payment, the year-round window. All of it was there. Amber just couldn’t see it without Firecrawl.

What It Shows

Firecrawl doesn’t just fix a technical limitation. It changes what you can ask Amber to do.

Without it, you copy-paste content into chat and hope the context window is big enough. With it, you hand Amber a URL and it does the reading itself — pulling out what matters, ignoring noise, and succeeding even when the site actively blocks standard requests.

Grant research, policy lookup, competitor analysis, tender reading — anything that currently involves you opening a tab and summarising a page by hand is now something Amber can do on its own.