Building the ReadyTools API Platform — Part 2: Affordable Infrastructure Without Being a Bad Citizen of the Web

A little over a week ago, we shared the first look at what we are building with the ReadyTools API Platform.

Since then, the platform has continued moving from individual API endpoints toward something much closer to the infrastructure we originally wanted to build.

The crawler has become more capable. Web Context now extends beyond individual pages. More of the production architecture is taking shape. Pricing is becoming easier to model. And perhaps most importantly, we have spent considerably more time thinking about something that is easy to overlook when building web infrastructure:

how our infrastructure affects the websites on the other side of every request.

The goal is still the same.

Build APIs that developers can depend on, make them extremely affordable, and do as little unnecessary work as possible.

But cheap infrastructure should not come at someone else’s expense.

We can finally talk about pricing — carefully

In the first article, we deliberately avoided publishing pricing estimates.

That was intentional.

You can make almost any API look inexpensive on paper before you have measured its actual infrastructure usage, retries, database load, network traffic, cache behavior, failures and production workload characteristics.

We now have a much better picture of where we want pricing to land.

These are some of our current pricing targets:

API

Target price

SEO Page

$0.15 / 1,000 requests

SEO Audit

$0.20 / 1,000 requests

SEO Crawl

$0.12 / 1,000 attempted pages

Web Context

$0.20 / 1,000 pages

Web Context Crawl

$0.32 / 1,000 attempted pages

Web Structure

$0.30 / 1,000 requests

Cache Warm

$0.02 / 1,000 attempted URLs*

Color conversion

Free

• Cache Warm is currently designed around a minimum job charge for smaller workloads.

There is an important word here:

target.

These are not final launch prices.

There is still a significant amount of production testing, benchmarking, cost measurement and large-scale validation ahead of us. Real infrastructure behaves differently from spreadsheets and controlled benchmarks.

Some prices may go up.

Some may go down.

Some products may change before release.

Our objective is to make these numbers possible — not to publish an artificially low number today and quietly change it once people depend on the platform.

If our engineering allows us to sustainably reach these prices, that is where we want to be.

Why are we trying to make it this cheap?

Because we think many web infrastructure tasks should be commodities.

Fetching a public webpage, extracting useful text, analyzing static SEO data or crawling a set of pages should not automatically become expensive just because a developer does not want to operate their own crawler fleet.

A major part of the work happening inside ReadyTools is therefore not adding more abstraction.

It is removing unnecessary work.

If two operations can safely reuse the same fetched representation, they should.

If content can be extracted during an existing crawl, we should not fetch the page again.

If a page can be safely served from an existing snapshot instead of contacting the origin again, we should use it.

If several requests need the same resource at the same time, we should avoid turning them into several identical requests to the target website.

Efficiency helps our costs.

But it also helps the web servers we interact with.

That connection has become increasingly important to how we are designing the platform.

Web Context is becoming one of the core building blocks

One of the APIs we are particularly interested in is Web Context.

The simple version takes a public page and turns it into clean, useful content without requiring an application to process the raw HTML itself.

That can mean Markdown, cleaned text, structured content or a compact representation intended for machine consumption.

But many applications do not need one page.

They need a website.

So we have also been building multi-page Web Context.

A developer should eventually be able to give ReadyTools a website, request a bounded crawl, and receive clean page-by-page context with the source URL and useful content preserved.

The important part is what does not happen.

We do not want to crawl a page and then make another network request just to generate its Markdown representation.

The crawler can extract context from the response it already received.

One network operation can therefore power several useful pieces of work.

That is one of the reasons our current target for Web Context Crawl is around:

$0.32 per 1,000 attempted pages.

That currently represents the targeted crawl cost plus the targeted context-processing cost, rather than adding another convenience surcharge on top.

Again: this is a target, not final pricing.

Cheap crawling must not mean aggressive crawling

Making crawling inexpensive creates an interesting responsibility.

If requesting 100,000 pages becomes affordable, the platform has to make sure that affordability does not translate into 100,000 careless requests hitting somebody else’s infrastructure.

Our answer to this is not simply a rate limit.

We are building protection at several layers.

The crawler can adapt to the health of the website it is accessing.

Shared work can be coordinated.

Repeated work can be reduced through caching.

Requests for the same resource can be collapsed where appropriate.

And when a website begins signalling that we should slow down, we should slow down.

This sounds obvious.

We think it should be.

ReadyToolsBot will identify itself as a bot

The crawler is being designed around a very straightforward principle:

ReadyToolsBot is a bot, so it should behave like one and identify itself like one.

We are not building an anti-bot bypass service.

We are not trying to make ReadyToolsBot appear to be a normal human browsing from Chrome.

We do not consider a website blocking our crawler to be a technical challenge that automatically needs to be defeated.

If an origin explicitly tells our crawler that it should not access something, our goal is to respect that decision.

That includes mechanisms such as robots.txt and crawl directives, but the philosophy extends further than that.

Explicit access restrictions matter.

Rate-limit responses matter.

Backoff instructions matter.

Repeated target failures matter.

A website should not need increasingly aggressive defenses simply because somebody behind an API requested more pages.

We will not rotate our way around a website’s decision

This deserves to be explicit.

Infrastructure redundancy and bypassing access controls are two very different things.

ReadyTools may use redundant infrastructure to remain reliable when our own infrastructure experiences a routing or connectivity problem.

But a target website saying no is different.

A denied request should not trigger a game of repeatedly changing network paths until the target finally accepts one.

We do not want to build that kind of crawler.

The objective is reliable access to public web resources where that access is permitted, not pretending that restrictions do not exist.

Origin Shield

Another part of this work is something we call Origin Shield.

The concept is simple even though the implementation underneath it is considerably more complicated.

ReadyTools may eventually have many customers interacting with the same website simultaneously.

From each customer’s perspective, their workload might be perfectly reasonable.

From the website’s perspective, however, all of those requests ultimately come from ReadyTools infrastructure.

That distinction matters.

Origin Shield is being designed to look at the combined pressure ReadyTools may place on a destination and prevent independent workloads from accidentally becoming one large burst against the target.

It works alongside caching, request coordination, adaptive crawling and other protections.

The idea is not to unnecessarily slow every crawl.

The idea is to recognize that our responsibility exists at the origin level, not only at the individual API-key level.

A single legitimate crawl should be allowed to operate efficiently.

Multiple workloads converging on the same website should not receive unlimited independent capacity simply because they came from different ReadyTools customers.

Cache first. Origin second.

There is another useful consequence of treating caching as infrastructure rather than just a performance optimization.

The fastest request to an origin is often the request you never have to make.

If ReadyTools already has a valid representation that can safely satisfy a request, another target fetch may be unnecessary.

If several callers request the same uncached resource simultaneously, coordinating that work may prevent several identical origin requests.

And if a target is temporarily under pressure, previously obtained data can sometimes be more useful than aggressively asking for it again.

This is good for latency.

It is good for infrastructure costs.

And it is good for the websites being accessed.

That combination is exactly the kind of optimization we want the platform to be built around.

What happens when a website starts struggling?

Our crawler should not operate under the assumption that every server on the internet has infinite capacity.

Small websites exist.

Shared hosting exists.

Temporary traffic spikes exist.

CDNs fail.

Applications become overloaded.

A crawler needs to react to those conditions instead of blindly maintaining its desired throughput.

ReadyTools is therefore being designed to progressively reduce pressure when a target begins showing signs that continuing at the current rate would be inappropriate.

That may mean slowing requests, reducing concurrency, deferring work, reusing existing cached data or waiting before trying the origin again.

The exact mechanisms will continue evolving as we test the system.

The principle will not:

target health takes priority over maximum crawl speed.

We don’t want to build a DDoS primitive

Low prices also create another problem: abuse becomes cheaper.

We are designing the API with that in mind.

A customer paying for API access does not receive an unlimited right to consume somebody else’s infrastructure.

API-level limits, workload controls, origin-level protections, resource budgets and abuse detection all exist partly to keep that boundary intact.

There will inevitably be cases where this makes ReadyTools less attractive to somebody who specifically wants unrestricted traffic generation.

That is fine.

We would rather build infrastructure for developers, applications, search tools, AI systems, researchers and legitimate automation than optimize the product for workloads whose primary requirement is bypassing other people’s controls.

Affordable does not have to mean unrestricted.

A crawler should be observable

Another theme since our first post has been explainability.

When something is slowed, rejected, deferred or fails, we want to know why.

Was the target unavailable?

Was access denied?

Was the request rate limited?

Did ReadyTools itself fail?

Was cached data used?

Was work delayed to protect the origin?

Those situations should not collapse into one generic “crawler failed” state.

This becomes particularly important as the system grows.

Infrastructure that automatically protects itself and other systems is much more useful when its behavior can also be understood.

The API is becoming more than SEO

SEO was one of the earliest practical workloads for the platform, but the architecture is intentionally broader.

The current direction includes building blocks such as:

SEO Page for deterministic page-level SEO data.

SEO Audit for ReadyTools’ evaluation and checks on top of those facts.

SEO Crawl for site-wide analysis.

Web Context for turning public webpages into clean machine-usable content.

Web Context Crawl for bounded website-level context.

Web Structure for applications and agents that need semantic information about a page rather than raw HTML.

Cache Warm for controlled cache/CDN warming workloads.

And smaller utility endpoints where an API is simply more convenient than rebuilding the same functionality in every application.

These products share infrastructure where doing so makes sense.

That is an important part of the economics.

We are not trying to operate seven completely independent systems and pass seven sets of duplicated costs on to developers.

Deterministic where deterministic is enough

Not every “AI API” needs an AI model behind it.

A recurring design decision in ReadyTools is to keep deterministic work deterministic.

Extracting headings does not require a language model.

Parsing metadata does not require a language model.

Converting HTML into clean structured content does not automatically require a language model.

Detecting links, forms and page structure does not require a language model.

There are excellent applications for generative models.

There are also many cases where adding one simply makes the result slower, more expensive and less predictable.

For infrastructure APIs, predictable behavior is often more valuable.

The production architecture is changing too

A large part of the work since the first article has happened below the API surface.

The production system is moving toward a Google Cloud-based deployment model built around the Go services we have been developing.

The objective is not simply to move from one hosting provider to another.

We want deployment, rollback, health checking and scaling to become reproducible parts of the system rather than manual operational knowledge.

Changes should be tied to exact releases.

Production should be able to prove that it is ready before receiving traffic.

Crawler workloads should be able to scale independently from normal API traffic.

And failure in one part of the system should not automatically mean failure everywhere.

There is still work left here.

A lot of it.

Which brings us to perhaps the most important disclaimer in this entire post.

None of these numbers are earned yet

We now have pricing targets.

We have increasingly mature architecture.

We have much more extensive crawler protection.

We have scale tests.

We have production plans.

But there is still a substantial difference between having the architecture for something and proving that architecture under real production workloads.

Before launch, we still need more testing around things such as:

production unit economics,

sustained crawling,

large concurrent workloads,

database pressure,

failure recovery,

cache behavior,

target-site protection,

billing correctness,

deployment and rollback behavior,

and the many edge cases that only begin appearing when systems run continuously.

That testing may change the product.

It may change limits.

And it may change the prices shown earlier in this article.

We are publishing them because they represent what we are engineering toward — not because we want to pretend the testing has already happened.

The target: infrastructure developers don’t have to think about

The long-term objective remains surprisingly simple.

We want a developer to be able to make a request like:

analyze this page.

crawl this website.

turn these pages into context.

understand this page’s structure.

warm these public URLs.

And then move on to whatever they were actually building.

They should not need to create their own queueing system, crawler fleet, origin coordination, caching layer, parsing pipeline, billing system and failure recovery architecture first.

And the website on the other end should not be punished for that convenience.

That second part matters.

A successful API platform should make infrastructure disappear for its customers without making itself a problem for everyone else.

That is the standard we are trying to hold ReadyTools to.

We still have a lot of testing ahead of us.

The pricing is not final.

The limits are not final.

Parts of the architecture will undoubtedly change again.

But the direction is becoming much clearer:

fast infrastructure, extremely low prices, predictable behavior, and a crawler that respects the web it depends on.

That is what we are building next.