When Bots Become Most of Your Traffic: A Practical Guide to Governing Scraping

The problem is not “bots”

A high bot percentage can describe several very different situations. A search crawler may help a page become discoverable. A monitoring service may support uptime checks. A partner may retrieve approved feeds. An aggressive scraper may copy content, consume capacity, or make it harder to understand what is happening on the site.

The first mistake is treating all automated traffic as one category. The better question is: what is this traffic doing, what does it cost, and what value does it create?

A recent first-person report is titled “A year of fighting scrapers on my 1.5 million-page website”. Its title describes a site where bots reportedly accounted for 99% of traffic. The supplied source does not provide the underlying evidence or the operator’s detailed methods, so that figure should be treated as a reported case signal—not a universal benchmark.

Start with evidence, not user-agent labels

User-agent strings are useful clues, but they are not proof of identity. A request can claim to be a major crawler without belonging to that crawler. Conversely, a legitimate service may use infrastructure or naming conventions that are unfamiliar to your team.

Build a traffic inventory using the evidence your systems already produce:

• request logs, including timestamps, paths, status codes, response sizes, and latency;
• IP or network information, where appropriate and lawful to retain;
• authentication or API credentials for approved automated access;
• cache hits, origin requests, and bandwidth consumption;
• crawl patterns across public pages, search functions, feeds, and expensive endpoints;
• referrals, indexing signals, conversions, or other business outcomes linked to the traffic.

The goal is not to identify every bot perfectly. It is to make a defensible decision about each important class of traffic.

Classify traffic by purpose and behavior

A practical classification can use four broad groups.

Useful or expected crawlers

These may support search discovery, accessibility, monitoring, syndication, or an explicit partner relationship. Their value should still be tested. “Legitimate” does not automatically mean “unlimited.” A useful crawler can create avoidable load if it repeatedly requests pages that rarely change.

Approved automation

Internal jobs, integrations, and partner access should be identifiable through an agreed account, token, network control, or other verifiable mechanism. If a business process depends on automated access, document its owner, purpose, expected volume, and review date.

Unknown automation

Some traffic cannot be confidently tied to a purpose or owner. Unknown does not mean malicious. It does mean the operator lacks enough evidence to grant special treatment. Monitor it and gather more context before making a permanent decision.

Abusive or extractive automation

Warning signs can include unusually high request rates, repeated access to costly endpoints, broad copying of content, attempts to evade controls, or traffic that creates measurable infrastructure cost without a corresponding business outcome. Any single signal can be misleading; the pattern matters.

Measure business value and operational cost together

Bot governance becomes easier when teams stop asking only, “How much traffic is this?” Volume is an incomplete measure.

For each traffic class, examine:

• Discovery value: Does it support indexing, referrals, or legitimate distribution?
• Content value: Is it accessing pages in a way that serves an approved use, or reproducing material elsewhere?
• Infrastructure cost: How much bandwidth, compute, cache churn, database work, or operational attention does it consume?
• Risk: Does it expose private information, create denial-of-service conditions, undermine access controls, or complicate compliance obligations?
• Predictability: Does the traffic follow an identifiable pattern and respect published limits?

A useful internal report should show these dimensions by bot class, route, geography or network where relevant, and time period. Avoid relying on a single daily percentage. A bot may account for a large share of requests while causing little harm, or a small number of requests may target an especially expensive operation.

Protect the expensive parts first

Operators often begin by trying to block every unwanted crawler. That can be the wrong first move. Protect the resources that create the greatest cost or risk, then refine the broader policy.

Priority areas commonly include search endpoints, authenticated pages, account workflows, dynamic filters, large downloads, and routes that trigger expensive database operations. Public content may remain accessible while sensitive or costly functions receive stricter controls.

Possible controls include rate limits, caching, access policies, authentication for approved integrations, request-cost monitoring, and clear terms for permitted automated use. The appropriate control depends on the site, legal context, users, and business relationships. A control that reduces scraping but also harms search discovery or legitimate accessibility is not a successful control.

Do not confuse robots.txt with enforcement

A published crawler policy can communicate intent and help cooperative crawlers behave appropriately. It is not a reliable enforcement boundary against a party that ignores it.

That distinction matters operationally. Treat crawler guidance as one layer of governance, not as proof that a resource is protected. Combine it with monitoring, technical controls, and a documented escalation path.

Use a graduated response process

A responsible process should make intervention proportional to evidence.

1. Observe: Establish a baseline for request volume, routes, response codes, and resource cost.
2. Verify: Investigate claimed identities and compare behavior with known relationships or expected purposes.
3. Segment: Separate valuable, approved, unknown, and abusive patterns rather than applying one global rule.
4. Reduce exposure: Protect high-cost or sensitive routes and use caching or rate controls where appropriate.
5. Measure the effect: Check infrastructure load, search visibility, referrals, errors, and user experience after each change.
6. Escalate carefully: Consider stronger restrictions only when the evidence shows material harm and softer controls are insufficient.
7. Review: Assign an owner and a review date. New crawlers, partners, content types, and search requirements can change the decision.

Keep an audit trail for major changes. Record what was observed, which hypothesis motivated the action, what was changed, and what outcome followed. This prevents emergency bot blocking from becoming undocumented infrastructure folklore.

Protect content without damaging discoverability

Scraping concerns often arrive alongside search concerns. Blocking too broadly can reduce legitimate discovery. Allowing unrestricted extraction can make content operations harder to manage and may create duplicate or misleading versions of a company’s material.

Content teams should know which pages are intended for broad discovery, which require controlled distribution, and which should not be publicly exposed at all. That is a content-governance decision, not only a security decision.

Useful questions include:

• Which content is strategically important enough to monitor for unauthorized copying?
• Which pages change frequently and should not be crawled repeatedly without reason?
• Are canonical, attribution, licensing, or syndication expectations clear?
• Can the team distinguish a copied page from a legitimate partner or customer use?
• Who can approve a restriction that might affect search performance?

The answer should be documented in terms the content, infrastructure, search, and legal teams can share.

Keep humans in the approval loop

Automated detection can prioritize investigation. It should not silently make every high-impact decision.

Human review is especially important when a proposed intervention could affect search visibility, public access, accessibility tools, customers, partners, or legal obligations. A review does not need to be slow or manual for every request. It does need clear thresholds for when automation can observe or apply a reversible control, and when a responsible owner must approve escalation.

For example, a system might flag a sudden request burst for review, apply a temporary protective limit to an expensive endpoint, and require an authorized operator before imposing a broad block. The exact policy should reflect the site’s risk profile; the principle is to make consequential actions explainable and reversible.

What operators should do next

Start with one recent period of logs and build a simple bot traffic register. For each meaningful class, record the apparent identity, purpose, behavior, resource cost, business value, current treatment, owner, and confidence level.

Then select one high-cost route and one content-governance question for review. This produces a narrower, more useful first decision than attempting to solve “scraping” across the entire site at once.

At SimplSolutions, we see this as part of a broader operating discipline: automation should make evidence easier to interpret and decisions easier to review, not hide uncertainty behind a dashboard or an automatic block. Teams that connect monitoring, search visibility, content operations, and human approval are better positioned to respond without trading one problem for another.

The practical takeaway is simple: classify before blocking, measure value alongside cost, protect the most sensitive resources first, and require accountable review for consequential changes.