Scraper Blocked? Diagnose DNS, TLS, HTTP and Page Errors

A scraper blocked message is an incident label, not a diagnosis. A name-resolution failure, a certificate error, an HTTP 403 and an empty data panel establish different facts. BotBrowser documents profile-backed browser surfaces and automation configuration, but it does not grant access or remove a server's policy. Find the last stage supported by the failing request's evidence before changing the browser, proxy or workload. The useful result may be a located failure with the server's reason still unknown. cross-run state guide separates cookie retention, snapshot coverage and server acceptance after a new context or process. For a vendor-specific HTTP response, the Cloudflare and DataDome response signals distinguish documented challenge markers from unknown server decisions.

When usable data changes into a challenge during the same job, diagnosing a CAPTCHA appearing during scraping connects the last usable response and the first challenge without treating their sequence as a cause.

TL;DR

• Identify the failing request, not only the page that looks blocked. A document and its data request can have different outcomes.
• Separate an absent HTTP response from a received refusal. HTTP 403 or 429 is still a response.
• Keep DNS, connection/TLS and page-execution errors in distinct categories; none alone identifies an anti-bot model.
• Treat a command-line request as a separate client observation, not a copy of the browser session.
• Stop at the documented access boundary. Attribute a server rule only when service documentation or authorized server records support it.

Contents

• Identify the request and the observation window
• DNS: distinguish a name from an HTTP responder
• Connection and TLS: retain the actual error
• HTTP: a refusal is still a response
• Page execution and data acceptance
• Five diagnostic rows and an attribution limit
• BotBrowser configuration is not a server verdict
• Practical conclusion

Identify the request and the observation window

Start with one authorized incident: the integration revision, browser and framework versions, request role, time and redacted route reference. Give the incident an opaque identifier such as requestId so a browser event can be matched to an owned server record without exposing credentials. The browser's top-level document, an application data request and an image are not interchangeable evidence. Record which one failed and whether the requested data was accepted before assigning a page-wide outcome. The SERP rank-tracking model separates a configured location, language or device input from the engine’s effective result context, and keeps a source-defined ordinal apart from Search Console average position.

For example, an HTTP 200 document can load an application shell while its data request receives 403. A screenshot of the shell cannot establish which request was refused. Conversely, a missing decorative image does not necessarily invalidate the data record. These are illustrative cases, not captured exchanges. Define the required field or application result in advance, then correlate that result with the specific request and response rather than judging a run solely from its final screenshot.

Use a bounded observation window and retain the first relevant failure. Do not let a library's retries replace the original event with a later success or a generic timeout. Keep an explicit distinction between no request dispatched, request dispatched without a response, response received and application result accepted. These are proposed incident states, not additional HTTP status codes. A log field such as lastEstablishedStage is a local classification and should never be presented as a server-provided explanation.

Record changes between observations. If the proxy, browser version, application state and concurrency all change together, a different result cannot identify which change mattered. Begin with the existing authorized configuration, preserve the failure and investigate one bounded question. Do not repeatedly raise traffic or change accounts to discover a service's limits. Missing evidence is a reason to keep the cause unresolved, not to replace it with a more confident label.

DNS: distinguish a name from an HTTP responder

MDN's DNS glossary describes the naming system that associates domain names with resources such as IP addresses. A failure to resolve a name occurs before an HTTP response from that destination has been established. It does not, by itself, prove that the site's application rejected the browser. Keep the hostname, resolver context and reported error category together; a typo, an unavailable resolver and a policy response are different possibilities requiring their own evidence.

The curl error reference distinguishes error 5, an unresolved proxy name, from error 6, an unresolved remote host. This distinction matters when a job has a proxy: an error involving the proxy endpoint is not evidence that the destination hostname failed. Retain the actual client and configuration used. A successful lookup in a separate terminal does not establish which resolver a browser or proxy used for the failing request.

An existing address in a local cache also differs from a fresh resolution observation. Record whether the diagnostic actually performed a lookup instead of assuming that every successful connection exercised DNS. Do not infer the resolver's internal state from one elapsed-time value. When the service is yours, correlate resolver or gateway records within the same window; when it is external, retain the client-visible error and avoid claiming an unseen infrastructure decision.

The bounded next action is to check the configured name and resolver path within the authorized environment. Preserve the original input before correcting an evident typo. If the problem cannot be reproduced outside the browser, describe that difference rather than declaring the browser fingerprint responsible. A DNS-stage incident has not yet supplied evidence about the content, status or policy of an HTTP response.

Connection and TLS: retain the actual error

Name resolution and connection establishment are separate observations. A resolved address does not prove that a connection to the configured destination or proxy succeeded. Curl error 7 reports failure to connect, while error 28 reports an operation timeout; the latter alone does not locate the timeout to one protocol stage. Preserve the client's error category and any stage-specific record instead of turning every connection failure into a website block.

MDN's TLS glossary describes TLS as protection for communications. A certificate-validation error establishes a trust problem observed by that client, not an HTTP authorization response. Curl's error reference distinguishes TLS connection error 35 from peer-certificate verification error 60. The browser connection-trust analysis covers certificate and browser trust boundaries in more detail; retain that distinction rather than treating every TLS error as evidence of fingerprint detection.

For an owned or explicitly authorized diagnostic endpoint, the following command records one separate-client observation. Set DIAGNOSTIC_URL to that permitted URL first. It performs a GET, discards the response body, keeps certificate validation enabled, imposes a finite timeout and does not follow redirects or retry automatically. The output is a proposed record format, not an example of a successful run:

code=0
curl --disable --silent --show-error --max-time 20 --output /dev/null \
  --write-out 'http_code=%{http_code}\ntime_namelookup=%{time_namelookup}\ntime_connect=%{time_connect}\ntime_appconnect=%{time_appconnect}\n' \
  "${DIAGNOSTIC_URL:?Set an owned or authorized diagnostic URL}" || code=$?
printf 'curl_exit=%s\n' "$code"

The timing fields are diagnostic inputs, not a proof that every named stage executed as a new operation. Connection reuse and the selected route affect their interpretation. The terminal client's cookies, resolver, proxy configuration and TLS behavior may differ from the browser's. Therefore, agreement can corroborate a shared symptom and disagreement can direct investigation, but neither makes this command a browser-session reproduction. Use the same declared observation window and retain the client identity in the record.

Do not add an insecure certificate option to make the diagnostic appear successful. On owned infrastructure, investigate certificate validity, expected hostnames and configured trust through the responsible operator. For a third-party service, preserve the failure and use its supported integration path. A completed TLS connection still does not imply that the next HTTP request will be authorized or that an application will accept its data.

HTTP: a refusal is still a response

The MDN HTTP status reference distinguishes response classes, including client and server errors. An observed HTTP 403 establishes that a responder refused that request; it does not identify a detector, fingerprint mismatch or account rule without further evidence. A 429 is a rate-limit response and belongs in its own category. The rate-limit decision record explains optional Retry-After guidance and finite-budget scheduling without inferring the counting key.

Playwright's requestfailed event documentation explicitly distinguishes a network failure from HTTP error responses. A received 404 or 503 is an HTTP response, so its status alone does not trigger requestfailed. The same distinction prevents a 403 from disappearing into a misleading network-error bucket. Attach listeners before the authorized action begins, then keep response status and request failure as different event types.

The following recorder attaches to a supplied Playwright Page. It does not launch a browser, navigate, retry or change settings. The caller owns emit and must redact any sensitive error text before retaining it. The pageerror reference supplies the separate uncaught-page-exception event:

export function attachDiagnosticListeners(page, emit) {
  page.on('response', response =>
    emit({
      event: 'response',
      status: response.status(),
      role: response.request().resourceType(),
    })
  );
  page.on('requestfailed', request =>
    emit({
      event: 'requestfailed',
      role: request.resourceType(),
      error: request.failure()?.errorText ?? null,
    })
  );
  page.on('pageerror', error =>
    emit({
      event: 'pageerror',
      name: error.name,
    })
  );
}

The event records intentionally omit URLs and credentials. The surrounding harness still needs a redacted request association and timestamp to connect events to the incident; a shared resource type alone cannot distinguish several data calls. Do not assume that event order across unrelated requests creates one causal chain. Likewise, a response event establishes response arrival, not that all response content has finished or that the application accepted it. Preserve those later outcomes separately.

Page execution and data acceptance

A document response is not a page-execution verdict. An uncaught script exception, a failed resource request and a missing application field are different observations. Use the separate pageerror, requestfailed and response records to locate the evidence, then check the declared data condition. An exception can interrupt a relevant operation or be unrelated to the required result; its presence alone does not establish that an anti-bot script stopped the collection.

For an owned fixture, define a stable readiness condition and a small expected record before observing the page. For an external integration, use its documented result contract rather than inventing success from a visible heading. A document with HTTP 200 can contain a sign-in screen, a service error or another application state. Keep the content classification alongside the HTTP status instead of rewriting the status to express the application's outcome.

Retain enough context to connect the missing field to its producing request or script stage. A data request may never have been dispatched, may have received a refusal, or may have returned data that the page did not apply. These require different next investigations. Do not treat a selector timeout as an observed server refusal: it establishes that the selected condition was not met within the configured window, not why. Record the bounded wait and the actual condition being awaited.

Comparison needs the same application revision, initial state and permitted request sequence. If one observation starts signed in and another starts without that state, the difference is not evidence of browser detection. Check whether a required authorized session or operation was present before attributing an empty panel to a device signal. Keep private account material out of public logs; an opaque state reference can connect an incident to access-controlled operational records.

Five diagnostic rows and an attribution limit

The first four rows below describe client-observable stages. The fifth concerns attribution to a service policy, including an alleged behavior rule; it is not a fifth protocol exchange that every request exposes. Only service documentation, an authorized support response or an owned server's decision record can establish the relevant rule for that incident. An unfamiliar challenge page or changed result is not a substitute for that evidence.

Diagnostic row	Retain	Bounded next decision
DNS	Client, configured name, resolver context and error	Check the named lookup; no HTTP cause yet
Connection/TLS	Route reference, connection error or trust error	Investigate the reported stage without disabling trust
HTTP	Responding request, status and selected headers	Apply documented response policy; do not name a detector
Script/data	Exception, resource outcome and required field	Locate the unmet condition without relabeling the response
Service policy	Documented rule or cause=unknown	Ask the responsible operator; stop if permission is absent

Store observations and interpretations in different fields. httpStatus=403 is an observed value; cause=unknown can remain the correct cause field even after the response has been located. A note such as “probably behavior scoring” is an unconfirmed hypothesis, not a fact column. If later server evidence identifies an access rule, add its reference and time rather than altering what the original browser record contained.

An owned server's record must still match the incident. Correlate request identifiers, time window, resource and the component that answered; a gateway refusal and an application refusal can be different decisions. A general rule document explains possible policy, while a matched decision record explains a particular response. For a third-party integration without that record, state the narrower conclusion and use the supported access or support route instead of probing unseen decision logic.

BotBrowser configuration is not a server verdict

The BotBrowser automation-consistency documentation describes profile-backed settings and framework configuration. The fingerprinting overview describes browser-engine surfaces controlled through the profile system. These are browser configuration capabilities, not a server's explanation for a refused request. Keep the configured binary, profile reference and framework settings in the incident, but do not mistake configuration intent for a verified remote outcome.

A browser-surface change cannot, by itself, establish that a DNS error was corrected, a certificate is trusted or a service quota permits another request. A proxy setting changes the declared route; a different result after that change does not reveal the original refusal reason. Preserve the original observation and seek evidence at the located layer. A configured browser neither supplies authorization nor promises an accepted application result.

FAQ

Does a 403 prove fingerprint detection?

No. It is an HTTP refusal. Identifying the rule requires service-side evidence or a documented explanation relevant to that request, not a guessed detector name.

Is an HTTP error a Playwright request failure?

Not solely because of the status. Playwright distinguishes received HTTP responses from failures to obtain a response. Retain both event types and the particular request association.

Does curl success prove the browser should work?

No. Curl is another client with its own state and configuration. Its result can corroborate an observation but does not reproduce the browser session.

Is a visible challenge enough to identify a behavior rule?

No. Preserve the visible state, response and request sequence. Keep the policy cause unknown unless the responsible service supplies supporting evidence.

Sources

• MDN: HTTP response status codes
• MDN: DNS
• MDN: TLS
• curl: Error codes
• curl: Command-line options
• Playwright: requestfailed event
• Playwright: pageerror event
• BotBrowser: Automation consistency
• BotBrowser: Fingerprinting overview
• Browser connection trust
• Scraping rate-limit decisions

Practical conclusion

Locate the failing request and preserve the last stage supported by its evidence. Distinguish DNS or TLS failure, an HTTP refusal and an unmet page-data condition before changing settings. Keep service-policy attribution unknown unless the responsible source establishes it. The next action is to retain the response or error and inspect that bounded stage within the authorized integration, rather than turning a vague blocked message into an invented detector verdict.