Cloudflare and DataDome: Scraping Response Signals

A Cloudflare or DataDome response during scraping is an observation of an access decision, not a report of every signal behind it. The immediate task is to distinguish a challenge response from the requested content, an origin error or an integration failure. BotBrowser documents browser and request-header controls; it does not expose either vendor's private decision state or promise acceptance. An authorized collection job should classify the response and hand unresolved access decisions to the site owner through its approved support path.

Watch the video

This English-language video explains how to classify a Cloudflare challenge page, a generic HTTP 403 and an unexpected HTML document during scraping, and why the hidden score or DataDome decision needs the operator's correlated record. The audio is English.

TL;DR

• Cloudflare documents cf-mitigated: challenge as a Challenge Page marker, with text/html even when another resource type was requested.
• A generic HTTP 403 does not identify a vendor, rule or browser signal.
• Cloudflare bot scores and DataDome server-integration decisions live at different observation boundaries from the browser response.
• Keep response classification separate from permission, application success and the cause of a refusal.
• Preserve a redacted request correlation record and ask the authorized site operator to inspect its logs.

Contents

• Watch the video
• Cloudflare scores and configured actions
• A challenge response inside a data job
• DataDome decisions behind the integration
• A response-to-action decision table
• Correlate the request without collecting secrets
• Browser controls and the access boundary
• Practical conclusion

Cloudflare scores and configured actions

Cloudflare's bot-score documentation defines scores from 1 to 99. A score of 1 means Cloudflare is quite certain the request was automated; 99 means it is quite certain the request came from a human. These are the vendor's classification endpoints, not a published percentage conversion. A value is not a permanent identity certificate for the client, and an outside client should not assign itself a score based on the page it received.

The same reference says site operators can use scores in WAF custom rules to block, challenge or allow requests, and can use them in Workers. Classification and action are therefore separate questions: an operator's configured policy turns available information into a response. Granular scores require the documented Enterprise Bot Management entitlement. Seeing a challenge does not establish that this particular product tier or one particular score-based rule was used.

Cloudflare describes heuristics, machine learning and JavaScript detections among its detection mechanisms. That public description does not enumerate the contribution of each browser value to a particular refusal. The automation-detection overview separates browser-exposed properties from the larger request context. Changing a property is not evidence that a vendor changed its classification, much less that the site granted permission.

Keep the inference narrow when comparing requests. A changed response can coincide with a different application path, session state or owner rule. Without an authorized operator's correlated record, it cannot isolate one detector input as the cause. Record the request and its visible result first; leave an unavailable score unavailable.

The Cloudflare score and log discussion covers operator-side score evidence in more detail. A score entry should be associated with the matching request rather than borrowed from another navigation in the same browser. A document request and its later data request can have different decisions. The useful comparison is the identified transaction against its own record, not an assumed verdict for the entire tab.

A challenge response inside a data job

Cloudflare's Challenge Page response reference gives a specific client-facing marker: cf-mitigated with the value challenge. It also states that the challenge's content type is text/html regardless of the requested resource type. A data endpoint that normally returns JSON can consequently deliver an HTML challenge instead of the requested data. Check the available response metadata before handing that body to a JSON parser.

An unexpected HTML body alone is insufficient attribution. An application login page, an origin error template or an intermediary's page can also be HTML. Prefer the documented marker when it is available, and retain the distinction between a header observed in network tooling and one readable by page JavaScript. The reference's fetch example illustrates the check; it does not make every response header readable in every cross-origin request.

If the response is a documented Challenge Page, stop the dependent extraction step. A parser failure after that response is a downstream consequence, not proof that the site's data schema changed. The layered blocked-scraper diagnosis explains why transport, HTTP and document-processing failures need separate records. A challenge response should not silently become an empty result set in a monitoring system.

The operational question for the site owner is whether this access path is approved and whether its configured action is expected. Solving a browser puzzle automatically, modifying challenge code or repeatedly changing client properties is outside that diagnostic task. The same boundary applies when a job already holds account cookies: client state does not establish continuing permission for the requested operation.

Preserve the response chain when a navigation redirects. A final HTTP 200 can describe a login document reached after the requested route changed. Record the final URL and the permitted redirect history alongside the content type, without retaining signed query values. Comparing a parser's requested endpoint with a different final document otherwise makes an application routing change look like a detector result. Do not follow an unexpected destination merely to complete the record if that destination is outside the authorized scope.

DataDome decisions behind the integration

DataDome's request-validation integration reference describes a server-side module calling its API and applying the resulting decision. Its prerequisites include access to request headers and the end user's IP address; it also recommends collecting incoming TLS fingerprint information where supported. Those are integration inputs. The public list does not disclose the decision weights for a particular request, nor does it establish that every deployment gathers every optional input.

The reference distinguishes the API's decision from the site's eventual client response. The integration forwards an allowed request to the backend, while a challenge decision returns the API-provided response according to the documented mapping. A browser visiting the site is not necessarily receiving that internal API exchange. Do not interpret the client's HTTP 200 as a direct reading of DataDome's internal allow decision: the requested application may have a different content or authorization outcome.

DataDome also describes a fail-open path for a missing decision header in that integration. This is guidance for the operator implementing the module, not an instruction for a client to treat an absent visible response header as permission. Missing client-visible metadata can result from the observation boundary itself. It is not proof that the module failed open, ran correctly or was present.

The reference requires handling upstream and downstream header mappings and warns against exposing internal pointer headers to end users. For an authorized incident investigation, ask the operator for the applicable decision and a redacted correlation result. Do not demand internal configuration or collect session tokens merely to classify a response. This separation prevents server implementation details from being mistaken for a public client-side detection API.

If the integration returns an unexpected document, its operator can compare the mapped response status, body type and downstream headers with the actual client observation. Review that mapping before blaming a browser API.

A response-to-action decision table

Use the table as a classification rule for an already authorized job. The entries describe documented conditions and ambiguous observations, not measurements from a vendor-protected target. Apply the most specific available evidence; if the required marker or operator record is absent, retain the uncertainty.

Available observation	Supported classification	Next authorized action	Unsupported inference
cf-mitigated: challenge	Documented Cloudflare Challenge Page marker	Stop data parsing and ask the operator about the intended access path	A particular bot score or browser property caused it
HTTP 403 without a specific marker	Access was refused at the observed response boundary	Correlate the request with authorized edge and origin logs	DataDome or Cloudflare necessarily made the decision
HTTP 200 with unexpected HTML	The response is unsuitable for the expected JSON data step	Check content type, available markers and the application outcome	A successful data collection or an internal vendor allow decision
Operator-correlated DataDome integration decision	The recorded module decision for that request	Compare the module action with the returned response and permitted application path	The client can read every internal API field
No readable decision metadata	Insufficient evidence for vendor-specific attribution	Preserve a redacted record and request operator correlation	No protection was present, or a fail-open path occurred

The table deliberately gives HTTP status and application success separate columns. A collected page can be an error document; a valid response may still contain no authorized data for that account. The scraping session-state discussion covers why a retained cookie jar does not settle that question. Treat an unavailable classification as an investigation result, rather than forcing it into a successful or failed detector verdict.

Keep extraction output distinct from incident output. If an endpoint returns text/html where the application's declared contract requires JSON, retain a classified job error instead of publishing an empty price list or deleting previously collected records. That choice concerns the consumer's data handling, not a new vendor detection claim. Resume the data pipeline only through the application's approved recovery process after the operator resolves the access or integration condition.

Correlate the request without collecting secrets

For a site you operate, retain the requested route, request time with its clock basis, HTTP status, content type and any permitted correlation identifier. Cloudflare's header reference describes Cf-Ray as a request identifier that can assist correlation. It is not a bot score or a permission token. A report should say where the identifier was observed and which operator log was consulted; seeing the identifier alone does not show that the origin handled the request.

Separate client observations from operator findings in the incident record. A client can record cf-mitigated: challenge; an authorized operator can add the applicable rule or correlated integration decision. If no operator record is available, the cause remains unconfirmed. Repeating the same status from several browser runs does not supply the missing internal evidence.

State which observation tool produced the record. A page's cross-origin fetch may not expose the same fields as authorized network instrumentation. If the page cannot read a header, record it as unreadable at that boundary rather than absent from the network response. The site owner can inspect its server logs or approved instrumentation without changing cross-origin policy simply to satisfy a collection script. A tool's visibility limit is a limit of the evidence, not a vendor verdict.

Redact cookies, authorization headers, signed challenge URLs and account data before retaining or sharing a response sample. Keep only the body fragment needed to classify the document, where collection is permitted. A response body can contain secrets even when its status is 403. An incident record needs enough context to reproduce the authorized failure, not a reusable account session or a full fingerprint of an unrelated user.

Record whether the response came from a fresh network transaction or an application-controlled cache when that information is available. A saved HTML error can remain visible after the operator changes an integration. Correlating its old Cf-Ray with the current time would send the investigation to the wrong request. Preserve the original observation time and the response provenance; a screenshot of a page alone cannot establish that the application made a new request.

Keep workload limits attached to the same application purpose across requests and routes. A refusal is not an instruction to raise concurrency, cycle addresses or replay tokens. After classification, the next useful decision is the operator's approved access method or a stopped job. That gives the investigation a concrete endpoint without pretending that every denial has a browser-side repair.

Browser controls and the access boundary

BotBrowser's custom-header documentation describes controls for setting, reading, updating and removing configured request headers. Those controls operate on browser configuration. They are not a channel for setting a vendor's server-generated decision, and copying a response marker into a request does not reproduce the trusted process that generated it. Keep the direction of each header explicit when reviewing configuration.

For an owned application fixture, those controls can help check whether the intended application header reached its authorized destination. The custom HTTP header guide covers that browser configuration boundary. A fixture result establishes what the fixture observed; it is not a measurement of Cloudflare or DataDome acceptance, their score changes or their private detection inputs. No such vendor runtime experiment is claimed here.

The browser, service integration and application each contribute a different part of the record. BotBrowser does not issue proxy capacity, vendor permissions or access decisions on a protected service's behalf. When a refusal is expected policy, preserve it as a stop condition. The managed rule groups that make a similar decision at another vendor compose differently, and AWS WAF Bot Control: what a client actually sees reads that structure rule by rule. When the site owner identifies an integration defect, repair the authorized integration rather than treating browser values as proof of identity.

FAQ

Does a Cloudflare challenge reveal the bot score?

The documented marker identifies a Challenge Page. The configured action and any score need the operator's relevant records; a client response does not provide a score conversion.

Does an absent DataDome header prove fail-open?

No. The server integration and the client response have different visibility. An operator must inspect the actual module decision before attributing that path.

Can a successful HTTP response still break extraction?

Yes. Check the returned document and application outcome as well as status. HTML returned where JSON was expected must not be recorded as a successful empty data result.

Can a custom request header set a vendor decision?

A browser configuration change is not a trusted vendor decision. Use header controls for the owned application's declared requirements, not as a claimed authorization mechanism.

Sources

• Cloudflare: Bot scores
• Cloudflare: Detect a Challenge Page response
• Cloudflare: HTTP request headers
• DataDome: Validate request
• BotBrowser: Custom HTTP headers

Practical conclusion

Classify the response before extraction: use a documented challenge marker when available, distinguish HTTP status from application success, and leave internal scores and causes unknown until an authorized operator supplies correlated evidence. Keep a minimal redacted record, stop a refused job and use the site's approved access path. Browser configuration is an input to that workflow, not permission to continue it.