Understand site traffic

  3 min read

Another characteristic to consider is your application’s traffic. Several aspects of your traffic might affect how you implement Bot management.

​​ Bot attacks

​​ Situations

If you are experiencing bot attacks, consider the nature of the attack.

These types of actions fall into two general categories:

• A bot submitting data through a form

  • Credential stuffing on login endpoints
  • Content spam
  • Inventory hoarding
  • Credit card stuffing

• A bot viewing data intended for human eyes only

  • Content scraping
  • Ad click fraud
  • Email address harvesting

​​ Implementation details

If bots are submitting data through forms, you should likely be creating focused rules to block traffic on specific endpoints.

If bots are viewing data intended for human eyes only, you may want broader rules limiting bot interactions across your application.

​​ Verified bots

​​ Situations

Cloudflare maintains an internal list of Verified Bots that are associated with search engine optimization (SEO), website monitoring, and more.

You can use this list to prevent any bot protection measures from impacting otherwise helpful bots, such as search crawlers.

For a partial list of verified bots, refer to Cloudflare Radar Open external link .

​​ Implementation details

Verified bots are blocked by default when you create firewall rules using cf.bot_management.score.

To allow verified bots, you would need to include not (cf.bot_management.verified_bot) as part of a firewall rule.

​​ Automated traffic

​​ Situations

By default, Bot management rules will block non-browser traffic.

This can be problematic if your application receives a lot of API traffic, or if you have tools related to:

• Indexing content for search.
• Auditing content (links, headers, etc.).
• Monitoring uptime.
• Forward proxying traffic, such as secure web gateways.

​​ Implementation details

Depending on your application, you may want to write rules that allow specific types of automated traffic or rules that allow all automated traffic to specific endpoints (/api, for example).

In some cases, APIs might be better suited for API Shield than Bot Management.

You should also take time to review Bot analytics to make sure you fully understand the automated traffic reaching your site. Often, you might discover services maintained by a different team or other surprises.

Pay specific attention to:

• Which endpoints are being targeted.
• The top non-Mozilla user agents.
• Traffic from Outlook or Office user-agents.
• Traffic from cloud-based Secure Web Gateways (ASNs labeled with the proxy provider).
• Traffic from on-premises forward proxies.
• Whether requests come from a predictable IP address and ASN, or have a similar JA3 fingerprint.

​​ Mobile app traffic

Because of how mobile applications send requests, Bot Management has the potential to score mobile traffic differently than browser-based traffic.

​​ Implementation details

​​ Review analytics

Take extra time to review Bot analytics to evaluate how your mobile application traffic is performing.

You can generally identify mobile traffic with common user agent strings, though these strings may differ between iOS and Android. Malicious actors might also try to impersonate your mobile application traffic with user agent strings.

Cloudflare recommends being as specific as possible when analyzing traffic and creating rules, usually including a combination of user-agent values, IP addresses or ASNs, and JA3 fingerprints.

​​ Adjust rules

If your application uses mTLS, you can also integrate that certificate’s presence into your firewall rules using the cf.tls_client_auth.cert_verified field.

Native or progressive web applications should also only use Block as a firewall rule action.