WP Engine Removes Bot Traffic from Overage Calculations – 2015 Recap

WP Engine Removes Bot Traffic from Overage Calculations

This post is part of a series focussing on the main changes and improvements made at WP Engine throughout 2015.

On the 13th October 2015, WP Engine removed Bot Traffic from their Overage Calculations.  In simple terms, WP Engine gives a page view allowance for each plan, and before this date, they used to count bot traffic as part of those page views.  

Bot traffic can include anything from Uptime Checkers, Google Bots, or the many 100’s of other spammy traffic you may receive on a daily basis.  For instance, we receive 100’s if not 1000’s of bot traffic each day (although thanks to Sucuri, many of these never hit our server).  This can quickly add up to many tens of thousands of bot traffic visits per month, and when your allowance may be just 100,000 visits this number may be significant.

The change resulted from much feedback WP Engine received from their customers who raised concerns over how they calculated their billable visitors and overages. WP Engine commented:

At WP Engine, we adhere to a set of core values that influences what we do as a company. Through those guiding principles we promise to be customer inspired and to do the right thing, which means we rely on feedback from our customers to improve our service and the value of our platform.

Why Did WP Engine Include Bot Traffic?

From a hosting perspective, WP Engine was previously charging for the resources used on their servers.  A Bot can use the same resources as a legitimate visitor, and as such they made the decision to charge the same.  Although, as we will mention later in the article, there are a few exceptions.

So Why Remove Bot Traffic from Calculations?

We have already mentioned that WP Engine made the change in part due to Customer feedback.  However, reading between the lines, it seems they are now taking a common sense approach as to what constitutes a visit based on the requisite value that visit gives to their customer.

In the case of Bot traffic, this is often of very little value, and as such WP Engine are taking on the cost of this traffic.  There are a few exceptions, but they are related to whether such traffic is of value to the user \ fair to include.

WP Engine further commented:

Our goal is to deliver a managed WordPress hosting platform that is fast, scalable, secure, and backed by 24/7 support from WordPress experts. We designed our platform to make your life easier and to be a valuable tool upon which you can build your business. Eliminating bot traffic from our billable visits and overage calculations is one way we deliver on that value and align with you for success.

We think this is a great move on their part and eliminates one of the major concern people have with being charged by the number of visitors.

How WP Engine Counts Visits

While the announcement seemed to imply that they were not going to count Bot Visits, in practice it is a little more complicated.  When deciding what does and what does not constitute a visit they asked two questions:

  1. How should a “visit” be defined?
  2. How do you measure “visits” in practice?

To answer this, they wrote down a long list of events that they thought should and shouldn’t be counted as a visit based on a “sensible approach.

  1. When a human being first loads the site and stays for at least 31 seconds.  This is a visit.
  2. If that same human then visits another page on your site, then that is the same visit, not a new visit.
  3. If that same human doesn’t have cookies or Javascript enabled then only one visit is counted.
  4. If that same human loads the site with different browsers, then that is part of the same visit.
  5. If that same human bookmarks the site, then 11 days later comes back to the site, that is a new visit.
  6. As of the announcement referred to in this post, Bots are not counted as a visit.
  7. If a robot scans 20,000 pages over the course of a month, that’s not just one visit. That being said, WP Engine believe it should not count as just one visit.  Equally, it shouldn’t be 20,000 visits. Something in the range of 100-1,000 visits is acceptable. This is because the same robot can come from different IP addresses.  This seems to conflict with the previous point 6, and we are seeking clarification on this point (we will update this post in due course).
  8. There are additional cases where the “right thing to do” is less clear. This includes a “quick bounce” where the human visitor may be on the site for less than a second. WP Engine argues that the servers still had to do the work to attempt to render the page. This is despite such a click not registering a Google Analytics visit due to the javascript code that records the visit not rendering.  Because WP Engine needs the notion of a “visit” to correspond to “the amount of computing resources required to serve traffic,” they argue that it should count.
  9. Image Traffic should not count towards Traffic charges.

Taking all these questions into account, what then constitutes a visit?

WP Engine has settled on the following definition of what constitutes a visit:

“We take the number of unique IP addresses seen in a 24-hour period as the number of “visits” to the site during that period. The number of “visits” in a given month is the sum of those daily visits during that month.”

WP Engine argues that this definition does satisfy the conditions \ thoughts they mention above, and while there are certain exceptions, those exceptions occur on both sides of the fence so as to be fair.  For example, one human could visit the website in two locations (i.e. home and work), and that would count as two visits.  Equally, two different humans could visit the same website from the same IP in an office, and it would only count as one visit.

How to View Your Visits at WP Engine

The great thing about WP Engine is that they have a very clear, customer dashboard that will show you all the metrics you need to know to ensure your keep within your usage limits.  You can see a screenshot of the dashboard below:

wp engine dashboard

You have an excellent summary of the usage for each installation of WordPress, as well as a summary of the visits for both this billing period and the previous billing period to the right of the dashboard.

We will be happy to see your thoughts

Leave a reply