About one in five sites did not see improvements after AMP implementations. There are a variety of factors for this:
- AMP can’t save a site on the decline.
While AMP can provide strong benefits, it can’t turn around a site that’s on a strong downward trajectory. Some of the websites evaluated as part of this study had a historical, downward trend and diminishing traffic. That said, for some of these sites we saw traffic flatten out and stabilize, and in some instances lead to an upward trajectory.
- Seasonality makes a difference.
Given the methodology of looking at 30 days before and 30 days after the implementation of AMP, the time of year, the promotion and other seasonal variables can have material effects on site traffic and engagement.
- Market conditions are a big factor.
No website lives in a vacuum, and competing sites may be making changes at the same time. These can lead to any potential gains from an AMP implementation being offset.
- The particular engagement metrics Google uses might not be impacted.
While we saw gains for a majority of AMP implementations examined, it doesn’t necessarily mean every search metric was impacted. For example, a site with an average search ranking of 1-2 is likely not going to see a major ranking improvement. Likewise, a site could see an improvement in CTR, but a decrease in impressions and clicks.
We started by creating Before and After time periods for each client, specifically:
- The Before period is 30 days before AMP went live; and
- The After period is 30 days after AMP went live.
We then looked at two different data sources in the Google Search Console:
- Aggregate mobile clicks and mobile impressions that listed at the top of Google Search Console; and
- Exported mobile query data from Google Search Console.
Some of the sites examined have both HTTP and HTTPS properties and, for these clients, we combined both data sets. To combine the aggregate clicks and impressions, we simply sum them. To combine query specific data (if a client surfaces both HTTP and HTTPS results for the same query), we sum the clicks and impressions, while keeping the highest CTR and position value.
We didn’t, however, average the CTR or position because if a client ranks both at #1 and #2 for a search term, it’s correct to count this as position #1, not position #1.5. We then do the same for the After period, combing both WWW and AMP properties in the same fashion. Again, if a client has both AMP and non-AMP results for the same query, we take the highest position of the two and combine the clicks and impressions.
Once we’ve combined Before and After datasets, we performed the following comparisons:
- Organic Traffic: We compared the total clicks after AMP launch (AMP plus non-AMP) against total clicks before AMP, and then took the average of those changes.
- Impressions: We compared the total impressions after AMP launch (AMP plus non-AMP) against total clicks before AMP, and then took the average of those changes.
- CTR: Measuring CTR is a challenge because Womp serves AMP pages from an amp-dot subdomain. Google Search Console (GSC) stores data from unique domains and subdomains separately. This artificially inflates impressions and artificially deflates CTR when comparing pre-AMP to post-AMP using GSC. The reason this happens is best explained with an example. Consider the search term “Audio Express Car Audio,” where the top four results are all for Audio Express:
As a result, in both instances, AMP and non-AMP traffic is split and impressions double, according to GSC.
Before AMP, Google would see multiple listings from www.audioexpress.com. Since they are from the same domain, Google only counts this as one impression. If users always click one of the top four listings, the Audio Express CTR will be 100%. After AMP, however, Google sees four listings, two from amp.audioexpress.com and two from www.audioexpress.com. Google counts this as two impressions, one reported to the www.audioexpress.com GSC property and one reported to the amp.audioexpress.com GSC property. Therefore, if users click one of the top four listings, it’s only one click – and when we aggregate the impressions and clicks for both properties, the Audio Express CTR will be 50%.
The above problem can be solved with GSC “Property Sets.” Property Sets, however, are not yet available in GSC Beta. For the purposes of this case study, we required GSC beta because it provides 16 months of data, instead of three months.
To work around this problem, we developed a way to estimate the actual post-AMP CTR.
We downloaded the top 999 queries for both the www-dot and amp-dot properties, and then measured how many queries appeared in both sets (looking at the after period, per client). For clients that had less than 999 queries, we downloaded all their queries. On average, there were 312 overlapping queries and 1,578 distinct queries in the after period, meaning that about 20% of the time, on average, a query will surface results from both amp-dot and www-dot, thus inflating impressions and deflating CTR about 20%.
Performing the above calculation, on a per client basis, allowed us to estimate a more accurate CTR.