As a publisher, we all know that monitoring the traffic and its behavior is very important for you to measure the performance of your website. Examining the data from various traffic sources allows you to strategize better and understand your audience.
But, what if the traffic captured is not even the complete picture?
In 2012, dark traffic was detected, which was nothing but the visits to the site whose source Google analytics could not tell. Over the years, it grew with the rise of HTTPS and an increase in private messaging apps such as Slack, WhatsApp, and other social media apps.
But, dark traffic was not the only kind because publishers could not get the complete picture of their traffic. Another wrench in the gears of the analytics tools, including Google Analytics, Adobe SiteCatalyst / Omniture, and the likes, is the shadow traffic.
Before we jump into understanding precisely what Shadow Traffic is, let us have a quick refresh of how web content and product analytics tools work together.
Every predominant analytics tool available in the market requires you to add code to the website. Only after the code is added, your analytics provider starts reporting on website visitors and events.
When someone visits your website, this code added will send “events”— little packets of information that summarizes the activity on your site—to your analytics provider. The most common web analytics events are page view events, email capture events, or video start events.
And if a browser or an adblocker somehow stops these events, the analytics provider won’t be able to record anything for you—not the user, source, or the pageview. These visits are still the actual traffic from your visitors, not tracked by your analytics tool that you won’t have any visibility into.
These failed visits are represented as shadow traffic.
Table of Contents
What is Shadow Traffic?
Shadow traffic is nothing but the visits to your website that the typical analytics software misses capturing. This usually happens when the analytics events are failing to reach analytics servers. Such failures can often deflate the publisher’s profits and websites’ performance.
How Does Shadow Traffic Affect Publishers?
In early 2020, while researching what proportion of visitors go untracked, Parse.ly found that at least 20% to 40% of traffic was untracked or shadow traffic. Because of losing such a percentage of your visitors’ data, you won’t be able to understand your audience’s route, profile, and value. Losing important data can lead you to make lousy decision-making, revenue loss, and worse business outcomes.
It’s a growing concern that can’t be ignored since it affects pretty much every major browser, analytics tool, and, thus, publishers.
So, to improve and measure your traffic quality and, in turn, your website’s performance, the first step is by addressing the shadow traffic problem while still fully respecting your audience’s privacy and user experience.
What Gives Rise to Shadow Traffic?
Ad-blockers, built-in browser privacy features, and other tools which stop events from reading analytics providers are some of the common factors that give rise to shadow traffic. These blockers were either used to prevent third-party personal information leakage or avoid absurd targeting.
In 2019, Global WebIndex reported that 48% of internet users use blockers to block served ads on the websites and keep their personal information away from third-party companies.
Ad-blockers: According to a report from Adobe and PageFair, there was an increase in the adoption of ad blockers by web users from 21 million in 2010 to more than 181 million in January 2020.
Based on the same report, by next year, $40 billion is what ad blocking could cost publishers. One of the primary reasons it’s going to happen is because the ad blockers prevent ads from rendering, with a lack of analytics data also playing a part here.
You can prevent such loss as few methods are available for working around the ad blockers and suppressing its effects.
Browser privacy features: Unlike ad blockers, the built-in browser-based blocking technologies do not actively seek to disrupt websites. As you know, these features stop the publishers from using cookies for showing targeted ads to the users.
Privacy features of known browsers, including Firefox’s Enhanced Tracking Protection, Safari’s Intelligent Tracking Prevention (ITP), and Microsoft Edge’s Tracking Protection, also act as built-in blockers of some first-party and third-party analytics services.
What are the other factors that are contributing to Shadow Traffic?
Many privacy tools used by users can also lead to this shadow traffic problem. Some of the tools include
- Network-level blocking tools
- VPN-level blocking tools
- Device DNS blocking tools
- Device App-based blocking tools
Long Answer:
- Network-level blocking is a problem for publishers as these kinds of tools block ads across the entire network. It works differently from a browser-based ad blocker; it also blocks ads in places like games and smart TVs. For example, Pi-hole.
- VPN-level blocking tool blocks ads stopping publishers from tracking any internet users. For example, NordVPN.
- Device DNS blocking tool blocks those devices or networks that request access. It also blocks communications between the users, leaving a big dent in your revenue, such as AdGuard.
- Device App-based blocking tool blocks the ads from all the apps that use browsers like Safari to display web pages. Such tools make it very difficult for publishers like you to reach out to your users as it not only blocks all the ads and blocks trackers, EU cookie, and GDPR notices. For example, Wipr.
How Can You Solve the Shadow Traffic Problem?
- Using Server Logs
- Leveraging First-Party Data servers
- Using First-party analytics providers
Long Explanation
On the brighter side, the shadow traffic problem can be solved if you use the right strategy. With better workarounds, you can gain visibility and solve the problem. But they require a heavy load of engineering or IT expertise.
With the approaches given below, you will be able to evade shadow traffic problems. These approaches can minimize the impact of ad blocking (added advantage is you can also display ads to Adblock users) and built-in browser features on analytics.
Server Logs
Every time any user visits your site, he/she connects with your web servers. Whenever that happens, it tracks and logs this interaction on your server. The idea is to rely on the server logs, consolidate it, and use the same to identify the actual traffic hits/data.
But, it’s easier said than done.
Adaption of such an approach can fall apart because it’s nearly impossible to restore and manage logs from different web servers, Content Delivery Network (CDN), or multiple sites. Even if we can manage logs with the server log management software’s presence, it still won’t give you the desired results that you are expecting from it.
Leverage First-party Server
Typically, ad blocking happens on the user’s web browser, and that restricts certain things. But the user still has to interact with the publisher’s server to retrieve content. So, the connection between the user’s browser and your content server cannot be blocked.
So, rather than connecting to a third-party analytics providers’ server, you can just use your server and capture the events. And via your server, you can send the data related to traffic sources. But it has its complications, especially when it comes to implementing it right.
On the contrary, the usual implementation of third-party analytics is pretty simple. All you need to do is to generate the analytics script and paste it on your website.
Use First-party Analytics Providers
You can use first-party analytics vendors’ (managed services) to help you access and track your website visitors and events via first-party servers. Such an approach can allow you to track your visible and shadow traffic on the site and still maintain your user’s privacy and experience.
What’s Next?
Today, with the users becoming smarter and the digital landscape becoming more complex, you have to continually find ways to stay on your toes to keep up with every problem, especially shadow traffic, as they indirectly impact your ad revenue earnings and user experience. How do you approach the shadow traffic issue? Have you tried any methods/tools that we haven’t listed? Let us know in the comments.
