I Tested Whether Popups Work: What I Found Shocked Me

Sorry for the headline.

I wanted to pay homage to modern news sites and the joy of clickbait. Clickbait headlines are a lot like another trend: Popups.

Not the old-school window-spawners, but the modal boxes that appear on so many websites. Some as interstitials, some appearing on scroll, some timed, and others when the user mouse tracks outside of the viewport (exit intent).

I’ve experimented with popups on a site that sells ebooks.

Why Use Popups?

Here’s how our design decisions get made:

“All the other sites use these popups – so we should be too…”

Not a great reason, but the Internet is a giant echo chamber. Very rarely are changes actually data driven. Many providers of popup technology claim increases in conversion rates (but they are trying to sell the product).

Popups to Avoid

popup
Google announced they will penalize pages containing certain kinds of popups. This includes:

  • Interstitials.
  • Popup that covers main content.
  • Fake interstitials – above-the-fold content looks like an interstitial (very confusing).

This interstitial greeted me on arrival at an article on a local tabloid newspaper. I tried swiping down and thought the site was broken as it wouldn’t scroll… then I realized I had to go searching for the little X… Google is about to punish this kind of thing.

img_1770

I also question ALL popups that appear on arrival at a webpage.

I find those kinds of popups annoying. Why would I signup for your newsletter when I haven’t yet seen your content? Signing up comes AFTER I’ve read something useful.

Advertisement

Exit Intent or Timed Popups

I have used these kinds of popups.

Exit Intent: When the user expresses intention to leave the webpage, show the popup.

Timed: After a set duration display the popup.

There are a lot of services out there that host these popups for you. To be honest they’re not cheap (typical many hundreds per year). OptinMonster, Privy, OptiMonk are a few.

There are loads of articles singing the praises, but offer very little to back up the claims. I also suspect popups may have reached (or passed) their effectiveness peak. Users are becoming blind to them.

Testing a Popup on Mobile

I’ve used an exit intent popup. They only work on desktop browsers. They use some JavaScript to track the mouse position – when it leaves the browser viewport, the popup is displayed.

Despite the claims of some companies – exit intent popups do not work on mobile or tablet (or any touch-based interface). That’s because you do not drag your finger out of the viewport on a mobile device. The only option for mobile is to use a timed popup.

mypopup

Measuring the Results

I implemented a timed popup on the most popular page on a site. After the user has stayed on the page for almost 3 minutes, the popup appears (the duration was chosen based on typical time on page). The idea is to alert users to the sales page, and entice them to click thru to the page. We’ll call this a conversion.

I checked with Google Analytics data (segmenting to Mobile and Tablet users), measuring how many times they viewed the popular page, and how many times they navigated to the sales page. I measured over a period of 83 days (before and after the change).

Caveat: This is not as robust as an AB test – the results are from two different time periods (so other factors are at play).

Page Views Conversions %
No Popup 176,788 7,935 4.49%
With Popup 152,342 5,359 3.52%

The results are disturbing.

During this period no other changes were made to the page. I expected MORE users to navigate to the sales page. Instead there were less – about 22% less page views.

A timed popup on mobile showed no increase in conversions – but instead decreased interaction.

Why? I don’t know. At worst the popup was such an annoyance that users gave up on the page altogether. At best it is randomness (to eliminate this I need more data – but the site does not have enough traffic for that).

Either way – timed popups on mobile are a complete fail for this site.

Timed Popups on Desktop

What happens if I check the data for desktop devices? It’s not good I’m afraid.

Page Views Conversions %
No Popup 80,235 4,547 5.67%
With Popup 64,342 3,057 4.75%

Consistent with other data across other sites I monitor – there is always more engagement on Desktop in general. Yet when it comes to timed popups – the results are disappointing.

Timed popups on desktop appear to discourage further engagement with the site.

Should You Use Timed Popups?

You should always try to get your own data (which always varies per site and niche). However these results shocked me. Could it be that most other users feel the same way as me? If something interrupts my reading of a page, I dismiss it immediately in frustration.

This is not the most scientific test, but it’s enough for me. Don’t use timed popups. Let your content do the persuading.

What about exit-intent popups? More on that soon.

Hi, I'm James, and for the last decade I've made a living by making my own blogs and websites.
Updated: September 21, 2016

1 Comment

  1. My 7-year-old niece plays a lot of free ipad games with full screen popups. She has become an expert at quickly finding the ‘X’, even when they try to be tricky about it.

    If you want to know the future of any technology, observe how the kids treat it because they are the future adult users.

Add a Comment