Measuring User Engagement - A New Metric

Measuring User Engagement - A New Metric

August 5, 2021
Marketing Two Cents
Data Analysis Skills | DEANLONG.io
10.54%

Monthly
Active User Rate

Daily Budget
$5000

Daily
Campaign Budget

Click-through rate increase
60%

Increase
Click-through Rate

an icon shows a lightbulb that indicates being creative | DEANLONG.io
15%

Growth
Return on Investment

Data Analysis Skills | DEANLONG.io
#1

Customer
Segmentation

Daily Budget
#2

Prioritisation of
Limited Resources

Click-through rate increase
#3

Competitive
Responses

an icon shows a lightbulb that indicates being creative | DEANLONG.io
#4

Consumer
Change

When it comes to measuring user engagement, Clickthrough Rate (CTR) is the first metric that springs to mind, but all is not as it seems. As a result, Pinterest has tweaked this metric in order to gain further information around user engagement.


The Importance of Understanding What Users Want


Understanding what users want is crucial to ensuring engagement levels, and ultimately conversation rates excel. The CTR metric will consider the number of clicks that an ad receives and then divides that by the number of times it was viewed. This provides clear data and it is an industry-standard which makes it more likely to be accepted. Furthermore, advertisers want high CTRs, which means both the interests of users and advertisers are met.


Despite this, everything is not as it seems with CTR. Position bias is a significant problem as ads are placed higher up, which means they are more likely to be clicked, but this is not taken into account. Furthermore, CTR does not consider other signals such as what a user likes or dislikes, providing data that is not a clear indication of the real situation. All of this can mean that CTR is not wholly reliable. 


As a result, a new metric was designed that incorporates solutions to the CTR problems as well as multiple engagements and takes into consideration position bias.


As a result, the metric is defined as :


Weighted Engagement on Ads/Weighted Engagement on Neighboring Organic Content


Weighted Engagement on Ads


This takes into consideration a range of actions such as clicks, hides, saves and other signals, unlike CTE. Therefore, as an example, it could be possible to measure user engagement on ads as  = CTR - 20* Hide Rate. 


In this instance, a larger negative number is taken and it is multiplied with the Hide rate. It is negative because “Hides” indicate dissatisfaction in users and it is large as this indicates strong dissatisfaction. 


So, it is possible to have two ads. Ad 1 sees more clicks than Ad 2, yet more users hide it.

CTR Hide Rate Weighted Ad Engagement
Ad #1 1% 0.01% 0.8%
Ad #2 1.2% 0.05% 0.2%

Image shows impact of hide rate on userengagement results.


If the CTR metric was considered here, it would suggest that Ad 2 is better than Ad 1. However, once the hides are considered, it shows that Ad 1 benefits from greater user engagement.


With this level of insight and analysis, it is possible to see how actions affect user retention as well as long-term revenue.


Dealing with Position Bias


Position Bias plays a significant role in CTR. However, take two groups known as Group 1 and Group 2. 


Using the CTR metric as a measurement of user engagement, users in Group 1 are shown ads on page 1 while users in Group 2 are shown ads on page 8. You could expect to see results like this:

CTR
Group #1 1.0%
Group #2 0.9%

Image shows CTR forads shown on page 1


Using the CTR metric, group 1 looks to be the better groups. As users scroll through, the CTRs decrease on ads and organic content and this makes Group 2 look better, although the CTR is only slightly lower and not what is expected.


So, this needs to be formalized and this is done by comparing the average engagement rate on ads to the average engagement rate of organic content in position prior and after. This is where the denominator is introduced.


If ads in Group 1 and Group 2 are delivered like this:

How ads are positioned in Group 1(control) and Group 2(Treatment)
How ads arepositioned in Group 1(control) and Group 2(Treatment)


Then it could be possible to see results that look like this:

Ad CTR Neighboring Organic CTR Weighted Ad Engagement//Neighboring Organic Engagement
Group #1 1% 2% 0.5
Group #2 0.9% 0.5% 1.8%


Image showing CTR results and Neighboringorganic CTR results

So, Group 2 looks better than group 1 for overall user engagement when it’s adjusted for organic engagement. This is down to the fact that not a lot of organic engagement is removed. Therefore, the position of bias of CTR is reduced when ads to organic content are compared.


As a result, it is clear to see that the new metric is effective but the user experience cannot be understood completely through engagement. Therefore, in-platform surveys, user interviews and ad relevance all play a significant role.

Source: (1)

PPC Tips & Tricks
Paid Social
Industry Update
Best Practices
Trends
Dean Long | Expert in Growth MarketingDean Long

Dean Long is a Sydney-based creative marketing and communication professional with expertise in paid search, paid social, affiliate and e-commerce advertising. He's also a distinct MBA Graduate from Western Sydney University.

Related Posts