Measuring User Engagement - A New Metric

Measuring User Engagement - A New Metric

July 31, 2023
Marketing Two Cents

Table of Contents (Click to show/hide)

Data Analysis Skills |

Active User Rate

Daily Budget

Campaign Budget

Click-through rate increase

Click-through Rate

an icon shows a lightbulb that indicates being creative |

Return on Investment

Data Analysis Skills |


Daily Budget

Prioritisation of
Limited Resources

Click-through rate increase


an icon shows a lightbulb that indicates being creative |


When it comes to measuring user engagement, Clickthrough Rate (CTR) is the first metric that springs to mind, but all is not as it seems. As a result, Pinterest has tweaked this metric in order to gain further information around user engagement.

The Importance of Understanding What Users Want

Understanding what users want is crucial to ensuring engagement levels, and ultimately conversation rates excel. The CTR metric will consider the number of clicks that an ad receives and then divides that by the number of times it was viewed. This provides clear data and it is an industry-standard which makes it more likely to be accepted. Furthermore, advertisers want high CTRs, which means both the interests of users and advertisers are met.

Despite this, everything is not as it seems with CTR. Position bias is a significant problem as ads are placed higher up, which means they are more likely to be clicked, but this is not taken into account. Furthermore, CTR does not consider other signals such as what a user likes or dislikes, providing data that is not a clear indication of the real situation. All of this can mean that CTR is not wholly reliable. 

As a result, a new metric was designed that incorporates solutions to the CTR problems as well as multiple engagements and takes into consideration position bias.

As a result, the metric is defined as :

Weighted Engagement on Ads/Weighted Engagement on Neighboring Organic Content

Weighted Engagement on Ads

This takes into consideration a range of actions such as clicks, hides, saves and other signals, unlike CTE. Therefore, as an example, it could be possible to measure user engagement on ads as  = CTR - 20* Hide Rate. 

In this instance, a larger negative number is taken and it is multiplied with the Hide rate. It is negative because “Hides” indicate dissatisfaction in users and it is large as this indicates strong dissatisfaction. 

So, it is possible to have two ads. Ad 1 sees more clicks than Ad 2, yet more users hide it.

CTR Hide Rate Weighted Ad Engagement
Ad #1 1% 0.01% 0.8%
Ad #2 1.2% 0.05% 0.2%

Image shows impact of hide rate on userengagement results.

If the CTR metric was considered here, it would suggest that Ad 2 is better than Ad 1. However, once the hides are considered, it shows that Ad 1 benefits from greater user engagement.

With this level of insight and analysis, it is possible to see how actions affect user retention as well as long-term revenue.

Dealing with Position Bias

Position Bias plays a significant role in CTR. However, take two groups known as Group 1 and Group 2. 

Using the CTR metric as a measurement of user engagement, users in Group 1 are shown ads on page 1 while users in Group 2 are shown ads on page 8. You could expect to see results like this:

Group #1 1.0%
Group #2 0.9%

Image shows CTR forads shown on page 1

Using the CTR metric, group 1 looks to be the better groups. As users scroll through, the CTRs decrease on ads and organic content and this makes Group 2 look better, although the CTR is only slightly lower and not what is expected.

So, this needs to be formalized and this is done by comparing the average engagement rate on ads to the average engagement rate of organic content in position prior and after. This is where the denominator is introduced.

If ads in Group 1 and Group 2 are delivered like this:

How ads are positioned in Group 1(control) and Group 2(Treatment)
How ads arepositioned in Group 1(control) and Group 2(Treatment)

Then it could be possible to see results that look like this:

Ad CTR Neighboring Organic CTR Weighted Ad Engagement//Neighboring Organic Engagement
Group #1 1% 2% 0.5
Group #2 0.9% 0.5% 1.8%

Image showing CTR results and Neighboringorganic CTR results

So, Group 2 looks better than group 1 for overall user engagement when it’s adjusted for organic engagement. This is down to the fact that not a lot of organic engagement is removed. Therefore, the position of bias of CTR is reduced when ads to organic content are compared.

As a result, it is clear to see that the new metric is effective but the user experience cannot be understood completely through engagement. Therefore, in-platform surveys, user interviews and ad relevance all play a significant role.

Source: (1)

PPC Tips & Tricks
Paid Social
Industry Update
Best Practices
Dean Long | Expert in Growth MarketingHongxin(Dean) Long

Dean Long is a Sydney-based performance marketing and communication professional with expertise in paid search, paid social, affiliate, and digital advertising. He holds a Bachelor's degree in Information Systems and Management and is also a distinguished MBA graduate from Western Sydney University.

Related Posts