Measuring User Engagement - A New Metric

When it comes to measuring user engagement, Clickthrough Rate (CTR) is the first metric that springs to mind, but all is not as it seems. As a result, Pinterest has tweaked this metric in order to gain further information around user engagement.

The Importance of Understanding What Users Want

Understanding what users want is crucial to ensuring engagement levels, and ultimately conversation rates excel. The CTR metric will consider the number of clicks that an ad receives and then divides that by the number of times it was viewed. This provides clear data and it is an industry-standard which makes it more likely to be accepted. Furthermore, advertisers want high CTRs, which means both the interests of users and advertisers are met.

Despite this, everything is not as it seems with CTR. Position bias is a significant problem as ads are placed higher up, which means they are more likely to be clicked, but this is not taken into account. Furthermore, CTR does not consider other signals such as what a user likes or dislikes, providing data that is not a clear indication of the real situation. All of this can mean that CTR is not wholly reliable. 

As a result, a new metric was designed that incorporates solutions to the CTR problems as well as multiple engagements and takes into consideration position bias.

As a result, the metric is defined as :

Weighted Engagement on Ads/Weighted Engagement on Neighboring Organic Content

Weighted Engagement on Ads

This takes into consideration a range of actions such as clicks, hides, saves and other signals, unlike CTE. Therefore, as an example, it could be possible to measure user engagement on ads as  = CTR - 20* Hide Rate. 

In this instance, a larger negative number is taken and it is multiplied with the Hide rate. It is negative because “Hides” indicate dissatisfaction in users and it is large as this indicates strong dissatisfaction. 

So, it is possible to have two ads. Ad 1 sees more clicks than Ad 2, yet more users hide it.

Image shows impact of hide rate on userengagement results.

If the CTR metric was considered here, it would suggest that Ad 2 is better than Ad 1. However, once the hides are considered, it shows that Ad 1 benefits from greater user engagement.

With this level of insight and analysis, it is possible to see how actions affect user retention as well as long-term revenue.

Dealing with Position Bias

Position Bias plays a significant role in CTR. However, take two groups known as Group 1 and Group 2. 

Using the CTR metric as a measurement of user engagement, users in Group 1 are shown ads on page 1 while users in Group 2 are shown ads on page 8. You could expect to see results like this:
‍

Image shows CTR forads shown on page 1

‍

Using the CTR metric, group 1 looks to be the better groups. As users scroll through, the CTRs decrease on ads and organic content and this makes Group 2 look better, although the CTR is only slightly lower and not what is expected.

So, this needs to be formalized and this is done by comparing the average engagement rate on ads to the average engagement rate of organic content in position prior and after. This is where the denominator is introduced.

If ads in Group 1 and Group 2 are delivered like this:

Then it could be possible to see results that look like this:

Image showing CTR results and Neighboringorganic CTR results

‍

So, Group 2 looks better than group 1 for overall user engagement when it’s adjusted for organic engagement. This is down to the fact that not a lot of organic engagement is removed. Therefore, the position of bias of CTR is reduced when ads to organic content are compared.

As a result, it is clear to see that the new metric is effective but the user experience cannot be understood completely through engagement. Therefore, in-platform surveys, user interviews and ad relevance all play a significant role.

‍

Source: (1)

‍