A headline is the most important part of any news story. It sets the tone for the rest of the article, and, for those readers who are only cursorily browsing the day’s news, the headline may be all they take away about an event. Research has shown that biased headlines can have a major impact not only on readers’ ability to make inferences from a news story but also on their ability to recall the facts presented in it. But how prevalent is such bias in headlines? Do certain news categories lend themselves more easily to such slant? And which publishers are more at fault for this? In this research blog post, we investigate whether and to what extent news headlines deviate from their article text in terms of their political lean.

To study this phenomenon, we dug into the core Media Bias Detector dataset: all thirty-four thousand articles published by ten major news publishers between January and March that were published in the top 10 slots on their landing page. We used GPT-4 to separately label the political lean of the headline and article text (minus the headline) on a scale of -5 to +5, with -5 representing strong pro-Democrat lean and +5 representing strong pro-Republican lean.

Figure 1 shows the result, broken down by publisher. Average headline lean is on the y-axis and average article lean is on the x-axis, with positive values representing increasingly pro-Republican lean (red) and negative values representing increasingly pro-Democrat lean (blue). The diagonal represents cases where the headline and article lean are equivalent, while darker shading represents regions where the titles have a stronger lean than the articles and lighter shading represents the inverse. The dark blue marker shows the mean over all articles, and the size of each scatter point represents the number of articles by that publisher.

Two trends are immediately apparent from this plot. First, headlines exhibit less lean than the articles themselves (almost all publishers find themselves in the lightly shaded areas). This runs somewhat counterintuitive to what we might have expected, given what we know about the role of ‘clickbait’ in the news industry for maximizing traffic to the source websites, and, by extension, ad revenue for the parent organization. Instead, it appears that major news publishers choose to write headlines in a seemingly unbiased style in order to increase their appeal to readers from across the political spectrum. However, for those readers that choose to read the articles through to the end, they may end up with a very different impression than those that only glanced at the headline. Consider, for example, Fox News’ headline “In State of the Union speech Biden showed America his entire presidency is based on one word”. Looks simple enough, right? The article behind it, however, is anything but: it devolves immediately into a tirade against the President, his policies, and his age. Such examples are not uncommon, and show us that sometimes a more careful read is needed before we can understand an article’s true angle.

The second trend we observe is that, overall, news headlines lean slightly more pro-Republican than the articles themselves, which is highlighted by the position of the dark blue marker representing the overall average being above the diagonal. In light of what we discussed earlier about the impact of unfaithful headlines, this can have troubling implications on the perceptions formed in the minds of news readers. Even though the article itself might favor Democrats or Democratic policies, the less-than-enthusiastic headline might leave readers with a different takeaway, more reflective of the neutral headline than the facts in the article. This effect would be far more significant for readers who come across such a headline while scrolling through their social media feed or while skimming across a newspaper’s homepage — they might never understand the true nature of the story without reading the full article. This problem is further exacerbated with the increasingly common usage of mobile app push notifications by news publishers which put the headlines directly on readers’ home screens.

To understand which kinds of news are most susceptible to this headline bias, we break the same results down by news category in Figure 2. The data, colors, and axes remain the same as before, only the scale has been magnified to enhance readability. Our attention is immediately drawn towards the economy category, which shows a significant vertical displacement relative to the diagonal. While articles about the state of the economy may be neutral to very slightly Democrat-leaning, their headlines lean far more in favor of Republicans (as has been noticed by some readers). We quantify this bias by computing the additive, i.e. how far a category places above the diagonal, defined as average article lean subtracted from average headline lean. Figure 3 shows the result, and it’s interesting to see that the top categories with headlines that lean more pro-Republican than their article text — Economy, Politics, and Business — are also some of the most important ones in the context of the upcoming Presidential Election. In turn, this “headline bias” might explain some of the irritation that readers of the Times feel towards it. Particularly in cases where the article itself, when read carefully and in full, strikes a liberal reader as reasonable, then it is very likely that the headline will seem biased in the wrong direction.

In conclusion, news headlines are not always faithful to the articles they represent. While they may exhibit less bias overall in absolute terms, they can still be misleading on individual topics by framing the story in a certain way. Our advice to news readers is to pay careful attention when they’re browsing the day’s news. Headlines can paint a very different picture of events, and a deeper look is often needed to understand what really happened. To journalists and editors, our message is to be especially cognizant of how choice of headline frames the article. Given the frequency with which readers see headlines and don’t read the text, these choices can impact on what people know and believe about the world around them..