News Recommendation using Advanced LLM Embedding Models

Abstract

Online news recommendation is a subset of recommendation systems used by online news portals to filter news articles according to user interest. News recommendation methods generally have three main components: news encoder, user encoder, and click predictor. After the success and popularity of BERT, many methods were developed that incorporate fine-tuned BERT for feature extraction. However, there have been further advancements in the field of LLM based embedding models, generalized embedding models, and decoder-based embedding models, that have not been studied yet for news recommendation systems. Most news recommendation studies consider only ranking a candidate set of news items for each user. In this study we compared this with other news recommendation tasks as well: user-news classification; all user classification for given news; candidate news classification for user; and candidate news ranking for user. We also studied multilingual news recommendation and the inclusion of a fake news classification component. In this study we investigated the effect of using five BERT-based models, five large decoder-based models, and one proprietary model as news encoders for news recommendation. We also studied the effect of different model inputs (title, category, and abstract) on news and user representation, as well as the effect of four user encoders:average pooling, attention pooling, dense layer + attention pooling, multi-head self-attention + attention pooling. Inclusion of the fake news classification component improved results for news recommendation tasks 1, 2 and 3, but reduced performance for task 4. For the first task, multilingual-e5-large-instruct with title, category, subcategory, and abstract input and dense layer + attention pooling for user encoder achieved the highest accuracy of 0.9149; for the second task, text-embedding-3-large with title input and multi-head self-attention + attention pooling for user encoder achieved the highest accuracy of 0.8902; for the third task, multilingual-e5-large-instruct with title as input and multi-head self-attention + attention pooling for user encoder achieved the highest accuracy of 0.8641; and for the fourth task, mxbai-embed-large-v1 with title, category and abstract as input and multi-head self-attention + attention pooling for user encoder achieved the highest AUC of 0.6648. Our findings indicate that frozen generalized embeddings can provide competitive news representations without embedding-model fine-tuning, although performance depends strongly on the recommendation task and user encoder.