Multimodal fusion for effective recommendations on a user-anonymous price comparison platform

Abstract

This study proposes a novel recommendation framework designed for a digital price comparison platform. The challenges arise from the absence of user login and gold labels in item variations, which make effective recommendations tricky. The proposed framework integrates three distinct modalities: product titles using a multilingual BERT model, product images through the CLIP model, and click data via a novel Word2Vec model named Prod2Vec. Three fusion methods were tested to obtain a single unified representation for a given product: early, intermediate, and late fusion. Offline evaluations showcased a significant performance boost when leveraging all three modalities and employing intermediate fusion. The proposed framework achieved an impressive 92% Adjusted Rand Index clustering score at the category level. Fusion with two modalities also proved to be competitively effective, yielding scores between 87% and 88%. The framework was shown to be scalable by maintaining good performance even when we increased the number of categories up to 50. For online evaluations, we selected three representative categories and deployed the best-selected fusion method on the platform through A/B testing against a click-text encoding baseline. Our framework resulted in a significant improvement by increasing the Click-Through Rate from 1.43% to 3.17% across all categories. These findings underscore the efficacy of the proposed framework in enhancing user engagement and interaction with the platform. © 2024 IEEE.