<link rel="stylesheet" href="styles.f3b1fba60ec7970c.css">

Publication:
Neurocache: Efficient Vector Retrieval for Long-range Language Modeling

Loading...
Thumbnail Image

School / College / Institute

Item type:Organizational Unit,
Item type:Organizational Unit,
Item type:Organizational Unit,

Program

Organization Authors

Co-Authors

Date

Language

Embargo Status

Journal Title

Journal ISSN

Volume Title

Alternative Title

Abstract

This paper introduces Neurocache, an approach to extend the effective context size of large language models (LLMs) using an external vector cache to store its past states. Like recent vector retrieval approaches, Neurocache uses an efficient k-nearest-neighbor (kNN) algorithm to retrieve relevant past states and incorporate them into the attention process. Neurocache improves upon previous methods by (1) storing compressed states, which reduces cache size;(2) performing a single retrieval operation per token which increases inference speed;and (3) extending the retrieval window to neighboring states, which improves both language modeling and downstream task accuracy. Our experiments show the effectiveness of Neurocache both for models trained from scratch and for pre-trained models such as Llama2-7B and Mistral-7B when enhanced with the cache mechanism. We also compare Neurocache with text retrieval methods and show improvements in single-document question-answering and few-shot learning tasks. We made the source code available under: https://github.com/alisafaya/neurocache © 2024 Association for Computational Linguistics.

Source

Publisher

Association for Computational Linguistics (ACL)

Citation

item.page.haspartof

Source

Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024

item.page.ispartofseries

item.page.edition

DOI

10.18653/v1/2024.naacl-long.50

item.page.datauri

item.page.link

Rights

Copyrights Note

Endorsement

Review

Supplemented By

Referenced By

Related Patent

Related Goal

Google Scholar
Scholar'da Ara ↗
Altmetric
Dimensions
PlumX Metrikleri
BIP! Indicators