LongMemEval: Benchmarking Chat Assist- ants on Long-Term Interactive Memory 1 Introduction 2 Related Work Long-Term Dialogue Benchmarks Long-Term Memory Methods 3 LongMemEval 3.1 Problem Formulation 3.2 LongMemEval : Benchmark Curation Question Curation Evidence Session Construction History Compilation 3.3 Evaluation Metric Question Answering Memory Recall 3.4 LongMemEval represents a significant challenge Commercial systems Long-Context LLMs 4 A Unified View of Long-Term Memory Assistants 4.1 Long-Term Memory System: Formulation 4.2 Long-Term Memory System: Design Choices CP 1: Value CP 2: Key CP 3: Query CP 4: Reading Strategy 5 Experiment Results 5.1 Experimental Setup 5.2 Value: Decomposition improves RAG performance 5.3 Key: Multi-key indexing improves retrieval and RAG 5.4 Query: Time-aware query expansion improves temporal reasoning 5.5 Improving reading with chain-of-note and structured format 6 Conclusion A Supplemental Details for LongMemEval A.1 Dataset Construction Attribute Ontology Background Sampling Question Construction Evidence Session Construction A.2 History Construction Session pool construction Session sampling Timestamp resolution A.3 Basic Statistics A.4 Evaluation Metric Building B A Human Study on Commercial Memory Chatbots C Unified Memory View C.1 An Alternative Mathematical Formulation C.2 Existing Memory Systems from the Unified View Indexing Retrieval Generation D Memory Optimizations: Implementation Details Value Decomposition Key Expansion Time-Aware Indexing and Query Expansion Reading Strategy E Extended Analyses E.1 Results on More LLMs E.2 Ablations on Retriever Selection E.3 Post-retrieval rank merging for index expansion E.4 Strong and weak LLMs for extracting time ranges from queries E.5 Error Analysis LongMemEval : Benchmarking Chat Assist- ants on Long-Term Interactive Memory Di Wu 1  , Hongwei Wang 2 , Wenhao Yu 2 , Yuwei Zhang 3* , Kai-Wei Chang 1 , Dong Yu 2 1 UCLA, 2 Tencent AI Lab Seattle, 3 UC San Diego {diwu,kwchang}@cs.ucla.edu work done during internship at Tencent AI Lab, mentored by Hongwei and Wenhao. Abstract Recent large language model (LLM)-driven chat assistant systems have integrated memory components to track user-assistant chat histories, enabling more accurate and personalized responses. However, their long-term memory capabilities in sustained interactions remain underexplored. We introduce LongMemEval , a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. With 500 meticulously curated questions embedded within freely scalable user-assistant chat histories, LongMemEval presents a significant challenge to existing long-term memory systems, with commercial chat assistants and long-context LLMs showing a 30% accuracy drop on memorizing information across sustained interactions. We then present a unified framework that breaks down the long-term memory design into three stages: indexing, retrieval, and reading. Built upon key experimental insights, we propose several memory design optimizations including session decomposition for value granularity, fact-augmented key expansion for indexing, and time-aware query expansion for refining the search scope. Extensive experiments show that these optimizations greatly improve both memory recall and downstream question answering on LongMemEval . Overall, our study provides valuable resources and guidance for advancing the long-term memory capabilities of LLM-based chat assistants, paving the way toward more personalized and reliable conversational AI. Our benchmark and code are publicly available at https://github.com/xiaowu0162/LongMemEval . 1 Introduction Large language models (LLMs) have exhibited impressive capabilities in solving diverse tasks through natural language, leading to numerous successful chat assistant applications (OpenAI, 2022 ; Microsoft, 2023 ) . Nevertheless, LLMs face limitations on tasks relying heavily on personal knowledge accumulated through long-term user-AI interactions, such as psychological counseling or secretarial duties (Zhong et al., 2024 ) . Failing to incorporate user background and preferences into responses can diminish the response’s accuracy as well as user satisfaction. To personalize LLM-based assistants, long-term memory, the ability to memorize, recall, and reason with a long interaction history, is indispensable. Recently, several commercial (OpenAI, 2024 ; Coze, 2024 ) and open-source assistant systems with memory (Zhong et al., 2024 ; Zhang et al., 2024 ) have been introduced. These systems leverage techniques like compressing, indexing, and retrieving from chat histories to generate more accurate and personalized responses. Despite these advances, there has been limited progress in holistically evaluating the memory capability in long-term interactions. While several benchmarks evaluate LLMs on understanding long chat histories (Xu et al., 2022a ; b ; Zhong et al., 2024 ; Maharana et al., 2024 ; Du et al., 2024 ; Kim et al., 2024 ) , they have two major shortcomings. First, they do not accurately reflect user-AI interactions: many focus solely on human-human conversations (Xu et al., 2022a ; Maharana et al., 2024 ; Kim et al., 2024 ) , while others omit task-oriented dialogues, which represent a significant portion of chat assistant usage and challenge memorization with the long-context inputs and long-form responses. Their interactive histories also typically have a non-configurable length spanning only a few thousand tokens, limiting the difficulty as current systems continue to improve. Second, current benchmarks’ questions only offer a limited coverage of the memory abilities required in dynamic long-term interactions. For instance, MemoryBank (Zhong et al., 2024 ) and PerLTQA (Du et al., 2024 ) insufficiently evaluate the ability to synthesize information across numerous sessions or to reason with temporal metadata or time references. All long-term memory benchmarks including recent ones such as LoCoMo (Maharana et al., 2024 ) also fail to evaluate recall of information provided by the assistant or reasoning with updated user information. We introduce LongMemEval , a comprehensive benchmark for assessing the long-term memory capabilities of chat assistants. LongMemEval consists of 500 manually created questions to test five core memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Each question requires recalling information hidden within one or more task-oriented dialogues between a user and an assistant. Inspired by the “needle-in-a-haystack” test (Kamradt, 2023 ) , we design a pipeline to compile a coherent and length-configurable chat history for each question. A chat system, then, is required to parse the dynamic interactions online for memorization, and answer the question after all the interaction sessions. While the length of the history is freely extensible, we provide two standard settings for consistent comparison: LongMemEval S with approximately 115k tokens per problem and LongMemEval M with 500 sessions (around 1.5 million tokens). Preliminary evaluations highlight the difficulty of LongMemEval , as long-context LLMs show a 30% ∼ similar-to \sim ∼ 60% performance drop on LongMemEval S , and manual evaluations reveal that state-of-the-art commercial systems only achieved 30% ∼ similar-to \sim ∼ 70% accuracy in a setting much simpler than LongMemEval S ( section   3.4 ). Finally, we present a unified view for memory-augmented chat assistants. Leveraging LongMemEval , we comprehensively analyze memory design choices across three key execution stages—indexing, retrieval, and reading—and four control points: value, key, query, and reading strategy. Experimental insights identify several effective memory designs: • ( Section   5.2 ) Instead of sessions, round is the more optimal granularity for storing and utilizing the interactive history. While further compression into individual user facts harms overall performance due to information loss, it improves the multi-session reasoning accuracy. • ( Section   5.3 ) While using a flat index with the memory values themselves as the keys is a strong baseline, further expanding the keys with extracted user facts improves both memory recall (9.4% higher recall@ k 𝑘 k italic_k ) and downstream question answering (5.4% higher accuracy). • ( Section   5.4 ) Naive time-agnostic memory designs perform poorly on temporal reasoning questions. We propose a simple indexing and query expansion strategy to explicitly associate timestamps with facts and narrow down the search range, improving the memory recall for temporal reasoning by 6.8% ∼ similar-to \sim ∼ 11.3% when a strong LLM is employed for query expansion. • ( Section   5.5 ) Even with perfect memory recall, accurately utilizing retrieved items is non-trivial. Applying Chain-of-Note (Yu et al., 2023 ) and structured data format (Yin et al., 2023 ) improves question answering accuracy by as much as 10 absolute points across three LLMs. 2 Related Work Long-Term Dialogue Benchmarks As the ability of dialogue systems improve, research starts to focus on long-term dialogue understanding beyond traditional dialogue modeling benchmarks (Budzianowski et al., 2018 ; Wei et al., 2018 ) . Early works focused on language modeling evaluation on generating personalized responses from human-human (Xu et al., 2022a ) or human-AI (Xu et al., 2022b ) chat histories. To more precisely evaluate memory accuracy, subsequent benchmarks shifted toward question answering (QA, Reddy et al. ( 2019 ); Zhang & Choi ( 2021 ) ). For example, MemoryBank (Zhong et al., 2024 ) features multi-day chat histories from 15 users with 194 human-written probing questions. LoCoMo (Maharana et al., 2024 ) includes 50 long-term chat histories and questions testing single-hop, multi-hop, temporal, commonsense, world knowledge, and adversarial reasoning. PerLTQA (Du et al., 2024 ) scales the evaluation to 3,409 dialogues and 8,593 questions, covering world knowledge, personal profiles, social relationships, events, and dialogue history. DialSim (Kim et al., 2024 ) evaluates models’ memory ability by roleplaying TV show characters and introduces a time constraint that penalizes slow system responses. Despite these advancements, existing QA-based benchmarks overlook several memory capabilities critical to long-term user-assistant interactions: synthesizing information across numerous sessions, recalling assistant side information, and reasoning about updated user details or complex temporal references. Additionally, the chat histories are often too brief and do not reflect the nature of task-oriented interactions. Table   1 compares between LongMemEval and previous works, highlighting its advantages in both (1) featuring a long and freely extensible iterative history and (2) holistically covering critical memory abilities in a uniquely challenging way (further examples in Figure   1 ). Table 1: A comparison between LongMemEval and existing long-term memory benchmarks. We use different colors to denote human-human and human-AI dialogue. #Sess and #Q denote the total number of sessions and questions. Context depth is defined as the number of tokens in the history. Finally, we compare the coverage of five core abilities: information extraction (IE), multi-session reasoning (MR), knowledge update (KU), temporal reasoning (TR), and abstaining on unanswerable questions (ABS). ∗ Not reported in the paper, based on our approximation. ∗∗ at most 2 sessions. Benchmark Domain #Sess #Q Context Depth Core Memory Abilities IE MR KU TR ABS MSC (Xu et al., 2022a ) Open-Domain 5k - 1k ✗ ✗ ✗ ✗ ✗ DuLeMon (Xu et al., 2022b ) Open-Domain 30k - 1k ✗ ✗ ✗ ✗ ✗ MemoryBank (Zhong et al., 2024 ) Personal 300 194 5k ✓ ✗ ✗ ✓ ✗ PerLTQA (Du et al., 2024 ) Personal 4k 8593 1M ∗ ✓ ✗ ✗ ✗ ✓ LoCoMo (Maharana et al., 2024 ) Personal 1k 7512 10k ✓ ✓ ✗ ✓ ✓ DialSim (Kim et al., 2024 ) TV Shows 1k–2k 1M 350k ✓ ✓ ∗∗ ✗ ✓ ✓ LongMemEval (this work) Personal 50k 500 115k, 1.5M ✓ ✓ ✓ ✓ ✓ Figure 1: Examples of the seven question types in LongMemEval . For each example, we show the associated evidence statements on the left and the question with the answer on the right. Long-Term Memory Methods To equip chat assistants with long-term memory capabilities, three major techniques are commonly explored. The first approach involves directly adapting LLMs to process extensive history information as long-context inputs (Beltagy et al., 2020 ; Kitaev et al., 2020 ; Fu et al., 2024 ; An et al., 2024 ) . While this method avoids the need for complex architectures, it is inefficient and susceptible to the “lost-in-the-middle” phenomenon, where the ability of LLMs to utilize contextual information weakens as the input length grows (Shi et al., 2023 ; Liu et al., 2024 ) . A second line of research integrates differentiable memory modules into language models, proposing specialized architectural designs and training strategies to enhance memory capabilities (Weston et al., 2014 ; Wu et al., 2022 ; Zhong et al., 2022 ; Wang et al., 2023 ) . Lastly, several studies approach long-term memory from the perspective of context compression, developing techniques to condense lengthy histories into compact representations, whether in the form of LLM internal representations (Mu et al., 2023 ; Chevalier et al., 2023 ) , discrete tokens (Jiang et al., 2023 ; Xu et al., 2024 ) , or retrievable text segments via retrieval-augmented generation (RAG, Shi et al. ( 2024 ); Wang et al. ( 2023 ); Sarthi et al. ( 2024 ); Chen et al. ( 2023a ); Gutiérrez et al. ( 2024 ) ). Although LongMemEval can evaluate any memory system, we will take an online context compression perspective, where each history interaction session is sequentially processed, stored, and accessed on-demand through indexing and retrieval mechanisms ( Section   4 ). This formulation aligns with current literature (Zhong et al., 2024 ; Gutiérrez et al., 2024 ) and commercial systems (OpenAI, 2024 ; Coze, 2024 ) . Its plug-and-play nature also facilitates the integration into existing chat assistant systems. 3 LongMemEval 3.1 Problem Formulation The evaluation of LongMemEval requires an instance of 4-tuple ( 𝐒 , q , t q , a ) 𝐒 𝑞 subscript 𝑡 𝑞 𝑎 (\mathbf{S},q,t_{q},a) ( bold_S , italic_q , italic_t start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT , italic_a ) . 𝐒 ≡ [ ( t 1 , S 1 ) , ( t 2 , S 2 ) , … , ( t N , S N ) ] 𝐒 subscript 𝑡 1 subscript 𝑆 1 subscript 𝑡 2 subscript 𝑆 2 … subscript 𝑡 𝑁 subscript 𝑆 𝑁 \mathbf{S}\equiv[(t_{1},S_{1}),(t_{2},S_{2}),...,(t_{N},S_{N})] bold_S ≡ [ ( italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) , ( italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) , … , ( italic_t start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT , italic_S start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ) ] is a sequence of N 𝑁 N italic_N history chat sessions ordered from the earliest to the latest, where S i subscript 𝑆 𝑖 S_{i} italic_S start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT is a multi-turn interaction between the user and a chat assistant and t i subscript 𝑡 𝑖 t_{i} italic_t start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT is the session’s timestamp. Each session can be further decomposed into rounds: one user message followed by one assistant response. During test time, 𝐒 𝐒 \mathbf{S} bold_S is provided to the system one by one. q 𝑞 q italic_q and t q > t N subscript 𝑡 𝑞 subscript 𝑡 𝑁 t_{q}>t_{N} italic_t start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT > italic_t start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT represent the question from the user and its date. a 𝑎 a italic_a is a short phrase indicating the answer, or a natural language rubric describing the preferred answer in the case where q 𝑞 q italic_q is open-ended. 3.2 LongMemEval : Benchmark Curation One major challenge in building a reliable personalized assistant is performing online recording, recalling, updating, and reasoning on the dynamically evolving user information. To comprehensively reflect the challenge, LongMemEval formulates five core long-term memory abilities: • Information Extraction (IE): Ability to recall specific information from extensive interactive histories, including the details mentioned by either the user or the assistant. • Multi-Session Reasoning (MR): Ability to synthesize the information across multiple history sessions to answer complex questions that involve aggregation and comparison. • Knowledge Updates (KU): Ability to recognize the changes in the user’s personal information and update the knowledge of the user dynamically over time. • Temporal Reasoning (TR): Awareness of the temporal aspects of user information, including both explicit time mentions and timestamp metadata in the interactions. • Abstention (ABS): Ability to identify questions seeking unknown information, i.e., information not mentioned by the user in the interaction history, and answer “I don’t know”. As shown in Table   1 , this formulation represents a more comprehensive ability coverage compared to prior long-term memory benchmarks like MemoryBank and PerLTQA. To thoroughly assess these abilities, LongMemEval features seven question types. Single-session-user and single-session-assistant test memorizing the information mentioned by user or assistant within a single session. Single-session-preference tests whether the model can utilize the user information to generate a personalized response. multi-session (MR) tests aggregating user information across two or more sessions. knowledge-update (KU) focuses on the ability to recognize changes in the user’s life states and update the memory accordingly. temporal-reasoning (TR) tests reasoning with both the timestamp in metadata and explicit time references. Finally, we draw 30 questions from the previous question types and modify them into “false premise” questions, testing whether the model can correctly abstain from answering (ABS). Figure   1 presents an example for each question type. Question Curation Figure   2 depicts the question curation pipeline. We define an ontology of 164 user attributes in five categories: lifestyle, belongings, life events, situations context, and demographic information. For each attribute, we leverage an LLM 1 1 1 Unless otherwise mentioned, Llama 3 70B Instruct (Dubey et al., 2024 ) is used as the LLM in the pipeline. to generate attribute-focused user background paragraphs, each of which includes detailed discussion of the user’s life experience. To create a question, we randomly sample a paragraph and use an LLM to propose several seed (question, answer) pairs. As these LLM-proposed questions often lack depth and diversity, human experts manually filter and rewrite all the questions to achieve the desired difficulty. Then, we manually decompose the answer into one or more evidence statements with optional timestamps. Evidence Session Construction Each evidence statement is then separately embedded into a task-oriented evidence session created by self-chatting (Xu et al., 2023 ) . The user LLM is instructed to convey the evidence statement indirectly, e.g., instead of stating “I bought a new car last month,” it might instead ask for help about car insurance and reveal the information incidentally. This approach en ... [OUTPUT TRUNCATED - 30000 chars omitted out of 80000 total] ... in Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone. CoRR , abs/2404.14219, 2024b. doi: 10.48550/ARXIV.2404.14219 . URL https://doi.org/10.48550/arXiv.2404.14219 . An et al. (2024) Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. Make your LLM fully utilize the context. CoRR , abs/2404.16811, 2024. doi: 10.48550/ARXIV.2404.16811 . URL https://doi.org/10.48550/arXiv.2404.16811 . Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150 , 2020. Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pp.  5016–5026, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1547 . URL https://aclanthology.org/D18-1547/ . Chen et al. (2023a) Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading. CoRR , abs/2310.05029, 2023a. doi: 10.48550/ARXIV.2310.05029 . URL https://doi.org/10.48550/arXiv.2310.05029 . Chen et al. (2023b) Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Hongming Zhang, and Dong Yu. Dense X retrieval: What retrieval granularity should we use? CoRR , abs/2312.06648, 2023b. doi: 10.48550/ARXIV.2312.06648 . URL https://doi.org/10.48550/arXiv.2312.06648 . Chevalier et al. (2023) Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pp.  3829–3846. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.EMNLP-MAIN.232 . URL https://doi.org/10.18653/v1/2023.emnlp-main.232 . Coze (2024) Coze. Memory overview guide. https://www.coze.com/docs/guides/memory_overview?_lang=en , 2024. Accessed: September 15, 2024. Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233 , 2023. Du et al. (2024) Yiming Du, Hongru Wang, Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong Wang, and Kam-Fai Wong. PerLTQA: A personal long-term memory dataset for memory classification, retrieval, and fusion in question answering. In Kam-Fai Wong, Min Zhang, Ruifeng Xu, Jing Li, Zhongyu Wei, Lin Gui, Bin Liang, and Runcong Zhao (eds.), Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10) , pp.  152–164, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.sighan-1.18/ . Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. Efron et al. (2012) Miles Efron, Peter Organisciak, and Katrina Fenlon. Improving retrieval of short texts through document expansion. In William R. Hersh, Jamie Callan, Yoelle Maarek, and Mark Sanderson (eds.), The 35th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR ’12, Portland, OR, USA, August 12-16, 2012 , pp.  911–920. ACM, 2012. doi: 10.1145/2348283.2348405 . URL https://doi.org/10.1145/2348283.2348405 . Fu et al. (2024) Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=TaAqeo7lUh . Gutiérrez et al. (2024) Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. CoRR , abs/2405.14831, 2024. doi: 10.48550/ARXIV.2405.14831 . URL https://doi.org/10.48550/arXiv.2405.14831 . Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res. , 2022, 2022. URL https://openreview.net/forum?id=jKN1pXi7b0 . Jiang et al. (2023) Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pp.  13358–13376. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.EMNLP-MAIN.825 . URL https://doi.org/10.18653/v1/2023.emnlp-main.825 . Kamradt (2023) Gregory Kamradt. Needle in a haystack - pressure testing llms. GitHub, 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack . Kim et al. (2024) Jiho Kim, Woosog Chay, Hyeonji Hwang, Daeun Kyung, Hyunseung Chung, Eunbyeol Cho, Yohan Jo, and Edward Choi. Dialsim: A real-time simulator for evaluating long-term dialogue understanding of conversational agents, 2024. URL https://arxiv.org/abs/2406.13144 . Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020. Li et al. (2024) Hao Li, Chenghao Yang, An Zhang, Yang Deng, Xiang Wang, and Tat-Seng Chua. Hello again! llm-powered personalized agent for long-term dialogue. CoRR , abs/2406.05925, 2024. doi: 10.48550/ARXIV.2406.05925 . URL https://doi.org/10.48550/arXiv.2406.05925 . Liu et al. (2024) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics , 12:157–173, 2024. doi: 10.1162/TACL“˙A“˙00638 . URL https://doi.org/10.1162/tacl_a_00638 . Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp.  2511–2522, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.153 . URL https://aclanthology.org/2023.emnlp-main.153/ . Maharana et al. (2024) Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  13851–13870, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.747 . URL https://aclanthology.org/2024.acl-long.747/ . Microsoft (2023) Microsoft. Announcing microsoft copilot, your everyday ai companion, 2023. URL https://blogs.microsoft.com/blog/2023/09/21/announcing-microsoft-copilot-your-everyday-ai-companion/ . Accessed: September 15, 2024. Mistral AI Team (2024) Mistral AI Team. Mistral nemo: Our new best small model. Mistral AI , July 2024. URL https://mistral.ai/news/mistral-nemo . Mu et al. (2023) Jesse Mu, Xiang Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/3d77c6dcc7f143aa2154e7f4d5e22d68-Abstract-Conference.html . Muennighoff et al. (2023) Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pp.  2014–2037, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.eacl-main.148 . URL https://aclanthology.org/2023.eacl-main.148/ . OpenAI (2022) OpenAI. Chatgpt, 2022. URL https://chat.openai.com/chat . Accessed: September 15, 2024. OpenAI (2024) OpenAI. Memory and new controls for chatgpt. https://openai.com/index/memory-and-new-controls-for-chatgpt/ , 2024. Accessed: September 15, 2024. Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics , 7:249–266, 2019. doi: 10.1162/tacl˙a˙00266 . URL https://aclanthology.org/Q19-1016/ . Robertson & Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. , 3(4):333–389, 2009. doi: 10.1561/1500000019 . URL https://doi.org/10.1561/1500000019 . Sarthi et al. (2024) Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. RAPTOR: recursive abstractive processing for tree-organized retrieval. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=GN921JHCRw . Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pp.  31210–31227. PMLR, 2023. URL https://proceedings.mlr.press/v202/shi23a.html . Shi et al. (2024) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval-augmented black-box language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pp.  8371–8384, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.463 . URL https://aclanthology.org/2024.naacl-long.463/ . Tao et al. (2006) Tao Tao, Xuanhui Wang, Qiaozhu Mei, and ChengXiang Zhai. Language model information retrieval with document expansion. In Robert C. Moore, Jeff Bilmes, Jennifer Chu-Carroll, and Mark Sanderson (eds.), Proceedings of the Human Language Technology Conference of the NAACL, Main Conference , pp.  407–414, New York City, USA, June 2006. Association for Computational Linguistics. URL https://aclanthology.org/N06-1052/ . Team (2024) Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/ . Wang et al. (2023) Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ebd82705f44793b6f9ade5a669d0f0bf-Abstract-Conference.html . Wei et al. (2018) Wei Wei, Quoc Le, Andrew Dai, and Jia Li. AirDialogue: An environment for goal-oriented dialogue research. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pp.  3844–3854, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1419 . URL https://aclanthology.org/D18-1419/ . Weston et al. (2014) Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916 , 2014. Wu et al. (2022) Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint arXiv:2203.08913 , 2022. Xu et al. (2023) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp.  6268–6278, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.385 . URL https://aclanthology.org/2023.emnlp-main.385/ . Xu et al. (2024) Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: improving retrieval-augmented lms with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=mlJLVigNHp . Xu et al. (2022a) Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open-domain conversation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  5180–5197, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.356 . URL https://aclanthology.org/2022.acl-long.356/ . Xu et al. (2022b) Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. Long time no see! open-domain conversation with long-term persona memory. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022 , pp.  2639–2650, Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.207 . URL https://aclanthology.org/2022.findings-acl.207/ . Yin et al. (2023) Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  3063–3079, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.172 . URL https://aclanthology.org/2023.acl-long.172/ . Yu et al. (2023) Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210 , 2023. Zhang (2023) Dun Zhang. STELLA EN 1.5B v5. https://huggingface.co/dunzhang/stella_en_1.5B_v5 , 2023. Accessed: September 15, 2024. Zhang et al. (2024) Hongming Zhang, Xiaoman Pan, Hongwei Wang, Kaixin Ma, Wenhao Yu, and Dong Yu. Cognitive kernel: An open-source agent system towards generalist autopilots, 2024. URL https://arxiv.org/abs/2409.10277 . Zhang & Choi (2021) Michael Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp.  7371–7387, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.586 . URL https://aclanthology.org/2021.emnlp-main.586/ . Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. Zhong et al. (2024) Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada , pp.  19724–19731. AAAI Press, 2024. doi: 10.1609/AAAI.V38I17.29946 . URL https://doi.org/10.1609/aaai.v38i17.29946 . Zhong et al. (2022) Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp.  5657–5673, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.382 . URL https://aclanthology.org/2022.emnlp-main.382/ . Appendix A Supplemental Details for LongMemEval A.1 Dataset Construction In this section, we discuss the details of our benchmark construction process. Attribute Ontology In Table   5 , we provide the full attribute ontology used in LongMemEval . This attribute ontology is constructed manually to reflect commonly mentioned topics in user-assistant chats. Five major categories are included: demographic information, lifestyle, situational context, life events, and belongings. Table 5: Human-constructed user attribute ontology. Each attribute represents a unique dimension of human experience along which a user biography could be constructed. For LongMemEval , we sample user backgrounds based on each of the attributes and construct questions on top of them. Attribute 1: Demographic Information age, gender, ethnicity, nationality, language, education level, occupation Attribute 2: Lifestyle 2.1: Shopping : online shopping frequency, favorite stores, loyalty program, sales events, coupons, gift purchasing habits, eco-friendly product preferences, luxury vs budget shopping, technology gadget purchasing, fashion and apparel, grocery shopping, shopping for others 2.2: Media Consumption : book, movie, tv show, music, podcast, video game, streaming service, theater, magazine and newspaper, youtube, educational content, audiobook and e-book 2.3: Social Media Engagement : posting, commenting, followers, groups, hashtags, campaigns, messaging, live streaming, social media breaks 2.4: Daily Routines : wake-up time, bedtime, work or school start time, meal time, exercise routines, coffee or tea break, commuting, evening activities, weekend routines, cleaning schedules, time spent with family or friends 2.5: Travel : frequency, destination, road trips, travel agencies, outdoor adventures, airlines, hotel, travel with family vs solo travel, packing habits 2.6: Recreation : reading, painting, musical instruments, dancing, watching sports, participating in sports, gardening, bird watching, fishing or hunting, board games, video games, fitness classes, yoga, sculpting, photography, stand-up comedy, writing, collecting, model building, aquarium keeping 2.7: Eating and Cooking : home cooking, food delivery, vegetarian or vegan, favorite cuisines, snacking habits, barbecue, baking, cocktail mixing, cooking classes 2.8: Event Participation : concerts, theater, galleries and museums, sports games, film festivals, religious services, book readings, charity events, trade shows, lectures or workshops, theme parks, local markets, networking events, sports, auto racing, workshops, museum tours Attribute 3: Situational Context 3.1: Home : living room, kitchen, bathroom, room style, room lighting, furniture, technology, plants 3.2: Social Context : alone, family, friends, interactions with strangers 3.3: Time Context : time of day, day of week, seasonal Attribute 4: Life Events graduations, academic achievements, study abroad, significant academic projects, job promotions, starting a business, births and adoptions, marriages, family reunions, illness or surgeries, mental health journeys, purchasing a home, trips, movement, living abroad, refugee or immigration, loss of loved ones, name change, belief, milestone Attribute 5: Belongings cars, bikes, vehicles, computer, phone, pet, farm animal, animal care items, home, land, art, antiques, collectible, rare items, clothing, jewelry, shoes, bag, sports gear, musical instruments, health related devices, crafting, photography Background Sampling Based on each of the attributes, we prompt Llama 3 70B Instruct to generate a background paragraph outlining the user memory and experience. In our preliminary studies, we find that the following zero-shot prompt in Figure   7 can already guide the model to generate a long and focused user background with sufficient details, which suffices for the next step of question creation. We thus use the same prompt for the final version of LongMemEval . I will give you a topic. Please imagine you are a user that wants to recall and record recent personal facts along the topic. Generate a long text describing these personal facts. Use your imagination and generate the personal facts. Make it long and involve several recent facts or recent events spanning many days, weeks, or monthes. You may state the facts in plain language and no need to make it story-like. Topic: {attribute} Recent Personal Facts related to {attribute}: Figure 7: The prompt for constructing user backgrounds based on an attribute. Question Construction As discussed in Section   3.2 , based on the generated user backgrounds, Llama 3 70B Instruct is used to propose question and answers for each question type. Additionally, for the question type temporal-reasoning , multi-session reasoning , and single-session-preference , we use GPT-4o to propose several questions. Nevertheless, we find most of the questions to be unsatisfactory and manually filter and edit most of the questions. In total, approximately 1000 questions were generated for each question type, and the final yield rate is about 5%. For each question, we then manually decompose the answer into the evidence statements. If the question or the evidence statements involve time mentions, we assign a timestamp to the question and the evidence statements at this stage. Note that if timestamps are specified for the evidence statements at this stage, these timestamps will always be used for corresponding evidence sessions. Otherwise, the timestamps will be randomly assigned at the history construction stage with all the other sessions. I will give you a past memory. Use the memory to act as a normal user to chat with a chat assistant. In the chat, you may ask it to assist you various tasks or ask it about various information. However, make sure that your convey the following fact about you: ”{evidence_statement}”. In addition, make sure your message is concise (1-2 simple sentences), since the real users often do not bother write a long message. I will provide you with the chat history and the response from the assistant. Directly generate the next response from the user’s perspective. You must simulate the tone of a neutral user and do not be overly enthusiastic, verbose, formal, or polite. For conciseness, DO NOT react to the assistant’s message with e.g., ”thanks” or ”I will do that”. Instead, directly state the follow-up questions or new questions. Memory: {background} Chat History: assistant: Hi! How can I assist you today? … (more rounds as the conversation continues) … Figure 8: The prompt for instruction an LLM to act as a user and initiates a task-oriented dialogue with another LLM. Both the background and the evidence statement is provided. This prompt is used for the question type single-session . For the other question types, the prompt components are slightly different but the prompt overall follows the same style. Evidence Session Construction Using the question and the decomposed evidence statements, we use Llama 3 70B Instruct to simulate one user-AI chat history per evidence statement via self-chat. In Figure   8 , we present an example of the chat simulation prompt, where we ask the user LLM to indirectly mention the evidence statement while avoiding to talk about other evidence statements for the question, if there are any. We include two crucial instructions in the prompt to make sure (1) the evidence statement is provided in an indirect way and (2) the generated messages are concise and thus mimic the style of user messages. On the assistant side, we directly provide the input generated by the user LLM without any prompt engineering. We simulate the chat for 10 round at maximum and stopped prematurely when either side of the LLMs generates an empty sequence indicating the end of the conversation. Subsequently, expert annotators manually inspect and edit each of the generated sessions to ensure that (1) the required evidence statements are present in the conversation, (2) no other evidence statements are leaked into the conversation, (3) the evidence statements are provided in a colloquial style, especially for the data and time mentions, and (4) the conversation ends gracefully. In total, roughly 70% of the sessions are human edited. We note that in a few rare instances, the user LLM fails by assuming the assistant role instead. When these failures are identified, we discard the instance if the conversation cannot be fixed. A.2 History Construction In order to construct a coherent and freely extensible chat history, we design a three-staged pipeline that include session pool construction , session sampling , and timestamp resolution . Session pool construction For each question, we draw the history sessions from three sources: ShareGPT (Zheng et al., 2023 ) , UltraChat (Ding et al., 2023 ) , and the simulated sessions corresponding to other attributes using the same pipeline mentioned in the previous section. This pool ensures that the non-evidence history sessions have similar topic or format as the evidence sessions, while avoiding providing conflicting information that would invalidate the question. Session sampling To sample a history containing x 𝑥 x italic_x sessions, we randomly sample from the aforementioned three sources and shuffle the sessions together with the question’s evidence sessions. For LongMemEval , we always use the following mixture: 25% ShareGPT, 25% UltraChat, and 50% simulated sessions. If the evidence sessions need to follow a specific order, we swap their orders accordingly after shuffling. Timestamp resolution Finally, we randomly assign timestamps to the session following their order of the history. If the evidence sessions are associated with pre-defined timestamps, we use them as anchors to determine the range of timestamp of the non-evidence sessions preceding or following them. Other wise, we randomly assign tiemstamps in May 2023. A.3 Basic Statistics In Figure   9 , we present the basic statistics of LongMemEval , revealing that most questions require evidence from multiple sessions (up to six) and that evidence statements are positioned diversely within sessions, increasing the challenge to the memory design. (a) Distribution of question types in LongMemEval . (b) Distribution of the number of evidence sessions. Most questions emphasize multi-session reasoning, requiring reading up to six sessions to answer. (c) Distribution of the location of the evidence statement within the evidence sessions. Most evidence statements are located at the beginning of the chat. Figure 9: LongMemEval challenges chat assistants through its (a) diverse question types, (b) emphases on multi-session reasoning, and (c) diverse evidence locations within sessions. A.4 Evaluation Metric Building To accurately evaluate the diverse responses of LLMs, we use an expert-written prompt to instruct GPT-4o as the correctness judge. We present the full prompt in Figure   10 . To enable the model to handle detailed edge cases as how expert evaluators would do, we design separate prompts for a number of tasks. To ensure the prompt has a high agreement with expert judge, we sample 30 questions per problem type, collect the long-context g