Book a demo

    The Second Phone Call Test: Does Your AI Actually Remember?

    JD Camden6 min read

    The real test of whether an AI system has memory is not the first interaction. It is the second one. Any system can sound competent on a customer's first call. The second contact is where you find out whether the AI retrieved what happened before or is starting from zero. That gap exists because most AI still separates "generate a good answer" from "remember this specific person," and research on both catastrophic forgetting (McCloskey & Cohen, 1989) and retrieval architecture (Lewis et al., 2020) shows those are genuinely separate engineering problems, not one problem solved by a bigger model.

    Why does the second call expose what the first call hides?

    A first call has no history to fail against. The AI answers the question in front of it. A well-tuned model will sound fluent regardless of whether it has any durable memory underneath. The second call is different. The customer brings context ("I called about this last month") that only a system with persistent, retrievable memory can use. Stanford and Google's research on generative agents demonstrated this directly: agents that logged and retrieved past observations behaved with continuity across simulated days, while agents without that retrieval loop treated each moment independently (Park et al., 2023). The second phone call is the commercial version of that same experiment, run on real customers instead of simulated agents.

    Isn't this just a CRM problem, not an AI problem?

    This is the accepted belief worth naming: memory is solved once you have a CRM record, and the AI layer just needs to read from it. That is incomplete. A CRM stores data. It does not decide what is relevant at the moment of a live conversation. Retrieval research shows that surfacing the right prior information, ranked correctly, is a distinct problem from simply having the information on file. A 2025 Frontiers in Psychology study found that poor retrieval ranking degraded an agent's use of its own memory store even when the underlying data was accurate (Hong & He, 2025). A business can have a perfectly populated CRM and still fail the second phone call test. Nothing forces the AI to retrieve the right record, weigh it correctly, and use it in real time.

    What's the counterargument — does memory always help?

    The honest counterargument is that more memory can make an AI worse, not better, if retrieval is undisciplined. An agent that pulls up every past interaction indiscriminately risks surfacing outdated or irrelevant context. It might mention a canceled order, a resolved complaint, or a past agent's mistake at the wrong moment. Hong and He's 2025 findings support this: memory volume and retrieval quality are not the same variable, and optimizing only for "store everything" can actively hurt response quality (Hong & He, 2025). The answer is not less memory. It is disciplined retrieval: relevance ranking, recency weighting, and a clear boundary on what gets surfaced versus what stays archived. NIST's AI Risk Management Framework similarly emphasizes that AI systems should be governed for reliability and validity in how they use data, not just for how much data they hold (NIST AI RMF 1.0, 2023).

    What's the operational consequence of failing this test?

    For call- and appointment-driven businesses, failing the second phone call test shows up as a hard operational cost. Customers repeat themselves. Staff re-diagnose issues already solved. AI systems sound competent but treat loyal, repeat customers like strangers. This failure compounds. Every failed second call is a data point telling the customer their history does not matter to you. The fix is not a scripting patch. It is building the retrieval discipline the research points to: a persistent record per customer, a ranking method that surfaces what is relevant to this call, and a strict boundary on what gets pulled forward versus set aside.

    How should a business apply this?

    Run the test yourself before you evaluate any AI vendor's memory claims. Have the same customer contact the system twice, days apart. Watch what the agent does with the second contact. Compare vendors on our AI platform comparison page using that exact scenario, or see how persistent memory is built into a working system on our platform overview. For the underlying research on why forgetting happens by default, see why AI forgets between sessions.

    Keep reading: persistent memory as advantage, AI call handling.