Agent Memory
From a timeout to a managed commitment. How evidence becomes durable, revisable guidance.
Ben Labaschin - Applied AI Conference 2026 (GDG Chapel Hill), July 18, 2026
Runtime 29:55

Machine-generated transcript, lightly corrected for names and technical terms.
Timestamps mark the start of each paragraph. [...] marks a spot where the
recogniser dropped words at a segment boundary.

[00:00] and it looks great. Okay, well, hi everyone. My name is Ben Labaschin. [...] to be speaking with you today about Agent Memory. I think this is going to be a pretty casual conversation here. I might even get through my slides. [...] than I'd like, but... [...] feel free to leave comments as I'm and I'll try to have the conference. [...] so I can see anything that you might bring up. But it's a pleasure to be able to speak with you today about Agent Memory. So I am actually writing a book on Agent Memory. I'll talk a little bit about that, but the conversation that we're going to have here, it's all content from the book.

[01:01] a little bit about me. I'm head of AI and engineering at Workhelix. We focus on measuring productivity gains from the usage of AI at enterprise companies. [...] engineer there and I built the platform for it and a lot of my experience comes from those integrations with those kind of Fortune 50 companies and how I've seen them use Agentic AI. From that I'm writing this book, Agent Memory for O'Reilly. Early release should be out in the next few weeks, so the next few chapters, the first few chapters of the book will be released to everyone here. And so the content of this conversation primarily from those first few chapters. So I'm just going to kind of go through those in the book. It's going to be a lot more detailed, but here it's just going to be sort of high level. So-- no other questions than really what I'm going to do and I'm just keep scanning just to see if there are any questions. I'm going to just get started.

[02:04] start with a problem. Okay. And the problem is that we see this Python function. It's standard Python function. You say, okay, we were making some sort of submit purchase request for a company. The company is called Very Important Company because it's very important. And we are trying to work with our vendor. So we've got HTTPS, buythings.importantvendor.com/purchases. We're sending a payload. [...] we're setting a timeout for two seconds and we're saying, hey, we want to make sure that we've made the right purchase for the vendor. [...] a few symptoms, it returns a timeout. The function itself is deceptively simple. This is a very straightforward Python function. And we get no error logs.

[02:57] pause here. I forgot to mention that we've got a mascot through the conversation and that's my dog Atlas who is also behind me right now. Maybe you can see him in the corner. So he'll be in the corner of the slides. That's who's in the corner with us. He's going to be along for the ride as we have this conversation. Okay. So now you know who that dog is. Let's talk about this function. So something's off. Hand it to me. [...] and we tell the agent, "Hey, something's wrong here. It's returning a timeout. Go fix it for us." So it goes and it tries to fix it for us and this is what it returns to us This is the kind of thing that any of us might do in our working lives. There's a problem, it's a PR, it's an error, whatever.

[03:52] paste or it's already in Claude Code and we just say go fix it. So it gives us this for attempt in range three, try return post makes the post request with the payload. [...] if attempt = = to raise otherwise try again. [...] gives us a retry. It says, try three times. Maybe it was just like a time out thing. Do it again. [...] out. [...] work. Try it. We submit it. The problem is that... [...] go home, you come back to work the next day and you're told by your colleague, okay, well now the vendor is reaching out to us and saying that they're getting multiple purchase orders.

[04:43] And it turns out that even though the timeout was being returned, the purchase order was actually going through. So your agent, by saying repeat three times, actually just caused you to make purchases for your vendor three times in a row. That's a problem. [...] turns. [...] problem into many problems. The setup timeout didn't mean failure. The request was accepted. The response was lost. You made blind retries and then you got duplicate charges. [...] the agent did. That's a problem. Cool. [...] happen. It happened because the agent didn't have enough context to provide you with the right solution, the correct solution. So imagine now, if you will, two different agents. One is the one we just had. That's the blank slate agent. I just call that the blank slate agent because it doesn't have any context. And you might imagine for a book called Agent Memory, I'm going to pitch to you the idea that perhaps there's be a memory capable agent.

[05:56] difference between the models. Blank slate agent might give you a retry after time up because it's got no context or it might give you something a little bit fancier, but it, you know, for this case, this is what's happening. It gives you a fresh key after each attempt, no reconciliation for the state, and duplicate charges possible. With a memory capable agent, what it might do grab information and then use that to solve the problem. We've seen that a lot if you use Claude or OpenAI. There's a lot of agents out there that have integrated memory. But we're going to talk more about where they can be better and where agents may be going today. Um.

[06:42] I talk more about that, though. I want to talk a little bit about the difference between context and memory, right? So there is a difference, I argue, between context and memory for agents. What are they? Well, here's a helpful little graphic that I started building out between context and memory. So on the left, you have got the present working material. [...] is what is available to the agent basically and memory is something much more intentional. I will be referring to this throughout the entire presentation as an investment. Memory as an intentional investment in the future. Just like my background's in economics, so I think about things in an economic sense. Just like in real life, where an investment is you're placing money into some sort of money generating scheme, stock, whatever, intentionally with the goal that one day you will pull it out and there will be more. That is what memory is. It is intentional. It carries things like preferences, prior incidents, constraints, et cetera.

[07:48] context is available to the model right now, memory is deliberately retained. Okay, so now we can go back to the incident, right? Actually, pieces of intentionally left memory that the original blank slate agent because it didn't have the ability to retain information. [...] it didn't pull in. And those are vendor docs, an incident with RCAs, a run book, and an engineer correction. All of those were available to the blank slate agent, but it didn't pull those because it didn't have the infrastructure for it. Whereas in our mythological, idealized agent memory harness, we're able to pull these types of memories in. [...] end.

[08:34] them as evidence, right? So someone at some point, I want to make this point clear, went out and decided, "Hey, I am going to retain vendor docs, I'm going to retain the RCA, etc. for the use that someday an agent or some person can use them for." get access to this information? Well, that's where a sort of chapter two starts to walk you through, right? Let's say someone writes a bunch of documents and they're in GitHub or they're in Google Drive or they're in a blob storage, an S3, wherever they might be in an organization. The problem is a lot of these agents that we have today, Claude, Codex, etc., they're using information that we have access to.

[09:28] generated. And oftentimes they'll automatically generate those memories for you, unless you tell them, "Retain this memory." Right now the technology is such that they're actually able to infer valuable memory. But if you're building an organizational agent memory, which I argue is much more powerful, then you're going to need something more, right? So here's an example figure of what that might look like. You've got, let's say, GitHub, Google Drive, and Asana. There are five stages that need to be gone through to start maintaining and retaining that information. That's a mapping for your agent, the connection for your agent, normalization, We're going to go through those very quickly at a high level. The book's going to go in a lot more detail than this.

[10:13] talking about what all this is. But basically, the first thing you want to think about The mapping right what even exists for the agent to see right so it The agent doesn't know that there is a GitHub or an Asana or a Google Drive that it should reach out to and seek. How is it going to go seek it there? It might look around, but if it's not clear, it might miss it or it might look in the wrong place. There's a lot of reasons why a mapping makes a lot of sense. [...] same of the previous figure it's just closer in, right? So let's say that there's a mapping and it says GitHub, Asana, and Google Drive.

[10:58] Then you have to make the connection, right? How do you connect? Well, I go through that in the book and we're not going to go through that here, but basically you need to be able to connect to your sources. [...] is even if you can connect to your sources and there are sources now and says, ah, okay, I have access to Google Drive. That doesn't mean... But every record that it looks at is useful. And how is it useful? How do you know whether it's useful? That's where normalization and curation come in, right? Because normalizing says, hey, some of these records might be Google Drive. Some of these records might be GitHub. Some of these records might be Asana. [...] pull them in when a system pulls them in it might be spending valuable tokens in context just trying to sort out all of the information it's seeing and if you have a very large organization that's going to take a lot more time a lot more context it's costly it's time-intensive it's inefficient it's not how a retrieval system work, especially for our agents.

[12:07] The next step is normalizing. And the idea of normalizing is let's place all of this information in a common schema. You might call it a memory record such that even if I'm looking at Asana or I'm looking at Google drive, et cetera, they're. [...] in a way where the agent knows this is where I should look, this is how I should know whether or not this record is useful. [...] to curation. Once you've normalized them into a common schema, you want to curate them. And curating them means making sure that the data is information rich and it belongs in the store that's being thought about. [...] is routing, right, which is let's get the most important information to the top of the system. So we've normalized it, we've curated and we've decided, ah, this information must be the most reliable, therefore it should go at the top and less reliable information should go to the bottom. All of this more detail of the book. I am really parsing through this or scanning through this because we don't have a lot of time, but That's kind of how you might think about it just for accessing information. There's a lot more to it.

[13:21] And so in doing so... Here your agent finds here a quote from from one of your engineers about the timeout problem that you saw The purchase calls must reuse the purchase intents idempotency key Bye. [...] interesting. That is useful context about our problem. The question is... Would you save this? Would you save this piece of information? Because part- and the thing that I like to insist upon is that not every piece of information should be kept. Let's say you did keep all that information. Can you imagine in an organization like Google, Apple, Facebook, or Meta now, all of the information that would proliferate on their systems? And let's say you did do that, and that was fine because we've got unlimited storage.

[14:19] Now the agent has to parse through that information. It's not useful, right? If we're talking about an organizational agent memory system, it's not useful to always save every piece of information. So the question is, would you save this? Would you use it now? Would you propose to save it? Or would you automatically commit it to working durable memory? I'm not telling you what to do. I'm making the point right now that-- should be made as agent goes through the system. And ideally, the agent is the one, not you, that is automatically figuring this out. And so part of the reason for this book is those systems such that the agents can do this for you, how they're doing it at OpenAI, how they're doing it in Anthropic, what a principled memory system is.

[15:13] How do you? [...] use. [...] which memories to save and which not, right? Well, first we have to understand what kinds of memories there are out there. So let's talk about three useful ways of breaking down thinking about memories, right? In psychology, they talk about different kinds of memory and we've actually just adopted that into agent memory because there's a lot of similarities between how we use memory in psychology and how we use it here. So there's semantic, episodic, and procedural. Let's go through them one by one. Semantic is basically a fact, right? So for instance, what we just said, what the agent, sorry, what the engineer just said about stable key reuse is required.

[16:03] So that's something you would call it a semantic memory if you chose to keep it. Another way of forming or articulating memory is episodic. That's an event. That's what happened before. In that case, it could be an engineer corrected the retry assumption. [...] the same quote that we had from before. Now articulated in two different ways, a stable key reuse is required or an engineer corrected the retry assumption. Both would be accurate. Then there's procedural. That's normative. That's how should you, right? How should an agent act? [...] you should reconcile an ambiguous purchase before retrying. That is an instruction, right? So three different ways of articulating the same memory.

[16:55] Sometimes they're all three, sometimes they're two, sometimes they're one. It just depends on how you categorize it. But part of the way of working through agent memories and memories in general is. [...] and having the agent articulate what kind of memory it is and therefore its value and where it should be stored. [...] so Another way of talking about memories instead of an investment is a forecast. It's just another way of saying that you are anticipating future use. So it's not just, you know, I'll get value in the future. It's that you are anticipating the use of a memory in the future. And there are different ways you can do this, but you can articulate them as memory candidates that-- attention decision and then who owns them, right? These are sort of three different ways of talking about durable memory.

[17:50] So in the case of the purchase calls must reuse the purchase intents and idempotency key, we can say, OK, that's a memory candidate. [...] give a retention decision articulation to our agent to autonomously and automatically decide, hey, given the schema, does the memory candidate to be chosen? And then if so, where is it going and who will maintain it? Again, I am gliding over a lot of the details, but I want you to think about memories as being very intensive. There's a process to it. Every new insert into a chat that you're having with Claude could be a memory candidate.

[18:36] aside, should I retain it? And then once it has been retained, the question is, is it owned? Should it be revisable? Should it be versioned? All of these things are things we have to think about. So in that vein... [...] about memories as actually having a life cycle, right? You encounter a memory, it is a candidate, you can choose to materialize and keep it, and then at some point you can reconsider its usage. You can evaluate it and adapt the memory, compress it, all of these are part of the life cycle. One I would say about this is that there are kind of three relationships. It just so happens that there are a lot of three categories here to consider when you're thinking about memories. And that is the memories relationship to the world, the memories relationship to a store where it's stored, and the memories relationship to few.

[19:36] work. So let's talk about that. Oops, I'm getting lots of alerts because I guess it's almost 12. Um relationship to the world. So a deployed behavior or organizational reality change, maybe there's a reorg or something. So a memory may become stale or false. Maybe we've changed the API's of where we vendor requests, right? So the memory that we retained is now-- old or defunct or it needs to be changed. The memory to store or the memory's relationship to the store is important because there might be corroborating evidence. Like before when I showed you that there were four different pieces, there are RCAs, there were vendor documents.

[20:29] there was engineering documentation, maybe a memory is related to other evidence that bolsters it, right? Or maybe there's duplicative evidence and then you've got a problem because you've got duplicative memory. Or even worse, there are memories that contradict each other. These are all possibilities. So what do you do? You need to have a lifecycle management system in the background for your agent memory to be able to handle these types of things. Of course, there's the memory's relationship to future work. Will it be useful in the future? How will it be useful? What's the reasoning behind its usefulness? Where should it be stored? Etc. So now Let's look at this again.

[21:16] You go back to very important company and something changes. [...] same event should not produce the same operation on every memory. Okay, so you might need to consolidate the fact demote it or supersede it. So something happened at very important company. And the fact that we originally retained has to be adjusted. This is part of that life cycle that I'm talking about. [...] call those consolidations, demotions, or being able to supersede the memory based on different events. All of these need to be built into your agent memory system in order to-- maintain an organizational agent memory operation at a professional company.

[22:11] Now, you might just say, well, you could put a time to live on it, right? You could put a storage clock. But that's not really one that's not very... [...] very refined to approach the problem. But it's not actually answering the right question. So Time to Live says, hey, let this live for a week and then the memory should just be degraded or something. How long should a memory last? Should it last forever? Not very long. It's not exactly a very refined question. A better way of thinking about it, I think, what I assert is a retention horizon. It's answering a life cycle question. What condition should reopen a decision to consolidate, demote, or super- a memory. That's the thing that I'm considering and what I think agents should and what agents do consider in refined memory system. So you want to give them a decision trigger to reopen a commitment to a certain memory. And this is happening in the background as memories are being processed and added and changed, et cetera. Having event triggers, basically, let's say, OK, given this memory-- with this kind of metadata, if this happens, go and return to it. That's why it's really important to have those standard schemas, because you have a standard schema with a lot of metadata associated with the memory. If a trigger happens, you just search the schemas and say, "Oh, all these schemas with these metadata, time for a reconsideration." And maybe you leave them as they are, or maybe you change them based on the context. That's what's happening.

[23:56] in the background. [...] asking me if I will share my presentation. Happy to do so. [...] thing is might choose to put scores on them, right? Important scores. I understand the urge to do so, right? An important score is something like, hey, given these features, let's say between zero and one, you know, if it's a 0.8, that means anything above a 0.8 is an important score that's awesome. So let's retain it. 0.8, you know, don't retain it. But it's the action that is triggered needs more than that. There needs to be constraints, right? Is an action allowed?

[24:43] So these are things like authority validity obligations What is the current evidence of something that is happening? What's changed? That's the relationship between the world, the store and the future work that I talked about before. And what's the smallest action that is justified relating to the memory? Is it demotion, consolidation? Thank you. [...] proper action here. An important score doesn't give a lot of memory. So... [...] make sure we have all of this information considered as we're thinking about changing a memory. A quick note about by the way. Some people might say, OK, you just delete memories. Probably not a great idea. And a lot of the systems that are out there like Zep and Mem0 and Letta, et cetera.

[25:37] You know, there's deletion, but the thing to note here is you're not actually deleting the file, right? You're deleting a relationship. You might remove, you know, a vector or something in a directional graph or something like that. But we're not actually deleting memories very often because we want to keep the information. We just... [...] don't want, you know, we might not want it in our memory system. So other ways of forgetting a memory, let's say that we've decided, hey, this memory is not super useful is we can consolidate it, right? So what we can do is we can just say, hey, these are two memories, let's consolidate them together. [...] demote it, right? So the metadata attached to the memory in our routing system all the way from the beginning, which says, hey, this memory is more important, this memory is less important, we might make it less important, right? Or we might say it's less authoritative, right? So you can give your agent a schema that says, these are more important, these are less important. And another thing you can do is just supersede it and say, hey, look at this memory, actually look at this memory, right? So you can think of it as a hierarchy. So this memory is more important. So just go look at that now.

[26:55] My kind of favorite part of this subject matter is compression. Compression is really interesting. Now-- The problem is compression is very... [...] difficult to do well, because the idea of like, let's say summarizing text. Language is not, they're not numbers, right? We make them into numbers, we make them into embeddings to be able to, for our models to interpret them. But a subtle shift between two words can change the meaning drastically, right? So you might want to be careful when you're using compression to change a memory if something has happened in your organizational system, right?

[27:42] That's all I'll really say about that here. One last thing I'll say because I know I'm running out of time here. This is really important when you have an organizational memory system that you're building out. [...] want to prove that remembering actually changed an action? [...] doing so well is actually incredibly... There are a lot of papers on this, but basically if like Claude or Codex or something pulls the memory into the system. [...] you're running into a causal problem and the causal problem is if agent didn't bring in a memory, would it have done the same thing or something similar? It's very hard to do so in systems that are so stochastic to determine this. So you have to isolate different factors and run experiments offline and online. And then from there be able to choose the be able to say, okay, our memory system is actually helping. There's a lot more about this in the book, but I just want to note here that proving this out, you can't just assume that because it pulled in a memory, the memory is the thing that fully improved it without...

[29:02] like an A/B test at the very least like we did in the beginning with a blank slate and then I'm the same agent but with a different harness that had the information. So let's just end this here because I know the presentation is kind of winding down. Next time, our timeout is going to end differently, right? We have the same intent, same key. We're going to reconcile, and it's going to be one charge, all because we pulled the right memory using our agent memory system. [...] this moved very, very fast. There's appendices and all of these other things that are part of my book that I would love to promote, et cetera. But I'm running out of time. So all I'll say is, you know, Check me out, you can check out O'Reilly, Agent Memory, where this book is coming from. If you have any questions, please let me know. And Atlas and I, thank you very much. Right, Atlas?

[29:52] Okay.