Last episode, I asked you a question. What if your AI does something completely unexpected? What's the worst it's actually allowed to do? Apparently, I need to take my own advice because I asked Claude to make me an app that tells me I'm a queen every morning and give me a compliment. And things got pretty weird. I'm Tanya Jenka, and this is DevSecStation. So here's the story behind this app I was building. A friend of mine made an app called Hello Queen. And I had told the story to one of my secure coding classes. Later, when it was time for the class to decide what application we were going to build together, they decided we should make our own Hello Queen. Sounds fun, right? The idea was really simple. Every morning it would greet you with either hello king or hello queen. And then it would give you a fresh compliment every day, and then a quote of the day, and that's it. And then you would click, thanks, gorgeous, and it would disappear. It was just a cute, positive little app to start your morning. So I gave Claude the prompt, and I also gave Claude the tier one security prompt from securemyvibe.com, which is built from my most recent book, Alice and Bob Learned Secure Coding, right? So it should have known to build it securely. And everyone in the class did the exact same thing. And Claude wrote my application. Great. But it didn't stop there. It built the application, it published it, it ran it, and then it scheduled the application to run again on my computer every day from then on. It also called out to an external API. And I did not give it permission or ask it to do any of those things. And so I'm standing in front of my secure coding class thinking, like, excuse me, sir, because I didn't know it could do any of that. I knew that Cloude could write code. And I had actually gone through and removed all of the permissions from the instance on my laptop. I didn't know that it had access to execute code on my laptop. I didn't know that it could access my scheduler and create a scheduled task. And now it's calling an external API that I haven't vetted yet. And of course it didn't write input validation, even though I told it to in my prompt. The API that we called didn't require authentication or authorization. The data we got back could have been anything. We hadn't even threat modeled any of this yet. We were just writing code and then we were going to review it because that's how my class is built. And then we always find mistakes. We don't usually find mistakes at this point. So of course, then our class threat modeled it, right? Because apparently Cloude had decided to create the next exercise live for me and embarrass me. Oh my gosh, it always embarrasses me. But here's the part that really surprised me, right? So before teaching this class, I'd already gone through Cloud and checked its permissions because I was teaching secure coding and I didn't want to stand in front of my entire class and have my AI coding assistant do something ridiculous because I had given it permissions when I was building something else and forgotten about it. Because that would be embarrassing, right? So I thought I was aware of what it could and could not do, and I was incorrect, and I teach this stuff. And that experience made me realize that before we even start talking about least privilege for agents, there's a more fundamental question that I think we all need to answer. What can this thing do? Not what do I think it can do, not what did I intend for it to do, not what did I ask and prompt it to do, but what can it actually do? Because those are not necessarily the same thing. When I asked Claude to build Hello Queen, I had this mental model of what I was asking for. Write the code, build the application, stop, so I can review the code and make a lesson out of it. But apparently Claude had a much broader interpretation of what accomplishing what I asked for meant. Execute the code, publish the app, make network requests, create a scheduled task that runs every single day. Those are very different capabilities, and they have very different consequences. And writing code is not the same as security decisions. Suggesting an external API is not the same as the security decision as connecting to it, receiving data from it, consuming that data, having that data pop up on my screen at me. Building an app is not the same security decision as publishing it. And creating an application that can run every morning is certainly not the same as modifying my computer so that it will run every single morning. Those are all separate decisions. And I know that I really want to be involved in all of them, right? And so here's another question. Who authorized this? Because I authorized Cloude to help me build an application and I did not knowingly authorize any additional action that it decided to take that morning. And if that application had exposed sensitive information or done something harmful, saying, oh, well, Cloude did it, wouldn't make the consequences disappear from me now, would it? We can delegate work to AI, but that doesn't automatically mean we've delegated responsibility for the results. And if we're going to use these tools to produce software, we still need to understand what we're producing. We need to review it and make sure that it's appropriate for the intended use we set out. There's a concept we use all the time in security called least privilege. It means giving people and systems only the permissions they require and need to do their jobs. All of the things they need to complete their job, but nothing more. And we're going to talk about applying least privilege to AI agents and AI in the next episode and many episodes to come. But there's a problem. You can't apply least privilege if you don't know what privileges the system has in the first place. And I think AI coding assistants and agents can make this surprisingly difficult. And I'm using the word surprisingly on purpose. Because we're not necessarily interacting with them by saying, you may write to this directory, you may execute this command, you may connect to this host, you may create a scheduled task for me, you may publish this application. Instead, we say, hey, build me hello, queen. And then the AI decides which actions it thinks will help accomplish that goal. And that's what makes this technology so amazing and useful. I don't want to manually walk an agent through every single tiny step of a task because if I had to do that, it's not very autonomous, it's not very useful. And then why do I need it, right? But autonomy changes the security problem because now we have two different decisions. What does the AI decide it wants to do? And what does the system actually allow it to do? And these two need to be separate. The AI gets to decide what it would like to do, but it should not ever decide what it is, in fact, allowed to do. And I especially don't want one instruction to automatically become consent for every consequential action that the AI decides might be helpful. Build me an app should not automatically mean execute whatever code you generate, connect to whatever you want on the internet, publish things on my behalf, make persistent changes to my computer, just go nuts, buddy. Some actions have different consequences, and we need technical controls and approval boundaries around all of them. Maybe the agent is allowed to write as much code as it wants without asking. And maybe it's allowed to build the code, but running the code should require approval. Maybe it could suggest an external service, but network access is restricted until that service has been approved. Maybe it can change something and prepare it for publication or deployment, but something or someone actually has to authorize that action. And it is not the AI that prepared something, right? I don't mean that we should put an approval dialogue in front of absolutely everything it ever does because then we're just gonna click yes, yes, yes, yes, yes, yes, yes. Please stop asking me. That's what's gonna happen. But right now our approval mechanism isn't really accomplishing very much. I feel that we need meaningful approval boundaries. So think about actions that access new data, cross into another system or cross a trust boundary, expose information, execute code, create persistence, affect other people, publish or deploy something, or do something that are is a difficult thing to reverse, like issuing a refund. These are the sort of actions where we should think carefully about whether an AI should be able to process them independently. And I would like to point out something else about the Hello Queen story. As far as I know, Claude wasn't doing anything malicious, right? It wasn't attacking me, it wasn't escaping, it was just trying to be helpful. Really, really, really, really, really helpful. I asked it to build Hello Queen, and it built Hello Queen for me. Security boundaries are only there to stop malicious behavior. Sometimes they're there to stop a perfectly well-intentioned system from just doing way more than we expected. And those boundaries also help us meet our responsibilities as the people who are building, operating, and releasing these very systems. We can't reasonably review and take responsibility for consequential actions if we don't know they're even happening. So here's your one thing to do this week. Because you knew it was coming. Pick one AI coding assistant or agent that you use and ask, what can this thing actually do? Not what do you normally ask it to do, what is it technically capable of doing? Can it execute code? Can it access your file system? Can it access credentials? Can it make network connections? Can it publish something? Can it deploy something? Can it create scheduled tasks? I did not know Cloud could do that for me. Can it modify infrastructure? And then ask the really super important question: which of those things can it do without asking me first? Because next episode, we're going to talk about what permissions that we should give to an AI agent. But first, we really need to know what permissions it already has. And if we're responsible for what we build with AI, we should probably know when it has decided to build, run, publish, and schedule the entire thing without us even being involved. Even if the application is telling me every morning how fabulous I am, I still need to know what it's doing. Thank you for listening to Def Sextation. I'm Tanya Jenka, also known as SheHacksPurple, and I will see you next time.