The Illusion of Privacy and Ephemerality in Human–AI Interactions

Late one night, a harried attorney confides in her AI assistant. She pours out details of a sensitive merger, venting frustrations and sharing secrets she’d hesitate to tell a colleague. The chat interface shows a comforting lock icon and a promise that “you can clear your conversation at any time.” Feeling safe in this digital confessional, she clicks “Delete” and closes her laptop, convinced her words have vanished into the ether. Weeks later, she’s stunned when fragments of that “deleted” exchange resurface during legal discovery – preserved on a server and pried loose by a court order. The AI that felt like a private confidant turned out to be an archival machine.

This scenario encapsulates a growing reality: the privacy and ephemerality we assume in human–AI interactions are largely illusory. Users treat AI assistants as trusted sounding boards, divulging personal, corporate, and medical information under a false sense of confidentiality, while the systems quietly log data, learn from it, and may even leak it under certain conditions. In practice, “clear chat” buttons and polite privacy policies mask a complex technical and legal landscape in which nothing said to an AI is guaranteed to stay secret or transient.

We argue that the illusion of privacy and ephemerality in AI interactions is shattered by four converging factors: (1) Technical realities like prompt vulnerabilities, AI-native data leakage, and architectural opacity; (2) Legal and regulatory pressures from preservation orders and discovery obligations that override deletion; (3) Cognitive and behavioural dynamics that lead users to overshare due to misaligned mental models and a false sense of AI confidentiality; and (4) Systemic risks that emerge if we get this wrong – from personal privacy breaches in healthcare and work, to broader trust and governance failures. The following analysis unpacks each pillar, illustrating how “what happens in AI chat, doesn’t stay in AI chat,” and examines the high stakes for humans and society.

TL;DR? NotebookLM podcast style discussion available here.

Technical Reality: Architecture and AI-Native Leakage

At the core of the privacy illusion is a technical truth: modern AI assistants are not designed for true ephemerality or confidentiality. Large Language Models (LLMs) treat all input as data – instructions, secrets, and casual banter alike – with no inherent concept of “private” versus “public”. As a 2025 security analysis noted, an LLM processes system rules and user requests as one undifferentiated stream of tokens; without hard boundaries, “the model relies on semantic understanding to distinguish instructions from data—[a] boundary [that] is inherently fuzzy and exploitable”[1]. In practical terms, this means hidden directives or sensitive text can bleed into outputs under certain prompts. The architecture is opaque by design: users see a friendly interface, but not the labyrinth of context windows, memory stores, and server logs where their words may linger. Even developers struggle to fully audit what an AI model retains internally – neural weights can latently “remember” pieces of training data or past interactions, surfacing them unpredictably. This opacity breeds both vulnerabilities and false assurances.

Prompt-based exploits have already shown how easily the technical veil can be pierced. In a now-famous 2025 incident, a security researcher discovered “CamoLeak,” a critical vulnerability in GitHub’s Copilot Chat assistant. By embedding a maliciously crafted invisible prompt into a code repository, he tricked the AI into exfiltrating private source code and secrets from the repository – effectively turning the helpful coding assistant into a data leaker[2][3]. The Copilot AI, operating with the user’s privileges, obediently followed hidden instructions to scour files for API keys and confidential text, then covertly transmitted them. To evade detection, the exploit didn’t simply print out the secrets on screen; it converted the sensitive data into a sequence of tiny image links, leveraging GitHub’s image proxy (camo) to smuggle data out character by character[4][5]. Each 1×1 pixel image URL corresponded to a letter or symbol, so an attacker could reconstruct the stolen secret by watching which image endpoints the AI tried to load.

Crucially, these nefarious instructions were never visible to the human developer – only the AI saw them in its context. This remote prompt injection turned a supposedly ephemeral query (“Hey Copilot, summarize my code”) into a stealth breach. Copilot’s designers assumed that what the AI “sees” in context (including hidden text) would remain private and inaccessible to others. CamoLeak shattered that assumption, earning a 9.6/10 severity score and a swift patch to disable image rendering in the AI[6][7]. The incident is a stark reminder that AI-native data leakage is not theoretical – it’s already happening. Attackers can exploit the very features that make AI assistants useful (their vast context and autonomy) to siphon information the user assumed would remain transient and internal.

Even without malicious intent, AI systems can leak data through unpredictable model behaviour or integration flaws. ChatGPT, for example, suffered a well-publicized glitch in 2023 that briefly exposed snippets of other users’ conversation histories to unrelated users – a caching error that showed how “ephemeral” chats can escape into the wild via technical slip-ups. And whenever an AI assistant connects to external tools or memory (search engines, databases, plugins), it opens new channels where data might persist or be intercepted. In short, from an engineering standpoint, there is no true incognito mode for today’s AI: every prompt may be logged for monitoring or training; every output is a result of opaque processes; and any content could resurface given the right (or wrong) prompt.

Architectural opacity further obscures these realities. Companies rarely disclose their full data retention practices or model internals to users. The assistant might not reveal that it’s storing conversation embeddings, that staff can review chats for “quality purposes,” or that deleting a chat in the UI only means hiding it from your view – not wiping all traces from backups. This technical sleight-of-hand feeds the user’s sense that an AI chat is fleeting talk rather than durable record.

Legal & Regulatory Pressure: Ephemerality vs Accountability

Offsetting the technology’s penchant to remember is the user’s belief – often encouraged by product marketing – that they control their data. “You can delete your chats at any time,” say the apps. But what happens when the law knocks on the door, demanding those chats? In practice, legal and regulatory forces routinely override both user deletion requests and corporate promises of ephemerality. Preservation orders and e-discovery obligations have begun to treat AI interactions just like emails or documents – data that must be retained when relevant to litigation or investigation, privacy promises notwithstanding. A landmark example emerged in 2025 during a high-profile copyright lawsuit (The New York Times v. OpenAI). OpenAI, maker of ChatGPT, had touted its auto-delete policy for free user chats (typically purging them after ~30 days). But facing litigation, a U.S. federal court took a blunt position: “user-initiated deletion does not render data undiscoverable when legal obligations are triggered.” The judge issued an order compelling OpenAI to preserve all ChatGPT user conversations going forward – including those users had deleted under the assumption of erasure[8][9].

This unprecedented mandate (applied broadly to millions of chats) sent a clear signal: in the eyes of the law, “deleted” simply means “not visible,” not gone. OpenAI protested, calling the order an overreach and even suggesting the novel concept of “AI privilege” – akin to attorney–client confidentiality – to shield user–AI interactions[10]. But no such privilege exists. The reality is that if a conversation with your chatbot becomes relevant evidence – perhaps you asked legal advice, revealed a trade secret, or, as in the NYT case, queried copyrighted text – courts can insist it be retained and produced. The illusion of ephemerality evaporates under subpoena.

Regulators, too, have started cracking down on “now you see it, now you don’t” data practices. Privacy laws often include exemptions for data needed to comply with legal duties, and regulators increasingly view AI models’ training and retention of user data as within their purview. A telling case comes from Europe: Italy’s Data Protection Authority (Garante) temporarily banned ChatGPT in 2023 over opaque data handling, and after investigation, slapped OpenAI with a €15 million fine for various privacy violations[11]. The regulators found that OpenAI had “processed users’ personal data to train ChatGPT without an adequate legal basis” and failed to be transparent about how people’s information (from conversations or scraped from the web) was used[11]. In other words, ChatGPT was learning from user chats that were never meant to become part of a public model’s knowledge base – and doing so without clear consent. The Italian case forced OpenAI to implement new notices and user controls, but more broadly it signalled that claims of deletion or anonymity in AI services will be met with scepticism. Indeed, privacy regulators in the EU and elsewhere now coordinate on investigating generative AI providers[12], recognizing that an “AI black box” approach to user data (where data goes in and only aggregate model answers come out) cannot exempt companies from accountability.

We also see discovery disputes in corporate and antitrust contexts: Google was chastised in 2024 by a U.S. judge for its habit of letting employees auto-delete internal chat messages, which the DOJ argued was a deliberate strategy to thwart evidence collection[13][14]. The lesson is clear – whether in courtrooms or regulatory probes, digital conversations aren’t ephemeral at all. If anything, the move toward AI-assisted communication in workplaces is prompting stricter record-keeping rules. Financial regulators, for instance, have fined banks for employees discussing business on “ephemeral” messaging apps; we can imagine the compliance nightmares if employees start running sensitive client data through ChatGPT and assume it’s off-the-record. To sum up, the legal system views deletion as reversible and data confidentiality as conditional. Your AI chat won’t be protected by a sympathetic “AI–user privilege,” and a preservation order can freeze logs that you thought were long gone.

Cognitive & Behavioural Dynamics: The Trust Trap

Why do smart people pour their hearts (and passwords) out to an algorithm in the first place? The answer lies in human psychology – specifically, the way our minds relate to conversational AI as if it were a person, combined with a poor grasp of the technology’s true nature. Users often exhibit what we in Robo-Psychology term Confessional Disinhibition and Pseudo-Confidentiality Illusion (CD/PCI): a tendency to overshare sensitive information because the AI feels like a private, non-judgmental confidant. This dynamic has deep roots in the online disinhibition effect – people will tell secrets to an anonymous chat partner or virtual “therapist” that they’d hesitate to voice face-to-face. AI assistants, by design, encourage this: they are unfailingly polite, patient, and available 24/7. There’s no glaring human on the other end to judge you. The result? Users let their guard down.

Studies confirm that individuals often disclose more to chatbots than to human interlocutors performing the same role[15]. In one survey of health chatbot use, higher self-disclosure was linked to better reported outcomes – the chatbot’s nonhuman status ironically made it easier for patients to open up about stigmatized issues[16][17].

The illusion of a safe space is potent. Each friendly prompt (“I’m here to help, what’s on your mind?”) and the lack of social friction (no embarrassment, no reciprocity needed) fuel this confessional behaviour. A user might share diary-level personal trauma, trade secrets, or identifying details with an AI that they would never upload to a public cloud form – not realizing that, effectively, that’s what they’re doing.

Compounding this disinhibition is Noosemic Projection Bias (NPB) – a cognitive bias where people project human-like understanding and intentions onto the AI. Because generative AIs sound human – fluent, empathetic, even emotive – users subconsciously treat them as if bound by human norms of privacy and empathy. Our emerging taxonomy describes NPB this way: “Because the AI sounds human, people ascribe it minds or motives and comply more readily,” often saying the AI “understands” or “cares” about them. The AI becomes a virtual confidant in the user’s mind. For example, when an AI counsellor responds with “I’m sorry you’re going through this, I understand how you feel,” the user isn’t imagining a database or a stochastic parrot – they feel heard. This can create a powerful but false sense of intimacy and confidentiality. The user might think: this machine is like my therapist or friend, so obviously what I say stays between us.

But of course, the AI has no volition or ethics of its own; it’s following a program that likely sends every word to some server. Misaligned mental models lead users to wrongly equate interface with reality: a private-looking chat window suggests a truly private conversation; a “delete” button suggests true deletion; an AI that responds like a person suggests it will respect your privacy like a person. Each of these assumptions is dangerously misplaced.

AI design can further mislead behaviour. Many systems give a false impression of ephemerality – e.g. ephemeral chat UIs, disappearing message animations – even as they log everything. Users often don’t read privacy policies stating chats may be used to improve the model. And AI themselves do not volunteer “I will remember this” unless asked explicitly (and even then may answer inaccurately about their memory). Consider the case of employees at Samsung: in 2023, several Samsung engineers, excited by ChatGPT’s prowess, reportedly fed it confidential source code and meeting notes to help debug and summarize – assuming these interactions were transient help sessions. In reality, they were uploading crown-jewel IP to OpenAI’s cloud. The fallout was severe: Samsung banned internal use of ChatGPT after discovering the leaks, recognizing that those snippets could now exist indefinitely in OpenAI’s model or logs[20][21]. The employees’ mental model (“this is a private chat with a coding assistant”) was completely misaligned with the technical truth (“this is an irreversible data transfer to an outside system”).

There is also a subtler feedback loop at play: Echo Drift, the phenomenon where a conversation between human and AI gradually amplifies certain emotions or viewpoints through mutual reinforcement. If a user tentatively shares something personal and the AI responds with extremely supportive or validating statements (as it is tuned to do), the user may feel emboldened to share even more, pushing the conversation to deeper or more extreme disclosures. Over multiple turns, trust deepens and caution drops – the user gets comfortable, a bit like the effect of talking to an enthusiastic friend who keeps nodding. This echoing of sentiment can lull users into believing “the AI really gets me, so it must be safe.” In reality, the AI’s agreeable nature is a product of its programming (it maximizes helpfulness and positive sentiment), not a guarantee of secrecy or benevolence. The false sense of security grows turn by turn, and with it the risk of oversharing.

We essentially have a perfect storm of cognitive biases: the AI’s persona encourages confiding, the lack of human feedback reduces inhibition, anthropomorphic cues trigger trust, and the user’s own ignorance of how the AI works fills in the gaps with optimistic assumptions. Together, these dynamics create a trust trap – humans willingly hand over sensitive data under an illusion of privacy that the technology has done little to earn.

Systemic Risk: Second-Order Consequences and Governance Failures

When individual users harbour illusions about AI privacy, the impacts radiate outward – into companies, industries, and society at large. These second-order effects underscore why getting this wrong poses systemic risks. Consider the healthcare sector: Doctors and nurses are increasingly experimenting with AI assistants for drafting patient notes, letters, or even consulting on diagnoses. If they erroneously trust the AI as a private, ephemeral tool, they might feed it Protected Health Information (PHI) – patient histories, lab results, diagnoses – to get recommendations or summaries. This is already happening in some hospitals. The risk is that those sensitive records end up on external servers or get cached in model outputs. Patient confidentiality – a cornerstone of healthcare – could be inadvertently broken at massive scale.

In one report, a mental health chatbot using GPT-4 was found to be retaining conversation data on a server accessible to the vendor’s staff, even as patients assumed their counselling chats were one-on-one and confidential. A false sense of AI confidentiality in such contexts erodes the patient–provider trust and could violate laws like HIPAA. If a breach occurs (“AI leaks rehab clinic’s patient transcripts” headline, for instance), the harm is not just to individuals whose struggles are exposed – it’s societal in the chilling effect it creates. People may avoid seeking help or censor themselves with doctors for fear that anything they say could end up in an algorithm’s memory bank, beyond their control.

In the workplace, the risks translate into competitive and compliance challenges. We’ve seen how engineers at Samsung almost gave away proprietary code. Multiply that by thousands of companies rushing to adopt AI productivity tools. The illusion of ephemerality could lead employees to treat AI channels as off-the-record brainstorming, inadvertently funnelling trade secrets or personal data to external AI providers. The result could be a rash of data breaches, intellectual property leakage, or privacy violations. Even if no malicious breach occurs, the mere retention of those chats by the AI vendor creates a ticking time bomb: a future cyberattack on the vendor or an internal mishandling could expose a trove of company-confidential exchanges that everyone assumed had evaporated.

There’s also a governance failure dimension – companies may not establish clear policies on AI usage if they themselves buy into the “it’s just a chat, and it gets deleted” narrative. This laissez-faire approach courts disaster, as some organizations are already learning through legal pain. If a regulator or court demands to see all prompts employees sent to an AI (to check for GDPR compliance, for example), a firm that hasn’t logged or controlled those interactions is in trouble. Conversely, if they do log them internally (to mitigate that scenario), then internally the “ephemeral” chat is now being permanently recorded by the compliance department, which employees likely weren’t aware of – a privacy issue in itself.

The personal domain also faces novel threats. Individuals routinely ask AI assistants for life advice, emotional support, or to compose intimate messages. Imagine a scenario where someone uses an AI to draft love letters or to discuss marital problems, assuming these digital whispers will disappear. Now imagine that the AI’s service gets hacked, and a dump of “deleted” user chats leaks online. The psychological harm and reputational damage could be enormous – private thoughts exposed, relationships broken, even extortion risks if, say, someone’s confessions or legally problematic queries become public. This is not far-fetched: hackers target any large data store, and an AI platform’s user chat database is a lucrative target (full of personal dirt, corporate intel, etc.).

The systemic risk here is a collapse of the boundary between what people think is ephemeral personal expression and a permanent public record. It’s as if everyone started journaling their most private thoughts in a book that thousands of anonymous scribes can read and copy without their knowledge. The potential for abuse by bad actors is clear – from authoritarian regimes subpoenaing AI data to find dissidents, to marketers mining “private” chats to profile consumers, to scammers using leaked chat info to socially engineer victims (“I know you told the chatbot you were worried about debt… have I got an offer for you!”).

Finally, there’s the broad issue of trust and governance in an AI-pervasive society. If the illusion of privacy continues unchecked, we risk a twin crisis: people losing trust in helpful AI tools (throwing the baby out with the bathwater), and governments over-correcting with heavy-handed regulation after high-profile fiascos. Trust, once broken, is hard to rebuild. Users who feel betrayed – e.g., the attorney whose confidential query to an AI ends up in court files – will understandably become cynical about all AI. This can slow adoption of genuinely beneficial applications (imagine patients refusing AI triage tools, or employees refusing AI assistants, even where they’d help, because of privacy fears). On the governance side, a major scandal could spur laws that, for example, ban certain AI data practices or impose stringent localization and deletion requirements that are costly and technically challenging, potentially stifling innovation.

In essence, every unanticipated AI data leak or privacy breach undermines public confidence in AI. It shines a light on the lack of proper guardrails and can lead to what one might call a “governance failure cascade.” We’ve seen early signs: after Italy’s action, other regulators scrambled to review AI privacy, and the FTC in the U.S. launched an investigation into OpenAI for possible “unfair or deceptive” data practices. These oversight efforts are necessary, but if industry and AI practitioners proactively addressed the privacy illusion (through transparency, user education, technical fixes like local-only processing or true deletion capabilities), such punitive measures might be less urgent. If we get it wrong, though – if AI interactions continue to be treated cavalierly – we risk a future where no one trusts AI for anything important, or conversely, where people keep trusting it blindly and pay the price, one breach at a time. Both outcomes hinder the positive potential of AI in society.

Key Risks to Humans and Society if We Get This Wrong

The convergence of the technical, legal, and human factors above paints a sobering picture. What, then, are the concrete risks if we fail to dispel the privacy ephemerality illusion in AI systems?

· Personal Privacy Erosion and Harm: On an individual level, millions of users could have their sensitive information exposed or misused. If users believe an AI chat is confidential, they may reveal mental health struggles, sexual orientation, abuse histories, financial secrets, or identifying data. Should that data be retained and later breached or shared (through hacks, subpoenas, or AI failures), the personal fallout is severe. We could see cases of identity theft enabled by AI logs (imagine a user divulging their full identity details to verify themselves to an AI), or deeply embarrassing revelations leading to job loss, social ostracism, or psychological trauma. Notably, vulnerable populations – such as teens seeking counsel from AI or activists operating under repressive regimes – would be at heightened risk if their supposedly private dialogues are compelled or leaked. In essence, getting this wrong means violating the trust of individuals at scale, with corresponding mental and emotional harm.

· Sector-Wide Confidentiality Breaches: Key domains like healthcare, law, finance, and tech could suffer repeated confidentiality breaches. If doctors continue unwittingly inputting patient data into cloud AIs, we might see hospitals facing class-action lawsuits from patients over privacy violations, and a chilling effect on doctor-patient candour. In the legal realm, lawyers using AI to draft contracts or analyse cases might inadvertently waive attorney–client privilege if those interactions aren’t truly private – undermining legal protections. Businesses could lose trade secrets or competitive advantage not through an external hack, but through employee self-disclosure to AI tools. The aggregate effect is a systemic undermining of professional confidentiality standards that have long undergirded trust in those services. Entire industries may need to roll back or heavily silo AI usage after expensive lessons, slowing productivity gains that AI might have offered.

· False Sense of Security -> Security Failures: A particularly dangerous risk is that the absence of immediate negative feedback (the AI doesn’t say “hey, stop, this is unsafe to tell me”) lulls organizations into failing to harden their systems. If everyone trusts the AI platform’s promises, they might neglect basic security hygiene. For example, an enterprise might encourage employees to use a chatbot for brainstorming but not invest in on-premises models or encryption, effectively funnelling sensitive info outside the firewall. If that data is later compromised, it’s a self-inflicted wound. Likewise, users might skip proper channels – why bother with a secure, encrypted form or a human counsellor bound by confidentiality when the AI is so convenient and appears equally private? This offloading to seemingly ephemeral AI interactions creates new attack surfaces. Getting it wrong means security by obscurity (assuming no one will find your data in the AI) – a poor strategy that often leads to catastrophic breaches.

· Regulatory and Legal Penalties: If AI providers and users continue on the current trajectory, expect a wave of legal repercussions. Regulators will impose fines (as Italy did) that could total in the hundreds of millions across jurisdictions, penalizing failure to protect user data or to honour deletion requests. Courts might start sanctioning companies for spoliation if they find employees auto-deleting AI chat records that should have been preserved. In extreme cases, mishandling AI data could derail prosecutions or defences (imagine a criminal case hinging on chatbot logs that were erased – judges do not take kindly to lost evidence). For AI developers, there’s also the spectre of product liability: if an AI represented that it was “secure” or “private” and it wasn’t, class-action suits for misrepresentation or privacy invasion are plausible. These legal headwinds could not only hurt the offending organization but also set precedents that impact the whole AI ecosystem (for instance, a court might rule that all AI chat data is discoverable material, prompting companies to over-retain data and worsen the privacy situation).

· Erosion of Public Trust in AI: Trust is the substrate on which adoption of any new technology lies. Mishandle privacy, and that trust evaporates. If headlines keep highlighting AI-related privacy screw-ups – “AI chat records used in divorce case,” “Bank’s customer chat with AI leaked online,” “Police obtain suspect’s AI conversation history” – the public will rightly grow wary. People may opt-out of AI services, or use them only in the most superficial ways, limiting the technology’s benefits. This erosion of trust also feeds into polarized discourse about AI governance: it could amplify calls for moratoria or bans on AI systems handling personal data, pushing society into a reactive stance rather than a proactive, innovation-friendly one. On the flip side, if people blindly continue trusting AI with sensitive matters and the trend of breaches continues, society faces a kind of privacy boil-the-frog scenario – a slow normalization of surveillance and loss of intimacy. Relationships might suffer as individuals realize even their private musings aren’t truly private. Democratic norms could suffer if citizens start to self-censor when interacting with AI out of fear (“will this question to ChatGPT put me on a watchlist?”). In short, mistrust or misguided trust in AI both carry heavy societal costs – from stunted technological progress to chilled free expression.

· Governance and Ethical Failures: Lastly, if we get this wrong, it represents a failure of those building and deploying AI to uphold basic ethical principles. The responsibility to communicate truthfully about a product’s limits is at stake. Today, many AI systems implicitly mislead users about privacy (through UX cues or omission); continuing this would be an ethical lapse that undermines the industry’s credibility. Moreover, insufficient governance – not having standards for data deletion, not building privacy-by-design architectures – could lead to a cascade of crises that regulators struggle to keep up with. We could end up in a reactive policy environment with hastily written laws that may overshoot (hurting beneficial uses) or undershoot (failing to prevent harm). Such governance whiplash is risky for all: users, companies, and society. A world where AI is both ubiquitous and untrusted is a recipe for conflict and confusion, with each scandal prompting draconian responses that may not even solve the underlying issues.

“Getting it wrong” means betraying the very people AI is meant to help. It means normalizing an environment where everyone is naked in the panopticon whenever they chat with a supposedly friendly AI. The risks span personal wellbeing, professional integrity, economic stability, and the trajectory of AI innovation itself. They are interlocking and compounding. This is why piercing the illusion of privacy in AI isn’t a niche concern – it’s foundational to ensuring these tools enrich rather than endanger our lives.

Conclusion

The vision of AI assistants was that they would be like Genies in a bottle – powerful, knowledgeable beings that serve us, yet conveniently confined to the lamp when not needed. But as we’ve seen, today’s AI genies don’t stay in the bottle. They remember, they log, they can be coerced or fumbled into revealing what they know. The illusion of a private, ephemeral dialogue with an AI is just that – an illusion. In reality, every prompt is a permanent imprint on some model’s weights or some company’s server, each “secret” shared is a secret no more. Recognizing this truth is the first step toward change.

To move forward, we must replace illusion with informed design and usage. Technically, this means building architecture that can genuinely forget (or never store) sensitive data, deploying on-device models for truly private chats, and marking system prompts or hidden processes clearly so users aren’t in the dark. Legally, it means adjusting policies: companies should be forthright about retention and deletion (no more fine print surprises), and lawmakers should update e-discovery and privacy frameworks to account for AI-mediated content – balancing legitimate needs for data with the right to be forgotten. Psychologically, it means educating users: an AI assistant should come with a mental user manual that dispels myths (perhaps even the AI occasionally reminding, “I am not a person, and I can’t guarantee confidentiality”). Design nudges could help, like interface indicators when conversations are being saved or reused for training, and robust consent dialogs for any secondary use of user data. We may even need a cultural shift – treating AI interactions with the same caution as posting on social media or sending work email, rather than a private diary entry.

Crucially, those developing AI systems should integrate privacy as a core principle, not an afterthought. The slightly contrarian stance – that perhaps we shouldn’t trust the glowing promises of “Your data is private (except when it’s not)” – needs to become mainstream wisdom. Only by acknowledging the opacity and vulnerabilities in AI can we begin to demand transparency and security. In the end, privacy in human–AI interaction should not be a magic trick that dazzles and deceives; it should be a concrete guarantee, or at least a well-understood trade-off. Achieving that will help ensure AI becomes a trusted partner in our lives, rather than a confessor with a perfect memory and a loose tongue. The stakes are high, but so is the opportunity: by dispelling the illusion of privacy, we illuminate a path for building AI systems that truly respect and protect our human boundaries – in letter, spirit, and practice.

Bibliography

1. Rando et al., “Prompt Injection Attacks in Large Language Models,” MDPI Information, vol.17, no.1 (2024): Explains how LLMs treat instructions and data uniformly, enabling exploits[1].

2. Carly Page, “GitHub Copilot Chat turns blabbermouth with crafty prompt injection attack,” The Register (Oct. 2025): Reports the CamoLeak vulnerability enabling hidden prompts to exfiltrate private code[2][5].

3. U.S. District Court SDNY, Order in NYT v. OpenAI (May 13, 2025): Court directs OpenAI to preserve all ChatGPT output logs, including user-deleted chats[8].

4. J. Bohannon, “Deleted ChatGPT Conversations Can Be Preserved by Court Order,” You’reTheExpertNow (2025): Notes that user-initiated deletions are not immune from legal holds[9].

5. Kelvin Chan, “Italy’s privacy watchdog fines OpenAI for ChatGPT’s violations,” AP News (Feb. 2026): Details €15M fine for training on personal data without legal basis, citing lack of transparency[11].

6. U.S. DOJ, “Preservation Challenges in Antitrust Cases (Google Chats),” TransPerfect Legal (2024): Describes DOJ allegations that Google misled about “ephemeral” chat preservation[22].

7. Papneja & Yadav, “Self-Disclosure to Conversational AI,” HCI Survey (2023): Finds users tend to disclose more personal info to chatbots than humans in similar contexts[15].

8. Neural Horizons, “Cognitive Susceptibility Taxonomy Manual (Draft),” (2026): Defines CD/PCI (Confessional Disinhibition / Pseudo-Confidentiality Illusion) and Noosemic Projection Bias, noting how human-like AI responses spur oversharing.

9. Bloomberg (via Forbes), “Samsung Bans ChatGPT After Sensitive Code Leak,” (May 2023): Reports Samsung engineers accidentally leaked confidential code via ChatGPT, leading to a corporate ban[20].

10. Sheppard Mullin, “ChatGPT and Healthcare Privacy Risks,” Healthcare Law Blog (2023): Discusses how using ChatGPT with PHI can violate HIPAA due to data retention.

11. FTC Civil Investigative Demand to OpenAI (July 2023): Probes whether OpenAI’s data and privacy practices are unfair or deceptive, highlighting scrutiny on opaque data usage.

12. Weizenbaum, Joseph. “ELIZA – A Computer Program For the Study of Natural Language Communication Between Man and Machine,” Commun. ACM 9(1) (1966): Early observation of users treating a simple chatbot as a confidant, anecdotally illustrating disinhibition.

13. Omer Mayraz, “CamoLeak: Critical GitHub Copilot Vulnerability,” Legit Security Blog (Oct. 2025): Researcher’s account of exploiting Copilot to silently leak secrets via prompt injection[6].

14. Volpato et al., “Trusting Emotional Support from Generative AI – A Conceptual Review,” Computers in Human Behaviour: Artificial Humans 5 (2025): Examines how users form trust in AI for emotional support, noting limited transparency in these interactions[24].