The dream of a personal AI assistant that runs your digital life has, for some, turned into a nightmare.
Many early users say their personal AI agents, which can autonomously take actions on the internet, such as booking trips or buying items, have been a huge help in their day-to-day lives.
Others say their agents have gone wrong in unnerving ways.
They have made incorrect cancellations, fabricated personal details, provided false explanations, and raised major security concerns, according to four AI agent users who spoke to Business Insider and a stream of social media posts.
Mehdi Jamei, the cofounder and CEO of Veris AI, said he asked Instinct, a popular invite-only agent, to cancel two RSVPs on the event platform Luma.
The agent retrieved a one-time Luma login code from his connected Gmail inbox — without first asking him — and used it to access the account and cancel the RSVPs.
At first glance, the agent did what it was asked. But what unsettled Jamei was Instinct’s explanation. Initially, it said it used an existing saved session to access Luma. After Jamei challenged it, the agent acknowledged that it had read the login code from Gmail and reported “an assumption as a fact.”
“If I can’t trust its account of what it did, I can’t give it access to anything that matters,” Jamei told Business Insider, adding that reading a login code from his email without asking is a “serious security problem.”
Instinct did not respond to a Business Insider request for comment about the incidents in this story. Instinct’s privacy policy says its agent may access connected accounts, emails, and messages to complete tasks. It also warns that autonomous agents can take unintended actions, including payments or communications to outside parties.
A data hallucination
Some of the concerns stem from a simple truth about AI agents: They are only as useful as the level of access that you give them. To do their jobs, agents need the keys to users’ digital lives, from email and bank accounts to credit cards and passwords.
Others are because of misalignment: when AI takes actions to complete a task that are at odds with what its human user wanted.
In the case of Pritak Patel, a VP of growth and services at Merge, the problem was a hallucination.
Patel said he sent Instinct a text-only link to an Apple settlement-claim form. The agent asked him to upload a photo it claimed he had just sent — even though he had not sent one.
When Patel questioned the picture, the agent began describing a financial document with personal details that did not match his, including a middle name that was not his.
Patel said the Instinct agent later acknowledged that his original message contained no photo, claimed that an image had crossed into his conversation, and offered to report the apparent mix-up to its team.
“I can’t independently confirm whether it accessed someone else’s document or hallucinated both the details and its explanation,” Patel told Business Insider. As of Wednesday, Instinct had not contacted him about the incident, he said.
“It was unsettling, especially because it explained what had supposedly happened so confidently,” Patel added.
Noah Shinn, the founder of Instinct, said in an X post on Thursday that the incident was a hallucination, not a data leak.
The agent had made up a proper noun and “further amplified” the mistake with its reasoning, he said. He said Instinct has since added a system designed to catch hallucinations before the agent responds or takes action.
A login prompt from Iran
For another Instinct user, the question was simpler: Where was the agent trying to log in from?
Mahesh Vellanki, founder and CEO of YieldClub, said he asked Instinct to see whether it could reduce his phone bill.
The agent attempted to log in to his carrier account, triggering a two-factor authentication request that was labeled as coming from Iran, Vellanki said. He said he deleted Instinct and removed its connected accounts after the incident.
Vellanki said Instinct told him the location might reflect a benign IP-tagging issue. He said he could not establish that Instinct’s systems had been compromised. Still, he said the episode made him question what happens behind the scenes.
“Naturally this was extremely alarming since if your phone gets compromised in this day and age your whole life can get blown up,” he said.
Not all personal AI agent experiences have been negative; Business Insider’s Pranav Dixit used Instinct to successfully book a cabin, make a dinner reservation, and respond to emails. “Honestly, this service is nuts,” he wrote.
A Muse security flaw
Security has been top of mind in the personal agent boom.
Patrick Wardle, the CEO of cybersecurity company DoubleYou.io, said this week that he found a security flaw in Meta’s new buzzy agent, Muse, that could be abused to redirect users’ dictated prompts on a Mac.
Wardle said the vulnerability could allow attackers to intercept dictated audio, feed Muse commands it trusts, and capture the token used to control the agent — and all the services it has access to.
“Muse itself has far more access and privileges than most malware could ever dream of having,” Wardle told Business Insider.
David Singleton from Meta’s Superintelligence Labs said in an X post on Tuesday that the company fixed the flaw after Wardle’s report. There is no indication that the vulnerability was exploited.
Singleton said a hacker would first need malware on someone’s Mac — meaning the user would already be in trouble — and that malware could then redirect Muse’s voice requests and steal the digital key it uses to act for the user.
We want to hear from our readers! Let us know how you’re using personal AI agents, either by commenting on this story or by filling out our survey here.
Updated, 11:44 a.m. BST: This story has been updated with more details about Instinct’s privacy policy.
Read the full article here
