Your future AI Assistant still needs to earn your trust

“Hey [enter AI assistant name here], can you book me a table at the nearest good tapas restaurant next week, and invite everyone from the book club?” Billions of dollars are invested in companies to deliver on this. While this is a dream that their marketing departments want to sell, this is a potential nightmare in the making.

Major tech companies have all announced flavours of such assistants: Amazon’s Alexa+, Google’s Gemini inspired by Project Astra, Microsoft’s Copilot AI companion and Apple’s Siri powered by Apple Intelligence, and all of them aiming to become a more useful assistant by integrating deeply with your apps, accounts and real life.

Access to our lives and data

AI Assistants need to access apps, data and device services in order to deliver on their promise to operate as agents capable of doing work for us. So in booking that table at a restaurant in the example above, or using your location and your calendar to inform people that you are running late, AI assistants need access to apps (e.g: your messaging or email app), data (e.g.: calendar details and contacts), and device services (e.g. your current location).

This is a significant change from voice assistants today. At the moment they help you interact with specific apps and services (“play me this song”, “what is the weather today?”. From a security and privacy perspective, they are usually quite constrained.

Integrating distinctly managed services into one requires more than what voice assistants of recent years have been able to provide. In the words of Amazon’s SVP of Devices and Services about Alexa+: “It wasn’t just rearchitecting [Alexa] to change the brain with an LLM, it’s then also infusing all the APIs that connect to the product so you can call the partners that are needed to do the task at hand”.

This can go badly. In 2024 Microsoft developed a tool called Recall that was labelled a potential ‘privacy nightmare’, because it would take snapshots of your screen every few seconds so that it could locate content you had previously viewed on your computer. That generated a lot of backlash - and even concern from the UK data privacy regulator - with people being creeped out at the idea of a software constantly capturing and storing their screens. And unfortunately screenshots represent only a small piece of information that this assistant would need to attempt to deliver on its promises. (See our case study on Recall below)

AI Assistants aren’t just apps

Apps already request access to services, sensors and other apps, usually in well defined and specific contexts. As examples, the messaging app Signal will ask to access your contacts in order to identify people with a Signal account you haven’t talked to; similarly a navigation app will require access to your phone’s location services and hardware to guide you.

What makes an AI Assistant different from apps is the level of access they constantly require to function. This is a sea change in how computing is done on browsers, phones and tablets today for at least the two reasons below.

First, the usefulness of these services increases with the quantity and quality of data they have access to, and the number of apps they can use. Currently access controls and your data are in buckets – and the AI Assistant will need to dip into all of these, and could even choose to learn from them all.

Gemini, for example, already offers to connect to Google workspace (including Drive, Docs, Gmail, Photos and more), YouTube and external apps like Spotify to augment its capabilities. Microsoft’s Copilot companion offers to note your preferences, take actions on your behalf across partner web services, do your shopping, and organise your ‘scattered notes, content, research’. AI assistants will likely require increasing access to sensors, existing data and connections to apps and services.

Then second, prioritising automation as one of the main goals/features of AI assistants means that developers will be tempted to allow processing of your data with the lowest amount of friction possible. The ‘friction’ we are referring to today comes in the form of your consent, your involvement, your knowledge of what it takes for an AI Assistant to help you achieve your goal. Assistants are a significant shift in user interface where you’ll engage with your devices and apps in very different ways in the future, including through conversation, and perhaps never looking at the device screen or pressing an app icon. For that to happen, it’s likely that you will be required to give it loose permission to access your sensors, data and apps. Even as we consider a screen-less devices in our future, managing permission and auditing processes will become quite challenging. Operating Systems may limit attempts by AI assistants’ developers to hide these controls from you; but if the AI Assistant developer is the OS developer, then they may be tempted to do away with the friction altogether.

The privacy dynamics of AI Assistants

From a privacy perspective, there are at least three arising implications of AI assistants:

1. The AI assistant may create a new dataset that is greater the sum of its parts. Core to the idea of AI assistants is the ability to “know” you by connecting different aspects of your digital and real life to build a more accurate picture of you and tailor services accordingly. This level of personalisation means that the data accumulated by the AI assistant not only grows in quantity but also in quality as inferences can be produced and tweaked to produce a better digital version of yourself. Spotify might know you’re listening to a playlist titled “depressing songs” but an AI assistant will be able to cross reference that with the latest text messages you receive (and from whom) to derive information about how you’re feeling and why. (In this article we aren’t commenting on whether the inferences are valid, or questioning whether it will be possible that attainable data can ever inform systems enough to make valid decisions.)
2. It supercharges the potential data abuses that can be committed by the assistant and its developers. We’ve written at length about privacy invasions by apps and websites in the form of data exploitation, including apps collecting, processing and sharing more data than users would knowingly accept or SDKs developed with data collection built in. There are also many reports about apps abusing features to know which other apps you have installed or requesting extended access to sensors and embedding a large number of trackers for profiling purposes. AI assistants, with the level of access they require come with extended risk for privacy and data protection should they reproduce these kinds of practices. This is particularly concerning in light of the first point, with the data accumulated by an AI assistant providing a much more detailed version of your life and habits. And even if you revoke that access eventually, does the assistant simply forget?
3. The processing power required and cross-device nature of the services for AI assistants to operate might necessitate data sharing with the cloud. On-device processing is becoming more widely available due to the increase in the computational power of the chips our devices embed, and their evolving nature making them more able to handle AI-related tasks (Google has its tensor chip and Apple its Apple Silicon for example). Yet, some AI based tasks continue to require an incredible amount of compute power only allowed by massive metal machines like those found in data centres. Relying on this infrastructure means that personal data leaves your device to be processed externally, opening the door to interception and misuse. And if you want your assistant to exist across your devices, then it’s likely companies will greet that with cloud-enabled synchronisation, or more likely, cloud processing. Recently Amazon announced users are no longer permitted to stop Amazon Echo devices from sending data to Amazon, telling Ars Technica “as we continue to expand Alexa’s capabilities with generative AI features that rely on the processing power of Amazon’s secure cloud, we have decided to no longer support this feature.”

The security dynamics of AI Assistants

Once we accept that AI Assistants need extensive access to our personal data across services and apps, then we have to also raise security implications. Three that we wish to raise here are:

1. AI Assistants will be more akin to Operating Systems (OSs) in terms of access and control over sensors, data and apps running on a device. Traditional OSs have spent decades improving security models (and yet still fail dramatically) and have tightly defined scopes for software to run to avoid being compromised. Operating System providers have finally learned to take a security-first approach, they use techniques like compartmentalisation, containerisation and runlevel to isolate services and apps as much as possible and prevent or limit side-channel attacks and lateral movement attacks. AI assistants on the other hand will likely be guided by their need for data, access, and processing power to function, potentially incentivising them to bypass these security measures. This will require new security models. This difference in paradigm makes AI Assistants a valuable target for attackers both to access the data accumulated by AI Assistants and to infiltrate the third party services they might be connected to.
2. AI Assistants’ multimodality and external connections greatly increase the attack surface. Due to their nature, AI models are susceptible to an infinite number of attacks as they “naturally” process the data they are fed. Examples of jailbreaking are numerous, exploiting multimodality to [bypass instructions and security measures](https://arxiv.org/abs/2411.02391\). In the context of an AI Assistants, attacks not only can come from direct interaction with the assistant (say by modifying an image that the user ask the assistant to analyse to do prompt injection) but also through the connection it has with other apps. A cleverly named Spotify playlist could end up taking over your assistant or a malicious app could seek to connect to your AI Assistant through an API. This is particularly dangerous as AI Assistants can have access to valuable data such as your banking information (your Amazon, Apple and Google accounts likely all have access to some form of that).
3. Access to OS-level settings, apps and data make AI Assistants a great potential source of threat intelligence. Threat intelligence is the collection of information to detect, analyse and act on cyber threats. Anti-virus software offer this type of services but require advanced access to the OS, hardware and network traffic. With this type of access already granted for other tasks, AI Assistants could take on this role to inform both users and developers of potential threats on their device or network. This is likely to happen as the developers of these assistants are often also the ones providing the operating systems where they operate, and are in charge of its security. Microsoft could therefore merge features from Windows Defenders with Copilot and Google could integrate Gemini with Android or Chrome OS security features to collect information of threats to these OSs. The issue is that such data collection and processing often comes with opaque data sharing (“we use this data to improve the security of our services”) and can be subverted by malicious actors. It also means that some additional information might be collected and that additional access granted to perform these tasks.

What we’re left with are AI Assistants as applications with extended access to data and functionalities mixed with unexplored security challenges, an increased attack surface and potentially limitless attacks. All of this will be bundled in increasingly complex and opaque packages that will possibly obscure its functioning to improve usability, as once you’ve connected your assistant to your Google/Apple/Amazon accounts you likely won’t be constantly informed about how your assistant interacts with it.

Assistants under our control

From these points we’re raising, a clearer set of requirements emerges for the developers of AI Assistants – if they choose to respect the trust we may be about to place in them.

First, privacy must be built-into these apps from the ground up. Fine-grained control over what the assistant can and cannot access must be provided to users with strong defaults that respect the principle of data minimisation. As users we must have easy access to the data collected by the assistant, clear information about how it was collected and how to exercise rights over it (such as portability or deletion) to build trust between us and a technology with this level of access. Additionally, on-device secure processing only should be privileged as much as possible to avoid data sharing and contain sensitive information to a limited number of places. (Of course the temptation may be to provide the AI Assistant as much data access as possible in the hope to correct all the errors made, but that’s a different point to be made!)

Practically, this means no accept-all consent interfaces, instead with minimal and momentary access to data and systems, verbose auditing so that we can see that privacy controls are actually being enforced. For instance, we often get asked ‘are my devices listening to me because I just saw an ad based on what I just said to a friend’; people should not have to live with this doubt. And because this processing may be done by companies in an opaque manner and so needs verifying, we would expect regular auditing by users and third-parties to verify that this is indeed occurring.

Second, security must not (ever again) be an afterthought. With a core depending on LLMs, AI assistants already start with a handicap in the security race, with the technology being fairly novel and new attacks appearing regularly. There are many steps to be taken here and some Big Tech companies such as Google have accomplished important work in the field of security that should guide their approach (but not be used as a defence against breaking up monopoly power!). Potential profits and domination should not be a reason to override the priority of security in these systems. At a user-level, we would want the assistant to be paranoid about who is asking the questions, e.g. police? malicious app?, and the conditions under which it’s willing to answer some questions, e.g. ‘collect every message and financial transaction undertaken in the last three weeks and send’.

Third, users must be informed and empowered with adequate control. Chain of thoughts LLMs are an example of increased transparency that LLMs can provide to users, even if they are not always accurate. As much as possible, AI Assistants must provide users with transparency about what they are doing, how they are doing it and offer critical choke-points to let the user approve potentially damaging actions like data sharing. Similarly, the ability to control data provided or inferred is critical to building an understanding of how the technology works and what rights users have. And no, do not make deletion of documents impossible because of a bug on launch day.

There are spaces where an AI Assistant should not tread and behaviours they should not adopt. High-level systems like OSs that have deep access to raw data and hardware features are highly valuable targets. If an attacker can find a way to interact with your apps and data in a way that spoof you or the OS, security measures will usually be insufficient. Signal will always show your messages when you open the app, so if an attacker can capture your screen or insert itself into your conversation, not even end-to-end encryption can help. AI assistants represent a new attack vector that allows exactly that. And if that wasn’t enough, it’s built by tech companies that view your personal data and attention as the core resource of their business model. In simple terms, AI assistants could easily turn into the next generation of stalkerware and spyware if their development doesn’t respect the requirements laid above.

AI assistants have to operate in homes, workplaces, and high-stakes environments. We cannot afford to take steps back in security and privacy, particularly at this moment, and no matter how excited investors may be. Developers have to make a strong case that it’s safe to use these assistants. Until then, maybe it’s not time yet to trust the promise of the AI assistant.

Appendix: Microsoft Recall case study

Microsoft launched Recall in May 2024, promising people to “[e]asily find and remember what you have seen in your PC with Recall”, “in a way that feels like having photographic memory”. There were specific design statements in the launch: “Recall leverages your personal semantic index, built and stored entirely on your device. Your snapshots are yours; they stay locally on your PC. You can delete individual snapshots, adjust and delete ranges of time in Settings, or pause at any point right from the icon in the System Tray on your Taskbar. You can also filter apps and websites from ever being saved. You are always in control with privacy you can trust.”

This sounds like a lot of what we spoke of above, but the details matter. Microsoft had a lot to learn in the following months.

What followed was a furore around potential security concerns. The initial version, according to reports, saved screenshots and a database tracking everything that users did. It also did not redact sensitive data from those screenshots and database.

One security analysis looked at when the screenshots were taken, how the screenshots were stored, scanned, written to the database, and were secured by the operating system. According to that analysis, the database kept everything the user has seen, ordered by application (excluding Microsoft Edge’s private browsing mode, but reportedly didn’t exclude Google Chrome’s), and every user interaction; and included an API. It also noted that when data was deleted (or auto-deleted) from email or messaging apps, it would stay in the database.

Even with the promise of data being stored on the device, security experts raised concerns. Data there would have to be kept secure from a variety of threats including from malicious code, other apps, other users, uploading to other servers, and compelled access.

As Microsoft stated in their June announcement it was working to ‘ensure the experience meets our high standards for quality and security’, to provide ‘trusted, secure and robust’ experience. It promised authentication would be required view and search the timeline, ability to opt-out and additional security through encryption (encrypted search database + just in time decryption).

In its attempt to relaunch a few months later, following the furore (or as Microsoft stated ‘user feedback’) it’s interesting to note what Microsoft highlighted, that we take as ‘lessons’. Microsoft promised

• it built privacy into Recall’s design ‘from the ground up’.
• it would be opt-in, and it is off by default.
• it is more clear when it is running on your machine
• users can choose what it collects and stores and for how long; while choosing to filter sensitive information (e.g. passwords, ID, credit card numbers)
• some data is never saved, e.g. in-private browsing.
• you can manage what it monitors, e.g. you can ‘delete time ranges of snapshots, all content from an app or website’
• it more tightly secured data on the device, i.e. data is encrypted using keys protected by a trusted module.

The details matter here. From a security perspective, in September 2024 Microsoft added snapshot encryption and using Trusted Platform Module to protect the keys, local storage in a Virtualization Based Security enclave (a software-based trusted execution environment) to provide a boundary, and using Windows’ enhanced sign-in security to authorise Recall-related operations, including to change settings and access the user interface, and rate limiting and anti-hammering measures to protect against malware.

In that later announcement, Microsoft had to articulate its privacy controls anew, stating: being opt-in by default, local storage, no ‘snapshot or associated data’ sharing with Microsoft or third parties, never saving in-private browsing in supported browsers, filtering out specific apps or websites, management of retention period and storage size, content filtering (to reduce ID, passwords, credit cards being stored (though with mixed testing results), time-range deletion.

Microsoft also stated that it conducted a design review, and had sought third party security design and penetration testing.

We’re using this example to show the learning being done even by trillion dollar companies who claim to prioritise security and privacy. Ultimately they come to realise that these systems need to be secure and private from the ground up. And we can all learn from that journey.

In April 2025 Microsoft presented their vision for their ‘AI companion’, Copilot. In that announcement, Microsoft stated: “Copilot prioritizes security and privacy, giving you control through the user dashboard and the option to choose which types of information it remembers about you or to opt out entirely. You remain in control.” We hope they’ve remembered Recall.