Are AI Assistants built for us or to exploit us? and other questions for the AI industry

Layla looks at her calendar on her phone. She’s in charge of planning her book club’s monthly meeting. After thinking for a second, she summons her AI assistant: “Hey Assistant, can you book me a table at that tapas restaurant I read about last week, and invite everyone from the book club? The restaurant should be in my browsing history. Let me know if the journey is more than 1-hour for anyone”. As the assistant compiles a response, she wonders if anyone else will hear this.

Technology has made certain things incredibly convenient and practical, whether it’s communicating instantly with people across the globe, archiving easily digital versions of our memories, or ordering quickly goods and services. Yet, as we float towards this friction-free world, huge companies have become embedded in our lives. The goal is to have us continuously relying on their services, software, devices and infrastructure.

AI Assistants represent a novel way in which the tech industry is seeking to embed itself into our lives, in deeper ways than ever. We have written about the underlying technology in more detail and in this piece we explore what the AI-assisted future will likely look like and the critical questions we all need the AI Industry to answer before they’re done building them. And before people get locked into their orbits.

What data is used and why?

As companies like Amazon, Google, Microsoft and Apple race to deliver a product capable of handling requests like Layla’s, everyone deserves to know what this will mean for their privacy: what data will these companies have access to, what is the data actually being used for, and what risks lurk around the corner?

Consider Layla’s situation in today’s context: her ability to control her own data would likely be compromised from the very beginning. For instance, if Layla was using Amazon’s Alexa+, the recording of her spoken voice would certainly have ended up in Amazon’s hands, as the company announced that it now processes all requests on its servers rather than locally on users’ devices. Does that mean that a team of Amazon employees would actually listen to Layla’s recordings? It may sound extreme but Amazon previously had employed workers to listen to conversations with Alexa . This does not bring peace of mind, on the contrary, it lays bare the lack of clarity regarding what and how data is processed and retained by the company for purposes other than answering specific user requests.

Let’s not forget that the industry building AI Assistants has already made billions of dollars honing the targeted advertising business model. They built their empires by drawing our attention, collecting our data, inferring our interests, and selling access to us.

AI Assistants supercharge this problem. First because they access and process incredibly intimate information, and second because the computing power they require to handle certain tasks is likely too immense for a personal device. This means that very personal data, including data about other people that exists on your phone, might leave your device to be processed on their servers. This opens the door to reuse and misuse. If you want your Assistant to work seemlessly for you across all your devices, then it’s also likely companies will solve that issue by offering cloud-enabled synchronisation, or more likely, cloud processing.

Once data has left your device, it’s incredibly hard to get companies to be clear about where it ends up and what it will be used for. The companies may use your data to train their systems, and could allow their staff and ‘trusted service providers’ to access your data for reasons like to improve model performance. It’s unlikely what you had all of this in mind when you asked your Assistant a simple question.

This is why it’s so important that we demand that our data be processed on our devices as much as possible, and used only for limited and specific purposes we are aware of, and have consented to. Companies must be provide clear and continuous information about where queries are processed (locally or in the cloud) and what data has been shared for that to happen, and what will happen to that data next.

Here are some questions you can ask AI firms:

• Is data leaving my device?
• Is my data used for other purposes than answering my requests?
• How is my data protected locally and on servers?

What control do we have over our Assistants?

Layla wakes up a bit groggy. Last night, she had an argument with a family member and it still bothers her. As she prepares to start her day with a cup of tea, she asks her AI Assistant for events in her calendar. After going through her schedule, the assistant asks her how she is feeling after yesterday’s heated discussion, and offers to order her favourite food for lunch to make her feel better. Layla is surprised and slightly angry, as she doesn’t remember allowing the microphone to be always on. Ignoring the question, she orders the food as she knows it will bring her comfort.

To achieve Silicon Valley’s dream of a helpful AI Assistant, developers have to succeed on two fronts. First, they will want to enable the Assistant to access as many sensors, services, apps and data as possible, since their plane of usefulness will increase with the amount of data they have access to. Second, they must lower the amount of friction to get anything done, so that users can get what they want simply and efficiently.

Your AI Assistant will seek access to an incredibly detailed picture of your life, with diverse types of data flowing from different sensors, apps and services into the Assistant itself. And after being in possession of that precious data the Assistant will additionally be in a position to infer more and novel data about you, allowing for personalised recommendations.

In Layla’s case, this led to a problematic situation. An entire private conversation with someone that was likely not consented to be recorded ended up being processed by her AI Assistant nonetheless. Navigating app settings and options to allow or deny access to certain data is already not straightforward, as that would hinder tech companies’ ability to profile users and leverage that data in other contexts. Due to the level of access they require, AI Assistants should be particularly transparent and clear about what data they need access to, in which occasions and for what purpose. Even then we must be able to continuously challenge and tweak these settings with ease, and be able to verify that our preferences are respected.

These companies had many opportunities over the years to learn how to implement meaningful consent, provide feedback to users about access to sensors, and offer better interfaces to control data sharing. AI Assistants will be a test of fire to see if they are actually capable of proposing a service that respect our privacy and our interests.

AI firms must affirm that you can assert your control by answering:

• How do I have granular control over access to sensors, data and apps?
• How can I easily access settings to retract consent?
• Where is the clear information on what data is used to respond to a query?
• How can I access and delete any data accessed and used by the Assistant?

How secure is it?

Watching a video on her laptop, Layla notices that her AI Assistant is asking her to confirm a purchase for an item she’s not interested in, on a site she’s never heard of. She quickly cancels the purchase and wonders what happened.

Large Language Models (LLMs) powering AI chatbots have already demonstrated their vulnerability to a wide range of cyberattacks, from jailbreaking to prompt injection, to data poisoning. The impact of these attacks can be limited to the context they operate in (say a conversation or a web browsing session) and mostly focus on getting the models to circumvent security measures and instructions. But when models are deployed in more critical environments, security becomes paramount to ensuring the technology doesn’t turn into a trojan horse.

With this technology at their core and a wide range of access to apps and data, AI Assistants may be an incredibly attractive target to attackers. Any time you allow your assistant to access a given app (say your banking app and emails) and to take certain action, it increases the value of the Assistant for a malicious actor, while broadening the attack surface e.g. someone sharing a maliciously named music playlist to access your emails.

In Layla’s case, adversarial attacks targeting speaker recognition technology have existed for a long time, allowing an attacker to hide a message (say a command to make a purchase on a given site) in an audio file that our ears will not notice. Similarly, visual attacks using hidden information in images can be leveraged to make an LLM take an action that’s different from what it should do when analysing the image.

With AI Assistants potentially accessing so much of our lives and managing so much critical data about us, security must be paramount to their design, with extensive testing, rapid patch deployment and tight security controls.

Finally, wherever the data resides, is it protected from others seeking to access it? We’ve seen a rise in concern regarding device searches by authorities at the border for instance; some AI implementations we’ve seen require recording all our transactions in real life. Are these complete records of our interactions actually secure?

What we expect AI Assistants’ developers to answer:

• Has the system been subjected to sufficient security testing?
• What controls are in place to prevent critical actions (such as payment) to be made without my clear approval?
• How are security vulnerabilities handled and how quickly are fixes deployed?
• How is the data secured from anyone else having access?

How is the industry profiting from this?

After registering for a new boxing class, Layla’s AI Assistant informs her that she has to buy boxing equipment for her first class on Monday. The Assistant offers a few choices that fits in her budget this month and can be delivered on time. Layla wonders for a moment whether those are truly good products or just the advertised ones. She also tries to remember if she allowed her assistant to access her monthly budget. As time is running out, and as the suggested equipment’s reviews look positive, she makes the purchase anyway.

Most of the major AI developers at this time also happen to be Big Tech companies that likely dominate digital markets, from search to social media to marketplaces. Our take is that their dominant position in those markets has allowed them to accumulate resources critical for the development and deployment of AI, namely compute power, data and labour. Two of these companies make their money through advertising (Google and Meta) and one through market placement (Amazon), so it’s fair to say that advertising will likely be key to the business model of these assistants.

How will this ad-driven model impact our agency? Search engines results, social media recommendations and other algorithmic feeds have demonstrated their ability to use our data (our social graph, our search queries, our history of purchases…) to study us, understand us, and get us to take desired actions, or, with dark patterns, to stop us taking actions they deem undesirable. AI Assistants will have far more data to work with, making them extremely potent agents to push targeted advertising and promoted recommendations. As users we should be able to know if the content presented to us by assistants has been manipulated to suit their interests, whether through a corporate partnerships, recommender systems, or paid promotions. We should also be able to access alternatives easily.

There are other ways that the propagation of AI Assistants developed by Big Tech companies will likely fuel their dominance over digital markets. Integration with other services has always been a key deciding factor for users, as demonstrated by Apple’s lock-in strategy, for example with iMessage. Big Tech almost all develop the Operating System (iOS, Windows, Android, ChromeOS, FireOS…) where their Assistant is available, giving them a privileged position to easily integrate with existing services and the OS itself. By running the Operating System, they can decide who else has access to your data across your apps. It’s possible that these operating systems will prevent assistants from accessing other apps for security and privacy reasons; while only permitting deep and broad priviledged access to their own Assistant.

To ensure fairness in those markets and user choice, an equivalent level of access must be offered to other AI Assistant developers so that competition can exist. Laws like the European Union’s Digital Markets Act provide a basis for this, potentially forcing gatekeepers to open up their services.

Nonetheless, at PI we’re still concerned with some of that. That’s because we think the question shouldn’t be ‘do all firms have the same access to our data and devices?’; but rather we should be asking ‘should any company have privileged access to our data and devices? and if so are they protecting us from abuse?’.

AI firms should build Assistants that put you in control. You can ask AI firms:

• Where are the settings to disable personalised recommendations?
• Can I easily understand why a recommendation was made?
• Can we all audit how the AI Assistant is making money from users?
• Is privileged access to my data and devices protected from abuse?

Conclusion: Our Assistants, Under Our Control

Have you ever had a conversation with someone and then shortly after seen an ad about what you discussed, and then wondered if your device was secretly listening to you and sharing that with advertisers and platforms? We shouldn’t even have to doubt this fact: we are supposed to be in control of our devices.

The AI Assistant of the AI Industry’s dreams will be the most knowledgeable and powerful application in your life. And it learns. It could have access to details you’ve forgotten or deleted, or wish you had. It becomes a data store of its own.

Can this be protected from access by the AI firm that runs the Assistant? Can it be protected from data brokers seeking further access to your life? Can it be protected from malicious attackers who have access to your device? What about family members, or colleagues? Employers, insurance firms, or financial institutions? Government? Governments?

Ultimately, people are being asked to blindly trust these AI Assistants and the firms that operate them. We are being asked to trust these companies with more and more of our daily lives. They may make assurances now but are quick to pivot for new political realities, as we’ve seen the shifts to national security and away from climate protection. With a mere software update or a change of policy, they can alter what data is processed, how and where.

To trust these firms and their products, we need assurances locked down in code and policy, with visibility over all operations, and the ability to audit how these companies process our data. Put simply, we need to be clearly in control over our data, our devices, and our assistants.

Will the AI industry’s marketing departments allow this? Will its business model permit it? Will Governments allow it? The answer must be yes, or not at all. Your future assistants need to be private and secure under your control, otherwise, whose assistant is it really?