The cybersecurity risks associated with AI/ML tools are slowly but surely being documented and communicated. People are learning about these risks and beginning to consider them before they use (typically Generative) AI tools. We still have a long way to go, but at least it's a start.
People are learning that inputting sensitive content, such as intellectual property or personal information, into third party hosted AI tools (eg: ChatGPT or CoPilot) carries the risk of leakage of that information because you are also sharing it with the AI tool vendor. They are also starting to grasp just how frequently ML models produce incorrect output or mis-classify input data.
Now, given all the mainstream AI hype, vendors are rushing to integrate AI into anything they can cram it into, even beer! This (frequently unnecessary) trend is creating a new cybersecurity problem (on top of exacerbating increasing environmental problems). It is now far less obvious that you might be using AI and not even know it. I'm going to call this threat: "Invisible AI"
Invisible AI - Case Study: Video Call
Picture this: Two people are having a private video conference using an online service (Zoom, Meet, Slack, Teams, etc... this is not about any specific service) The purpose of this call is to discuss some highly sensitive information. Perhaps a HR or Legal matter. Perhaps about some new technology involving intellectual property that they intend to patent.
Now, one of the parties enters the sensitive content into the chat window, intended only for the other person on the call to see. Here is where Invisible AI may come into play. It is possible (and lately, quite likely) that the tool has integrated AI in the background, capturing everything in that chat. Did anyone actually read the entire 20 page EULA that describes exactly how that captured information will be used? I doubt it. The presence of this Invisible AI now means that the chat carries the exact same risks as something more obvious like inputting the content into ChatGPT, Except in this case, it is completely invisible to the user.
Taking this example a step further, let's assume the individuals are more cautious and choose not to put anything into the chat. They feel it is safer to only to discuss the topic verbally; hoping this will ensure the call is actually private. Meanwhile, unbeknownst to them, the videoconference tool also has AI busy in the background transcribing the entire conversation and making even their spoken words susceptible to those same risks.
Invisible AI - Case Study: Browser Extension(s)
You are collaborating on a sensitive document in your web browser. You're using an online service (eg Microsoft O365, Google Docs, etc... the service is not relevant). The service itself may have AI running in the background (probably) that is consuming every word you type. Let's assume the service has been vetted and therefore you're confident that this is not a risk (I wish I could share your optimism if so!)
The problem is that you also have a bunch of browser extensions installed, some of them have been lingering for a while, perhaps they have changed hands a few times and are not even owned by the original developer any more. The are just getting silent updates in the background. It's important to understand that browser extensions have complete access to everything on the web pages you view in your browser. It again is possible (even likely) that one or more of these extensions are leveraging AI tools in the background (possibly for nefarious purposes) and you wouldn't even know.
Taking this example also a step further, let's assume you're very careful about what extensions you install in your browser, you audit them regularly, or don't even have any at all. The question is, are you sure that's the case for every single one of the other collaborators? It only takes one of them to have one such extension and again, it has access to everything in that document the moment they open it to view or edit.
Invisible AI - Case Study: Your Car
If you haven't already read up on the abominable state of privacy from major auto manufacturers you really should. (Do it now and come back when you've caught up and had a drink to recover... ready... ok let's crack on)
So, you're sitting in your car with a passenger, having a private discussion... well probably not all that private since the car could easily be listening to everything you say. Oh and I really hope you didn't connect your phone to that car, because then it's slurping up even more of your data. Text to speech for texting while you drive? They have all your messages too in that case. All this data sent back to the manufacturer (and whomever they choose to share/sell it to) for processing by Invisible AI.
The step further for this case is similar to the others, no matter what precautions you may choose to take, if you're having a conversation with someone else who hasn't, their exposed systems will happily slurp up your side of the conversation just as readily.
Where should you expect to find Invisible AI?
Herein lies the real problem. With the proliferation of AI, it is quickly becoming impossible to know where Invisible AI may be lurking. It can be anywhere in the chain so that even the vendor you deal with directly, who may not be using it or capturing data for training, may be unaware that one of their service providers is. The video conference example above is currently almost ubiquitous. Most systems have added to their terms of service that your content will be used for AI training and processing. Some of the only exceptions are enterprise licenses (the pay for privacy model). The same licensing and terms of service changes are true for the online collaboration suites and the majority of social media type platforms. Except that with social media, you can't even pay for privacy.
It's more than just information exposure
Feeling ok because you're in the "I have nothing to hide" camp? think again. Keep in mind that AI/ML includes far more than the much-hyped generative AI tools. Information disclosure is only one of the potential risks. As we begin to see AI integrations that perform tasks like classification, transcription, and/or inference many other factors must also be taken into account. Invisible AI risks in these cases can lead to other serious issues even when no sensitive information is involved. Case in point, the example documented by the Morris II paper (explainer for the paper here). Researchers show an AI email assistant (a software interpreter that processes un-sanitized inputs) was weaponized into a malware delivering worm that is triggered, without user action, simply by sending an innocent looking image in an email. Potential here: your computer now has ransomware because you received an email, and your computer also resent that (seemingly from you) to all your contacts.
What can individuals do about Invisible AI?
Thankfully many of the standard approaches for basic cybersecurity hygiene will help.
- Think before you store or share sensitive information
- Remove apps and extensions you don't need
- Review and configure appropriate privacy options
- Disable features you don't require (turn off transcription, if possible, any time you don't need it)
- Read the terms of service and/or EULA (search for a few key terms like "user data" and "AI") - Get cheeky: paste the EULA into an AI and ask it to summarize the document for you or identify specific privacy concerns.
- Keep in mind what browser extensions have access to - consider editing sensitive documents offline
- Prefer tools with end-to-end, zero knowledge encryption.
- Use only those tools approved by your employer when accessing work information (if your employer didn't check into these tools then at least it's not your fault)
Finally, taking into account the environmental impacts of needless AI assistants and integrations: start to question vendors about Invisible AI. Eventually this hype will die down and hopefully more sensible minds will prevail. At least if we collectively begin to question (and not use) ill-conceived AI integrations. Companies will begin to think twice before putting AI into all the things or giving it agency to do things for which it is incapable or the risk of failure is too high.