"I think my phone is listening to me"... What should CTOs do when AI knows what you never told it? "I think my phone is listening to me"... What should CTOs do when AI knows what you never told it? "I think my phone is listening to me"... What should CTOs do when AI knows what you never told it?

In a recent Stanford study, a user asks an AI assistant for heart-friendly dinner ideas. The model recognises a pattern and classifies the user as ‘health-vulnerable’. That inference then flows into optimisation systems, ad tech and analytics pipelines. Eventually, it can even reach insurers or financial assessors.

No sensitive data was disclosed, yet sensitive knowledge was created. The problem is, modern AI can create knowledge that users never intended to share. Existing privacy laws were built around what people knowingly disclose, yet AI is defined by what it can infer, and the gap between these two realities is where trust often breaks down.

Privacy incidents are already rising. Stanford's 2025 AI index reported a 56% year-on-year increase in AI-related privacy incidents, with 233 documented cases in 2024. Over the same period, public trust in AI companies declined because of systems failure.

If privacy can no longer be perfectly guaranteed, trust needs to be deliberately engineered into every stage of the AI lifecycle. Where should organisations start?

For decades, the commercial privacy contract was relatively simple. Organisations collected data, users consented (often imperfectly, but knowingly), and the risk came from misuse, breaches or over-collection. However, that contract is now broken.

Inference doesn’t happen as a side effect of advanced models, it’s the product itself. Modern AI systems take fragments of behavioural data, including search queries, mobility traces, device metadata and language patterns, and transform them into conclusions users never intended to share, such as health risks, political alignments, financial stress and emotional states.

Personality traits, demographics, socioeconomic indicators and even loan repayment likelihood can be predicted from mobile phone signals alone - with no clinical data, no income disclosure, and no explicit consent.

This creates an ‘inference economy’, where value is no longer generated from first-order data alone, but from second- and third-order conclusions drawn from it. A postal code becomes a proxy for income, typing cadence becomes a signal of stress, a shopping habit becomes a health prediction.

The International AI Safety Report suggests that advanced pattern recognition enables AI to infer sensitive personal attributes from seemingly unrelated data points, making privacy violations both easier to scale and harder to see. The worrying part is that the damage may not emerge until years later.

CTOs now need to recalibrate, as regulators scramble to catch up. The EU AI Act, for instance, asks an uncomfortable question: if a model can reliably infer personal attributes, can it ever be considered anonymous?

Stanford HAI distinguishes between two types of AI privacy risk: on the input side (training data) and the output side (inference and model behaviour). The issue is that most companies govern the former, and almost none govern the latter.

The techniques we rely on were designed for a different era. Anonymisation assumed sparse data and expansive cross-referencing. But neither assumption holds anymore; with high-dimensional behavioural data, re-identification isn't an edge case as it's statistically inevitable.

As far back as 2014, the Obama administration warned that anonymisation wasn’t fail-safe against foreseeable re-identification techniques. AI has only amplified that failure. And consent models aren't any safer, as all major frontier AI developers train on user interactions by default. ‘Best practices’ of the past are now woefully insufficient: retention policies vary, de-identification isn’t done consistently, people can opt out (but it’s rarely obvious how), and although consent lives on paper, in practice it disappears.

This is where technical leadership needs to be direct, as anonymisation alone is no longer a safety net. Consent buried in pages-long terms-of-service is not meaningful, and compliance checklists designed for databases don’t govern probabilistic systems. My suggestion for CTOs is to reject any assumption that ‘de-identified’ data is low risk, and treat ‘behaviourally rich’ datasets as sensitive by default. If your privacy posture still assumes that risk lives only in raw data, not in model outputs, you’re not governing the full stack.

But CTOs still face regulatory hurdles: GDPR, HIPAA and CCPA weren’t designed for inference-driven risk. While compliance may satisfy regulators today, it no longer aligns with user expectations or with technical reality.

Machine ‘unlearning’ is often positioned as a fix, with techniques intended to remove the influence of specific data from trained models, but it misses the point. Cornell-based researchers find that approach is immature, and that it can actually introduce new attack surfaces while offering no guarantees at scale.

On top of this, not every inference is equal, so treating them as such is an avoidable mistake. Some inferences like fraud detection and safety, clearly serve users, while others, such as profiling, secondary monetisation, silent health or risk classification, primarily serve companies.

The EU AI Act's risk-based framework formalises this distinction between users and organisations, especially in healthcare, finance, public safety, and is - at the moment - the regulatory future.

Inference governance means explicitly deciding which conclusions an AI system is allowed to draw, regardless of technical capability; in practice, it can’t live in policy documents alone. It has to be enforced through system design, which means making inference something you can see, challenge and explain, before models reach production and long before regulators intervene.

A simple test for CTOs here is to ask the following questions:

• Was this inference the original purpose of the collection?
• Would a reasonable user expect it?
• Does it introduce material legal or reputational risk?
• Does the benefit justify making it at all?

CTOs can make this review a default step in the machine learning lifecycle, alongside security and reliability reviews. If an inference can’t pass the anticipation test or justify its downstream risk, it should be not merely discouraged, but in fact technically constrained.

There are immediate actions every CTO can take:

• Map inference risk across the stack: document what your models are trained to predict and what they can plausibly infer, treating secondary inference pathways as serious risks.
• Set inference boundaries early: the cheapest time to constrain inference is at design time, not after deployment, as retrofits can be costly and brittle.
• Design for deletion limits: be honest about what unlearning can and can’t do. While forgetting matters, favour architectures that minimise learning in the first place.
• Make trust a KPI: build internal metrics for explainability, inference scope and user control - not just model accuracy.
• Prepare for disclosure: assume you’ll eventually need to explain not only what data you collect, but what conclusions can be derived from them.

This is really a seachange in the CTO’s set of priorities. Trust is no longer something a CTO can delegate to legal, compliance or comms. It’s an architectural decision. The first step is not deploying new tooling, it’s understanding what your systems already know. That means auditing AI outputs, not just inputs, as most organisations can list the data they collect, but very few can list the inferences their models generate, and until that changes, privacy risk will remain invisible to users. That’s why trusting in reputable companies (like Nearform, with 15 years of heritage working with the world’s most complex regulated enterprises) is so important when market dynamics are nascent and quick to change.

As legal scholar Daniel Solove notes, AI amplifies privacy harms in ways existing law wasn’t designed to manage. Organisations that define these boundaries themselves will systematically outperform and build better trust than those waiting for regulators to enforce them.