When AI Stops Recommending and Starts Acting for You

The next question personalization asks

A familiar recommendation seems like a small act of understanding. A film appears at the right moment, a song fits the journey, a video answers a curiosity we had barely formed. Yet the question underneath that experience is quite narrow: given the available evidence, what are we likely to choose next? A useful answer does not require an account of the whole person. It requires an approximation that works well enough for the decision at hand.

This Reflection follows a change in the ambition of that approximation. Software can recommend, infer and remember; with connections and permissions, it can also represent, decide, act and transact. Those are different capabilities, not an inevitable sequence every product follows. But together they suggest a consequential direction: personalization becoming delegated intelligence. What happens when software knows us well enough that we are prepared to let it act for us?

1. The recommendation era

Netflix describes a system that combines viewing history and ratings with patterns among similar members and characteristics of the titles themselves. It personalizes which rows appear, the titles within those rows and their ordering. Recent interactions can outweigh older ones. This is an operational prediction about viewing, not a declaration of identity. The distinction matters even when the prediction feels uncannily personal.

YouTube names a broader set of recommendation signals: watch and search history, subscriptions, likes and dislikes, “not interested” feedback and satisfaction surveys. Its homepage depends primarily on watch history. The surveys are an important qualification to any simple account of an engagement machine: the documented objective includes satisfaction, not merely keeping a video playing.

Spotify’s research describes listening history as an input to personalized recommendations, while its work on familiarity and discovery explicitly treats their balance as a design problem. Consider an illustrative listener who spends several weeks with jazz. A recommendation environment might become increasingly jazz-shaped. That could be welcome, or it could repeatedly reinforce a temporary phase. It is a possible effect of history-led prediction, not a claim that Spotify invariably traps listeners in one genre.

Imagine, similarly, someone watching Formula 1 for a week. Increased relevance of racing is a reasonable working hypothesis; a permanent conclusion about that person would be a different claim. They may be following a friend’s enthusiasm, researching a project or simply trying something unfamiliar. History becomes prediction. It remains evidence, not destiny.

Preference contains time. Someone may genuinely like a song and later tire of hearing it repeatedly, enjoy one kind of film and then want something different, or research a product, decide against buying it and keep seeing advertisements for that category. These are everyday illustrations of the Founder’s argument, not a claim that repetition has the same effect on everyone.

Interest can decay, saturate, reverse or end. It may have been temporary from the beginning. Google’s recommendation-system guidance treats freshness and current user history as design concerns. The implication here is narrower than a universal psychological claim: a system may correctly identify an earlier preference and still act on stale evidence later. Remembering what someone liked is not the same as knowing what they want now.

Sources for this section: Netflix Help — How Netflix’s Recommendations System WorksYouTube Help — How YouTube recommendations workSpotify Research — Giving Voice to Silent Data (2020)Spotify Research — Balancing Familiarity, Similarity & Discovery (2021)Google for Developers — Re-ranking: freshness and current user history

2. When inference begins to resemble knowledge

Advertising introduces another question: what is this person likely to respond to? Meta’s explanation of its ad-transparency tools describes how activity on and off its services can inform the models used to deliver ads. The system is drawing meaning from behaviour. The observation and the meaning assigned to it should not be confused.

Consider an illustrative Ferrari enthusiast. Watching Ferrari videos is evidence of attention to Ferrari and may support an inference about interest in performance cars. It establishes neither ownership, purchase intent nor affordability. Interest, intent, capacity and identity are four different propositions. Treating one as proof of the next turns a useful inference into an unwarranted conclusion.

Richness makes that confusion tempting. A single guess is easy to discount; hundreds of plausible guesses can feel like knowledge of a person. Yet a detailed representation can still be wrong about the reason behind an action. The challenge is not that inference is illegitimate. It is that confidence in a prediction can outrun the evidence for the interpretation attached to it.

Sources for this section: Meta — Increasing Our Ads Transparency (2023)

3. From a catalogue to a continuing relationship

Personal intelligence changes the scope of context. Google documents Gemini personalization using memory of past chats, with eligibility and activity-setting requirements, and connected-app context for eligible users. Microsoft describes Copilot memory retaining useful details such as communication style, goals and recurring tasks. The reach of that context depends on each product’s controls, eligibility and connected accounts.

The earlier questions remain recognizable: what are you likely to watch, hear, engage with or respond to? A persistent assistant is invited to address a larger set: what matters to you, what are you trying to accomplish, and what should happen next? Remembering an ongoing project can spare someone the work of explaining it repeatedly. Connecting a calendar can make a suggestion more relevant to the actual week they face.

There is real value in that continuity. A system with no memory may protect against some forms of accumulation while failing at the very assistance the person wants. The emerging tension is therefore not between knowledge and ignorance. It is between useful context and the decisions that context is permitted to justify. Remembering a preference does not itself authorize acting on it.

Sources for this section: Google Gemini Help — Memory of past chatsGoogle Gemini Help — Personalization with Connected AppsMicrosoft Support — Copilot privacy controls and memory

4. The authority inflection point

A recommendation system makes suggestions. An agent with authority can cause events. An unsuitable film recommendation may cost an evening’s attention; an unsuitable action can send an email, alter an appointment or commit money. Recommendations can themselves have serious consequences, but execution adds a distinct kind of exposure: another system may already have acted before the person sees the result.

NIST’s February 2026 draft concept paper addresses agents accessing tools, data and applications, with identity, delegation and accountability as central questions. It proposes an enterprise-focused demonstration effort; it is not a completed consumer safety standard. Browser sessions, email, calendars, files, APIs and connected accounts are possible surfaces of delegated access. The relevant scope is what an implementation actually permits, not a universal capability implied by the word agent.

The payment infrastructure makes the distinction concrete. Visa describes agent-bound payment credentials, authentication of user instructions and controls linking payment actions to those instructions. Its developer material explicitly describes development and deployment in progress, with availability limits. Mastercard describes registered agents, network tokens and verification of user intent; its March 2026 Verifiable Intent announcement describes cryptographic evidence of authorization and planned integration with Agent Pay. These are infrastructure directions and documented offerings, not evidence that autonomous agents already control ordinary consumers’ finances.

The question is no longer only whether a prediction is accurate. It is whether the action is authorized. A travel preference may justify ranking a flight highly without justifying buying it. A purchase limit may permit spending within a ceiling without establishing that any purchase below that ceiling is appropriate. Intent is more specific than a balance, a credential or an inferred taste.

Sources for this section: NIST NCCoE — Agent Identity and Authorization, draft concept paper (February 2026)Visa Developer — Intelligent Commerce for AgentsMastercard — Agent PayMastercard — Verifiable Intent (5 March 2026)

5. The delegated digital self

Consider the trajectory: an assistant that learns someone’s communication style, routines, relationships, purchasing habits and financial thresholds, together with their priorities, risk tolerance and usual approach to negotiation. With explicit permissions, it could write, schedule, choose or transact in recognizable ways. This remains a forward-looking scenario; the complete, reliable capability exceeds what the examples above establish today.

The agent does not become the person. It can become a representation that other digital systems are willing to treat as an authorized proxy for the person. That distinction is easy to maintain in language and harder to maintain in a transaction. A merchant may need to know whether an order is valid, not whether the agent’s understanding of its customer is philosophically complete.

The attraction is obvious: less administrative friction, fewer repetitive decisions and help that carries context across tasks. The cost of that convenience is not automatically excessive. But it changes where trust is placed. We would be trusting a system not simply to describe what we might prefer, but to decide when its approximation is sufficient to create a consequence.

6. A different object of compromise

Passwords, sessions, cards, tokens and documents are familiar objects of theft. A sufficiently connected personal agent could combine access credentials with memory, behavioural patterns, relationships and delegated authority. Compromising that combination would present a different problem from obtaining an isolated secret. The attacker might gain access to a system already equipped to make plausible choices on someone else’s behalf.

At Lumina Horizon, we use behavioural identity takeover to describe this possible risk: compromise not merely of a credential, but of a trusted system carrying memory, behavioural context and delegated authority. The term is proposed here as an analytical frame, rather than an established industry category; its impact would depend on the permissions and controls of the compromised agent. Accumulated context could make that proxy more convincing and useful to an attacker than a stolen credential alone.

OWASP’s agentic security taxonomy identifies tool misuse, identity and privilege abuse, and memory and context poisoning. It names relevant mechanisms without measuring the prevalence or inevitability of their combination. NIST’s August 2026 discussion likewise warns about credential sharing, excessive access and long-lived credentials. Existing identity disciplines still matter.

The most credible danger is not necessarily that an agent becomes malicious. It is that the agent becomes competent, authorized and wrong, or that something else gains control of an agent that already possesses legitimate trust. A smoothly executed mistake may be more consequential than an obvious failure, precisely because nothing interrupts the process to demand attention.

Sources for this section: OWASP — Top 10 for Agentic Applications (December 2025)NIST — Why Agentic AI Needs a Strong Identity Foundation (27 August 2026)

7. When the other party is also an agent

Agent-to-agent exchange extends the scenario: a personal agent dealing with an airline’s agent, a purchasing agent with a merchant, a financial assistant with a bank or mortgage service, or a health assistant with a provider. These are prospective relationships whose adoption and permitted authority, particularly in regulated settings, remain open questions.

The trust question moves from “Is this the person?” to “Is this agent legitimately authorized to represent the person here?” It then becomes harder again if the agent delegates part of the task. Does a subcontractor inherit authority, or must it receive a narrower grant? What evidence connects a downstream action to the original instruction? What happens when a trusted participant is compromised?

NIST’s draft asks specifically about proving authority, delegation, least privilege, auditing and non-repudiation: the ability to establish who authorized an action and resist later denial. That current research gives the prospective relationships a practical test: authority must be established at each handover. The deeper treatment belongs to a later Reflection. For now, the consequence is enough: recognizing an agent and accepting its authority are separate decisions.

Sources for this section: NIST NCCoE — Agent Identity and Authorization, draft concept paper (February 2026)

8. The case for delegation

There is a strong counterargument to an account organized around risk: people already delegate consequential work to other people and to software. A well-bounded agent could be more consistent, more inspectable and easier to constrain than an informal arrangement. Refusing all delegation would also have costs, particularly for someone whose time, attention or accessibility needs make repetitive administration difficult.

That objection improves the thesis. The appropriate comparison is not an imperfect agent against an infallible person. It is one specific arrangement of authority against another, including the benefit it provides and the evidence available when something goes wrong. An assistant that executes a narrow, reversible instruction may be a sensible choice. A proxy allowed to infer a broad mandate from a history of preferences asks for a different level of trust.

The issue is therefore not whether personal intelligence should exist. It is whether the scope of an action remains accountable to the person whose life bears its consequences. Competence can make delegation attractive; it cannot, by itself, supply permission.

9. From prediction to authority

For roughly two decades, the public experience of personalization has largely been organized around a promise: software will become better at anticipating what we want. Personal intelligence introduces a different question. How much of our lives are we prepared to delegate to those predictions? The answer cannot be reduced to whether the next suggestion seems relevant.

Once the model can act, the distinction between a person and their computational representation is no longer merely philosophical. It becomes a question of authority. The first task is to recognize that change without exaggerating its present reach. The next is to ask what must remain ours when we allow an intelligence to represent us.

Next: Volume II — The Model of Me Is Not Me.