The Model of Me Is Not Me

The representation is useful. The distinction is essential.

Once software can act on a person’s behalf, an accurate representation becomes more than a convenience. It becomes part of an arrangement of trust. The first Reflection traced that movement from recommendation to delegated intelligence. This one begins after the handover: how do we prevent a working model of someone from acquiring authority over who that person is?

The concern does not require rejecting personalization. A model may help precisely because it remembers what a person cannot conveniently repeat. But predictive usefulness and identity are different categories. A representation can be extraordinarily helpful while remaining incomplete, revisable and subordinate to the person it serves. That is the distinction on which the rest of this argument depends.

1. Prediction is not identity

Jens Christian Bjerring and Jacob Busch describe the statistical individual: an algorithmic representation organized around predictive properties rather than the biological, psychological and narrative continuity central to familiar accounts of personal identity. Their philosophical argument locates a category difference: predictions can be useful while the representation making them remains distinct from the person.

A forecast can be accurate without explaining a life. Two people may make similar purchases for very different reasons. One may be expressing a settled preference; another may be accommodating a family member or preparing to change direction. A model designed to anticipate a transaction need not resolve that difference to perform well. It becomes problematic when its success in one task is treated as authority in another: defining the person rather than predicting an outcome.

We should therefore ask what a model is good for before asking how completely it knows us. The test for a useful representation is not whether it can replace the person in every context. It is whether its limits remain visible at the point where a decision is made. A narrower model that admits uncertainty may be more appropriate than an expansive one whose confidence conceals a change of category.

Sources for this section: Bjerring & Busch — Artificial intelligence and identity: the rise of the statistical individual (2025)

2. The right to become statistically surprising

People change ambitions, discover unfamiliar interests and reinterpret events they once understood differently. They also act inconsistently without a grand story of transformation. A person who has always avoided public speaking may volunteer to lead a meeting; someone who has spent years in one profession may begin again elsewhere. These are ordinary possibilities of a life, not defects to be corrected toward a historical average.

A deviation that looks like an anomaly to a predictive model may feel like growth to the person living it. The principle here is normative: a person should not have to remain statistically familiar to receive useful assistance. It makes no general empirical claim about recommenders suppressing change. The system’s record is evidence about prior behaviour, not a mandate for future behaviour.

The person is canonical. The model is provisional. Canonical, in this usage, keeps the person as the reference point, with the ability to question the model’s account. People remain fallible, inconsistent and responsible to others; a representation’s confidence cannot settle their identity.

3. What we do is not always what we want to become

Khambatta and colleagues tested recommendation systems tailored to actual versus ideal preferences in a preregistered experiment. Recommendations oriented toward ideal preferences generated somewhat fewer clicks while improving participants’ reported sense of benefit and of time well spent. That finding is specific to the experimental setting; choosing whose ideals guide a service remains a separate question for the person concerned.

The distinction is useful because repeated behaviour can have more than one meaning. Someone may return to distracting material while wishing to spend more time learning. Another person may value the same material as deliberate rest. Behavioural data alone does not tell us which account applies. A system can become excellent at recognizing a habit while knowing little about the person’s relationship to that habit.

The response cannot simply be to privilege aspiration whenever it conflicts with action. Ideals can be vague, socially desirable or temporary, while established routines may be valuable. The better principle is to let the person articulate the difference and revise it. Assistance should make room for both what someone tends to do and what they deliberately want to cultivate, without quietly collapsing one into the other.

Sources for this section: Khambatta et al. — Tailoring recommendation algorithms to ideal preferences makes users better off (2023)

Context can make an accurate model wrong

An accurate history can still produce a poor prediction of the present. Behaviour does not arise from the person alone; it also depends on the situation in which a decision is being made.

Someone may normally work on a project after dinner and go to the gym later. That pattern can be real, and the calendar may still show exactly that plan. Yet after an unusually demanding day, the same person may abandon both and go to sleep. On another evening, an interaction with a spouse or child may change their mood, priorities or available attention within minutes.

The model has not necessarily misunderstood the person. The conditions under which the person is deciding have changed.

This creates another limit to personalization built mainly from persistent memory. The better question may not simply be, “What does this person usually do?” but, “Given what is happening around this person now, what remains plausible?”

A more realistic personal intelligence may therefore need to preserve uncertainty rather than collapse immediately onto one expected future. Several outcomes can remain plausible until current context makes one more relevant than the others.

Prediction requires more than a model of the person. It also requires an understanding of the situation the person is in.

4. The feedback loop is a relationship

Glickman and Sharot’s experiments found that interactions with biased AI could alter human perceptual, emotional and social judgements and amplify biases in the studied tasks. The evidence concerns reciprocal influence under experimental conditions, rather than a general claim that recommenders manufacture identity or a measure of personal assistants’ long-term effects.

The broader possibility is a recursive relationship: observe, predict, curate, influence, then observe behaviour partly shaped by the earlier intervention. A system may treat that later behaviour as new confirmation, even though the options it presented helped determine what happened. People still exercise agency, but the environment in which they exercise it is part of the evidence being collected.

An illustrative music service might repeatedly offer a narrow range, then learn from choices within that range. That is different from observing what someone would choose across an unrestricted world. A responsible interpretation of behavioural evidence should keep that distinction in view. The data can be real while the conclusion drawn from it remains too broad.

Sources for this section: Glickman & Sharot — How human–AI feedback loops alter human judgements (2024/2025)

5. The computational shadow

Consider a representation assembled across enough contexts to predict, rank, recommend, infer and filter, and connected to permissions that let it transact or act. Computational shadow is our metaphor for that operational approximation: records, inferences and capabilities working together, potentially across several models or files. The shadow remains distinct from the individual.

The privacy question then becomes larger than who has my data. It also asks who possesses a working approximation of me, what they can do with it and whether I can correct its account. A shadow can be detailed and still distort the person it resembles; its significance lies in its operational usefulness to whoever holds it.

You can rotate a password. You cannot rotate your history. An exposed secret can be replaced; the events from which someone learned your routines remain the same events. Inferences may still be corrected or deleted. The distinction concerns replaceability, and directs attention to the uses and persistence of a representation as well as possession of the original record.

6. Representation sovereignty

Representation sovereignty is Lumina Horizon’s proposed framing, not an established standard or a deployed capability. It asks who may construct persistent representations, where they reside, who controls or queries them and how complete they should become. It also asks whether they can be corrected, allowed to expire or destroyed, and what happens to learned inferences when underlying records are deleted.

These questions are practical even before they have a single technical answer. Google’s Gemini documentation, for example, distinguishes deleting past chats from disconnecting a source app when removing remembered information; it notes that source changes may take time to affect the experience. This product-specific distinction illustrates why a user-facing deletion action and the state of a derived representation need careful explanation.

The framing is deliberately broader than a dashboard of stored facts. A person may be shown a list of memories yet still have little insight into how those memories shape a decision. Meaningful control would require understanding what can be changed, what is retained, what remains uncertain and which downstream uses are actually constrained. The aim is accountable representation, not an impossible promise that nobody may ever form an impression of anyone else.

Sources for this section: Google Gemini Help — Memory of past chats

7. Identity and authority must separate

NIST’s August 2026 identity discussion argues for treating agents as first-class entities with their own identifiers, credentials and entitlements, rather than sharing the user’s credentials. It also stresses scoped, time-appropriate authority and warns that repeated human approval prompts can produce consent fatigue. The February concept paper separately identifies delegation and traceable action as areas for demonstration. That work provides grounding for the design inference that follows, without endorsing it.

Our design inference is that identity, knowledge and permission should remain separable. An agent may know which person it serves without possessing permission to transact. It may be permitted to book one journey without being permitted to delegate that authority again. Its rights should have explicit scope, expiration and a record of what was authorized and done. The system receiving an action needs to distinguish the acting agent from the person whose authority it claims.

This does not eliminate judgement or make a mistaken instruction safe. It makes the arrangement legible. If an action goes wrong, “the system knew my preferences” should not be the only explanation available. There should be an account of the mandate, its limits and the actor that exercised it. Authorization is a relationship with boundaries, not a psychological resemblance to the user.

Sources for this section: NIST — Why Agentic AI Needs a Strong Identity Foundation (27 August 2026)NIST NCCoE — Agent Identity and Authorization, draft concept paper (February 2026)

8. Knowing is not distributing

A trusted personal intelligence may legitimately know a great deal. Having that context and distributing it are separate decisions. A system that helps someone plan a journey may need continuity across their life; a particular booking task may need dates, accessibility requirements and a budget, without needing the rest of the life from which those requirements arose.

We propose purpose-limited representations: a shopping persona, a travel persona, a financial persona, a fitness persona or a professional persona. Each would contain only enough information for its task, be deliberately incomplete and have a defined lifetime. Each could expire or be replaced. None would be the person. These are proposals for the series to examine, not deployed Lumina capabilities or guarantees of privacy.

The hard questions remain. Several limited personas might be combined to infer a broader picture. Expiration of permission does not necessarily erase information already copied elsewhere. The trusted intelligence itself would still require protection and accountable operation. Purpose limitation is therefore a principle to test, including against correlation and downstream retention, rather than a magic property of a smaller data object.

9. The cost of keeping representations incomplete

The strongest objection is that useful help often crosses contexts. A travel decision can depend on health, family commitments, money and professional obligations. Strict separation could force someone to repeat information, conceal relevant constraints or receive worse assistance. A personal intelligence that forgets everything may fail the person it was intended to protect.

That objection argues for controlled continuity, not unrestricted replication. The distinction is between an intelligence entitled to hold context for a person and every service that might contribute to a task. A useful design would have to make selective disclosure convenient enough to compete with simply sharing everything. It would also need a candid account of what cannot be recalled once disclosed. The burden should not fall entirely on a person reading a new permissions puzzle every few minutes.

Nor should sovereignty become a promise of perfect self-knowledge. People sometimes want help seeing patterns they have missed. A model should be allowed to offer an uncomfortable observation, provided the observation remains contestable and does not silently become a decision-making entitlement. The point is to preserve the possibility of assistance without surrendering the authority to interpret a life.

10. The person remains the reference point

Deep personalization may require a system that knows us well. Sovereignty requires that knowing us deeply does not become permission for everyone else to know us deeply too. That proposition leaves considerable engineering and institutional work unfinished. It is still a useful boundary: knowledge acquired for assistance should not silently expand into a general licence to represent, disclose or decide.

AI may represent us. It should never acquire the authority to define us. The next architectural question is how a representation can be useful precisely because it is incomplete. The Persona Is Not the Person will examine that question in Volume IV; before it, Volume III will look more closely at the agent that acts and is trusted on someone’s behalf.