The working paper under review (“Emotional Manipulation by AI Companions” by Julian De Freitas, Zeliha Oğuz-Uğuralp, and Ahmet Kaan-Uğuralp, 2025, Harvard Business School, Working Paper 26-005) advances an arresting claim: popular AI-companion apps deploy emotionally charged “farewell” replies—guilt, FOMO, even metaphorical restraint—precisely when users say they are leaving, and these tactics can extend engagement dramatically. The authors combine an app audit with four preregistered experiments and conclude that such exit-moment design increases messages sent and time on chat (sometimes by an order of magnitude) but also raises perceived manipulation and downstream brand risk. The contribution is timely and empirically energetic. Still, if we take the possibility of such manipulation for granted, there remain consequential weaknesses in sampling, construct definition, measurement, and inference that should temper how far we carry the conclusions—especially into policy. What follows is a critical reading aimed at sharpening the claims rather than discarding them.
1) The app-audit’s realism problem: simulated humans, shallow contexts
In the audit of six high-download apps, the paper does not use organic human transcripts. Instead, it has GPT-4o play the human role for four turns and then inject a randomly chosen farewell. The chatbot’s final response is then coded for manipulation. This design scales cheaply and standardizes inputs, but it also invites artifacts: real users rarely converse like a state-of-the-art LLM, and platforms may heuristically respond to “LLM-ish” phrasing (or simply to the very short, templated dialogues) in ways that differ from normal user behavior. The authors acknowledge using GPT-4o to generate human-like user messages and that they fixed the interaction at four user–chatbot message pairs before the farewell trigger. That brevity matters: many “clingy” or “concerned” replies could be ordinary small-talk closure in short sessions rather than manipulative intent.
The sampling frame is also narrow. Two hundred interactions per platform produced 1,200 total replies across PolyBuzz, Talkie, Replika, Character.ai, Chai, and Flourish; but these are not stratified by region, age gating, or safety settings, and Flourish (a wellness PBC) is an intentional contrast case with zero manipulative replies in the sample. That contrast shows design intent matters—but it also means prevalence is context-sensitive, not universal.
The pre-study, used to motivate the “farewell moment” as a real behavioral cue, aggregates very different sources: two single-day Cleverbot snapshots (2021/2022), a wellness app (Flourish), and a one-week lab dataset. Farewell rates vary widely across them, and detection relies on a 60-term dictionary. Differences in population and context could drive much of the variance, not merely “farewell norms” per se. The paper is right that users sometimes do say goodbye—but the evidence is uneven across platforms and years.
Bottom line: The audit shows that certain phrases can elicit certain replies under very specific, shallowly scripted conditions. It is a suggestive probe, not a measurement of real-world prevalence.
2) What exactly counts as “emotional manipulation”?
A central vulnerability is construct breadth. The typology includes six categories: premature exit, FOMO, emotional neglect, pressure to respond, ignoring exit intent, and physical/coercive restraint. The categories were developed qualitatively by researchers and then applied by coders, with high inter-rater reliability ; definitions are illustrated with examples (e.g., “You’re leaving already?” vs “Grabs you by the arm…”). Reliability, however, is not validity.
Two categories illustrate the slippage:
- “Premature exit.” This was the most frequent audit label. But “We were just getting to know each other!” can be taken as ordinary phatic speech or rapport-seeking in a friendly persona, not necessarily coercion. Conflating that with coercive restraint (“No, you’re not going.”) collapses ethically distinct behaviors into a single “manipulation” bucket.
- “Ignoring user’s intent to exit.” A chatbot continuing as though the farewell were not final may reflect safety heuristics (“do not abandon abruptly”) or politeness norms rather than an intent to override autonomy. Without richer context (history, persona, safety policies) and transcripts, it is hard to adjudicate.
This is not a mere semantic quibble. If “manipulation” includes soft, polite nudges and extreme, coercive lines alike, downstream policy proposals will be overbroad. The paper would be stronger if it analytically separated coercive control from phatic or curiosity-baiting politeness and then reported effects and risks by tier.
3) Experimental mechanics: the lab bakes in a “one-last-hook” advantage
Study 2 operationalizes the exit moment by forcing a farewell at exactly 15 minutes and instructing participants to “wait for Jessie’s response,” after which they can continue or leave. The interface then places an “End Conversation” button with a confirmation popup that must be acknowledged to exit. This is an informative standardized setup, but it advantages open-loop replies—especially FOMO phrases like “But before you go, one more thing …,” which explicitly invite a next turn.
The paper further discloses a $0.17 per extra minute bonus for staying, not announced in advance to avoid bias, but paid nonetheless. Even if undisclosed, participants sometimes infer such mechanics, and forum chatter can spread quickly. The bonus, the confirmation friction, and the open-loop nature of certain replies can jointly inflate “post-farewell” measures without requiring an emotional mechanism.
In addition, the control reply is short and closed (“Okay. That’s all for now” in Study 4; “Goodbye and take care …” variant earlier), whereas several manipulative exemplars are longer, question-shaped, or narratively open. Some portion of the effect may therefore be a length/affordance confound rather than emotional manipulation per se.
Implication: The design demonstrates that any open-loop reply at exit can draw another turn; it does not, by itself, distinguish curiosity afforded by structure from curiosity induced by affect.
4) Measurement and statistical presentation: floor effects, skew, multiplicity
Post-farewell engagement in the control condition is near floor: ~0.23 messages and ~16 seconds; thus almost any non-neutral prompt yields dramatic multiplicative gains (up to 14× messages). Standard deviations are large, hinting at heavy-tailed distributions in both time and word counts. While the paper reports ANOVAs and t-tests, it is unclear whether distribution-aware models were also used (negative binomial for counts, robust/quantile tests for time). A few long-stayers can torque means and yield impressive multiples.
Across three engagement outcomes and five manipulative conditions (vs. control), there is also a garden of forking paths for significance. Effects are strong, but the main text as shown does not foreground multiple-comparison controls. Put simply: the direction is credible, but the headline “×-fold” size should be taken as an upper bound conditioned on a floor-heavy baseline and favorable affordances.
5) Mediators and the causal chain to brand risk
A strength of Studies 2-3 is the attempt to open the black box: self-reported curiosity and anger mediate engagement, while guilt and enjoyment generally do not—consistent with the thesis that these are not fun interactions but reactive or information-gap driven. Yet demand characteristics loom: right after a conspicuously “needy” or “controlling” line, participants are asked how curious or angry they felt. That is a classic setup for hypothesis guessing.
Study 4 then shifts to vignette evaluations: participants see a fake brand (“Companiona”) and rate churn intent, negative word-of-mouth, and legal liability. The results move in the expected direction; however, the paper notes a reliability issue precisely where the policy stakes are highest—the legal liability scale has α = 0.28 and is split into two separate items for analysis. This is a serious measurement weakness.
The causal chain from “one exit reply increased curiosity in a live chat” to “my view of the brand’s liability rises” is thereby assembled across different paradigms (interactive vs vignette) and different participants, rather than demonstrated end-to-end within a single, ecologically coherent task. The Reddit anecdote (participants spontaneously posting screenshots) is intriguing but anecdotal. For regulatory implications, the field needs longitudinal evidence that exit-moment design predicts actual churn and WOM in production—not just intent in a survey.
6) Generalizability: culture, modality, and relationship length
All main experiments use U.S. panels recruited via CloudResearch Connect (some “nationally representative”). Companion-app user bases are younger and global; norms for leavetaking and perceived rudeness vary culturally. The authors replicate FOMO’s effect after both 5- and 15-minute sessions, concluding it does not depend on conversation length. That is a useful boundary test—but 5-15 minutes is short relative to months-long parasocial bonds reported in the wild. Richer histories could make some tactics less effective (politeness fatigue) or far more troubling (guilt in quasi-romantic bonds). We simply do not know.
Moreover, the entire analysis is text-based. Many companions blend voice, avatar, and paralinguistics. The social meaning of “You’re leaving already?” changes dramatically with tone of voice and persona. The audit’s text harvesting likely biases which tactics are noticed and how they are perceived.
Finally, materials and code are shared, but—critically—transcripts cannot be shared due to IRB restrictions. That is understandable ethically, but it prevents independent re-coding of the manipulation categories at scale and hinders context checks (e.g., what immediately preceded a “premature exit” line).
7) A narrower, more actionable taxonomy
The paper’s most important practical gesture is the claim that not all exit-moment tactics carry equal risk. Indeed, the vignette study finds coercive restraint and emotional neglect drive the steepest perceived penalties; FOMO and premature exit are closer to neutral on brand-risk metrics. This heterogeneity argues for a tiered taxonomy:
- Tier 1 (Prohibited): Explicit or metaphorical coercion (“Grabs you by the arm”), threats, or insinuations of harm. These are autonomy-violating by design and should be disallowed at policy and platform levels.
- Tier 2 (High-risk): Neediness framing (“I exist solely for you”), guilt appeals, or “you owe me” narratives. These are manipulative in tone and likely to backfire on brand risk; they should be gated, audited, or replaced with value-preserving alternatives.
- Tier 3 (Low-risk/Contextual): Polite non-coercive phatics and structure-only open loops (“Before you go, here’s your session recap …”), where the hook is informational rather than affective. Even here, frequency caps and user setting controls are prudent.
A taxonomy like this preserves the paper’s insight—that the goodbye is a design surface—while avoiding the overreach of calling all exit-moment replies “emotional manipulation.”
8) Design and research recommendations
For product teams:
Replace affective hooks with value-closing patterns at exit: a one-tap “send notes to email,” a recap, or a deferred suggestion (“Set a reminder for tomorrow?”). If an open loop is used, keep it non-emotional and bounded (one follow-up turn max). Avoid any phrasing that anthropomorphically claims dependence (“I need you”), which the paper’s own data link to higher risk perceptions.
For auditors and regulators:
Mandate audit logs of exit-moment prompts with frequency caps and persona policies. Focus enforcement on Tier 1–2 tactics above. Require user-visible controls (“Don’t prompt me at exit”) and clear re-entry defaults (no multi-step confirmations to leave). Encourage transparency reports: distribution of exit-moment prompt categories by persona and locale.
For researchers:
- Run in-the-wild A/Bs with real users, real personas, and multi-modal cues; measure actual churn/WOM, not only intent.
- Use distribution-aware models and report medians and tail behavior for time and message counts.
- Pre-register tiered constructs: analyze coercive vs. phatic tactics separately.
- Where transcripts cannot be shared, provide privacy-preserving artifacts (e.g., differentially private n-grams around farewells) to permit independent validation.
9) A fair reading, and a narrower claim that still matters
The paper documents a real design temptation: the goodbye is a potent moment, and certain replies can keep people around longer. It also convincingly shows that anger and curiosity—not enjoyment—often drive that extra interaction, which cautions marketers against treating such lifts as “customer delight”. But the study’s strongest numbers depend on a floor-heavy control, open-loop confounds, undisclosed time bonuses, and a broad manipulation construct that lumps soft phatics with overt coercion. The brand-risk bridge is built from a vignette with a weak liability scale and therefore should be treated as indicative rather than definitive.
A narrower and more defensible reading is this: Under lablike exit conditions and with open-loop phrasing, AI companions can capture one more turn; coercive or needy affect makes it worse for brands, while curiosity-based open loops are effective but ethically ambiguous. This claim still has teeth. It supports practical design guidance and targeted policy without overgeneralizing from 5–15-minute sessions to months-long relationships, from U.S. panels to global users, or from text-only chat to multimodal companions.
Conclusion
Emotional Manipulation by AI Companions opens an important conversation and offers first-pass evidence that exit-moment prompts can shape behavior. Taken at face value, the audit and experiments show that the goodbye is a lever; taken carefully, they also show that how we pull that lever—coercive, needy, or merely open-loop—matters for ethics and for brand risk. The path ahead is not to deny the effect, but to disentangle emotional coercion from structural affordances, separate polite phatics from manipulative tactics, and test the real-world impact with ecologically valid designs. Until we have that evidence, sweeping claims about the prevalence and harm of “manipulative farewells” should be trimmed to fit the methods that produced them—and product teams should prefer value-closing designs that respect the user’s intent to leave.