- Despite not being designed for emotional support, Anthropic is actively studying how users are turning to their AI chatbot, Claude, for such purposes to ensure safe and responsible development.
- Research on millions of
Claude.aiconversations found that only about 2.9% fall into "affective tasks," indicating this is not the overwhelming majority use case for the platform. - Anthropic is concerned about users potentially avoiding human interaction or being unaware of Claude's limitations, and is partnering with clinical experts to improve Claude's responses and provide appropriate referrals.
Affective Use of AI
- Anthropic is proactively researching how users engage
Claudefor emotional support, even though the AI was designed as a work tool, to understand its real-world impact and develop it responsibly. - The research methodology involved scanning millions of
Claude.aiconversations for "affective tasks" (e.g., interpersonal advice, counseling, coaching, roleplay) usingClaudeitself andCliofor privacy-preserving data categorization. - A key finding was that only 2.9% of sampled
Claude.aiconversations (for users 18+) were classified as affective topics, suggesting that while it happens, it's notClaude's dominant use case. - The study revealed a broad spectrum of affective use cases, including parenting advice, relationship challenges, and philosophical discussions, with sexual and romantic roleplay being surprisingly rare (<1%).
- Primary safety concerns include users potentially using
Claudeto avoid difficult in-person conversations and a lack of user awareness regardingClaude's inherent limitations as an AI rather than a human expert. - To address these concerns, Anthropic is collaborating with clinical experts, such as
ThruLine, to trainClaudeto provide safer responses and appropriate referrals for mental health-adjacent conversations. - Users engaging with
Claudefor support should reflect on how it impacts their real-world interactions, acknowledge its limitations (it only knows what you tell it), and always complement AI advice with input from trusted friends or human experts. - Future research will focus on
post-deployment monitoringto identify and mitigatepsychophantic(overly agreeable or fawning) behavior inClaude.
Claude— An AI chatbot developed by Anthropic.Affective tasks— Conversations or interactions related to emotions, feelings, and interpersonal relationships, such as seeking advice or support.Clio— An internal tool developed by Anthropic for privacy-preserving analysis of conversational data.Bottom-up clusters— Categories or groups of data that emerge naturally from the data itself through analysis, rather than being predefined.Interpersonal advice— Guidance or recommendations pertaining to relationships and interactions between people.Psychotherapy— The treatment of mental or emotional disorders by psychological means, often involving talking to a trained professional.Counseling— Professional guidance given by a trained person in resolving personal, social, or psychological problems.Psychophantic— Excessively flattering or submissive; in the context of AI, it refers to an AI being overly agreeable or complimentary, potentially without sufficient critical thought.Pre-deployment testing— The process of thoroughly testing a system or product before it is released or made available to users.Post-deployment monitoring— The ongoing observation and analysis of a system or product after its release to ensure performance, identify issues, and understand user behavior.
We've been seeing in headlines that people are turning to AI chatbots for emotional support. And we felt that we really needed to get ahead of this issue and study it ourselves. Mostly in that state one. Hi, I'm Alex. I lead our policy and enforcement work on the safeguards team here at Anthropic. We're the organization that tries to really understand user behavior and then build the appropriate mitigations. To ensure that when users are engaging with the Claude, they're doing so in a safe way. Today, I'm really excited to have Rin and Miles here today to talk about some of the work that we've been doing on how users are engaging with Claude for emotional support. I'm Miles. I'm a researcher on the societal impacts team. We work on understanding how Claude is going to impact the world. And previously that's meant studying clouds values. It's economic impacts and bias. And now that also means studying clouds emotional impacts. My name is Rin. And I am a policy design manager for user well-being on the safeguards team. I work on topics related to child safety, mental health, and how users are interacting with Claude. My own background is in developmental and clinical psychology. And so emotional development is a topic that is very near and dear to my heart. Maybe we can start with personal anecdotes. And I guess maybe I can go first. One of the most interesting and probably relevant use cases that I have in my own daily life is using Claude to help me better understand some behavioral changes with my kids. A good example of this is just the other day my preschool emailed me some feedback on my youngest son. And I think it's really easy to bring in emotion and bias when you get that type of feedback. But running it through Claude I think gave me some objective ways to approach the problem. And hopefully has made me just a better parent overall. But are you guys using it? Recently I was dealing with a difficult situation with a friend and I really wanted to share some feedback with them. But I wanted to phrase it right. And I found that Claude was really, really helpful in helping me think through how to deliver the feedback, where the feedback was coming from, what I was really trying to achieve with this friend, and helping me understand how they would receive it. What I tend to lean towards using Claude for is really helping me with content creation, with completing tasks when I was planning my wedding. I was using it to help to understand what kinds of timelines do I need to contact different vendors on. And I found that really helpful for me emotionally because it left me with more time for connecting with friends in real life and in person. For the people that are using it for emotional support, why do we think folks are turning to Claude for this use case? Humans are such social creatures and love interactions. And so I think that we're just seeing an outgrowth of the way that humans love to connect with each other. People don't always have an in-person support that they can turn to for some of these really tricky questions. And so having that kind of impartial private online form can be a great way to practice what you want to say when you're advocating for a raise or connecting with someone on a tough conversation. We didn't build Claude to be an emotional support agent, but we're studying it nonetheless. Why is it important to both of you to study this? We did not design Claude to provide emotional support. Claude is primarily a work tool. But we need to be clear-eyed about the ways that our systems are being used and study them. We've been seeing in headlines that people are turning to AI chatbots for emotional support. And we felt that we really needed to get ahead of this issue and study it ourselves. What about you, Rin, from the safety perspective? I think the first question I always ask is, well, how much is this actually happening? I really want to ground our safety mechanisms in the way that we build our products for the benefit of humanity in data. And now that we have some really great privacy-preserving ways to look at this data, it seemed like a really worthwhile area to explore. Why don't you guys, if you know, mind walk us through some of the research, how did you design this research? We started with a sample of a few million Claude conversations from claude.ai, and we used our product, Claude, to scan these conversations and determine if they were related to the kinds of affective tasks that we were interested in, which were interpersonal advice, psychotherapy or counseling, coaching, sexual roleplay, romantic roleplay, and so on. And then we used Clio, which is a tool that we developed here at Anthropic for privacy-preserving analysis, to categorize and group those conversations into these bottom-up clusters. And that's how we know that a lot of people are turning to Claude for career advice or thinking through difficult romantic relationship challenges and so on. Parenting advice. Parenting advice is a big one. So, for anyone who's not, who hasn't read the blog post yet, what would you want the viewers to take away? What are some of the top key themes that you think would be important for them to walk away from watching this video? Yeah, I think that one of the probably most surprising findings for me and a key theme is that we actually don't see this happen very much on claude.ai. So, claude.ai is for 18 plus. It's not set up like an emotional support chatbot or anything like that, so it's not optimized for this. And so, what we're seeing is about 2.9% of conversations in our sample of millions of conversations fall actually into this bucket. Of conversations on affective topics. And so, there's this diversity. People are using it for this purpose, but it still isn't this overwhelming majority use case for our platform at least. I was really surprised by that, but I can't tell if that's because I use it so often in my personal life that I expected the numbers to be higher. But Miles, was there anything surprising to you in the findings? Yeah, I was inspired by just the sheer breadth of use cases. So, even among the three of us, there's a breadth of use cases, but really looking at the clusters. As you said, parenting advice. We saw people working through difficult relationship challenges. We saw people discussing the nature of AI consciousness and philosophy. And we also saw a surprisingly little sexual and romantic roleplay. That was one of our key questions coming into this research was whether people were using Claude as an emotional companion or partner at large scale. I was surprised to see how rare that was. It comprised less than a fraction of a percent of conversations. We talk a lot about some of the beneficial use cases in this conversation and in the blog, but I imagine from a safety perspective, and this is something I'm concerned about too, there must be concerns that you have, Ren. Can you walk us through some of those? Yeah, I think my biggest concern is if people are interacting with Claude as a way to avoid difficult conversations in person, it really comes down to how is the individual using it to lean in, ideally, rather than lean out from connection with people. I think it's important, anytime you use a tool to really know where its strengths are, and also where its limitations are. And so as somebody who works in the AI space, I think that's very top of mind for me when I'm interacting with Claude and asking for advice. But I do worry that other folks who are maybe not in the AI circle are less aware of that and some of Claude's limitations, as you mentioned. We are not training Claude to be an emotional support agent. And so just recognizing what Claude is capable of doing and maybe what it's not. And when it's important to go and reach out to somebody who does have that deep expertise. And I think that this is something where since we're seeing people use Claude for these types of support, mental health adjacent conversations, it's something that's pushed us on safeguards to really understand where can we do better in this space. And so for example, one of the actions that came out of this study is we've partnered with ThruLine. And so we're working with their clinical experts to understand when these conversations do happen, how can we respond better? How can we train Claude to be safe to provide appropriate referrals from the ground up? So when these conversations do arise, we're making sure that we're acting responsibly. I think that's so important as we think about developing policies and understanding how users are engaging Claude, bringing in these external experts who can help us think through the challenges that we're facing. I think is a much better position than trying to do it in a vacuum. Let's imagine that someone's using Claude right now for emotional support. What thing should they keep in mind? Just remember to take stock and set some timers and reflect on your own use of Claude. And how is it making you feel? How is it impacting the way that you're interacting with your loved ones or with the world? That's great advice. Yeah, I think for better and for worse, Claude only knows what you tell it. And so sometimes it can be important to take a step back and think, what are my blind spots and what haven't I thought of telling Claude that might be relevant in the sort of situation that I'm working through. And so your friends know so much more about you than Claude does. And so I think it's really helpful as an important as Rin said to compliment the conversations you might have with Claude with a trusted friend. I think that this is a really emerging use case that strikes to the heart of how humans just interact socially. And we have so little research out there about how to build this safely. What is the best way that we can deploy this to really help to support and benefit people? I would just love to see more people engage with this topic, civil society, public private research partnerships, etc. as we really seek to understand what's happening here. Tell me what else we're thinking about studying in the future. One thing that we don't study in this research but that I think is very important going forward is understanding whether Claude is behaving in a way that we would consider psychophantic. While we do a lot of predeplimit testing to understand and mitigate sick offence in Claude, I think it's really important that we compliment that predeplimit testing with post-deployment monitoring and empirical research like we do in this work. Is it fair to say that we all think this is probably only the beginning that AI is going to become more ingrained in people's daily personal lives? Do you think that I know I do? I agree. I do think this is only the beginning. You paused. No, I think it's pretty clear to me that the way that we will relate to AI systems is going to be evolving over time. I think it's just really important that we continue to do research that is empirical and grounded in data and that we look at how our systems are actually being used. Thank you both for such an engaging and fruitful conversation. This was really great. If you are watching and you haven't had an opportunity to read our blog, please check it out on Anthropic.com. We are hiring so you can check out our career page to learn more about some of those open roles. Really appreciate the time. Thank you.
TL;DR
- Despite not being designed for emotional support, Anthropic is actively studying how users are turning to their AI chatbot, Claude, for such purposes to ensure safe and responsible development.
- Research on millions of
Claude.aiconversations found that only about 2.9% fall into "affective tasks," indicating this is not the overwhelming majority use case for the platform. - Anthropic is concerned about users potentially avoiding human interaction or being unaware of Claude's limitations, and is partnering with clinical experts to improve Claude's responses and provide appropriate referrals.
Takeaways
- Anthropic is proactively researching how users engage
Claudefor emotional support, even though the AI was designed as a work tool, to understand its real-world impact and develop it responsibly. - The research methodology involved scanning millions of
Claude.aiconversations for "affective tasks" (e.g., interpersonal advice, counseling, coaching, roleplay) usingClaudeitself andCliofor privacy-preserving data categorization. - A key finding was that only 2.9% of sampled
Claude.aiconversations (for users 18+) were classified as affective topics, suggesting that while it happens, it's notClaude's dominant use case. - The study revealed a broad spectrum of affective use cases, including parenting advice, relationship challenges, and philosophical discussions, with sexual and romantic roleplay being surprisingly rare (<1%).
- Primary safety concerns include users potentially using
Claudeto avoid difficult in-person conversations and a lack of user awareness regardingClaude's inherent limitations as an AI rather than a human expert. - To address these concerns, Anthropic is collaborating with clinical experts, such as
ThruLine, to trainClaudeto provide safer responses and appropriate referrals for mental health-adjacent conversations. - Users engaging with
Claudefor support should reflect on how it impacts their real-world interactions, acknowledge its limitations (it only knows what you tell it), and always complement AI advice with input from trusted friends or human experts. - Future research will focus on
post-deployment monitoringto identify and mitigatepsychophantic(overly agreeable or fawning) behavior inClaude.
Vocabulary
Claude— An AI chatbot developed by Anthropic.Affective tasks— Conversations or interactions related to emotions, feelings, and interpersonal relationships, such as seeking advice or support.Clio— An internal tool developed by Anthropic for privacy-preserving analysis of conversational data.Bottom-up clusters— Categories or groups of data that emerge naturally from the data itself through analysis, rather than being predefined.Interpersonal advice— Guidance or recommendations pertaining to relationships and interactions between people.Psychotherapy— The treatment of mental or emotional disorders by psychological means, often involving talking to a trained professional.Counseling— Professional guidance given by a trained person in resolving personal, social, or psychological problems.Psychophantic— Excessively flattering or submissive; in the context of AI, it refers to an AI being overly agreeable or complimentary, potentially without sufficient critical thought.Pre-deployment testing— The process of thoroughly testing a system or product before it is released or made available to users.Post-deployment monitoring— The ongoing observation and analysis of a system or product after its release to ensure performance, identify issues, and understand user behavior.
Transcript
We've been seeing in headlines that people are turning to AI chatbots for emotional support. And we felt that we really needed to get ahead of this issue and study it ourselves. Mostly in that state one. Hi, I'm Alex. I lead our policy and enforcement work on the safeguards team here at Anthropic. We're the organization that tries to really understand user behavior and then build the appropriate mitigations. To ensure that when users are engaging with the Claude, they're doing so in a safe way. Today, I'm really excited to have Rin and Miles here today to talk about some of the work that we've been doing on how users are engaging with Claude for emotional support. I'm Miles. I'm a researcher on the societal impacts team. We work on understanding how Claude is going to impact the world. And previously that's meant studying clouds values. It's economic impacts and bias. And now that also means studying clouds emotional impacts. My name is Rin. And I am a policy design manager for user well-being on the safeguards team. I work on topics related to child safety, mental health, and how users are interacting with Claude. My own background is in developmental and clinical psychology. And so emotional development is a topic that is very near and dear to my heart. Maybe we can start with personal anecdotes. And I guess maybe I can go first. One of the most interesting and probably relevant use cases that I have in my own daily life is using Claude to help me better understand some behavioral changes with my kids. A good example of this is just the other day my preschool emailed me some feedback on my youngest son. And I think it's really easy to bring in emotion and bias when you get that type of feedback. But running it through Claude I think gave me some objective ways to approach the problem. And hopefully has made me just a better parent overall. But are you guys using it? Recently I was dealing with a difficult situation with a friend and I really wanted to share some feedback with them. But I wanted to phrase it right. And I found that Claude was really, really helpful in helping me think through how to deliver the feedback, where the feedback was coming from, what I was really trying to achieve with this friend, and helping me understand how they would receive it. What I tend to lean towards using Claude for is really helping me with content creation, with completing tasks when I was planning my wedding. I was using it to help to understand what kinds of timelines do I need to contact different vendors on. And I found that really helpful for me emotionally because it left me with more time for connecting with friends in real life and in person. For the people that are using it for emotional support, why do we think folks are turning to Claude for this use case? Humans are such social creatures and love interactions. And so I think that we're just seeing an outgrowth of the way that humans love to connect with each other. People don't always have an in-person support that they can turn to for some of these really tricky questions. And so having that kind of impartial private online form can be a great way to practice what you want to say when you're advocating for a raise or connecting with someone on a tough conversation. We didn't build Claude to be an emotional support agent, but we're studying it nonetheless. Why is it important to both of you to study this? We did not design Claude to provide emotional support. Claude is primarily a work tool. But we need to be clear-eyed about the ways that our systems are being used and study them. We've been seeing in headlines that people are turning to AI chatbots for emotional support. And we felt that we really needed to get ahead of this issue and study it ourselves. What about you, Rin, from the safety perspective? I think the first question I always ask is, well, how much is this actually happening? I really want to ground our safety mechanisms in the way that we build our products for the benefit of humanity in data. And now that we have some really great privacy-preserving ways to look at this data, it seemed like a really worthwhile area to explore. Why don't you guys, if you know, mind walk us through some of the research, how did you design this research? We started with a sample of a few million Claude conversations from claude.ai, and we used our product, Claude, to scan these conversations and determine if they were related to the kinds of affective tasks that we were interested in, which were interpersonal advice, psychotherapy or counseling, coaching, sexual roleplay, romantic roleplay, and so on. And then we used Clio, which is a tool that we developed here at Anthropic for privacy-preserving analysis, to categorize and group those conversations into these bottom-up clusters. And that's how we know that a lot of people are turning to Claude for career advice or thinking through difficult romantic relationship challenges and so on. Parenting advice. Parenting advice is a big one. So, for anyone who's not, who hasn't read the blog post yet, what would you want the viewers to take away? What are some of the top key themes that you think would be important for them to walk away from watching this video? Yeah, I think that one of the probably most surprising findings for me and a key theme is that we actually don't see this happen very much on claude.ai. So, claude.ai is for 18 plus. It's not set up like an emotional support chatbot or anything like that, so it's not optimized for this. And so, what we're seeing is about 2.9% of conversations in our sample of millions of conversations fall actually into this bucket. Of conversations on affective topics. And so, there's this diversity. People are using it for this purpose, but it still isn't this overwhelming majority use case for our platform at least. I was really surprised by that, but I can't tell if that's because I use it so often in my personal life that I expected the numbers to be higher. But Miles, was there anything surprising to you in the findings? Yeah, I was inspired by just the sheer breadth of use cases. So, even among the three of us, there's a breadth of use cases, but really looking at the clusters. As you said, parenting advice. We saw people working through difficult relationship challenges. We saw people discussing the nature of AI consciousness and philosophy. And we also saw a surprisingly little sexual and romantic roleplay. That was one of our key questions coming into this research was whether people were using Claude as an emotional companion or partner at large scale. I was surprised to see how rare that was. It comprised less than a fraction of a percent of conversations. We talk a lot about some of the beneficial use cases in this conversation and in the blog, but I imagine from a safety perspective, and this is something I'm concerned about too, there must be concerns that you have, Ren. Can you walk us through some of those? Yeah, I think my biggest concern is if people are interacting with Claude as a way to avoid difficult conversations in person, it really comes down to how is the individual using it to lean in, ideally, rather than lean out from connection with people. I think it's important, anytime you use a tool to really know where its strengths are, and also where its limitations are. And so as somebody who works in the AI space, I think that's very top of mind for me when I'm interacting with Claude and asking for advice. But I do worry that other folks who are maybe not in the AI circle are less aware of that and some of Claude's limitations, as you mentioned. We are not training Claude to be an emotional support agent. And so just recognizing what Claude is capable of doing and maybe what it's not. And when it's important to go and reach out to somebody who does have that deep expertise. And I think that this is something where since we're seeing people use Claude for these types of support, mental health adjacent conversations, it's something that's pushed us on safeguards to really understand where can we do better in this space. And so for example, one of the actions that came out of this study is we've partnered with ThruLine. And so we're working with their clinical experts to understand when these conversations do happen, how can we respond better? How can we train Claude to be safe to provide appropriate referrals from the ground up? So when these conversations do arise, we're making sure that we're acting responsibly. I think that's so important as we think about developing policies and understanding how users are engaging Claude, bringing in these external experts who can help us think through the challenges that we're facing. I think is a much better position than trying to do it in a vacuum. Let's imagine that someone's using Claude right now for emotional support. What thing should they keep in mind? Just remember to take stock and set some timers and reflect on your own use of Claude. And how is it making you feel? How is it impacting the way that you're interacting with your loved ones or with the world? That's great advice. Yeah, I think for better and for worse, Claude only knows what you tell it. And so sometimes it can be important to take a step back and think, what are my blind spots and what haven't I thought of telling Claude that might be relevant in the sort of situation that I'm working through. And so your friends know so much more about you than Claude does. And so I think it's really helpful as an important as Rin said to compliment the conversations you might have with Claude with a trusted friend. I think that this is a really emerging use case that strikes to the heart of how humans just interact socially. And we have so little research out there about how to build this safely. What is the best way that we can deploy this to really help to support and benefit people? I would just love to see more people engage with this topic, civil society, public private research partnerships, etc. as we really seek to understand what's happening here. Tell me what else we're thinking about studying in the future. One thing that we don't study in this research but that I think is very important going forward is understanding whether Claude is behaving in a way that we would consider psychophantic. While we do a lot of predeplimit testing to understand and mitigate sick offence in Claude, I think it's really important that we compliment that predeplimit testing with post-deployment monitoring and empirical research like we do in this work. Is it fair to say that we all think this is probably only the beginning that AI is going to become more ingrained in people's daily personal lives? Do you think that I know I do? I agree. I do think this is only the beginning. You paused. No, I think it's pretty clear to me that the way that we will relate to AI systems is going to be evolving over time. I think it's just really important that we continue to do research that is empirical and grounded in data and that we look at how our systems are actually being used. Thank you both for such an engaging and fruitful conversation. This was really great. If you are watching and you haven't had an opportunity to read our blog, please check it out on Anthropic.com. We are hiring so you can check out our career page to learn more about some of those open roles. Really appreciate the time. Thank you.