Are We Training Our Replacements?
The AI isn’t documenting the therapy. It’s delivering it.
That’s not a prediction about where AI might be headed.
An AI behavioral health company called Limbic has developed Unpacked, a voice AI agent that delivers structured CBT to people with depression and anxiety through 20-minute phone sessions, with clinician oversight.
And now it’s being tested with Medicare beneficiaries.
The FDA selected Unpacked for a pilot connected to a Medicare program run by the Centers for Medicare & Medicaid Services (CMS). The program is testing new technology designed to expand care for people with chronic health conditions.
Meanwhile, therapists are increasingly inviting AI into the therapy room.
We let it listen to sessions. Transcribe conversations. Summarize clinical information. Write progress notes. Assist with intake. Identify patterns.
Some of these tools are genuinely useful. I use AI myself.
But while we’ve been asking, How much time can AI save me? there’s another question we need to be asking:
What are we giving these companies while they’re helping us?
Your Clinical Data Is Valuable
Think about how much happens during a therapy session.
We assess, question, reflect, redirect, interpret, challenge, validate, educate, evaluate risk, choose an intervention, watch the client’s response, and decide what to do next.
That’s an enormous amount of information about how to do therapy.
Collect it across thousands or millions of clinical interactions and you have more than information about clients.
You have information about therapists.
How we think. What we ask. Which interventions we choose. How clients respond. What we do when something doesn’t work. How we make clinical decisions.
That’s valuable information if you’re developing technology capable of doing some of the same things.
Limbic offers an interesting example.
In research published in Nature Medicine, Limbic tested AI designed to conduct CBT interactions. Twenty-two expert clinicians evaluated therapy conversations generated by AI agents and human clinicians. The researchers also evaluated the technology in real-world use by analyzing almost 20,000 therapeutic conversation transcripts involving nearly 9,000 users.
Those transcripts were used to evaluate the technology. The published research doesn’t tell us that they were the data originally used to train the AI.
But Limbic’s Terms of Use tell us something else that’s important.
Users retain ownership of their data. Limbic, however, may create an anonymized copy, meaning it can no longer be connected to an identifiable person, own the intellectual property rights in that copy, and use it for commercial purposes, including training a model used by its technology.
Put those pieces together and the concern becomes much bigger than one company or one AI therapist.
Clinical interactions have become a resource for developing, testing, refining, and training AI.
At the same time, behavioral health companies are moving beyond technology that helps clinicians provide therapy.
They’re developing technology that provides therapy.
That should get our attention.
“It’s HIPAA Compliant” Isn’t Enough
When therapists evaluate technology, we tend to ask one big question:
Is it HIPAA compliant?
That’s an important question, but it’s just the first of many.
A company may be permitted to anonymize or de-identify information and use it in ways that have nothing to do with treating your particular client.
So before you allow an AI scribe, EHR, documentation program, therapy platform, or other technology access to your clinical work, ask some additional questions:
- Are session recordings or transcripts retained?
- Can my data or my clients’ data be anonymized or de-identified and used for research?
- Can it be used to develop or improve products?
- Can it be used to train AI?
- Can I opt out?
- Is information shared with affiliates, vendors, or other companies?
- Who owns information created from the data?
- What happens to the data when I close my account?
Then read the Terms of Use and Privacy Policy.
Nothing says Saturday night like curling up with 14 pages of Terms and Conditions.
Read them anyway.
Or, in a fitting use of the technology, give them to AI and ask it to identify anything that allows the company to use your data for AI training, research, product development, or other commercial purposes.
The unsettling part is that a company doesn’t necessarily have to hide what it’s doing. These uses may be perfectly legal, plainly disclosed in the Terms of Use, and agreed to by the therapist who never read what they clicked.
Sometimes all it takes is clicking I Agree to become both the customer and the raw material.
Use AI. Just Know What You’re Feeding It.
Look at the company, too.
Who owns it?
Who funds it?
What other products is it developing?
How does its leadership talk about therapists and the future of mental healthcare?
Is the company’s goal to make clinicians’ jobs easier, expand access to clinicians, reduce the need for clinicians, or some combination of all three?
You may not find every answer. Looking is still worthwhile.
Find Out Who You’re Feeding
Because the same industry selling therapists technology to help us do therapy is now developing technology that provides therapy.
Which brings us back to Limbic.
The FDA hasn’t approved Unpacked. Its effectiveness for this intended use hasn’t yet been evaluated by the FDA. The pilot allows the technology to be used with Medicare beneficiaries while real-world data about its performance are collected.
And Unpacked isn’t intended for everyone. The FDA listing includes an extensive list of contraindications, including suicidality, psychosis, bipolar disorders, personality disorders, primary substance use disorders, and eating disorders.
Which raises some very therapist-like questions: How are those conditions identified before the AI starts providing the therapy? What happens if one emerges during treatment? And if someone is determined to be inappropriate for AI-delivered therapy, what prevents them from continuing to use it?
Assessment and differential diagnosis are part of the clinical work. So is recognizing when the person in front of you doesn’t fit neatly into the diagnosis or treatment you started with.
The AI therapist is no longer hypothetical.
The questions surrounding it aren’t either.
Know What You’re Agreeing To
I’m not anti-AI.
I use it. I think it has enormous potential to reduce administrative burden, make valuable contributions to documentation, and give therapists back time we’d much rather spend doing almost anything other than writing progress notes.
But free, cheap, and convenient technology usually has a business model somewhere.
Before giving a company access to your therapy sessions, clinical records, documentation, or clinical thinking, find out what you’re getting.
Then find out what they’re getting.
The information generated in psychotherapy doesn’t simply describe therapy.
It can help teach technology how therapy is done.
And technology is already being tested to do it.
That’s a pretty good reason to know what we’re feeding it.
If you’re using AI in your practice, my AI Disclosure includes a guide to vetting AI companies, with questions to ask about privacy, security, data use, and what happens to your information after you hand it over.
Ready to know what you’re agreeing to before you hand over your clinical work?
My AI Disclosure & Safety Checklist walks you through disclosing AI use to clients and vetting any AI tool before it touches your practice. Because you can’t protect your clients, or yourself, from something you never read the fine print on.
Beth Rontal, LICSW, a private practice therapist and the Documentation Wizard® is a nationally recognized consultant on mental health documentation. Her Misery and Mastery® trainings and accompanying Essential Clinical Forms help therapists create clear, clinically meaningful documentation that captures the thinking behind their clinical decisions without unnecessary detail. Beth’s Documentation Wizard training program helps clinicians turn their clinical skill and intuition into a systematic review of treatment that helps to pass audits, protect income, maintain professional standards of care, reduce documentation anxiety and increase self-confidence. Beth’s forms have been approved by 2 attorneys, a bioethicist, and a billing expert and have been used all over the world. She mastered her teaching skills with thousands of hours supervising and training both seasoned professionals and interns when supervising at an agency for 11 years. Her newest initiative, Membership Circle, is designed to empower psychotherapists to master documentation with expert guidance, efficient strategies, and a supportive community.
