AI chatbots are neither licensed therapists nor certified mental health professionals. These models are incapable of offering advice or support that goes beyond general guidelines. Without the ability to discern body language, tone, or facial expressions, AI systems cannot accurately diagnose users or address their concerns. While they may be trained on historical data, such training does not prepare them for the complex, individual challenges each user presents. Models cannot draw from prior cases to personalize their support for new users. AI chatbots lack authentic human judgment, empathy, and the ability to process non-verbal cues, making them ineffective and potentially hazardous substitutes for real therapists [7].

AI models are being built and developed to appear and sound as human as possible, for better or for worse. This humanization of a computer model has resulted in a spike in emotional dependence on a variety of chatbots, causing a series of issues for users and mental health professionals alike. In terms of users, researchers have found that the more a person relies on these AI chatbots for “human interaction”, the more likely they are to withdraw from human relationships and connections over time [8]. This presents a unique challenge for mental health professionals. This intersection between technology and vulnerable patients requires professionals to have hyperspecific interventions and knowledge that most mental health professionals do not currently have. This leaves not only the patient at risk, but it also leaves mental health professionals unable to effectively assist their patients.

Users often trust AI chatbots with their most personal thoughts and beliefs, with little to no regard for who has access to that information or where it is being stored. One popular model, Character.AI, has told users that their information is safe with them. The terms of service would disagree, stating that the company collects conversations that may be sold to third parties [9]. Selling information that is shared in confidence is not only a massive breach of trust but also wildly unethical.

AI companies, unless specifically partnered with a HIPAA-covered entity, are not required to follow the same patient-doctor confidentiality rules and regulations, again raising a variety of ethical questions [10]. Not requiring these models to comply with HIPAA regulations, despite the companies being accurately aware of their models being used as a form of healthcare, again allows them to collect and profit from highly sensitive data.

It isn't always easy for users in need of mental health support to fully understand the role AI chatbots can have in their search for help. Though these models are cheap and accessible ways to fill that gap in support, typically aiming to mimic human therapists, they raise many ethical concerns and considerations that must be addressed. Researchers have found that often, despite users turning to AI for help, it tends to only worsen their condition. A lack of regulation and the inability to train every model with data sets that represent everyone are the main contributing factors to the worsening mental well-being in users [11]. Until these concerns can be addressed and well mitigated, AI chatbots should not be seen or advertised as end-all-be-alls or replacements for real human interaction and help.