Scribe and Prejudice
The Ada Lovelace Institute in the UK has released a new report looking into the use of AI transcription and note taking services in social care.
The report is based on interviews they conducted with a cohort of social workers from 17 different English and Scottish Local Authorities as well as interviews with senior local authority staff involved in procurement and evaluation to find out how and why they selected the tools.
The picture I got from reading through the report is of a sector that is severely resource constrained, so is looking towards these tools to help bridge some gaps, both to become more efficient and to improve the outcomes of their clients (interestingly, the two aren’t necessarily the same thing apparently).
“Managers shared how the profession faces such acute challenges that it would be worth trying anything to improve the on-the-ground context for social workers. As outlined by one manager: ‘For like 15 to 20 years now it’s been austerity, and freezing, and [doing] more with less and all that kind of stuff, so there is always a productivity push […] if there is a potential for a new technology or new way of working or new service to provide us with efficiencies, then we have to explore them.’” [ Scribe and Prejudice – pg24 ]
Another issue highlighted in the report is the disconnected nature of the adoption of these tools. Each local authority is basically feeling their own way through the process.
“Some local authorities with well-established and resourced evaluation teams have been able to pilot multiple tools with different teams, before moving to official procurement procedures for formal adoption. Others have piloted a single AI transcription tool and adopted it quickly without significant oversight, after an initial pilot phase, to alleviate systemic challenges in social care as soon as possible.” [ Scribe and Prejudice – pg27 ]
This is very different to the way that these tools are treated in the Health sector for instance.
If you want to setup an AI transcriber (or “Ambient Voice Technology” as the NHS refers to it) in a medical setting, then if it’s anything more than JUST a transcriber (ie can the tool offer summaries?) it’s treated as a medical device and regulated as such.
The NHS has developed a detailed Guidance for the use of these tools in medical settings.
There is no such Guidance for the use of AI transcription services in the Social Care sector, leaving each authority to basically make it up as they go along.
Then there are the questions of accuracy and LLM hallucinations. There is no such thing as a perfect transcription tool, they can misidentify words or miss them completely, or it can get it completely wrong and hallucinate a word, words or whole portions.
“‘Any transcription that I’ve looked at from one of these meetings, I am in absolute tears with laughter half the time because I don’t even have one clue as [to] some of the […] what they’re saying. It is so badly transcribed, I find, and I think, again, I think it’s to do with the language and understanding of the dialect.’ – Social worker K” [ Scribe and Prejudice – pg55 ]
As an aside that comment about language and dialect is a huge issue. It’s no use having a state of the art service if it can’t match the words you’re saying in your accent to something in its database.
What’s worse than a bad transcript is a is a hallucinated summary.
“For example, one social worker recounted an instance where, when using an AI transcription tool to create a summary, the tool had incorrectly ‘indicated that there was suicidal ideation’, but ‘at no point did the client actually, you know, talk about suicidal ideation or planning, or anything’."
The social worker emphasised that without appropriate oversight, this error could impact the person’s care significantly: ‘If I hadn’t checked that and it had gone into the case note […] that could have had implications further on down the road’.”
These are issues that are live and need to be addressed properly.
One thing this report makes clear is that we really don’t have a good picture of how these tools are being used, of their impact on the quality of care for the clients of the social care services. There needs to be a much more structured approach to how this technology is introduced to a sector that is both struggling with a lack of resources, while at the same time dealing with some of the most vulnerable people in our society.
