AWS HealthScribe Uses Generative AI And Real-Time Doctor-Patient Conversations To Update Medical Charts

Generative AI (GenAI) has demonstrated profound usefulness in almost every industry segment. Amazon Web Services recently announced several AI-driven products, which I will cover in future articles. In this article, I focus on an AWS application called HealthScribe, a HIPAA-eligible service that combines speech recognition and generative AI. AWS is now releasing HeathScribe as a preview product in the eastern U.S., and there is no information yet on its general availability.

HealthScribe’s target market is large healthcare organizations and related software vendors. It automatically creates preliminary clinical notes using natural language processing and generative AI to analyze conversations between patients and clinicians.

HealthScribe can improve the quality, efficiency and accuracy of medical charting, a tedious and predominantly manual process that documents detailed information about each patient visit. Charting is an unpopular and time-consuming task in medical practices, in part because it reduces face-to-face time with patients. Nevertheless, it is essential for capturing historical patient interactions and administering quality programs and insurance billing.

Benefits of HealthScribe for medical charting

An illustrative example of the application experience that healthcare developers can provide users ... [+] with AWS HealthScribe.

HealthScribe uses generative AI techniques to manage and transform healthcare data. Considering HealthScribe’s potential to replace a massive number of manual charting hours across the industry, it could be one of the most promising and useful medical applications this year for various healthcare providers, payers, IT vendors and software developers. Its use in medical practices could enhance data-driven decision making, increase collaboration, reduce administrative labor and decrease the cost of care.

The graphic above shows how HealthScribe summarizes clinical notes from a conversation between a patient and a clinician. It also separates summaries into sections such as chief complaint, history of present illness, exam and treatment plan.

A helpful feature is its ability to extract specific medical terms, medical conditions, medications and treatments from the conversation. Medical terms that suggest reading topics can be included in a drop-down form for custom applications if desired.

The future of HealthScribe

Large medical organizations with thousands of medical practitioners will see significant productivity gains by using HealthScribe. Healthcare software vendors such as 3M, ScribeEMR and Babylon are already putting AWS HealthScribe to use in their clinical applications.

As interest in GenAI grows, software vendors that build high-performance applications for clinical settings—things like telehealth applications or centralized electronic health record (EHR) platforms—are considering how best to leverage HealthScribe. Integrating HealthScribe into clinical software will also help software vendors use AI responsibly. HealthScribe references each original patient transcript to its AI-generated sentence in the clinical notes, making it easier for users to validate the accuracy of summaries.

Wrapping up

AWS HealthScribe can potentially reduce the number of human hours associated with charting patients’ visits. Besides saving the healthcare industry a staggering amount of time in charting, its automated features promise to streamline patient care and improve clinical workflows for medical practitioners.

As technology advances and more healthcare software vendors integrate HealthScribe into packaged solutions, it should have a positive impact on individual medical practices and the healthcare system as a whole. Tedious documentation tasks will be minimized and physicians will be able to focus their attention where it matters most—caring for patients.

Here are the overall benefits of HealthScribe as I see them:

Moor Insights & Strategy provides or has provided paid services to technology companies like all research and tech industry analyst firms. These services include research, analysis, advising, consulting, benchmarking, acquisition matchmaking, and video and speaking sponsorships. The company has had or currently has paid business relationships with 8×8, Accenture, A10 Networks, Advanced Micro Devices, Amazon, Amazon Web Services, Ambient Scientific, Ampere Computing, Anuta Networks, Applied Brain Research, Applied Micro, Apstra, Arm, Aruba Networks (now HPE), Atom Computing, AT&T, Aura, Automation Anywhere, AWS, A-10 Strategies, Bitfusion, Blaize, Box, Broadcom, C3.AI, Calix, Cadence Systems, Campfire, Cisco Systems, Clear Software, Cloudera, Clumio, Cohesity, Cognitive Systems, CompuCom, Cradlepoint, CyberArk, Dell, Dell EMC, Dell Technologies, Diablo Technologies, Dialogue Group, Digital Optics, Dreamium Labs, D-Wave, Echelon, Ericsson, Extreme Networks, Five9, Flex, Foundries.io, Foxconn, Frame (now VMware), Fujitsu, Gen Z Consortium, Glue Networks, GlobalFoundries, Revolve (now Google), Google Cloud, Graphcore, Groq, Hiregenics, Hotwire Global, HP Inc., Hewlett Packard Enterprise, Honeywell, Huawei Technologies, HYCU, IBM, Infinidat, Infoblox, Infosys, Inseego, IonQ, IonVR, Inseego, Infosys, Infiot, Intel, Interdigital, Jabil Circuit, Juniper Networks, Keysight, Konica Minolta, Lattice Semiconductor, Lenovo, Linux Foundation, Lightbits Labs, LogicMonitor, LoRa Alliance, Luminar, MapBox, Marvell Technology, Mavenir, Marseille Inc, Mayfair Equity, Meraki (Cisco), Merck KGaA, Mesophere, Micron Technology, Microsoft, MiTEL, Mojo Networks, MongoDB, Multefire Alliance, National Instruments, Neat, NetApp, Nightwatch, NOKIA, Nortek, Novumind, NVIDIA, Nutanix, Nuvia (now Qualcomm), NXP, onsemi, ONUG, OpenStack Foundation, Oracle, Palo Alto Networks, Panasas, Peraso, Pexip, Pixelworks, Plume Design, PlusAI, Poly (formerly Plantronics), Portworx, Pure Storage, Qualcomm, Quantinuum, Rackspace, Rambus, Rayvolt E-Bikes, Red Hat, Renesas, Residio, Samsung Electronics, Samsung Semi, SAP, SAS, Scale Computing, Schneider Electric, SiFive, Silver Peak (now Aruba-HPE), SkyWorks, SONY Optical Storage, Splunk, Springpath (now Cisco), Spirent, Splunk, Sprint (now T-Mobile), Stratus Technologies, Symantec, Synaptics, Syniverse, Synopsys, Tanium, Telesign,TE Connectivity, TensTorrent, Tobii Technology, Teradata,T-Mobile, Treasure Data, Twitter, Unity Technologies, UiPath, Verizon Communications, VAST Data, Ventana Micro Systems, Vidyo, VMware, Wave Computing, Wellsmith, Xilinx, Zayo, Zebra, Zededa, Zendesk, Zoho, Zoom, and Zscaler. Moor Insights & Strategy founder, CEO, and Chief Analyst Patrick Moorhead is an investor in dMY Technology Group Inc. VI, Fivestone Partners, Frore Systems, Groq, MemryX, Movandi, and Ventana Micro., MemryX, Movandi, and Ventana Micro.

AWS’ head of health AI on Amazon’s new clinical documentation play

Amazon Web Services generated significant buzz earlier this week when it announced HealthScribe, its own clinical documentation tool that uses generative artificial intelligence to summarize doctor-patient visits.

With the launch, Amazon joined a number of other tech companies including Microsoft in the space, jockeying for a share of the market as providers look for ways to cut documentation burden on physicians.

In an interview on the sidelines of the AWS Summit on Wednesday, AWS Head of Health AI Tehsin Syed told Healthcare Dive what differentiates HealthScribe from other medical notetakers, how the technology is expected to drive client demand and how Amazon is approaching pernicious accuracy, ethics and liability issues around generative AI in healthcare.

Editor’s note: This interview has been edited for clarity and brevity.

TEHSIN SYED: HealthScribe transcribes the conversation between a doctor and a patient. It identifies the speakers and segments the transcript. If you start with some chitchat, it’ll recognize, okay, this is not relevant to the clinical note. It extracts specific medical information — things like medications, tests and references to anatomy — to take those as structured elements. This is where the generative AI capability comes in. That information, and the large language model we’ve trained, is used to generate summary documentation, in the shape of chief complaint, history of present illness, exam and plan.

So we take a Titan model — that’s an Amazon first-party model — and we do what’s called grounding. We adapt the model to this specific purpose of learning how to generate this kind of documentation.

Finally, we show the end user the transcript and the summary, and cite conclusions in the summary back to the dialogue. So it’s a really natural and easy review process for clinicians.

SYED: HealthScribe is an API that Amazon is selling to its partners for them to provide an end user experience. That’s a very different business model. We’re saying, you don’t have to have all the machine learning expertise and all of the infrastructure that’s needed to manage a large language model.

AWS head of health AI Tehsin Syed

Permission granted by AWS

AWS is going to do that part of the work, and maintain the security and privacy of the data. We provide this capability to anyone who wants to use it, and we have public pricing based on usage, so it’s clear what’s going on.

What we’re trying to do is say, whoever you are, whatever software builder you are, you get to market much faster with this capability. You can concentrate on the end user experience.

There’s also innovation in the technology itself. Some of the grounding techniques are patent-pending. Underlying HealthScribe itself — at a high level everyone’s using an LLM and whatever — but part of this is the expertise that we bring.

SYED: There’s a number of others who are interested. I absolutely anticipate new customers coming to the table who previously didn’t have the ability to do this [work] themselves. We’ve seen a lot of interest from software vendors, because we’re providing an API and you do have to have some expertise to build. When we talk to provider systems, they’re interested, and they usually ask who’s a good partner, and how do we work with them to incorporate this.

SYED: The best way to show accuracy and build trust in the system is the method of showing end users what in the summary came from the dialogue. I don’t think showing them an accuracy stat is going to help as much.

Internally, obviously, we use a number of approaches. Some are about factual completeness — is what was in the dialogue reflected in the summary? Did the model come up with something that wasn’t in the dialogue? There’s a couple other metrics we track internally but it’s not shared externally.

SYED: I think we’re directly addressing the concern about quality by showing the user every time why a decision was made.

SYED: This is part of why we’re announcing in preview, and we’re working with our partners to understand what’s necessary to do here. I don’t have a definitive answer for you. There’s a lot of complexity to it, right? You have an ambient listening device, but maybe it’s too far away, or the air conditioner’s too loud. It’s not as simple as HealthScribe sharing accuracy numbers, because there could be an issue with the recorded conversation that we’re getting.

I agree that there’s a lot of think about in doing this responsibly.

What we’re saying is here’s what we can do, and as HealthScribe is deployed in the workflow there are other things that our customers have to do to ensure it’s the right level of accuracy. We are very clear that it’s generating the summary but it is not automating the documentation. The idea is, this is a tool, and then the tool is applied in a workflow. And the workflow has a lot of those checks and balances in it.

SYED: The way we look at that is, we’re providing a capability that has to be incorporated in an end-to-end workflow, and that workflow has to deal with those aspects you’re talking about. It’s the vendor who’s incorporating the capability that has to make sure that workflow is appropriate, and the clinician also has to make sure.

I worked at Cerner a long time before this. The dictation thing has been around for a very long time. It’s the same thing, right? Is it accurate or not? So folks have mechanisms in place to handle the workflow around this. But across the board with AI, there’s lots of new issues.

SYED: I don’t have an opinion on moving too quickly or not quickly enough. We work with our customers on very specific use cases. With HealthScribe, we’re iterating during this preview period to make sure it is actually fit for its purpose, and it meets criteria that would result in real use in a responsible way.

AWS is ready to power AI agents that can handle busywork instead of just chatting

By Emilia David, a reporter who covers AI. Prior to joining The Verge, she covered the intersection between technology, finance, and the economy.

During the AWS Summit in New York on Wednesday, Amazon launched Agents for Bedrock, which will let companies build AI apps that can automatically do tasks on their own, like booking a flight for users instead of just telling them about it. AI agents are the assistant that actually gets a restaurant reservation instead of just giving suggestions on where to eat.

“I believe this will supercharge developers who wanted an easier way to build agents and at the same time customize the data the models read,” Swami Sivasubramanian, vice president of data and machine learning at AWS, tells The Verge. “Building agents took so much time even with how advanced generative AI is now, but we’re making it so developers can access exactly the models they need.”

Generative AI models like GPT-4 or Llama 2 are powerful but do not actually automate some tasks for users without extra help, like plug-ins. With agents, a travel company could use generative AI to suggest trips, then build one agent that will take in a user’s travel history and interests, another agent that finds flight schedules, and finally, an agent that books the chosen flight.

Amazon, along with other AI companies like Meta, Google, and OpenAI, signed an agreement with the White House last week pledging to develop responsible AI. 

Amazon itself hasn’t been a big player in the generative AI arms race in the way Meta, Google, OpenAI, and Microsoft have been. Still, it has positioned itself, via AWS, as a key infrastructure provider to the space. StabilityAI and Anthropic released new versions of their large language models, Stable Diffusion XL 1.0 and the chatbot Claude 2, that are now available on Bedrock. 

Agents for Bedrock is aimed at startups and enterprises, letting companies use their own data to teach foundational models, like image-to-text models or large language models, and then build additional apps to finish tasks. A developer can choose which foundational model to use, provide some instructions, and choose which data for the model to read.

Other tech companies are also working on agents; Meta CEO Mark Zuckerberg told investors there’s an opportunity to bring AI agents “to billions of people in ways that will be useful and meaningful.” OpenAI CEO Sam Altman, in a wide-ranging interview with The Atlantic, ruminated on AI agents and how to approach their creation best. 

Bedrock, a platform where organizations can access foundational generative AI models like Stable Diffusion, Claude, Jurassic, and Amazon’s own Titan language, was launched in April this year. 

Other AWS announcements included bringing generative AI to healthcare through a new service called AWS HealthScribe and a partnership with Nvidia allowing AWS to handle larger amounts of memory and data by using Nvidia H100 Tensor Core GPUs.

AWS HealthScribe automates clinical documentation, basically notes summarizing a patient’s complaint, and makes medical notes for diagnosis. AWS partnered with 3M Health Information System to deploy the tool. Sivasubramanian said using generative AI to create clinical documentation saves time for healthcare professionals and lets them interact with more patients. 

Bringing AI to healthcare, though, will be fraught. Healthcare is one of the most regulated spaces in the US. AI companies face questions about privacy protection and security. Data privacy is one of the main focuses of the Federal Trade Commission’s investigation into OpenAI. AWS reiterated that HealthScribe is HIPAA-eligible and will not use results from HealthScribe to train the tool. 

“Healthcare is one of the spaces where AI can dramatically reduce the burden on people,” Sivasubramanian said.

AWS made its machine learning-powered analytics tool, Entity Resolution, that helps people analyze and match documents across applications generally available. AWS also made its code assistant CodeWhisperer available in Glue Studio — a platform that simplifies coding for data analysts — to allow access for data engineers and analysts. It gives code suggestions in real time.