Q&A: Insurance exec says AI nearly perfect when processing tens of thousands of documents

Nearly a year after rolling out a generative artificial intelligence (genAI) tool to help it process thousands of claims documents, global insurance claims management firm Sedgwick says the technology’s accuracy is nearly perfect.

Sedgwick, which operates in 80 countries, receives about 1.7 million pages of digital claims-related documents a day. The documents then go through an arduous vetting process by examiners who must decide whether they’re valid and how they should be handled.

In April 2023, Sedgwick unveiled a genAI tool called Sidekick to help with document summarization, data classification, and claims analysis. Sidekick uses OpenAI’s GPT-4, giving the company “an unlimited number of large language models to be created for varying purposes.”

In December, Computerworld spoke with the company’s global chief digital officer, Leah Cooper, about the challenges and purposes of the genAI rollout. At that time, Sedgwick’s Sidekick genAI technology had combed through 14,000 documents and was “shockingly good” at accurately spitting out summaries. Five months later, Cooper said more than 50,000 documents have been processed by Sidekick, and those documents have been evaluated by more than 1,000 examiners who reported a 98%-plus accuracy rate in the document summarizations.

Sedgwick

Computerworld revisited the topic with Cooper to ask her what she and the company have learned about genAI and its capabilities for reducing workloads and increasing document processing efficiencies.

Tell me about the kinds of documents you had Sidekick evaluate. Are they all medical-related insurance documents or do they run the gamut? And how long are they? “Medical documents in the workers compensation space were the core focus for our initial pilot, but we have since expanded to other types of documents and photos for validation. The average length of the documents in the testing phase was six to seven pages, but some were much longer, ranging from 25 to 30 pages.”

Tell me a little about Sidekick, how you developed it using OpenAI’s GPT-4, and how it connects to your document management system? “We developed Sidekick so that we can leverage the best of what genAI has to offer, giving our claims professionals an advantage in their daily work.

“If we can drive efficiencies by taking the busywork out of claims administration, and allow our people to focus on taking care of our customers, we can transform not only the process but the experience of having a claim. Our first initiative was to summarize documents that are received in support of a claim. Those were the basics: deploying ChatGPT into the Sedgwick environment so that our data stayed secure, and then evaluating a first use case to see if genAI could be successfully implemented.  

“We are thrilled to say that we did that successfully, incorporating over 50,000 documents during our pilot phase of Sidekick. We’ve just wrapped up our second phase, where we integrated Sidekick technology into our proprietary claims admin systems. This was a huge part of the productivity driver in our business.

“Now, how do we make this tool more relevant to our business? How do we drive productivity, decrease resolution time, and shift from a tactical application to a strategic and conceptual one? This is where we are uniquely positioned: it’s not one tool, it’s several capabilities pulled together to create a scalable, rapidly deployable platform in a unique way. If we combine generative AI with 50 years of understanding and refining how the claims model works and a best-in-class data science program, Sedgwick will pivot into a business model that transforms the claims industry.

Last time we spoke, you told me about 500 employees were using Sidekick. Did that number remain the same and how did they use genAI in their work? “Now we have over 1,100 employees who have used this tool. Examiners are using this technology to summarize claims documents and expedite the entire process.”

What kind of feedback did employees using Sidekick give you – positive and negative (if both)? “Employees are actually asking us to make this product widely available more quickly. People who have used this solution are telling colleagues about the accuracy of summarizations and time saved on claims, fostering a culture of excitement around a new tool which hasn’t existed before.”

How did you determine the 98%+ accuracy rate? “In Sidekick’s pilot phase, we constantly asked for feedback from employees who were evaluating the output results of Sidekick. Examiners would be prompted to say whether the document was successfully summarized or whether something was missing.

“One key to defining a strong AI program is to set the expectation of outputs so users can understand what they are judging as part of this new process. By identifying what examiners are looking for and defining our output results, we were able to set a standard for what is deemed successful and what is not. 

“It’s incredibly important to obtain real feedback from the users who are ‘boots on the ground.’ Individuals who would normally create these summarizations manually were the ones who graded the AI. In the first few months, we did a lot of tweaking based on feedback and it took multiple iterations of prompt engineering to simulate what goes through an examiner’s mind.

“Once we nailed it down and were satisfied with initial testing scores, we rolled it out more widely to 1,100 employees, who ultimately scored Sidekick with the 98%+ accuracy rating. Business partner involvement in determining success is crucial to adoption of the technology. If the people that support these claims are not behind it, companies will not realize a successful engagement with technology.”

Can you explain the time savings and potential productivity increases genAI created? “This technology has created and will continue to learn efficiency gains throughout our organization. We’ve been trying to find a way to automate tasks associated with claims administration that are not complex for business operations, they’re just necessary steps in a process. From my perspective, we want to focus on how we can recognize areas that need lesser attention (e.g., the simpler claims) and allow our examiners to really focus on the other types of claims that could benefit from faster and more collaborative engagement with our customers. If we can take busywork out of claims that need minimal investigation and instead direct adjusters to a claim that needs more attention, clinical resources and time with customers, then we have changed the model for care throughout a difficult time in somebody’s life.”

You receive about 1.7 million claims-related documents every day, so the 50,000 documents handled by genAI is a relatively small percentage. What kind of methodology did you use to ensure these documents were an accurate representative of the whole? “During this rollout, we didn’t select only documents that had certain attributes. We wanted the big picture. In the end, we worked with clients who were anxious to collaborate with us on new technology-forward programs to analyze claims. We worked with clients directly to receive approval to use this new technology during our claims process and ran every document through the generative AI solution once they were on board. This gave us a solid exposure to every type of claim and document, ensuring that genAI was thoroughly vetted for every scenario.”

What are some things you’ve learned from the project? “When we first integrated genAI into our operations, we had to learn quick about how and when to best apply it. GenAI technology is rapidly evolving, it seems like on a daily basis. While this is transformative, it also means that we have a perpetual learning curve and challenge to understand the best application of genAI. It gives us an opportunity to work in the most agile environment imaginable: this is a very exciting, and overwhelming, time for leadership. 

“A great deal of articles and media around this topic have compared genAI to the introduction of the internet, and while they’re a bit different, there are some major similarities. Sedgwick and companies around the world are investing in these tech tools, but the companies that will see the most success are the ones that think critically about how to best leverage this technology. By identifying how genAI best fits into operational models — which can be challenging as much of the tech is developed in a vacuum — it is vital to identify the best use-cases for each tool. As these challenges and opportunities continue, we’re excited to see what new opportunities arise as innovation continues.”

Did you encounter anything unexpected throughout this trial of Sidekick? “We encountered a number of positive takeaways from this trial of Sidekick. Namely, the reliability of the genAI products currently available were much higher than we expected.

“Initial accuracy rates were staggeringly high, and they have proven to be incredibly useful for our claims management teams. In the past, tech tools ‘out of the box’ have not been this effective. That initial success highlights the rapid innovation, which will continue, of artificial intelligence right now and the current and potential use-cases for it across business models and operations.”

What’s the next step in your genAI journey? Will you be instituting a larger rollout, or are you considering other areas of the business for its use and, if so, what are they? “The next steps … involve a focus on transforming workflows through the combination of new tech tools, along with data science, decision engines, and dynamic API outreaches. This combination of tools into a new platform will enable low-touch automation on simple claims like never before. Our operational model and understanding of the industry climate has already set the stage for our ‘must haves.’ We understand that better than anybody out there. 

“However, our latest genAI release lets us recognize, ‘OK, we just learned this from new data or supporting documents. What does that mean to the claim?’ That’s where data science steps in: through our years of best-in-class operations and collaboration with clients, we have a data set that lets us know what happens next. Taking the information that we learned from generative AI, and combining that then with the analytical AI of predictive modeling, we can drive the advancement of a claim and provide prescriptive recommendations to our claims examiners. 

“And that’s an important point here. The goal is never to replace the critical thinking and judgment calls that our people do so well. It’s to inform them with relevant, rapid data so that they can make those decisions even better. To put it simply, we will be able to say, ‘This claim has taught us this, so expect that.’

“We can address the next step in claim lifecycle, and we can think further out to find an optimal path for resolution. Claim resolution time will recede, early adopters will reap benefits of technology enablement more rapidly, and the experience of those who we serve will improve. But this triangle that brings together genAI, data science, and 50 years of knowledge has uniquely positioned us to make that claims model more intelligent and more situation specific than ever before.”

Do you have any tips or recommendations for others considering rolling out genAI? “The AI sector has been funded and developed beyond the knowledge and understanding of how to use these tools, and companies are struggling to find a way to integrate this technology with legacy processes. It’s important to focus on meaningful ways to transform the way companies do business by adding in resources that help shape judgment calls without removing a human from the claims experience. 

“I would say that facilitating a strong partnership between tech and operations is key to figuring out this genAI journey. You have to approach the work as if no processes are sacred, and companies and employees can’t be afraid to leverage AI to find new innovations and efficiencies.

“And for a last tip, I like to call this ‘digital triage,’ in that there cannot be the assumption that there is a blanket application out there that will be useful across an entire business model. Take our work, for example, with complex claims, it is necessary for a human to be there to partner with the person submitting their claim, and by leveraging a tech solution such as genAI, the groundwork can be laid for humans to focus on the most important aspects of the process.”