5 things to consider when evaluating a benefits AI vendor

Editor’s note: This blog post is a direct follow-up to Episode 3 of Benefits Innovation Lab. If you haven’t checked it out yet, be sure to register to view or listen on demand and download the accompanying free worksheet to get the most out of this article.

When it comes to integrating artificial intelligence (AI) into your employee benefits practice, choosing which problem to solve is only half the battle. You’ll also need to secure buy-in from your organization and choose an AI partner that suits your needs.

There certainly isn’t a lack of options. There are more than 80,000 companies offering AI services worldwide¹ and, according to a 2024 HealthEquity survey, nearly 70% of benefits leaders say they’re approached by AI vendors at least once a month. That can feel overwhelming—especially if you’re not a tech whiz.

Fortunately, there are a few high-level points of consideration that can guide you through the process. Asking the five questions outlined in this article will help you create a shortlist of AI vendors and compare those offerings against each other.

When you’re done, you should have a good idea of the right partner to make your AI dreams a reality.

First: Get your team together

Choosing an AI vendor isn’t a one-person job. As the benefits team leader, you’ll drive many of the conversations, but you’ll also want the advice and expertise of people in your organization who will be impacted by the end result.

Here are a few folks to consider:

• Your AI committee. If your organization already has a formal AI committee—where you can lean on its members to provide input on vendor choice—this is a prime example of their value! They may even have vendor suggestions if you’re having trouble getting started. And, of course, they’ll be instrumental in helping you make a final call. If you don’t have an AI committee, be sure to include IT, Data privacy and security, Benefits leaders, and the primary user group of the tool.

• Your IT team. If you don’t have an AI committee in place, you’ll want to reach out to members of your security and technology teams to provide technical input on your choice.

• Your benefits broker. Your broker works with many clients and industry organizations, and may have knowledge of the best AI offerings.

• The primary user group of the AI tool. Consider putting together a focus group of people who will be using the AI tool most often. This could be your benefits team, your larger Human Resources (HR) team, or your employees.

Question 1: What are your business requirements?

Before you start shopping around—and certainly before you start any sales conversations—you should define your business requirements. These are the “must haves” that will allow your AI project to deliver the business outcomes you want to reach.

As the benefits team leader, you’re in the best position to start this process. Ask yourself about the opportunity you’re tackling and which features you’ll need to be successful.

You’ll also want to ask your technology and security team the same questions. They may have security or data maintenance standards that a vendor must meet to even be considered as a candidate.

Our example: Let’s illustrate this further by considering a real AI use case in the benefits world. According to the HealthEquity survey, the number one opportunity benefits leaders see with AI is health plan selection for employees. To find a vendor that can help accomplish that task, we should make a list of business requirements that includes:

• The ability to provide recommendations based on simple personal questions.

• The ability to highlight plan differences.

• Data protection for any health-related information provided.

• Integration with any health plan platforms.

• Mobile accessibility for employee convenience.

Question 2: What is the vendor’s expertise level?

There are two areas of vendor expertise you may want to know about: industry expertise and technical expertise.

Not all AI use cases require benefits industry expertise, but it can be useful. Knowledge of the many laws and regulations that govern healthcare and employee benefits can assuage potential risk management concerns, while familiarity with the subject matter can provide a better user experience for benefits team members and employees.

To gauge industry experience, consider whether the vendor specializes in benefits or HR tasks. Look to long-term client relationships, case study availability, or presence in industry forums and communities to see if the vendor occupies a meaningful place in the benefits world.

Your AI committee or IT team will also want to know about the vendor’s technical expertise. Depending on your project, these folks may want to know about the learning frameworks or large language model (LLM) technologies used by the vendor. Your organization’s experts can help translate any “tech talk” and help you understand if the vendor is a good fit for your specific business requirements.

Our example: To build a health plan recommendation tool for employees, we’d likely want to work with a vendor that has industry expertise. Knowledge of HIPAA requirements and other data privacy regulations will boost confidence in the tool, and likely provide a better user experience for employees. Our company technical experts can provide further input to ensure any AI vendor is able to handle the requirements of health plan comparison and data security.

Question 3: How does the vendor address relevant risk?

Because AI is still relatively new to most benefits teams, you likely have concerns you’ll want the vendor to address. According to the HealthEquity survey, over one-third of benefits leaders want to know more about how AI handles data security and privacy. More than 20% also want to know about the accuracy and reliability of data.

These are concerns that should be addressed when evaluating vendors.

When it comes to data security and privacy, your company technical experts can confirm the solution matches the regulatory landscape. They can also evaluate data handling and storage, as well as security practices like encryption methods, access control, and incident response plans.

As a benefits professional, you can focus on higher-level questions. One good question to ask is how the vendor would handle a scenario where something goes wrong—what support and leadership can you expect from the vendor?

Data quality is also crucial to the success of any AI tool. While company technical experts can look into the details, you can ask for a general explanation of how the vendor might handle biased or underrepresented data sets. Ask for real-world examples based on your specific use case.

Our example: Data security, privacy, and quality are all extremely important in a health plan selection tool. We’d want to have specific conversations with vendors to ensure any health information employees provide is private and protected, and that data sets are robust enough to support quality plan recommendations that meet employees’ true health needs rather than perpetuate past, potentially biased healthcare outcomes.

Question 4: What is the sales process like?

You can tell a lot about a potential vendor partner from your experience during the sales process.

Evaluate the vendor’s company culture and collaborative style. Determine if communication meets your needs and preferences—are you hearing from them with the frequency and care you expect? Does their style match your own?

One way to really explore this question is to conduct a pilot project with the vendor. If you can negotiate a small, one-off project with the vendor, you can get a sense of their problem-solving capabilities and your synergy as a team.

You should also consider the degree of flexibility you have in contract negotiation to ensure your organization’s time and budgetary needs are met.

Our example: A health plan recommendation tool would require a high degree of trust between a benefits team and its AI vendor. We’d want to feel comfortable in how the vendor communicates, both with our team and the eventual end use: our employees. We’d want to ask for samples of employee communication and understand their strategy in how they ensure consumer satisfaction.

Question 5: Will you have the right support during implementation?

Finally, be sure to look ahead to AI platform implementation.

Technically, you need to understand if the vendor is able to handle any bespoke requests or adjustments your project may require. You should also ask about platform integration—will the vendor platform be able to speak to the other key systems in your technology stack?

Culturally, onboarding and training of your benefits team members is key. Your team will want to be up to speed quickly on how to use the tool and educate any other users in turn. This will set you up for success and speed up crucial project timelines.

Ask about the vendor’s use of success metrics or key performance indicators (KPIs) as well. You’ll likely have ideas on how you’ll measure the outcome of your project, but the vendor may have useful input on how to get you the data-based insights you need.

Our example: To implement a health plan recommendation tool successfully, we’ll want high quality onboarding and a simple user interface for employees. We’ll also want to be able to measure employee outcomes like confidence in plan selection or satisfaction with the tool.

Making a final call on your benefits AI vendor

Still struggling to identify a clear frontrunner in your search for the right AI vendor? Consolidate your observations in our vendor checklist.

The checklist allows you to take notes on your vendor options as well as assign a rating of importance for each consideration. Your project may require more benefits industry expertise and less implementation handholding, for example. Considering feature prioritization like this could make your final decision easier.

Of course, you’ll also want to rely on the opinions of your AI committee and other designated organizational experts. You may even have other members of your benefits team weigh in.

Regardless of how you approach your ultimate decision, the time you spent evaluating your options will make you more confident in your vendor selection and your ability to reach your benefits goals together.

Stay tuned to the Remark blog and to Benefits Innovation Lab for more insights on how to improve AI use in your benefits team.