How Specialty Practices Vet and Adopt AI: A Physician’s POV

Learn how to vet AI tools for your practice the same way you’d appraise new clinical evidence

In a recent interview on The Podcast by KevinMD, Eric Jennings, MD, Senior Medical Director of Ophthalmology at ModMed, argues that the smartest response to AI isn’t refusal — it’s cautious skepticism. He says to treat a new AI tool the way you’d treat newly published evidence: Pin down the risks, the benefits, and whether it actually helps you care for patients.

Cautious skepticism vs. paralysis: Here’s the difference

There’s a fine line between productive skepticism and getting stuck.

“I think caution, if you’re being careful, you say, ‘OK, but prove it. Show me the data. Show me the validation. I will consider it,’” says Dr. Jennings.

His advice translates the decision back into the language every specialist already speaks. A new drug comes out for a condition you treat. You review the literature. You start with a small trial. Then you decide whether it fits your patients. Do the same with AI.

Accept the premise. Demand validation. Take it from there.

How to tell a real AI tool from fancy window dressing

Walk any conference expo floor, and you’ll see it. AI-powered this. AI-powered that. So how do you separate substance from marketing? Dr. Jennings offers a filter.

1. Look for tools that solve an actual problem. “There’s a big push to say, ‘I’m going to develop this cool AI tool that looks great, and we have a great logo,’ but it actually creates more work for you to do,” says Dr. Jennings. 
2. Ask what “AI-powered” even means for that product. Was it built natively inside your system, or bolted on? How was it trained? You don’t need to become an AI engineer. You just need to ask better questions.
3. Interrogate the training data. This is where many tools quietly fall apart.

“If you look at the N, and the N is 12, that’s not a really good training set,” Dr. Jennings says. If you treat an elderly rural population, but the algorithm was validated on healthy 20-somethings from a major city, it’s not going to work for your patients. Ask how many encounters trained the algorithm. Ask whether the validation was specific to your specialty. Ask who conducted the validation.

Dr. Jennings references the case of a pneumonia-detecting algorithm that reached 93% accuracy in the lab and then failed in clinical use. Turns out it hadn’t learned to spot pneumonia. It had learned that portable X-rays, used for patients too sick to leave their beds, tracked with the diagnosis. Lab performance didn’t hold up in a real clinic.

The lesson isn’t that AI can’t work. It’s that a tool has to be validated on the patients and the setting it will serve.

Start where the risk is lowest: Administrative workflows

Dr. Jennings points to the administrative side as the smarter place to begin. “Frankly, if you’re dipping your toe into AI,” he says, “it’s the safest sandbox to build some trust because you still have a lot of oversight into what eventually goes out the door and comes in.”

Where does he see the fastest operational ROI? Four areas stand out:

• Prior authorizations. Because prior auths hit every request, offloading that volume is where the time savings add up.
• Claim assessments. AI can evaluate claims before submission and flag potential issues, helping you fix problems before the payer says no.
• Patient communication. Rescheduling, questions, appointment changes. An AI tool can handle the routine requests and free your staff for the conversations that need their attention.
• Payer rule changes. Payer policies shift on a rolling basis, and chasing them takes time. AI can check claims against those rules to help your practice get paid.

Implementing AI: What the first week should look like

Rushing implementation is how good tools get abandoned. Dr. Jennings lays out the guardrails for Week 1 — and Month 1.

• Establish a human in the loop. No matter how a tool is marketed, human oversight comes first.
• Start with low-stakes tasks. Build trust before you scale.
• Overcommunicate with staff and patients. This is new, not a judgment on how anyone works today.
• Build a feedback mechanism. Nothing ships perfectly. Track what’s working and what isn’t.

Getting staff on board with AI

Staff resistance is one of the first signs that a rollout is going too fast. When a team hears “AI” and thinks “replacement,” they go quiet or complain nonstop. Dr. Jennings says the answer is to make sure they understand the tool is there to work alongside them. 

Forcing adoption from the top down creates friction. Instead, build ownership by naming super users: a front-office lead, a clinical lead, and a back-office lead. They champion the tool, become the go-to experts, and bring feedback to the team.

Dr. Jennings applies the same idea with his own technicians. “You’re not just here to click buttons.” The same mindset applies to any AI tool. Involve the team early, and you get far less pushback.

How to measure ROI beyond the P&L

Direct revenue is the easy metric. Are claims getting processed accurately? Are billing workflows more efficient? Check the P&L. But Dr. Jennings argues there’s a quieter return worth tracking, too.

Physician burnout has a cost. So does staff turnover. When AI takes on the routine work, your people can spend more time on the work they were hired to do, which can improve the patient experience.

“There’s a hidden ROI, but it’s real, and it can be tracked,” Dr. Jennings says. Practice leaders just have to choose to measure it. The clock is a good place to start: how late does your team stay to finish charts and follow-up work?

The future: One connected platform, not a pile of bolt-ons

Ask Dr. Jennings to look into the crystal ball, and he’s confident about the direction.

It won’t be a single tool. It’ll be an integrated solution.

“Right now, what’s becoming a problem for some of you is that a lot of the AI tools are bolted on,” he says. One solution here, another there, none of them talking to each other.

The practices that thrive will invest in platforms built for interoperability and in systems designed to work together rather than a patchwork of point solutions. 

Here’s what that could look like: a network of AI Assistants for clinical notes, billing and claims, and patient outreach — all designed to work together and hand off tasks in the background. That’s the idea behind The AI-Powered Practice™ from ModMed: AI built into the tools your practice uses every day, shaped by practicing physicians, with your team in control.

Dr. Jennings’ view: ” If you’re going to thrive, be skeptical and move forward.” 

The AI-Powered Practice: It’s your call

Dr. Jennings’ advice comes down to this: ask for the evidence, keep a human in the loop, and start with what’s low risk. The AI-Powered Practice is built to meet practices where they are: specialty-trained, fully integrated, and always leaving the final call to you.

See how ModMed can help your specialty practice adopt AI with a personalized demo based on your specialty. Get started here →

FAQs

What is the difference between native EHR AI and generic bolt-on AI tools?

AI operates within ModMed’s EHR, Practice Management, Patient Engagement, and RCM solutions, eliminating the need for staff to constantly move information between disconnected applications. Generic bolt-on AI tools operate in separate third-party software, often requiring manual copy-pasting, generic note rebuilding, and redundant workflow steps.

What questions should I ask an AI vendor before buying?

Ask whether it was built natively or bolted on, how many encounters it was trained on, whether the training data reflects your specialty and patient mix, and who validated it. Then test it in a limited rollout before going practice-wide.

How do you prevent staff friction and fear of job displacement during AI implementation?

To overcome internal resistance, practice leaders should avoid top-down directives. Instead, encourage peer ownership by identifying front-office, clinical, and back-end super users to champion the technology. Frame AI tools as assistants that help staff off-load administrative drudgery and do more of the work that matters.

Why does AI training data matter for specialty practices?

AI algorithms trained on small or unrepresentative datasets can fail in clinical environments. If an AI tool was trained on general medical data rather than specialty-specific cases, its note and workflow suggestions may require significant manual corrections when applied to complex  patient populations. Practices must verify that a vendor’s training set reflects their specific specialty, patient demographics, and acuity mix.

Results may vary based on practice size, product usage, specialty, and other factors.

This page includes “forward-looking statements,” including information about solutions and features that are not yet available. The decision and timing regarding release and development of solutions and features may be subject to change. Actual solutions and features may differ materially from any of those expressed here or in other forward-looking statements. Any purchasing decisions made by you should be solely based on ModMed’s existing solutions and functionality.

This blog is intended for informational purposes only and does not constitute legal or medical advice. Please consult with your legal counsel and other qualified advisors to ensure compliance with applicable laws, regulations, and standards.