I spent October 22 at ProductFest 2026, rvatech’s annual product conference at the Science Museum of Virginia, with product managers, designers, and engineers from CarMax, Capital One, CoStar Group, Dominion Energy, and MyFitnessPal. COLAB was a conference sponsor this year.
Most sessions came back to one idea. AI is taking on work that used to fall to people: logging a meal, drafting a ticket, summarizing a customer interview. The teams getting value from it decide on purpose which work to hand over, and they keep people accountable for the judgment. Here is what I heard on the main stage and what I think it means for the organizations we work with.
Dominion Energy: decide where AI will not go before you scale it
Dr. Nell Meshcheryakov, Strategic Advisor for Responsible AI at Dominion Energy, asked organizations to start with their biggest risks, the same first step the NIST AI Risk Management Framework recommends, and decide where they will not use probabilistic technology at all. She called change management the step most teams forget. Generative tools are approachable enough that anyone with access can use them, so responsible use depends on educating people as much as on technical controls.
Jason Gambill, Senior Strategic Advisor at Dominion Energy, split AI work into experiments and production. Experiments can wander and fail without an immediate return. Production work has to tie back to value and cost, and mixing the two is how a sound project loses its business case. His advice on where to begin: “Find the pebble in your daily task. Find the pebble in your customer’s task.”
When an engineer in the audience asked how to keep teammates from skipping code review, Dr. Meshcheryakov warned against cognitive offloading, which means handing your thinking to a tool. Review, testing, and the usual development lifecycle controls still apply before anything reaches production. Speed gains can also be smaller than they feel. In a 2025 METR study, experienced open-source developers took 19% longer on tasks with early-2025 AI tools while believing the tools had sped them up by 20%.
MyFitnessPal: the cheapest processor is no longer the user
Megan Overton, Head of AI Nutrition Coach at MyFitnessPal, gave a detailed, practical talk on rebuilding a 20-year-old app around AI. Software has long asked users to do tedious work because they were the cheapest place to do it. Large language models can now read messy input, remember context, and notice patterns, which lets that work move into the product.
The harder decision is how far to take it, and Overton weighed automation against what the user gains. Waymo delegates the entire ride because the outcome, arriving safely, is easy to judge. ChatGPT’s Instant Checkout, she said, went too far by automating a trusted moment with the retailer that people wanted to keep. OpenAI scaled the feature back in March 2026. For MyFitnessPal, logging food is how people pay attention to what they eat, and that attention supports weight loss. Photo and voice input and confirm-or-edit prompts make logging faster. Removing logging entirely would remove the value.
Her team’s measurement practices were just as useful. The model is scored against a “golden set” of anonymized real meals, photos, and conversations, with automated judges built from failure cases. A holdout group of about 10% of users tracks week-over-week retention, and edit rate shows whether people are still noticing what they eat. Overton also reads 100 full conversations a week with her content designer, UX designer, and data scientist, grading trust, safety, tone, and quality.
Did AI break product discovery? It made weak discovery faster
The panel’s answer was that AI speeds up whatever discovery a team already does. Teams that skip customer research now skip it faster, and AI summaries can hide thin interviews behind a confident report. Teams that research carefully get faster prototypes and more chances to test with real people.
Beth McHugh, Product Leader and Innovation Strategist at Tillta Consulting, focused on switch moments, the event that makes someone change providers. She complained about her gym for years and kept paying, until an instructor failed to show for a class she had rearranged her schedule to attend. Complaints show what is top of mind. Switch moments show where demand sits, and finding them takes judgment that pattern matching lacks.
Matt O’Connell, Founder and CEO of Vistaly, described the failures his AI discovery tool works against, including quotes from two people merged into one. His point on accountability stayed with me: if you share a synthesis, your name is on it, whether you wrote it or a model did. Tyler Pease, a lead product designer most recently at CoStar Group, added that cheaper prototypes raise the bar for deciding what is worth testing. Andrew Ilnicki, Director of the University of Richmond’s Innovation Studio, hosted.
Five years of ProductFest: wider team scope and a new kind of visitor
Jessica Allison of CoStar Group moderated the founders’ panel with Travis Bjorklund of Boldin, Abhinav Agarwal of the University of Virginia, and Victoria Lawson, AVP of Product at CarMax. Lawson said CarMax’s 40 to 50 product teams look much the same as five years ago. Each one now covers a wider scope because delivery is faster.
She also raised agentic shopping, where some shoppers hire an AI agent to shop on their behalf, and compared it to the early days of SEO. Teams still need to understand what the shopper is trying to do, and they also need to know what agents reward and how their data is structured. Her cautionary example was an agent that reserves all 40 matching cars to give its user a choice, which serves the shopper and hurts the business. One of the panelists raised an important design concern: generative tools raise the floor and lower the ceiling, which pushes products toward sameness.
Dean Peters: ask every AI claim for its receipts
Throughout the day I heard a general sense of existential angst about AI and product roles. Dean Peters, Principal Consultant and Trainer at Productside, landed the day on a more practical note. His closing keynote introduced the AI Product Manager Baloney Detection Kit, a free set of AI skills on GitHub that also works with pen and paper.
The kit runs a claim through a funnel. Rewrite the request as an outcome with a direction, metric, context, and user. Ask for receipts: where the claim came from, whether it is fact, inference, or assumption, and how old it is. Check whether it applies to your product, then run the smallest test that returns a decisive answer and decide whether to pivot, punt, or pursue.
His working agreement for teams fit in one line: the AI recommends, and the team decides. On choosing a model for patient records, he reminded the room that “the large language model is not the one getting deposed.”
What this means for our clients
Most of the nonprofits, healthcare organizations, and member organizations we work with will not run a holdout test on an AI coach or staff 50 product teams. The lessons still apply at their scale.
- Start with one manual burden. Look for work your patients, members, or donors do only because nothing else could, such as re-entering information or calling to check a status. Automate the tedious part and keep the steps that carry the value.
- Keep experiments and production separate. Pilots can fail cheaply. Anything that touches member accounts, patient information, or donations needs review, testing, and an owner before it ships. I build much of COLAB’s own website with AI, and changes still go through traditional software testing before they go live.
- Keep talking with the people you serve. AI summaries of interviews and surveys are a starting point. Decisions should trace back to what people said and did, with the source attached.
- Plan for agents as visitors. If CarMax expects some shoppers to send an agent, your members and donors may do the same, and your site can be made ready for them. At our table, we ran attendees’ websites through COLAB’s AI agent-readiness scanner, which checks whether an agent can find, read, and complete a task on a site.
- Label what you know. Before a recommendation goes to your board, mark which parts are facts, inferences, and assumptions.
If you are working out where AI fits in your website or digital services, see how we approach AI services, or let’s talk.
