How Mona Chitnis Defended Building Custom AI Models When Foundation Models Weren't Enough

Episode Summary:

In this episode of Engineering Choices You Have to Defend, host Nicola Onassis sits down with Mona Chitnis, engineering leader at Boon and former engineering manager at Apple, to discuss one of the most important AI engineering decisions facing modern software teams: knowing when foundation models are no longer enough for production use.

Working on AI systems for the commercial construction industry, Mona quickly realized that general-purpose foundation models struggled to accurately understand complex construction plans and domain-specific workflows. Rather than immediately building custom models, her team first invested in developing a comprehensive evaluation framework to identify performance gaps, measure business outcomes, and determine whether fine-tuning proprietary models would deliver meaningful improvements.

One of Mona's most significant engineering decisions was choosing to build a hybrid AI architecture instead of relying entirely on off-the-shelf foundation models. By combining proprietary computer vision models with large language models, her team created a system capable of understanding construction documents, extracting measurements, and helping customers generate more accurate project estimates. She explains why data quality, customer collaboration, and continuous evaluation ultimately proved more valuable than simply adopting larger or newer AI models.

The conversation also explores Mona's experience leading privacy-preserving machine learning initiatives at Apple, where federated learning enabled AI models to improve without collecting sensitive user data. She discusses why enterprise AI success depends on representative datasets, thoughtful evaluation frameworks, and engineering systems designed around real customer outcomes rather than benchmark performance alone.

For engineering leaders building AI products in specialized or regulated industries, this episode offers practical lessons on evaluation strategy, production AI architecture, data quality, privacy-preserving machine learning, and building AI systems that deliver measurable business value.

Key Takeaways:

• Evaluation frameworks should be established before deciding to build custom AI models.
• Foundation models often require domain-specific fine-tuning to solve specialized business problems.
• Business outcomes matter more than benchmark scores when evaluating AI systems.
• High-quality, representative datasets are critical to successful production AI.
• Hybrid AI architectures can outperform relying on a single foundation model.
• Customer collaboration plays a key role in improving enterprise AI performance.
• Privacy-preserving machine learning enables AI improvements while protecting sensitive user data.
• AI benchmarks should complement—not replace—real-world product evaluation.
• Engineering leaders must balance technical performance with customer value and business objectives.
• Successful AI products are built through disciplined engineering, continuous evaluation, and strong data strategies.

Connect with Mona Chitnis:

LinkedIn: linkedin.com/in/monachitnis

Company: Boon

Listen Now & Subscribe:

Apple Podcasts, Spotify, Amazon Music, YouTube, iHeartRadio, Captivate, or wherever you get your podcasts.

Engineering Choices You Have to Defend explores the real technical decisions behind AI systems, enterprise architecture, and scalable software engineering.