From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence

Microsoft Research has described an internal system for testing Copilot without touching customer data. In a blog post dated October 30, 2026, a team including Harshita Sahijwani, Jenny Chen and Sam Ade Jacobs introduces the Industry Evaluation Data Generation Platform, which builds fully synthetic enterprise workspaces with simulated users, emails, meetings and documents that develop over time, allowing tests of time-sensitive questions.

The team has created three environments so far, covering pharmaceuticals, capital markets and an IT/software company, each holding hundreds of users. Realism comes from public sources: the U.S. Department of Labor's O*NET occupational database, online job postings and SEC filings, all cleared by Microsoft's legal and policy organization. A six-stage automated pipeline turns generic workplace tasks into industry-specific queries, rewrites them to sound natural and validates them for accuracy and uniqueness. Automated checks screen the generated content for harmful or personally identifiable material.

Microsoft reports that its composite quality score, an internal measure of how closely the data mirrors real organizations, rose 39% over its initial approach, with some environments scoring above 0.9 out of 1.0. It estimates savings of about $100,000 per dataset and $300,000 overall, and says multi-day manual work is now handled by automated runs that reproduce identical results from the same configuration. These figures are self-reported and were not independently verified in the post.

The datasets now sit in an internal catalog and, according to the authors, form the first standardized evaluation baseline for Copilot. The team says its next challenges include assessing multi-turn agentic interactions and AI systems that take actions.