The Center for Data Innovation recently spoke with Christian Vestergaard, co-founder of The Agentic Data Company, a San Francisco-based company that collects real-world speech data to help developers train AI systems. Vestergaard explained how authentic conversations can help systems understand context, interpret vocal cues, and respond more reliably to complex tasks.
David Kertai: How does your approach to speech collection differ from other methods?
Christian Vestergaard: Many speech datasets train AI systems on structured recording sessions where participants read scripts, respond to prompts, or act out conversations. Although these methods produce clear recordings, they often fail to capture how people communicate naturally. Real conversations often depend on relationships, intentions, context, and shared experiences. Asking strangers to imitate everyday interactions can change how they speak and respond to one another.
We take a different approach by collecting speech during activities people already engage in. We build applications around those activities, starting with a calling app similar to WhatsApp and later expanding into tools such as meeting recorders. Rather than asking people to perform for a recording session, we aim to capture conversations that occur for genuine reasons. This gives us examples of the interactions that voice-AI systems need to understand to function effectively.
Kertai: What types of speech data do you collect?
Vestergaard: We collect speech from a wide range of real-world settings, including phone calls between family members, workplace meetings, customer-service interactions, and conversations at drive-through ordering points. Each setting involves different communication styles, vocabulary, goals, and social dynamics.
The common requirement is authenticity. We collect conversations in which people communicate for a genuine purpose, whether they are solving a problem, sharing information, making a decision, or completing a task. These recordings preserve natural speech patterns and conversational dynamics that scripted exchanges may miss.
Kertai: How do you design collections to reflect everyday conversations?
Vestergaard: We design the recording experience around activities people already perform. For a phone conversation, for example, we can provide a calling application. For a fast food drive-through interaction, we use recording hardware at the ordering point. The goal is to let people focus on their conversation rather than having to perform for a recording session.
We also capture audio in a format that helps researchers train AI systems. We record each individual on a separate audio channel, sometimes called a stem, so researchers can listen to and analyze each person’s voice independently. We synchronize these channels so researchers can reconstruct the full conversation, including interruptions, overlapping speech, and pauses. We record at 48 kilohertz, meaning the system samples the audio 48,000 times per second, to capture detailed sound information that can help researchers effectively train AI models.
Kertai: How do you tailor a dataset to address a specific weakness in a speech-AI model?
Vestergaard: We start by identifying exactly where the model struggles. Knowing that a model performs poorly during flight bookings, for example, does not tell us enough. We need to identify the underlying problem. The model might misunderstand a request, retrieve incorrect information, fail to handle a correction, or struggle to complete an action in another software system.
Once we identify the failure, we look for real-world situations in which people successfully perform the same task and collect examples of those interactions. If a model struggles to maintain a conversation while checking information and helping a customer, we collect conversations that naturally involve those challenges. We build the dataset around the specific skill the model needs to improve, rather than simply collecting more speech about the same general topic.
Kertai: What can AI models learn from interruptions, pauses, and changes in tone?
Vestergaard: These features convey information beyond the words themselves. Someone saying “I love it” might express enthusiasm or sarcasm, depending on their tone of voice. A pause might indicate uncertainty or reflection. An interruption might correct a misunderstanding or change the direction of a conversation. People often interpret these signals instinctively, but many speech datasets do not capture them in enough detail to help AI systems respond appropriately.
We believe this understanding will become essential as voice interfaces—systems that let people interact with computers through speech—become more common. To communicate naturally, AI systems must recognize hesitation, unfinished thoughts, corrections, changes in intent, and the broader context of a conversation. This matters especially in situations where mistakes carry consequences, such as emergency services and customer assistance.
The next challenge is teaching AI systems to speak, reason, and act at the same time. For example, a banking assistant must maintain a conversation while checking records, evaluating information, and deciding what action to take next. To do this reliably, models need examples of how people coordinate conversation, decision-making, and action in real-world situations. By capturing these interactions, we aim to help developers build voice-AI systems that can handle more complex tasks while communicating naturally and responding appropriately to the people they assist.
