Conversational AI data collection and platform development

Client Overview

Our client, a leading technology company specializing in conversational AI solutions, sought to expand their existing product to support conversations in several new languages. The goal was to ensure seamless communication for a global user base and enhance the capabilities of their product.

Challenges

The primary challenges faced by our client included the need to:

1. Source and onboard a large team of native language experts proficient in several languages.
2. Create a custom platform to collect audio data tailored to their specific requirements.
3. Generate a substantial and high-quality dataset in each of the new languages for training and improving their conversational AI.

Services Provided

To address these challenges and help our client achieve their objectives, we provided the following services:

1. Native Language Expert Sourcing and Training: We rapidly sourced and onboarded a team of over 300 native language experts. This diverse team was essential to ensuring the authenticity and fluency of the conversations in each language.
2. Custom Audio Data Collection Platform Development: We designed and developed a custom data collection platform tailored to our client's specific needs. This platform allowed for the efficient gathering of audio data, including natural conversations and interactions in the target languages. The platform was designed to be user-friendly, ensuring seamless data collection by the native language experts.
3. Data Collection and Curation: Leveraging our extensive network of language experts and the custom platform, we embarked on a data collection process. This involved collecting a substantial amount of audio data in each of the target languages. The collected data was meticulously curated to ensure its quality and relevance for training the conversational AI system.

Results

Our comprehensive approach to expanding the client's conversational AI to several languages yielded significant results:

• The client successfully integrated support for conversations in the new languages, enabling them to cater to a more diverse user base.
• The custom data collection platform streamlined the process of gathering audio data, saving time and resources while ensuring data quality.
• The curated dataset provided a solid foundation for training and improving the conversational AI system's language capabilities in the new languages.

Overall, our services helped the client achieve their goal of expanding their product's language support and enhancing its global reach. With our assistance, they were able to offer a more inclusive and effective conversational AI solution to users worldwide

Multilingual NLP entity recognition guidelines crafted by expert linguists

Client Overview

Our client, a prominent Fortune 500 tech company specializing in natural language processing (NLP) products, embarked on a significant entity recognition project. The project aimed to advance their NLP capabilities in English and extend entity recognition to 10 additional languages. To achieve this, the client needed assistance in defining entity categories and creating comprehensive annotation guidelines to maintain consistency in data labeling.

Challenges

The challenges faced by our client included:

1. Entity Category Definition: Defining and refining entity categories across multiple languages required linguistic expertise and a principled approach.
2. Annotation Guidelines: The client required precise and principled annotation guidelines that would serve as a reference for annotators across English and 10 additional languages, ensuring consistency in data labeling.

Services Provided

To address these challenges and support our client's entity recognition project, we provided the following services:

1. Linguist Team Assembly: We assembled a team of experienced linguists, each with expertise in multiple languages, to work on this project. This diverse team was crucial in ensuring linguistic accuracy and cultural sensitivity during the entity recognition process.
2. Linguistics-Based Entity Definition: Our linguists used a linguistics-based approach to streamline and solidify entity categories. By leveraging their linguistic expertise, they ensured that entity categories were well-defined, culturally relevant, and consistent across languages.
3. Annotation Guidelines Development: We produced a comprehensive set of annotation guidelines that contained precise instructions and principled principles for entity annotation. These guidelines were crafted to serve as a foundational resource for annotators in English and the 10 additional languages, enabling consistent and high-quality data labeling.

Results

Our services yielded significant results for the client's entity recognition project:

• Linguistically Informed Entity Categories: The client received a set of linguistically informed entity categories that were well-defined and consistent across languages, enhancing the accuracy of their NLP applications.
• Comprehensive Annotation Guidelines: The provided annotation guidelines served as a valuable resource for annotators, ensuring precise and principled data labeling in both English and the 10 additional languages.
• High-Quality Data Annotation: With the support of our linguist team and annotation guidelines, the client achieved high-quality data annotation that was culturally sensitive and linguistically accurate.

Overall, our services empowered the client to establish linguistics-based entity recognition guidelines and maintain data consistency across multiple languages. These guidelines were instrumental in training and testing their NLP applications, enhancing their capabilities and ensuring success in diverse linguistic contexts.

Computational model optimization and language expansion with grammar authoring support.

Client Overview

Our client, a major technology company, requested support for improving two related existing computational models of natural language and expanding them to support input in several world languages. Their goal was to provide a tool for use in automated scheduling and customer service applications.

Challenges

The primary challenges faced by our client included the need to:

• Identify areas to expand existing coverage of the models to include microvariation, ambiguity, and dialectal differences in everyday language usage.
• Improve efficiency of the language representation to optimize model performance.
• Generate comprehensive datasets in a variety of target languages according to specific usage domains and project specifications.

Services Provided

To address these challenges and help our client achieve their objectives, we provided the following services:

1. Native Language Expert Sourcing and Training: Utilizing our multilingual network of linguists and language experts, we were able to rapidly source and identify native speakers in each of the target languages that would be able to collaborate on providing high-quality data for the requested language usage domains. We modified existing onboarding processes from previous similar projects and provided the client with a pilot study.
2. Data Collection and Curation: Leveraging our extensive network of language experts, we embarked on a data collection process. We curated large-scale, comprehensive datasets from the collected native speaker data and passed them to our computational linguists who specialize in mathematical models of language and linguistic theory.
3. Computational Model Authoring and Evaluation: We began by working closely with the client's engineers to familiarize themselves with the underlying logic of the client's existing approach to language modeling, in this case a rule-based approach to recognizing domain-specific language usage. Drawing on our expertise in formal mathematical representations of natural language, we were able to identify numerous improvements to the models’ efficiency and existing coverage. We used the curated cross-linguistic data samples to localize the models into more than 30 languages.

Results

Our comprehensive, data-driven approach to improving and expanding upon our client's existing model led to the following key results:

• The client accomplished both facets of their goal for the project, namely the improvement of existing coverage and the expansion of coverage to include new languages and everyday microvariation. The client was further able to improve the efficiency of their product based on input from our linguists.
• The client was able to use our feedback about evaluation methods to better ensure that both related models achieved identical data coverage.
• We worked closely with the client to develop a pipeline for continued improvements to model performance and additions to model coverage across all included languages.

Overall, our services helped the client achieve their goal of expanding their product's efficiency and language support and enhancing its global reach.