Evaluate large language model outputs for tone and fluency. Assess quality and correctness, providing ratings and rationales.
Join Project Spearmint, a multilingual AI response evaluation project reviewing large language model (LLM) outputs in different languages, focused on either Tone or Fluency. Native-level fluency in a target language, along with strong English comprehension, is required.
As an evaluator, you will review short, pre-segmented datasets and assess model-generated replies based on specific quality dimensions. Your input will help validate evaluation frameworks and establish baseline quality metrics for future model development.
Key Responsibilities:
- Evaluate model replies in your native language based on either Tone or Fluency.
- Assess the overall quality, correctness, and naturalness of responses.
- Read the user prompt and two model replies, then rate each using a five-point scale.
- Provide brief rationales for any extreme ratings.
Project Breakdown:
Batch 1 – Tone: Determine whether replies are helpful, insightful, engaging, and fair. Flag formality mismatches, condescension, bias, or other tonal issues.
Batch 2 – Fluency: Assess grammatical accuracy, clarity, coherence, and natural flow.
This is a project-based opportunity with CrowdGen, where you will join the CrowdGen Community as an Independent Contractor. If selected, you will receive an email from CrowdGen regarding the creation of an account using your application email address. You will need to log in to this account, reset your password, complete the setup requirements, and proceed with your application for this role.
Make an impact on the future of AI – apply today and contribute from the comfort of your home.
Similar Jobs
Artificial Intelligence • Software • Conversational AI
The Account Manager will engage small customer segments through automation, maintain account plans, develop onboarding content, and drive customer success initiatives.
Top Skills:
MarketoOutreachSalesforce
Blockchain • Software • Financial Services • Cryptocurrency
Manage all projects related to core products, ensuring timely completion and high-quality deliverables through effective communication and Agile methodologies.
Top Skills:
Agile DevelopmentBlockchainProject Management SoftwareZero-Knowledge Cryptography
Blockchain • Software • Financial Services • Cryptocurrency
As an SDET at Alpen Labs, you'll develop test frameworks, manage test environments, conduct various tests, and advocate for quality software delivery.
Top Skills:
APIsBlockchainCi/CdKubernetesTest Automation
What you need to know about the Edinburgh Tech Scene
From traditional pubs and centuries-old universities to sleek shopping malls and glass-paneled office buildings, Edinburgh's architecture reflects its unique blend of history and modernity. But the fusion of past and future isn't just visible in its buildings; it's also shaping the city's economy. Named the United Kingdom's leading technology ecosystem outside of London, Edinburgh plays host to major global companies like Apple and Adobe, as well as a growing number of innovative startups in fields like cybersecurity, finance and healthcare.
