Data Scientist (AI Quality & Evaluation)
About the Role
We're looking for a Data Scientist to own the quality, reliability, and trustworthiness of our clinical AI outputs. You'll build the systems that ensure our AI "knows what it doesn't know" — developing evaluation frameworks, calibrated confidence scoring, and automated quality assurance that physicians can actually trust.
What You'll Do
Design and implement automated evaluation pipelines that assess AI output quality, accuracy, and safety at scale
Develop uncertainty quantification systems where confidence scores meaningfully correlate with accuracy
Build comprehensive evaluation frameworks combining automated assessment with clinician-validated test cases
Implement feedback loops that continuously improve model outputs based on validation signals
Establish scalable quality gates that catch errors before they reach end users
Contribute to model alignment and fine-tuning efforts
Qualifications
Required
Strong foundation in deep learning frameworks (PyTorch) and LLM architectures
Experience with model evaluation, benchmarking, and quality metrics
Proficiency in Python and modern ML development tools
Strong statistical foundations
Ability to read, implement, and extend research papers
Excellent communication skills
Preferred
Master's degree in Computer Science, Machine Learning, Statistics, or related quantitative field (PhD preferred)
Publications in top ML/AI venues (NeurIPS, ICML, ICLR, ACL)
Experience with RLHF, DPO, or preference optimization techniques
Background in healthcare AI or regulated industries
Experience building evaluation systems for production LLM applications
Average salary estimate
If an employer mentions a salary or salary range on their job, we display it as an "Employer Estimate". If a job has no salary data, Rise displays an estimate if available.
Lead the measurement and experimentation strategy for Scribd's user-generated content ecosystem, turning ambiguous product questions into rigorous, high-impact analytics and AI evaluation.
Mirage, a leading AI short-form video company based in NYC, is hiring a Marketing Data Scientist to build measurement foundations, improve acquisition performance, and inform go-to-market strategy.
TMEIC is hiring an AI/ML Applications intern to help design, build, and deploy data analytics, visualizations, and machine learning solutions for its Energy and Infrastructure business.
Contribute to production-ready machine learning and analytics at Spring Venture Group as a Junior Data Scientist, applying Python, SQL, and modern AI tools to improve KPIs and automate workflows.
Suno seeks a Content Data Scientist to drive experimentation, content-health tracking, and discovery optimization for its AI music platform in New York City.
Lead the design and production of generative AI coaching systems at BetterUp, shaping product direction while mentoring engineers and partnering across product, design, and research.
Lead the development of a production-ready, multi-asset optimization and real-time bidding platform to enable automated ERCOT market decisions at scale for Intersect.
Lead the architecture and production deployment of low-latency, explainable machine learning systems that drive personalized next-best-action decisioning across digital and assisted channels at Humana.
Suno is hiring a Senior Data Scientist to define creator and artist success metrics, run experiments, and inform programs that accelerate creator growth on its AI-driven music platform.
Senior AI Engineer / AI Technical Architect sought by a top-ranked hospital to design and operationalize cloud-based AI/ML solutions using Epic and Azure to improve clinical and operational outcomes.
🔮
Hi, I'm Risa! Your AI Career Copilot
you, just ask me below!
