Senior Software Engineer II - Applied AI and Evaluations (Remote Eligible)
Smartsheet seeks a Senior Software Engineer II to own agent quality for SmartAssist, an AI-powered work management platform. This role involves deep technical work in LLM evaluation, prompt engineering, and RAG, with high autonomy and immediate impact on production agents.
Responsibilities
- Own agent quality end-to-end: diagnosis, improvement, and validation across SmartAssist's orchestrator and subagents
- Identify failure modes across quality dimensions and prioritize what to fix
- Drive quality improvements through prompt engineering, context engineering, and RAG retrieval tuning
- Extend and mature our evaluation framework: scorers, golden datasets, regression gates, and online evaluation
- Close the feedback loop to ensure every change has a measurable, attributable quality signal
About the role
For over 20 years, Smartsheet has helped people and teams achieve–well, anything. From seamless work management to smart, scalable solutions, we’ve always worked with flow. We’re building tools that empower teams to automate the manual, uncover insights, and scale smarter. But more than that, we’re creating space– space to think big, take action, and unlock the kind of work that truly matters.
Because when challenge meets purpose, and passion turns into progress, that’s magic at work, and it’s what we show up for everyday. Smartsheet is building the next generation of AI-powered work management through SmartAssist, our intelligent agent platform.
As we scale from early demos to production-grade agents, quality is the critical frontier and we are looking for a Agent Quality Engineer to own it. This is not a QA role. It's a deeply technical, high-autonomy position at the intersection of LLM evaluation, prompt and context engineering, and retrieval-augmented generation.
You will diagnose why our agents fail, design the systems that catch regressions, and drive measurable improvements across our orchestrator and subagent fleet. You will work closely with our Agent Engineering and AI Platform teams, embedded in a team that has already shipped evaluation infrastructure on Databricks/MLflow and is building toward a mature Agent Development Lifecycle (ADLC). S. S.
(role specific) Smartsheet provides a competitive base salary range for roles that may be hired in different geographic areas we are licensed to operate our business from. Actual compensation is determined by several factors including, but not limited to, level of professional, educational experience, skills, and specific candidate location.
In addition, this role will be eligible for a market competitive incentive opportunity. US Base Salary Pay Range $175,000 — $245,000 USD Get to Know Us: At Smartsheet, your ideas are heard, your potential is supported, and your contributions have real impact. You’ll have the freedom to explore, push boundaries, and grow beyond your role.
We welcome diverse perspectives and nontraditional paths—because we know that impact comes from individuals who care deeply and challenge thoughtfully. When you’re doing work that stretches you, excites you, and connects you to something bigger, that’s magic at work. Let’s build what’s next, together.
Equal Opportunity Employer: Smartsheet is an Equal Opportunity (EEO) employer committed to fostering an inclusive environment with the best employees. It is our policy to provide equal employment opportunities to all qualified applicants in accordance with applicable laws in the US, UK, Australia, Germany, Costa Rica, Japan, Bulgaria, and India.
All qualified applicants will receive consideration without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veteran or disabled status, or genetic information. If there are preparations we can make to help ensure you have a comfortable and positive interview experience, please let us know.
About Smartsheet
Visit job-boards.greenhouse.io for more.