Senior Prompt Engineer - LLM Evaluation & AI Role in Columbus
Career & Market Insight
Prompt Engineering is one of the fastest-growing niches in AI, and this role sits at a technically sophisticated end of the spectrum — focused on autorater systems, model evaluation metrics (F1, precision, recall), and optimization workflows rather than simple chatbot tuning. That's a meaningful differentiator on a CV.
Skills developed here are in high demand:
- LLM evaluation methodology (gold data benchmarking, shadowbot monitoring)
- Automated Prompt Optimization (APO) — still rare expertise in the market
- Chain-of-thought and few-shot prompting for structured classification
- SQL-based model performance analysis
Compensation concern: No salary range is listed, which is unusual for a senior role (4+ years experience required). In Germany, senior Prompt Engineers or AI specialists typically earn €70,000–€100,000+ annually. Candidates should probe this in interviews.
Culture signals: The mention of "client's internal tools" and migration workflows suggests this is a vendor/contractor role supporting a larger tech company rather than an in-house position. This may limit long-term career growth within the organization but can build highly specialized, portable expertise.
Green flags: Genuine technical depth; exposure to proprietary enterprise LLM infrastructure; dual-language (German/English) requirement adds scarcity value. Watch out for: No benefits listed, no salary transparency, and the contractor framing — clarify employment type and stability before accepting.
We are seeking a Prompt Engineer to be responsible for the end-to-end technical migration workflow for transitioning templates to LLM autoraters. The role requires using client's internal tools to leverage prompt engineering techniques to maximize model performance.
Responsibilities
- Utilize Automatic Prompt Generation (APG) tools to create baseline prompts for complex parent-child template clusters.
- Run and supervise Automated Prompt Optimization (APO) tool, review the outputs, and flag when the APO reaches deadlocks or plateaus.
- Manually draft, test, and refine prompts to navigate complex template architectures, overcome anti-patterns, and handle edge cases where tooling is lacking or broken. Solve edge-case scenarios by designing and refining manual prompts.
- Monitor shadowbot runs to ensure sufficient disagreements (between human and LLM ratings) are registered, generated, and tracked.
- Run prompt versions against established gold data to continuously measure autorater quality against the human crowd baseline, calculating accuracy metrics such as F1 scores, precision, and recall.
- Draft technical launch readiness justifications (Launch Certification Documentation) for final approval.
Requirements
- Language Skills: Native fluency in German and fluent in English.
- Location: Must be based in Germany.
- Education: Bachelor's, Master's, or Doctorate degree in Computer Science, Data Science, Computational Linguistics, Human-Computer Interaction (HCI), Cognitive Science, or a related analytical field.
- Prompt Engineering & AI Expertise: At least 4 years' experience as a Prompt Engineer. Proven experience tuning Large Language Models (LLMs) for strict, structured outputs, complex classification tasks, and familiarity with chain-of-thought and few-shot learning.
- Data Analysis: Strong proficiency in identifying error patterns, analyzing model performance, and using SQL or other data analytics tools.
- Technical Agility: Ability to quickly learn and master proprietary tools with minimal supervision.
- Communication: Excellent verbal and written communication skills.
Optional / Preferred Skills
- Familiarity with enterprise-grade LLM interfaces like the Goose API.
- Experience in AI model evaluation, data science, computational linguistics, or software engineering.
- Hands-on experience with Automated Prompt Optimization (APO) systems or tuning workflows.
- Linguistic expertise, including an understanding of semantics and logic.
Frequently Asked Questions
Is this role fully remote or office-based?
What language skills are required for this position?
What level of experience is needed to apply?
Is salary information available for this role?
What technical tools or skills should I highlight in my application?
