Senior AI Engineer - Agentic Systems & LLM Ops Role in Columbus
Career & Market Insight
High-Signal Role in a Rapidly Maturing AI Niche. This position sits at the intersection of agentic AI, LLM operations, and data engineering — three of the fastest-growing specializations in tech right now. Experience shipping production agents (not just prototypes) is increasingly the differentiator between mid-level and senior AI engineers.
Compensation benchmark concern: No salary is listed, which is a mild yellow flag. Senior AI/ML engineers with 7+ years and production LLM experience command $160K–$230K+ annually at comparable companies. The defense/government sector sometimes pays below pure-tech market rates — ask early.
Skills you'll develop:
- Production agentic system design (MCP servers, Agent Skills, multi-agent orchestration) — rare and highly marketable
- LLM FinOps and eval engineering (Langfuse, golden datasets) — an emerging discipline with few experts
- GraphRAG and hybrid retrieval pipelines — differentiated from commodity RAG work
- FedRAMP/DoD compliance engineering — opens doors to a large, stable government tech market
Green flags: Small, senior team; early-stage impact; real production systems (not demos); explicit "AI-native" culture. The roadmap into graph + agents territory is technically ambitious.
Red flags to probe: Defense customers can mean slower release cycles and bureaucratic overhead. "Small team" with incident response duties may imply on-call load. The citizenship requirement narrows talent competition (good for applicants) but may limit future team growth.
Overall, a strong career bet for engineers who want to build serious AI infrastructure and don't mind regulated-industry constraints.
Headquarters: Minneapolis, MN
URL: http://collaboration.ai
Who We Are
Collaboration.Ai is a mission-focused, AI-powered software and services company based in Minnesota, with employees, partners, and customers around the world. We unite people, technology, and purpose to accelerate breakthroughs that transform industries, empower communities, and create a more sustainable future. We collaborate with organizations across the defense ecosystem, helping them navigate complex challenges and drive transformative change.
Our Products
NetworkOS — An AI-powered platform that aligns people, purpose, ideas, and expertise in real-time, generating actionable insights to propel movements forward.
CrowdVector — An integrated solution marketplace and innovation management platform that rapidly uncovers new ideas and advances breakthroughs to fuel movements.
To learn more about us, visit collaboration.ai.
About the Role
You'll build the agentic systems and data pipelines behind NetworkOS's AI capabilities: production agent workflows built on industry-leading agent SDKs and harnesses, MCP servers, and Agent Skills standards; the eval and observability layer that keeps LLM quality measurable; and the ingestion pipelines that turn messy, diverse data sources into queryable knowledge.
This is an execution seat, not an ivory tower. You'll commit code every week, ship agents as product capability rather than demos, and help shape a roadmap heading deep into graph + agents territory — for customers in defense, public sector, and regulated enterprise.
Agents in production. Pipelines that hold. Evals that keep everyone honest.
What You'll Do
- Ship production agent systems — design, build, and operate agentic workflows (agent SDKs, MCP servers, Agent Skills standards) powering AI-driven matching, analysis, and data intelligence
- Operationalize LLM quality — build the eval and observability layer with Langfuse, golden datasets, LLM-as-judge patterns, and FinOps-style tracking so every workflow has measurable quality, cost, and latency
- Engineer data pipelines — robust ingestion of documents, structured data, and external sources into searchable knowledge bases with quality validation, deduplication, and incremental updates
- Own retrieval quality — hybrid search combining vector, keyword, and metadata retrieval, continuously improved through reranking, query expansion, and contextual compression
- Accelerate with AI — build custom MCP tools and Agent Skills that make the whole engineering team measurably faster
- Execute alongside the team — pair with full-stack engineers on AI integration points, contribute to incident response for AI services, and keep your hands in the code
Our Tech Stack
- Languages: Python (primary); Kotlin (core platform language at CAI); TypeScript/Node.js and other modern languages (secondary)
- AI/ML: FastAPI, Pydantic; multi-provider LLM SDKs (Anthropic, OpenAI, and others)
- Agentic Tooling: Claude Code/Codex/etc.; industry-leading agent SDKs and harnesses; MCP servers; Agent Skills standards
- LLM Operations: Langfuse + evals (golden datasets, LLM-as-judge); in-house FinOps tracking (token usage, latency, cost); multi-provider orchestration including AWS Bedrock
- Search & Retrieval: Vector databases, OpenSearch, embedding models
- Data: PostgreSQL, Amazon S3; streaming pipelines (Kafka/Kinesis) where needed
- Infrastructure: Docker, Kubernetes (AWS EKS); DataDog + OpenTelemetry observability
What We're Looking For
Must-Haves:
- 7+ years of professional software engineering experience, with 3+ years focused on AI/ML or data engineering
- Production agentic/LLM application experience — built and operated systems around LLM APIs (Anthropic, OpenAI) serving real users: agents, tool-use, or orchestrated LLM workflows
- Data engineering background — robust, scalable pipelines for AI/ML workloads
- LLM operations experience — evals and observability for production LLM systems (quality, cost, latency)
- Production retrieval experience — vector databases and/or search engines (OpenSearch, Elasticsearch)
- Modern Python stack proficiency — FastAPI, Pydantic, async/await, modern dependency management
- AI-native workflows — demonstrated ability to leverage Claude Code/Codex or similar agentic coding tools to accelerate development
- Experience with Docker, Kubernetes, and AWS
- US citizenship required (DoD contracting — IL4/IL5 environments — and FedRAMP compliance)
Nice-to-Haves:
- Deep agentic ecosystem experience — Agent Skills standards, custom MCP servers, agent SDKs across major vendors
- Advanced RAG expertise — GraphRAG, agentic RAG, contextual retrieval, reranking strategies
- Graph data experience — knowledge graphs, graph databases, or graph-based retrieval
- Model selection & rightsizing — matching models to domain-specific use cases across quality, cost, and latency tradeoffs
- Streaming data experience (Kafka, Kinesis) for real-time knowledge base updates
- Research background, open-source contributions, or an advanced degree in ML/IR/NLP
Why Join Collaboration AI?
- Real AI engineering, not a wrapper shop. Production agents, hybrid retrieval, continuous evals, and a roadmap heading into graph + agents — with the autonomy to shape how it's built.
- AI-native by default. We build with AI, not just for AI. Agentic coding tools, agent SDKs, MCP servers, and Agent Skills standards are how we work daily — you'll both use and build them.
- Work that matters. Defense, public sector, and regulated industries — SOC 2 and NIST compliance, FedRAMP readiness, and customers whose missions demand AI they can trust.
- Small, senior team. Early-stage impact with your work visible from week one. You'll help set the bar for how AI engineering is done here.
To apply: https://weworkremotely.com/remote-jobs/collaboration-ai-senior-software-ai-engineer
Frequently Asked Questions
Is this role truly fully remote or is there an office requirement?
Why is US citizenship required for a remote AI engineering role?
What level of experience is needed for this position?
What is the compensation range for this Senior AI Engineer position?
Do I need prior defense or government sector experience to apply?
