AI Engineer, Inference

ROLE SUMMARY The (Senior) AI Engineer (Inferencing) will build and improve the AI & Applications team’s inference capability, making models available as reliable, secure, scalable, and high-performance endpoints for internal products, external customers, and future Inference-as-a-service offerings. The role will establish the engineering foundation for self-hosted model serving in the organization’s AI-factory environment. This includes model onboarding, deployment, endpoint provisioning, runtime selection, performance benchmarking and optimization, observability, capacity management, security, and operational lifecycle management. The objective is to provide users with predictable and efficient access to models while maintaining control over performance, cost, data handling, deployment configuration, and infrastructure utilization. The role is a key contributor to the Model-to-Grid product and agentic applications roadmap. It will convert model and runtime characteristics into benchmarked, repeatable inference recipes and endpoint profiles that can inform workload scheduling, topology-aware placement, capacity planning, performance recommendations, and operational decision-making. It will also provide the governed and fit-for-purpose model endpoints needed by agentic systems for reasoning, retrieval, tool use, diagnosis, recommendation, and controlled automation. KEY RESPONSIBILITIES • Build, operate, and continuously improve self-hosted AI inference services for internal applications, customer-facing products, and future Inference-as-a-service offerings. • Define and implement standard model-onboarding workflows covering model intake, compatibility validation, packaging, runtime selection, optimization, deployment, endpoint registration, testing, release, and lifecycle management. • Provision and manage secure, scalable inference endpoints for common AI application patterns, including interactive generation, RAG, embeddings, reranking, batch processing, multimodal use cases, tool calling, and agentic workflows. • Develop reusable deployment templates, APIs, SDKs, configuration standards, and self-service workflows for users to request, configure, access, monitor, update, and retire model endpoints. • Work with leading inference frameworks and toolkits, such as TensorRT-LLM, TensorRT, SGLang, vLLM, Triton Inference Server, NVIDIA Dynamo, NVIDIA NIM, CUDA, cuDNN, NCCL, and related serving, profiling, and observability tools. • Optimize model-serving performance using appropriate techniques, including quantization, compilation, batching, continuous batching, request routing, KV-cache management, prefix caching, speculative decoding, load balancing, model routing, memory optimization, and distributed parallelism. • Build and validate reusable inference recipes that specify compatible model versions, framework and runtime versions, precision formats, GPU configurations, topology requirements, scaling approaches, scheduler profiles, benchmark results, and expected performance envelopes. • Use quantization and optimization approaches such as NVFP4, FP8, INT8, TensorRT compilation, kernel optimization, efficient attention mechanisms, and memory-management techniques while maintaining agreed model-quality targets. • Design distributed inference configurations for large models, including tensor, pipeline, expert, context, and data parallelism where appropriate. • Work with the Kubernetes and proprietary scheduler team to define endpoint resource profiles, placement requirements, topology preferences, priority classes, quota models, autoscaling rules, capacity reservations, and workload-management policies. • Contribute inference workload characteristics, benchmarks, and performance profiles to the Model-to-Grid product so that endpoint placement, scheduling, capacity planning, and AI-factory operations can make more informed decisions. • Build benchmarking and qualification workflows using controlled experiments, reproducible baselines, load tests, latency tests, throughput tests, concurrency tests, scaling tests, performance profiling, regression testing, and internal or industry-standard benchmark methodologies where relevant. • Measure and improve key inference indicators, including time-to-first-token, inter-token latency, tokens per second, requests per second, end-to-end latency, concurrency, GPU utilization, memory efficiency, cache hit rate, scaling efficiency, power efficiency, and cost efficiency. • Establish automated performance-regression testing and release qualification for model versions, runtime and toolkit upgrades, CUDA and driver changes, Kubernetes releases, scheduler changes, networking and storage changes, and new GPU platforms. • Build operational observability for inference services, including endpoint availability, request volume, latency, queueing, errors, GPU utilization, GPU memory use, cache behavior, capacity, cost, power, and service-level objectives. • Partner with the agentic applications team to provide fit-for-purpose self-hosted endpoints for agent planning, retrieval, tool use, summarization, diagnosis, recommendation, optimization, and AI-factory operations. • Expose governed inference, benchmark, recipe, performance, and capacity information to agentic systems, allowing them to recommend suitable models, identify degradation, diagnose bottlenecks, plan optimization experiments, and validate results. • Work with Product, UX, DevOps, Platform, Infrastructure, Security, and Global Operations teams to ensure that inference provisioning, model selection, endpoint configuration, performance visibility, quota management, and troubleshooting are clear, secure, and operationally supportable. SKILLS AND EXPERIENCE • 5+ years of software engineering experience, including 3+ years in AI inference, model serving, ML systems, high-performance computing, distributed systems, or comparable performance-critical environments. • Demonstrated experience b…