Job Purpose

The GPU Cluster System Engineer will play a role at the forefront of optimizing performance for GPU clusters and
help ensure the healthiness of the cluster. The ideal candidate will have a strong background in GPU architecture,
parallel computing, and hands-on experience in system level performance tuning, debug methodologies, and data center
management.

Key Responsibilities:

• Performance Optimization: Collaborate with hardware and software teams to enhance the overall performance of
  GPU clusters, focusing on aspects such as throughput, latency, and GPU utilization.
• Benchmarking and Analysis: Develop and execute comprehensive benchmarking strategies to assess baseline
  performance, analyze bottlenecks, and identify areas for improvement within GPU cluster environments.
• Cluster Stability: Evaluate the stability of GPU clusters by conducting thorough testing under various
  workloads, ensuring optimal performance across of the cluster at the large scales.
• Lead and manage the growth of our open-source community from projects such as LLM360.
• Performance Profiling: Utilize profiling tools and methodologies to analyze and identify performance
  bottlenecks, providing actionable insights for improvement.
• Performance Tuning: Implement optimization strategies, including but not limited to protocol enhancements,
  load balancing techniques, and parallel processing optimizations.
• Documentation: Create detailed documentation of performance analysis, tuning efforts, and outcomes, providing
  clear and concise reports for internal teams and stakeholders.
• Collaboration: Work closely with cross-functional teams, including hardware engineers, software developers,
  and system architects, to integrate performance improvements into the GPU cluster architecture.
• Continuous Learning: Stay current with the latest developments in GPU architectures, parallel processing, and
  emerging technologies to drive continuous improvement in GPU cluster performance.

Academic Qualifications

• Bachelor’s Degree or higher in a related technical field (computer science, high performance computing) or the
  equivalent professional experience.

Professional
Experience

• Proven experience in optimizing the performance of GPU clusters.
• Strong understanding of GPU architectures, parallel computing concepts, and network protocols.
• Proficiency in scripting languages (e.g., Python, Bash) for automation and performance analysis.
• Experience with system level performance analysis tools and methodologies for GPU clusters.
• Analytical mindset with excellent problem-solving and debug skills.
• Familiarity with cluster management tools and systems.
• Excellent communication and collaboration skills for effective teamwork.
• RDMA network configuration, troubleshooting and performance tuning.
• Linux kernel networking expertise
• Machine learning and/or HPC system design