Data Engineer, EDM III

Position Overview:Autodesk is looking for a talented and motivated Sr Data Engineer to join our Analytics team to develop robust and scalable data pipelines using and improving existing platforms.The successful candidate will drive performance enhancements, build pipelines and collaborate with other Data Engineering teams across Autodesk. You will work with cutting edge technologies in big data space.Responsibilities:Deploy and configure Apache Spark clusters on AWS (particularly EKS), optimizing for performance, resource allocation, and scalability.Develop, automate, and support robust and reliable Spark data pipelines, focusing on high performance and low latency.Design and implement highly optimized queries to improve data processing efficiency, streamline analysis, minimize latency, and enhance overall system performance.Collaborate to make the Data Platform an effective, scalable, and resilient system.Gather requirements from stakeholders, prioritize work, and document technical solutions clearly and effectively.Engage with data engineers, data scientists, product managers, and internal stakeholders to align project goals and implementations.Contribute to a team culture that values quality, robustness, and scalability while fostering initiatives and innovationMinimum Qualifications:5+ years of data processing experience in large cloud-based infrastructure (AWS is must)Hands-on software development experience in Python, with strong proficiency in Py Spark for data engineering tasks and data pipeline developmentExpert understanding of SQL, dimensional modeling, and analytical data warehouses, such as SnowflakeUnderstanding of Data Engineering best practices for medium to large scale production workloadsExpertise with data pipeline orchestration tools, such as AirflowFamiliar with processing semi-structured file formats such as json or parquetTeam player with good communication skillsProblem solver with excellent written and interpersonal skillsStrong problem-solving skillsAbility to learn new technical skillsBachelor s degree in Computer Science, Data Science, or related fields Preferred Qualifications:Experience with Jinja, Shell scripting, DBTDeveloping in Cloud platform using serverless technologies such as glue, lambda, EMR is a plus.Experience with remote development using AWS SDK is a plusExperience with ELT pipelines - DBTREST API design and implementationFamiliarity with containers and infrastructure-as-code principlesExperience with automation frameworks - Git, Jenkins, and Terraform