Senior Data Engineer

Position Title: Senior Data Engineer

Department: Data Science

Reports To: Team Lead

Shift: Flexible

Position Summary:

The Senior Data Engineer will design, build, and maintain our data infrastructure, ensuring seamless data integration and optimizing data processing for high-performance analytics. The ideal candidate will have a strong background in data engineering, database management, and collaboration with cross-functional teams to drive data-driven decision-making across the organization.

Essential Duties and Responsibilities:

The essential duties of the position include the following, and other duties may be assigned:

• Ensure the seamless integration of data from various sources, enabling consistent and accurate data flow across platforms.
• Optimize data processing and query performance by fine-tuning data pipelines, configuring databases, and implementing effective data partitioning strategies.
• Establish and enforce data quality checks and validations to identify and resolve data inconsistencies, ensuring high-quality, reliable data for downstream applications and analytics.
• Work closely with cross-functional teams, including data scientists, analysts, software engineers, and business stakeholders, to understand data requirements and deliver solutions that support business objectives.
• Design and develop robust data infrastructure, including data warehouses, data lakes, and scalable data pipelines, to support the organizations growing data needs.
• Conduct regular design process reviews and ensure adherence to development standards within the data engineering team.
• Monitor and optimize query performance, identify performance bottlenecks, and implement solutions to enhance overall database efficiency and reliability.
• Create and maintain comprehensive documentation related to database design, configuration, and processes, ensuring clarity and continuity for ongoing development and maintenance efforts.
• Stay updated on industry best practices and emerging technologies in data engineering, and proactively implement improvements to enhance data architecture and processes.

Qualifications:

To perform this job successfully, an individual must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform essential functions.

Education/Experience:

Bachelors or Masters degree in Computer Science, Information Systems, or a related field. At least 5+ years of experience as a Data Engineer or in a similar role. Experience with data governance, data security, and compliance considerations in data engineering. Proven experience in data engineering, including data integration, pipeline optimization, and database management. Strong knowledge of data modeling, ETL processes, and data warehousing concepts. Proficiency in SQL and experience with database systems such as MySQL, PostgreSQL, or NoSQL databases. Experience with big data technologies and cloud platforms like AWS, Azure, or Google Cloud is a plus. Excellent problem-solving skills and the ability to work effectively in a collaborative, fast-paced environment. Strong communication skills, with the ability to convey complex technical concepts to non-technical stakeholders.

Language Ability:

Strong verbal and written communication skills, enabling effective collaboration with technical and non-technical stakeholders. Ability to create clear, concise, and comprehensive technical documentation, including design specifications, user guides, and operational procedures. Proficiency in articulating complex data concepts and solutions to a variety of audiences, ensuring alignment across teams.

Math Ability:

Proficient in mathematical concepts essential for data engineering, including linear algebra, probability, and statistics. Capable of applying mathematical techniques to optimize data storage, query performance, and data processing tasks. Skilled in analyzing data trends, patterns, and anomalies using statistical methods to support data-driven decision-making.

Reasoning Ability:

Demonstrated ability to approach data challenges with logical, structured thinking, and to develop innovative solutions to complex problems. Strong analytical skills for troubleshooting and resolving data issues, optimizing processes, and improving system performance. Ability to evaluate the impact of design choices on system scalability, efficiency, and data integrity, ensuring robust and reliable data architecture.

Computer Skills:

Advanced proficiency in SQL, including complex queries, indexing, and query optimization within various database systems (e.g., MySQL, PostgreSQL, Oracle). Hands-on experience with data pipeline tools and big data technologies, enabling efficient data processing and integration. Expertise in cloud computing platforms (e.g., AWS, Azure, Google Cloud), with experience in deploying and managing data infrastructure in a cloud environment. Proficient in scripting languages (e.g., Python, Bash) for automating data workflows and performing data transformations. Familiarity with version control systems (e.g., Git) and collaboration tools (e.g., Jira, Confluence) to manage code and track project progress.

Certificates and Licenses:

Relevant certifications are highly desirable.

Travel:

This position does not require travel.

Work Environment:

The work environment characteristics described here are representative of those an employee encounters while performing the essential functions of this job in an office environment. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. A typical, climate-controlled office environment exists. The worker is not substantially exposed to adverse environmental conditions.

ISM and Privacy Statement:

• Data Security and Encryption: Ensure that sensitive data, especially personal and financial data, is encrypted and securely stored within systems.
• Access Control and Authorization: Implement role-based access controls and ensure that system permissions align with job functions.
• System Security: Perform regular security audits, vulnerability testing, and maintain a secure development environment to avoid unauthorized data access.
• Compliance with Privacy Laws: Ensure compliance with privacy regulations (e.g., GDPR) and data retention policies when handling personal data.