Data Testing - Spark

As a Data Testing Specialist focusing on Spark, you will play a crucial role in ensuring the quality and integrity of our data processes. Your expertise in working with big data technologies and Spark framework will enable you to design, implement, and enhance testing strategies that ensure the reliability and accuracy of the data pipelines we create.

You will collaborate closely with our data engineering teams to validate data flows, perform regression tests, and verify that all data transformations meet specified requirements. Your role is pivotal in identifying data discrepancies, issues, and performance bottlenecks, providing valuable feedback to improve our data architecture and processes.

Key Responsibilities:

• Serve as the automated quality assurance specialist within the team, supporting and collaborating with Engineers on testing strategies and their technical implementation.
• Work alongside developers and data engineers to grasp intricate system requirements and convert them into comprehensive automated test plans.
• Assist in early identification of dependencies, risks, and quality discrepancies throughout the development lifecycle.
• Design and develop automated test cases, including integration, end-to-end, load, and performance testing for data-driven applications.
• Integrate testing practices within CI/CD pipelines to facilitate swift and dependable deployments.
• Actively engage in:
• Decomposing deliverable components into valuable increments.
• Defining success metrics, including the Definition of Done (DOD).
• Demonstrating business value in frequent ceremonies.
• Maintaining live documentation of the domain and platform to promote knowledge sharing and distribution.
• Participate in design discussions, technical evaluations, and contribute to the ongoing enhancement of testing standards and practices.
• Guide team members on testing frameworks, industry best practices, and strategies for effective test automation.
• Possess substantial experience with Data Engineering testing frameworks and strategies.

Required Qualifications:

Required Skills:

• Proficiency in distributed computing technologies (such as Spark) and software development.
• Adept at working with databases, with a preference for Postgres.
• Solid grasp of Object-Oriented Programming principles and development methodologies.
• Experience in Agile development environments, including Scrum or Kanban.
• Familiarity with version control systems, preferably Git.
• Knowledge of Continuous Integration and Continuous Deployment (CI/CD) pipelines.
• Experience in automated testing, encompassing integration, delta, load, and performance testing.
• Extensive experience in database testing, ideally with Postgres.

Good to Have Skills:

• Proficiency in working with Docker and Containers.
• Experience with Spark and Scala.
• Background in Data Engineering.
• Familiarity with Python.
• Experience in .NET Core.

We Offer

• Flexible working hours (self-managed)
• Competitive salary
• Annual bonus, subject to company performance
• Access to Udemy online training and opportunities to learn and grow within the role

About Mindera

At Mindera we use technology to build products we are proud of, with people we love.

Software Engineering Applications, including Web and Mobile, are at the core of what we do at Mindera.

We partner with our clients, to understand their product and deliver high performance, resilient and scalable software systems that create an impact in their users and businesses across the world.

You get to work with a bunch of great people, where the whole team owns the project together.

Our culture reflects our lean and self-organisation attitude. We encourage our colleagues to take risks, make decisions, work in a collaborative way and talk to everyone to enhance communication.

We are proud of our work and we love to learn all and everything while navigating through an Agile, Lean and collaborative environment.

Follow our Linkedln page - https://tinyurl.com/minderaindia

Check ot our Blog: http://mindera.com/ and our Handbook: http://bit.ly/MinderaHandbook

Our offices are located: Aveiro, Portugal | Porto, Portugal | Leicester, UK | San Diego, USA | San Francisco, USA | Chennai, India | Bengaluru, India