Open Source Engineer

About LanceDB

LanceDB is a developer-friendly, open-source database for multimodal AI. From hyper-scalable vector search to advanced retrieval for RAG, from streaming training data to interactive exploration of large-scale AI datasets, LanceDB is the best foundation for your AI application, and powers some of the most groundbreaking applications and challenging requirements today.

About the role

Join our world-class team as an experienced Open Source Engineer, where you'll drive the evolution of high-performance multimodal databases. You'll leverage your expertise in Java/Scala and Rust to expand the reach of Lance and LanceDB within the broader Data Infrastructure ecosystem, contributing to cutting-edge open source projects.

As an Open Source Engineer at LanceDB, your responsibilities will include:

• Driving OSS community effort to integrate Lance format into Spark, Hive Metadata Store, Presto, Trino, Ray and other Data Infrastructure systems.

• Helping promote Lance format in Big Data conferences and meetups.

• Designing and maintaining efficient distributed Lance dataset operations.

• Designing efficient indices to power predicate push down in Spark, Ray or Trino.

• Work on table format, data encodings and various aspect of the Lance format in Rust.

• Operating on in-house data processing infrastructure.

Requirements:

• You have at least five years of experience building high-performance databases, big data systems, or web-scale data services.

• Experience with internals of Open Source Big Data or AI training systems, such as Hadoop, Spark, Flink, Ray, Iceberg, Delta-lake, Hudi, Clickhouse, Trino, Presto, PyTorch or JAX.

• Hands-on experience with high-performance computing in Java or Scala.

• You like working with a small, high-caliber team with a lot of autonomy and drive, and you can iterate fast

Nice to have:

• You are an open-source veteran, committer or PMC of large Open Source systems in the Apache community.

• You fearlessly challenge the status quo and dismiss mediocre engineering as unacceptable.

• You have a proven record driving large features in Apache projects.

• You are familiar with Java, Rust, C++, Apache Arrow, Apache DataFusion, Apache Parquet, Apache Iceberg, and Delta Lake.

About the LanceDB team:

LanceDB was created by experts with decades of experience building tools for data science and machine learning. From co-authors of pandas to Apache PMC of HDFS, Arrow and Delta, the LanceDB team has created open source tools used by millions world-wide.