Job Summary:
As a Databricks Lead, you will be a critical member of our data engineering team, responsible for designing, developing, and optimizing our data pipelines and platforms on Databricks, primarily leveraging AWS services. You will play a key role in implementing robust data governance with Unity Catalog and ensuring cost-effective data solutions. This role requires a strong technical leader who can mentor junior engineers, drive best practices, and contribute hands-on to complex data challenges.
Responsibilities:
* Databricks Platform Leadership:
* Lead the design, development, and deployment of large-scale data solutions on the Databricks platform.
* Establish and enforce best practices for Databricks usage, including notebook development, job orchestration, and cluster management.
* Stay abreast of the latest Databricks features and capabilities, recommending and implementing improvements.
* Data Ingestion and Streaming (Kafka):
* Architect and implement real-time and batch data ingestion pipelines using Apache Kafka for high-volume data streams.
* Integrate Kafka with Databricks for seamless data processing and analysis.
* Optimize Kafka consumers and producers for performance and reliability.
* Data Governance and Management (Unity Catalog):
* Implement and manage data governance policies and access controls using Databricks Unity Catalog.
* Define and enforce data cataloging, lineage, and security standards within the Databricks Lakehouse.
* Collaborate with data governance teams to ensure compliance and data quality.
* AWS Cloud Integration:
* Leverage various AWS services (S3, EC2, Lambda, Glue, etc.) to build a robust and scalable data infrastructure.
* Manage and optimize AWS resources for Databricks workloads.
* Ensure secure and compliant integration between Databricks and AWS.
* Cost Optimization:
* Proactively identify and implement strategies for cost optimization across Databricks and AWS resources.
* Monitor DBU consumption, cluster utilization, and storage costs, providing recommendations for efficiency gains.
* Implement autoscaling, auto-termination, and right-sizing strategies to minimize operational expenses.
* Technical Leadership & Mentoring:
* Provide technical guidance and mentorship to a team of data engineers.
* Conduct code reviews, promote coding standards, and foster a culture of continuous improvement.
* Lead technical discussions and decision-making for complex data engineering problems.
* Data Pipeline Development & Optimization:
* Develop, test, and maintain robust and efficient ETL/ELT pipelines using PySpark/Spark SQL.
* Optimize Spark jobs for performance, scalability, and resource utilization.
* Troubleshoot and resolve complex data pipeline issues.
* Collaboration:
* Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.
* Communicate technical concepts effectively to both technical and non-technical stakeholders.
Qualifications:
* Bachelor's or Master's degree in Computer Science, Data Engineering, or a related quantitative field.
* 7+ years of experience in data engineering, with at least 3+ years in a lead or senior role.
* Proven expertise in designing and implementing data solutions on Databricks.
* Strong hands-on experience with Apache Kafka for real-time data streaming.
* In-depth knowledge and practical experience with Databricks Unity Catalog for data governance and access control.
* Solid understanding of AWS cloud services and their application in data architectures (S3, EC2, Lambda, VPC, IAM, etc.).
* Demonstrated ability to optimize cloud resource usage and implement cost-saving strategies.
* Proficiency in Python and Spark (PySpark/Spark SQL) for data processing and analysis.
* Experience with Delta Lake and other modern data lake formats.
* Excellent problem-solving, analytical, and communication skills.
Added Advantage (Bonus Skills):
* Experience with Apache Flink for stream processing.
* Databricks certifications.
* Experience with CI/CD pipelines for Databricks deployments.
* Knowledge of other cloud platforms (Azure, GCP) is a plus.
Rydal Park, a premier Life Plan Community located in Jenkintown, is hiring part-time Certified Nursing Assistants (CNA) to join our Medical Center team! Our Certified Nursing Assistants actively incorporate our Philosophy for Person-Directed Care. You would develop...
...Azure and Snowflake Data Engineer Technology Lead - US Technology|Analytics - Packages|Python - Big Data, Technology|Cloud Integration|Azure Data Factory (ADF), Technology|Data on Cloud-DataStore|Snowflake Company ITL USA Requisition ID 133466BR Infosys...
Starbucks Barista-Part Time Job Category: Food & BeverageRequisition Number: STARB053530Location: GA, Westin Duluth GAWEDU | 293 6450 Sugarloaf Parkway Duluth, GA 30096, USADescription We are hiring a Barista / Bistro Server. Join our team as we are currently looking...
...organizational and time management skills. Ability to prioritize tasks and manage multiple responsibilities. Requirements: Valid LPN license in the state of practice is required. Minimum 1 year of experience in a healthcare setting, preferably in assisted living...
...giving back to our communities, and pursuing personal growth. Come grow with us and be part of... ...Banker III, who is an experienced advisor in establishing, developing and retaining... ...relationships with members to help meet their financial goals. The right candidate will have the...