Senior Data Engineer

Job Description

<p><p><b>Number of Positions: 1</b> <b>Duration: 12 months initial contract</b></p>
<p><b>Must be eligible to work in Canada</b></p>
<p><b>Hybrid position, 2-3d/w onsite in Toronto mandatory, 7hr/d billing, no exceptions</b></p>
<p>The <b>Senior Data Engineer</b> needs to have strong cloud technologies and a minimum of 5 years of hands‑on Databricks experience. As well, this individual needs to have strong ETL background. The resource will be working on a major data platform migration and transformation initiative.</p>
<h3>Key Responsibilities</h3>
<ul>
<li>Design and implement robust batch and streaming ingestion pipelines on Databricks using Apache Spark (PySpark/Scala).</li>
<li>Utilize advanced Databricks features like Delta Live Tables (DLT) or Workflows, and Auto Loader for scalable file ingestion.</li>
<li>Model and manage datasets with next‑generation open table formats, preferably <b>Apache Iceberg</b>, registered in Unity Catalog.</li>
<li>Implement end‑to‑end data governance, lineage, and access controls using <b>Unity Catalog</b>.</li>
<li>Build robust, observable data pipelines, including implementing monitoring, alerting, and logging mechanisms.</li>
<li>Take a leadership role in defining Software Engineering best practices for code standardization, branching strategy, optimization, and design patterns.</li>
<li>Collaborate with Solution and Enterprise Architects to design complex data transformation solutions.</li>
<li>Optimize cluster performance and cost using appropriate cluster types and tuning Spark jobs.</li>
</ul>
<h3>Must have skills/experience/qualifications</h3>
<ul>
<li>At least 10 years of strong experience in Data Integration and ETL pipeline development.</li>
<li>Minimum 5 years of hands‑on experience building data pipelines with <b>Databricks</b> and <b>Apache Spark (PySpark/Scala)</b> in cloud environments.</li>
<li>Strong understanding of open table formats, with a preference for <b>Apache Iceberg</b> (snapshots, time travel).</li>
<li>Experience with data governance using <b>Unity Catalog</b>.</li>
<li>Proficiency in orchestration/scheduling and familiarity with Delta Live Tables or similar frameworks.</li>
<li>Solid SQL and data modeling skills.</li>
<li>Experience with streaming ingestion (Spark Structured Streaming) and version control (Git) in a CI/CD environment.</li>
<li>Familiarity with cloud object storage (AWS S3 / Azure Data Lake / GCS).</li>
</ul>
<h3>Seniority level</h3>
<p>Mid‑Senior level</p>
<h3>Employment type</h3>
<p>Contract</p>
<h3>Job function</h3>
<p>Information Technology</p>
<h3>Industries</h3>
<p>IT Services and IT Consulting, Insurance, and Financial Services</p>
<p>Toronto, Ontario, Canada CA$80,000.00-CA$130,000.00</p></p>
#J-18808-Ljbffr