EY - GDS Consulting - AI And DATA -Azure Databricks-Senior

At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.

EY GDS – AI & Data Databricks Senior Data Engineer AI-enabled data platform engineering   The opportunity Our AI & Data practice helps clients build trusted, modern data platforms that power analytics, operational decisions, and responsible AI. As a Databricks Senior Data Engineer, you will own end-to-end data products and technical workstreams, from design through production operation. You will combine strong engineering fundamentals with practical use of native AI capabilities on Databricks, always prioritising business value, security, and reliable production operation.   Your key responsibilities Data engineering and platform delivery

Build lakehouse solutions using Delta Lake, Lakeflow, Databricks SQL, Unity Catalog, and medallion or domain-oriented data-product patterns. Engineer secure batch and streaming pipelines using Spark, Delta Live Tables / Lakeflow Spark Declarative Pipelines, Auto Loader, workflows, and declarative quality expectations. Design and own scalable batch and streaming data products, selecting appropriate data models, processing patterns, quality controls, and service-level expectations. Troubleshoot complex performance, reliability, and cost issues; establish reusable engineering patterns and guide code reviews and delivery standards. Translate business requirements into reusable, tested data products with clear ownership, contracts, documentation, data-quality checks, and service-level expectations.

AI-ready data and native AI capabilities

Use Mosaic AI, AI/BI Genie, Agent Bricks, vector search, model serving, MLflow, and Databricks Assistant / Genie Code capabilities where appropriate to accelerate governed AI delivery. Design AI-ready data products and implement production patterns for retrieval, semantic search, evaluation, safety, and reliable operation. Assess when a governed RAG or agent workflow is justified and implement the supporting ingestion, metadata, access-control, evaluation, and deployment foundations. Develop and test scoped system prompts, context-assembly patterns, agent skills, and approved MCP integrations with clear tool contracts, least-privilege access, input/output validation, and traceability. Partner with data scientists, analytics teams, security, and business stakeholders to select the right pattern: deterministic analytics, semantic layer, retrieval-augmented generation (RAG), agent workflow, or model-based solution. Ensure AI solutions have documented data sources, access controls, quality thresholds, evaluations, human oversight where needed, and clear release controls.

Governance, security, and engineering excellence

Apply Unity Catalog governance for data, AI assets, lineage, access control, discovery, auditability, and cost-aware compute design. Implement automated testing, source control, code reviews, CI/CD, release controls, monitoring, alerting, incident learning, and clear runbooks. Work in Agile teams and communicate progress, dependencies, risks, and design decisions clearly to technical and non-technical stakeholders. Lead technical delivery for a workstream, mentor engineers, and communicate design choices and delivery risks to client stakeholders.

Skills and attributes for success

Hands-on Python, PySpark, SQL, Delta Lake, Spark performance tuning, data modelling, and Git-based engineering practices. Databricks workspace administration concepts, jobs/workflows, compute policies, CI/CD, infrastructure as code, and Azure, AWS, or GCP integration. Practical experience implementing or supporting RAG, semantic search, agent workflows, context