Senior Databricks Security & AI Automation Engineer

Responsibilities

Databricks Platform Security Architecture & Operations

• Own the end-to-end security architecture, configuration, and hardening of the Databricks platform across account and workspace administration, Unity Catalog, compute, Databricks SQL, Mosaic AI / Model Serving, and Databricks Apps.

• Design and maintain network isolation strategy, including VNet/VPC injection, secure cluster connectivity, Private Link / PrivateLink, IP access lists, serverless network policies, and egress controls across Azure and AWS deployments.

• Architect identity and access: SSO and SCIM provisioning from the corporate IdP, group-based entitlements, service principal standards, OAuth token policy, and elimination of long-lived personal access tokens.

• Build and maintain the Unity Catalog governance model (metastore, catalog, schema, and volume layout; privilege inheritance; row filters and column masks; attribute-based access control) aligned to data classification, business-unit ownership, and regulatory scope (banking regulatory, GLBA, PCI-DSS, SOX).

• Define and enforce compute guardrails through cluster policies, compute policies, init script and library allow-lists, secret scopes, and customer-managed keys for storage and workspace encryption.

• Deliver the platform as code: Terraform-managed workspaces, policies, and grants with drift detection, so every security control is reviewable, repeatable, and auditable.

• Serve as the Databricks security subject-matter expert for the security organization, reviewing new data and AI use cases, managing vendor escalations, and evaluating new Databricks features and security releases.

AI Agent Development for Data Platform Security Automation

• Design, build, and operationalize AI agents that automate routine Databricks security workflows: access request review, entitlement recertification, privilege-drift triage, secrets and token hygiene, policy exception processing, and audit evidence assembly.

• Develop prompt-engineered investigation workflows that correlate Databricks audit logs and system tables with identity data, data classification, and threat intelligence to produce prioritized, analyst-ready findings.

• Build monitoring agents that detect and triage risky behavior such as anomalous bulk data access, unauthorized external sharing, over-permissioned service principals, public or misconfigured storage locations, and policy bypass attempts.

• Create AI-driven posture agents that continuously assess workspace and account configuration against internal standards and industry benchmarks, open remediation tickets, and track them to closure.

• Build automated reporting agents that generate executive dashboards, regulatory and audit summaries, and business-unit-level data-platform risk scorecards without manual intervention.

• Secure the AI workloads themselves: define guardrails for model serving endpoints, vector search, agent frameworks, and MCP / tool integrations, including prompt injection defenses, least-privilege tool access, and output logging.

• Continuously tune agent logic based on false-positive analysis, remediation outcome data, and feedback from data engineering and data science teams to improve accuracy and reduce noise.

• Evaluate and integrate the Databricks REST API, SDK, and system tables into AI orchestration pipelines for programmatic configuration review, access analysis, and automated evidence collection.

Data Platform Security Program Operations

• Partner with data engineering, data science, analytics, and cloud platform teams to embed security into pipelines, notebooks, and model lifecycles, and to drive remediation of critical and high-risk findings within defined SLA timelines.

• Develop and maintain data-platform security KPIs, dashboards, and trend analyses (coverage of Unity Catalog, token and secret hygiene, privileged access, policy exceptions) that give leadership actionable risk visibility.

• Instrument Databricks audit logs and system tables into Splunk for correlation with broader security telemetry, enabling enriched detection, investigation, and response workflows.

• Own data protection controls on the platform in partnership with the data governance and DLP teams: classification tagging, masking and tokenization patterns, Delta Sharing and external-sharing review, and data exfiltration monitoring.

• Manage the exception and risk acceptance process for data-platform controls, ensuring documentation meets regulatory and audit requirements.

• Maintain runbooks, architecture diagrams, and operational documentation that enable the broader security team to support Databricks security operations and incident response.

• Track the data and AI security landscape (Databricks security bulletins, CVEs, CISA KEV, vendor advisories, emerging AI threats) and translate emerging risks into actionable control and monitoring priorities.

Qualifications:

• 5+ years in cloud security, data security, or security engineering, including 2+ years of deep, hands-on Databricks experience in a production enterprise environment.

• Expert-level knowledge of Databricks security features: Unity Catalog and fine-grained access control, workspace and account administration, cluster and compute policies, secrets management, audit logging and system tables, and encryption with customer-managed keys.

• Demonstrated experience with Databricks identity integration (SSO, SCIM, service principals, OAuth) and with the Databricks REST API or SDK for programmatic configuration, auditing, and automation.

• Strong scripting and automation skills in Python (strongly preferred) for building integrations, data pipelines, and reporting automation, plus Infrastructure as Code experience (Terraform preferred).

• Solid understanding of cloud networking and security on Azure and/or AWS: private connectivity, segmentation, storage security, key management, and IAM.

• Experience managing data security and access governance in regulated financial services environments (banking regulatory, GLBA, PCI-DSS, SOX) with demonstrated ability to drive remediation accountability.

• Working knowledge of data protection techniques (classification, masking, tokenization, DLP) and of common risks in data and analytics platforms, including over-permissioned access, credential and token sprawl, and data exfiltration.

• Experience with SIEM integration and log engineering (Splunk preferred) and with ticketing systems (ServiceNow, Jira) for workflow automation.