Job Description

Prudential's purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured, for our people, customers, and partners. We provide a platform for our people to do their best work and make an impact to the business, and we support our people's career ambitions. We pledge to make Prudential a place where you can Connect, Grow, and Succeed.
We are seeking a highly motivated Cloud Intelligence Engineer to join our Cloud Economics and Intelligence team under Group Technology Infrastructure Engineering organization. In this hands-on role, you will design and develop AI-driven, agentic solutions on the Databricks platform to transform how cloud financial data is understood, tagged, and cost-optimized. You'll build scalable ML models, intelligent automation pipelines, and advanced analytics frameworks that deliver actionable insights across Azure and GCP environments.
Your primary focus will be on:
- Databricks Genie for interactive AI querying
- Prompt Engineering for optimized model performance
- Mosaic AI Framework for end-to-end AI workflows
- RAG (Retrieval-Augmented Generation) solutions for context-aware, accurate responses
You will collaborate with cross-functional teams to build robust AI pipelines, integrate APIs, enable agent-to-agent communication, and drive innovation in agent decision support
Key Responsibilities

• Design and implement generative AI solutions using Databricks Genie, focusing on natural language querying and interactive AI experiences.

• Build RAG-based applications integrating retrieval mechanisms with LLMs for improved accuracy and relevance.

• Develop and maintain AI workflows using the Databricks Mosaic AI Framework (training, fine-tuning, inference).

• Engineer end-to-end AI pipelines: data ingestion (PySpark, SQL), processing (DLT), and deployment (CI/CD, Asset Bundles).

• Deploy AI products with intuitive UIs using Streamlit or Dash.

• Integrate APIs and build advanced agent systems using ReAct, Reflection, Planning, and Deep Agent frameworks.

• Optimize performance and minimize cloud costs on Azure and GCP through monitoring and cost-saving strategies.

• Leverage LangChain/LangGraph for orchestrating complex AI chains and multi-agent workflows.

• Ensure code quality, testing, documentation, and production readiness.

Technical Skills & Qualifications
Required

• 3-5 years of hands-on experience in AI/ML engineering with a focus on generative AI.

• Proficiency in PySpark and SQL for large-scale data processing.

• Hands-on experience with Databricks Genie for AI querying and interaction.

• Expertise in Prompt Engineering and optimizing LLM performance.

• Strong knowledge of Databricks Mosaic AI Framework (model development, serving, monitoring).

• Experience building RAG solutions (vector stores, embeddings, retrieval pipelines).

• Proficiency with LangChain and LangGraph for AI orchestration and agent workflows.

• CI/CD for AI/ML using Databricks Asset Bundles and automated deployment pipelines.

• API integration and Agent-to-Agent communication patterns.

• Advanced agent frameworks: ReAct, Reflection, Planning, Deep Agent systems.

• Databricks Lakehouse architecture and Delta Live Tables (DLT) for real-time pipelines.

• Deployment of AI apps using Streamlit or Dash.

• Good understanding of Azure and GCP infrastructure and cost optimization strategies (e.g., auto-scaling, spot instances, resource tagging).

• Bachelor's degree in Computer Science, Data Science, Engineering, or related field (or equivalent experience).

Preferred

• Experience with MLOps practices and production monitoring of AI systems.

• Familiarity with Databricks Unity Catalog, Model Registry, and Feature Store.

• Background in vector databases (e.g., FAISS, Pinecone, Databricks Vector Search).

Professional & Soft Skills

• Excellent communication skills - ability to explain complex AI concepts to technical and non-technical stakeholders.

• Strong problem-solving and analytical thinking in fast-paced environments.

• Collaborative team player with experience in cross-functional projects.

• Attention to detail in code, documentation, and system design.

• Proactive learner - stays updated with advancements in generative AI and Databricks features.

• Ownership mindset - takes responsibility for deliverables from development to production.

Prudential is an equal opportunity employer. We provide equality of opportunity of benefits for all who apply and who perform work for our organisation irrespective of sex, race, age, ethnic origin, educational, social and cultural background, marital status, pregnancy and maternity, religion or belief, disability or part-time / fixed-term work, or any other status protected by applicable law. We encourage the same standards from our recruitment and third-party suppliers taking into account the context of grade, job and location. We also allow for reasonable adjustments to support people with individual physical or mental health requirements.

Skills Required