Data Engineer Job

Data Engineer with AI/ML Experience
Experience: 4 Years
Responsibilities
•Design, develop, and maintain scalable data pipelines and ETL/ELT processes to support analytics, machine learning, and Generative AI applications.
•Develop data ingestion and transformation pipelines using Python, SQL, PySpark, Apache Beam, and Apache NiFi.
•Integrate data from multiple sources including databases, APIs, cloud platforms, enterprise applications, and structured/unstructured data sources.
•Build and optimize data pipelines using GCP Dataflow, BigQuery, Azure, AWS, and Databricks to support large-scale data processing.
•Perform data cleansing, transformation, validation, and quality checks to ensure reliable and accurate data for analytics and AI applications.
•Develop data models and optimize SQL queries for analytical workloads using BigQuery, Snowflake, and cloud-based data platforms.
•Prepare and transform structured and unstructured data for Machine Learning, NLP, and Generative AI applications.
•Build data pipelines to support LLM and RAG applications, including document ingestion, text extraction, chunking, metadata processing, embeddings, and vector database integration.
•Work with Azure OpenAI, LLMs, LangChain, LlamaIndex, and vector databases to support AI-powered search, document processing, and chatbot solutions.
•Develop pipelines for processing documents and enterprise content used in Retrieval-Augmented Generation (RAG) applications.
•Work with vector search technologies such as Azure AI Search and Cosmos DB to enable semantic search and retrieval of relevant enterprise information.
•Collaborate with Data Scientists and ML Engineers to prepare high-quality datasets and feature pipelines for machine learning and AI model development.
•Support NLP and AI use cases including text classification, sentiment analysis, entity recognition, topic modeling, and document summarization.
•Implement data processing solutions using PySpark and MLlib for large-scale data transformation and analytics.
•Develop and maintain cloud-based data solutions across Azure, AWS, and GCP.
•Implement automated CI/CD workflows using GitLab CI/CD for data and AI application

Agile IT Systems Inc. is a U.S.-based technology company specializing in AI, Data, Cloud, and Enterprise IT solutions. We help organizations transform complex and fragmented data into actionable business intelligence through innovative technologies and enterprise-grade consulting. Our expertise spans AI & Machine Learning, Data Engineering, Data Integration & ETL, Cloud Services, Business Intelligence, and Digital Transformation. Our flagship Unified Intelligence Framework (UIF) is designed to enable AI-native, real-time data integration across cloud, on-premises, and hybrid environments.