Software Developer job PISCATAWAY New Jersey
Software Developer  job PISCATAWAY New Jersey
My Spider Scam Awareness Contacting Us F. A. Q.
Job Seekers
Search Jobs
Browse Jobs
Post a Resume
Job Alerts
Search Resumes
Browse Resumes
Post a Job

Software Developer Job

Employer Name: SpiderID: 8836122
Location: PISCATAWAY, New Jersey Date Posted: 7/26/2019
Wage: Category: Information Technology
Job Code:

Job Description:

Software Developer needed to perform the following duties :

∑ Participate in business and functional requirements gathering, which involves a review of existing systems configurations and operating methodologies as well as understanding the evolving business needs.

o Extraction of data structures and data types stored in current and proposed architecture in NoSql databases such as HBase, Cassandra etc using Big Data querying tools such as Pig, Hive and Impala.

o Validate Hive, HBase, Cassandra, Spark on Hive databases compatibility for various datatypes on regular basis using static meta comparison tools such as IBM CDC, Attunity, etc.

o Development of the code on HBase API calls to analyze, load and extract data between relational databases such as Oracle, Teradata over Hive and HBase.

o Create the sqoop scripts (MR Jobs) to sqoop data from the source (DB2, Oracle, SQL) to raw layer.

o Create the scripts to create the Hive tables on top of the raw data and schedule the jobs.

o Create scripts to create the snapshot for the day partition.

o Schedule all jobs in the Oozie server (workflow.xml, co_ordinator properties, job_properties).

∑ Design and develop data models, data processing and transformation rules for data representation.

o Design utility jobs using Spark-API and MR for retrieving data in cluster very quickly and maintain health status.

o Develop custom reusable scripts which are responsible for sending alerts based on source priority, clean up, and archival scripts which abides with custom data retention policy.

o Participate in weekly architecture meetings with internal team wherein we discuss the scope of DataLake framework and related Spark, Kafka libraries we use in the current version of product and analyze its pros and cons.

∑ Sqoop the data from multiple source systems if data is not available in Azure Data Lake and coordinating source deployments and monitoring the functioning, design Hive on HBase snapshot tables which acts as replica of source relational database using Spark-SQL, Hive and Oozie scheduler

o Co-ordinate with the source teams and consumption teams accordingly, source team deployments and consumption team requirements.

o Develop and support Map reduce utilities in MapR distribution for Message Queue handling with semi structured data like XML messages, JSON messages etc.

o Prepare and support Spark API calls which provides the Hadoop MapR production queue utilization statistics which helps us on resource utilization.

o Customize Spark sql parameters to reduce the latency in creation of views on audit tables built on Hive and also to meet adhoc requests from management on cluster health status reports.

o Design utilities using Apache Drill, Kafka, Splunk, Hadoop and Spark-API in MapR distribution which are used for creation of on-demand test data for POCs and tool performance evaluations.

∑ Perform analysis of the data to understand the source data from multiple source systems in the project and validate design with business requirements to ensure completeness of data integration solutions.

o Troubleshoot production support issues post-deployment and come up with solutions as required.

o Prepare Jenkins pipeline for new projects in different environments which aids automatic creation of jars using Maven plug-in.

o Develop Docker image of jars built through Jenkins for deployment into Open shift which is a containerizing platform.

o Automate process of deployment in cloud using Jenkins and Docker tools.

∑ Performing Unit testing and system integration testing based on established best practices and creating automation jobs to monitor all the bad and rejected folders. Developing ETL Talend Data Fabric components and used features of Context Variables, MySQL, Oracle, Hive Database components. Customize and monitor Kibana dashboard which captures current status of ingestion process with minimal delay.

o Develop ad-hoc Spark-API, HBase-API, MQ, MR programs and jobs to support daily operational duties.

o Design email alerts to team members on the status of deployment into cloud.

o Build dashboard using elastic search on the logs created during the user activity with services deployed on the cloud.

o Work with Integration team closely for creation and support of Jenkins pipelines for Continuous Integration and Continuous Deployment (CI/CD) of new features to framework using Open Shift tool.

∑ Review different components in the development tools to ensure they meet the requirements as design and common data Acquisition from disparate Trading Partners and centralized Data Storage and standard data provisioning. Participate in biweekly meetings which involve multiple integration teams and discuss on deployment tool compatibility with current and future MapR versions, Red hat patches, Attunity patches.

o Design on demand trigger job using Hive, HBase and Spark for finding real time status of ingestion in the Hadoop cluster.

o Update to management and consumers on the availability of data for consumption 24/7 on the Kibana dashboard and maintain its health status using hive and Spark scripts.

o Participate in Scrum daily stand up calls and provide updates on the progress of assigned tasks for the sprint in Agile methodology.

o Practice in collocated scrum team methodology and participate proactively in sprint demos and sprint retro meetings held bi-weekly.

Bachelorís degree Required in Computer Science or Computer Engineering

Job Criteria:
Start Date:
Position Type: Full-Time Permanent
Years of Experience Required:
Education Required: Bachelors
Overnight Travel:
Vacation Time:

Contact Information:
Contact Name: Datametrics Software Systems inc Company Type: Other
State: New Jersey
Zip: 08854

Send ad to a friend
Report a Problem


© 2020 Job Spider
Privacy Policy | CC Marketing Sites | Site Map | Links