The #1 Site for Management Jobs - search all Management jobs.

Senior Big Data ETL Developer

Job Description

Senior Big Data ETL DeveloperBaltimore, MDFUNCTIONThis position is an opportunity to use Informatica Big Data Management(BDM) with HDFS(Cloudera), Spark and Kafka technology for batch and streaming analytics. Seeking a Big Data Engineer who possesses a base of programming skills and have a passion to learn new tools and techniques in a big data landscape that is endlessly changing. Data processes in BDM, if appropriate, include ingestion, standardization, metadata management, business rule curation, data enhancement, and statistical computation against data sources that include relational, XML,JSON, streaming, REST API, and unstructured data. BDM has provided a metadata injection layer for data ingestion across six SQLServer sources that will be leveraged for additional source ingestion. RESPONSIBILITIESThe role has responsibility to understand, prepare, process and analyze data to drive operational, analytical and strategic business decisions. The Senior Big Data Engineer will work closely with big data engineers, product owners, information engineers, data scientists, data modelers, infrastructure support and data governance positions. This role requires exceptional BDM developer skills who can mentor big data engineers in the appropriate solution architecture and BDM development practices. While you do not have to have big data skills, we are seeking BDM engineers who start with a base of programming skills but who also love to learn new tools and techniques in a big data landscape that is endlessly changing.
  • Build end to end data flows from sources to fully curated and enhanced data sets. This can include the effort to locate and analyze source data, create data flows to extract, profile, and store ingested data, define and build data cleansing and imputation, map to a common data model, transform to satisfy business rules and statistical computations, and validate data content.
  • Produce data building blocks, data models, and data flows for varying client demands such as dimensional data, data feeds, dashboard reporting, and data science research & exploration
  • Create, modify and maintain BDM code and complex SQL for BI/DW data flows
  • Learn to program in Spark and Python where BDM Spark co-generation is not adequate Produce automated tests of data flow components
  • Use knowledge of the business to automate business-specific tests for data content quality
  • Automate code deployment and promotion
  • Build automated orchestration with BDM and error handling for use by production operation teams •Provide technical expertise to diagnose errors from production support teams
  • Collaborate with team members in an Agile team (e.g., Scrum)
  • Participate as both leader and learner in team tasks for architecture, design and analysis
  • Coordinate within collocated on-site teams as well as with work plans for off-shore resources

MINIMUM QUALIFICATIONS
Minimum of two years college with at least ten years involvement in systems analysis and application development, or an equivalent combination of education and experience.


Experience with development of metadata-driven and fully parameterized BDM environments

Advanced SQL coding skills for data transformations, profiling, and query tasks

Unix commands and scripting

Experience in agile environments such as scrum and Kanban

Preference for experience in Hadoop fundamentals and architecture: HDFS, map-reduce, job performance

Preference for open source big data skills in tools such as Hive, HBase, parquet, Spark SQL

Programming in a language such as Python (preferred), Scala, etc.



IDEAL QUALIFICATIONS PREFERREDFive years of ETL tool experience which can be from other tools such as Talend, DataStage, and Informatica Bachelor’s degree in Computer Science or related discipline preferred

ABOUT M&T
 At M&T, we understand what's important when you're considering a career change: a company with a solid history of responsible growth and positive earnings. A company committed to the communities it serves, one that rewards performance and provides employees with opportunities to direct their own career paths. M&T is a company with a focus on the future.



Job Posting:  Nov 6, 2018, 11:08:31 AM
Unposting Date:  Ongoing

Job Requirements

 

Job Snapshot

Location US-NY-Cheektowaga
Employment Type Full-Time
Pay Type Year
Pay Rate N/A
Store Type Other
Apply

Company Overview

M&T Bank

M&T Bank is a multi-state community-focused bank serving New York, Maryland, New Jersey, Pennsylvania, Delaware, Connecticut, Virginia, West Virginia and Washington, D.C. Founded in 1856, M&T Bank provides banking, investment, insurance and mortgage financial services to consumer, business and government clients. Learn More

Contact Information

US-NY-Cheektowaga
Snapshot
M&T Bank
Company:
US-NY-Cheektowaga
Location:
Full-Time
Employment Type:
Year
Pay Type:
N/A
Pay Rate:
Other
Store Type:

Job Description

Senior Big Data ETL DeveloperBaltimore, MDFUNCTIONThis position is an opportunity to use Informatica Big Data Management(BDM) with HDFS(Cloudera), Spark and Kafka technology for batch and streaming analytics. Seeking a Big Data Engineer who possesses a base of programming skills and have a passion to learn new tools and techniques in a big data landscape that is endlessly changing. Data processes in BDM, if appropriate, include ingestion, standardization, metadata management, business rule curation, data enhancement, and statistical computation against data sources that include relational, XML,JSON, streaming, REST API, and unstructured data. BDM has provided a metadata injection layer for data ingestion across six SQLServer sources that will be leveraged for additional source ingestion. RESPONSIBILITIESThe role has responsibility to understand, prepare, process and analyze data to drive operational, analytical and strategic business decisions. The Senior Big Data Engineer will work closely with big data engineers, product owners, information engineers, data scientists, data modelers, infrastructure support and data governance positions. This role requires exceptional BDM developer skills who can mentor big data engineers in the appropriate solution architecture and BDM development practices. While you do not have to have big data skills, we are seeking BDM engineers who start with a base of programming skills but who also love to learn new tools and techniques in a big data landscape that is endlessly changing.
  • Build end to end data flows from sources to fully curated and enhanced data sets. This can include the effort to locate and analyze source data, create data flows to extract, profile, and store ingested data, define and build data cleansing and imputation, map to a common data model, transform to satisfy business rules and statistical computations, and validate data content.
  • Produce data building blocks, data models, and data flows for varying client demands such as dimensional data, data feeds, dashboard reporting, and data science research & exploration
  • Create, modify and maintain BDM code and complex SQL for BI/DW data flows
  • Learn to program in Spark and Python where BDM Spark co-generation is not adequate Produce automated tests of data flow components
  • Use knowledge of the business to automate business-specific tests for data content quality
  • Automate code deployment and promotion
  • Build automated orchestration with BDM and error handling for use by production operation teams •Provide technical expertise to diagnose errors from production support teams
  • Collaborate with team members in an Agile team (e.g., Scrum)
  • Participate as both leader and learner in team tasks for architecture, design and analysis
  • Coordinate within collocated on-site teams as well as with work plans for off-shore resources

MINIMUM QUALIFICATIONS
Minimum of two years college with at least ten years involvement in systems analysis and application development, or an equivalent combination of education and experience.


Experience with development of metadata-driven and fully parameterized BDM environments

Advanced SQL coding skills for data transformations, profiling, and query tasks

Unix commands and scripting

Experience in agile environments such as scrum and Kanban

Preference for experience in Hadoop fundamentals and architecture: HDFS, map-reduce, job performance

Preference for open source big data skills in tools such as Hive, HBase, parquet, Spark SQL

Programming in a language such as Python (preferred), Scala, etc.



IDEAL QUALIFICATIONS PREFERREDFive years of ETL tool experience which can be from other tools such as Talend, DataStage, and Informatica Bachelor’s degree in Computer Science or related discipline preferred

ABOUT M&T
 At M&T, we understand what's important when you're considering a career change: a company with a solid history of responsible growth and positive earnings. A company committed to the communities it serves, one that rewards performance and provides employees with opportunities to direct their own career paths. M&T is a company with a focus on the future.



Job Posting:  Nov 6, 2018, 11:08:31 AM
Unposting Date:  Ongoing

Job Requirements

 
HeadHunter Advice

For your privacy and protection, when applying to a job online: Never give your social security number to a prospective employer, provide credit card or bank account information, or perform any sort of monetary transaction.Learn More

By applying to a job using headhunter.com you are agreeing to comply with and be subject to the workinretail.com Terms and Conditions for use of our website. To use our website, you must agree with theTerms & Conditionsand both meet and comply with their provisions.
Senior Big Data ETL Developer Apply now