Azure Developer - Brussels, België - Code Convergence ltd

Code Convergence ltd

Geverifieerd bedrijf

Brussels, België

1 week geleden

Geplaatst door:

Sophie Dubois

beBee Recruiter

Will be primarily responsible for developing, deploying and operating the Azure Databricks and Data Factory/ Data Flow product.

Also, build and extend the Databricks cloud offering, which is based on a microservice architecture and includes: a cluster management platform Spark job scheduling, notebook services deployment and automation.
Expert in Azure Data Factory, Azure Data Lake Analytics, Python, Spark SQL, Azure Data Bricks
Strong SQL skills with experience in Azure SQL DW
Experience handling Structured and unstructured datasets
Experience in Data Modeling and Advanced SQL techniques
Experience implementing Azure Data Factory Pipelines using latest technologies and techniques
Use the interactive Databricks notebook environment using Apache Spark SQL, Examine external data sets, Query existing data sets using Spark SQL
Visualize query results and data using the builtin Databricks visualization features, Perform exploratory data analysis using Spark SQL.
ETL processing and data extraction using Azure Databricks to write a basic ETL pipeline using the Spark design pattern, Ingest data using DBFS mounts in Azure Blob Storage, Ingest data using serial and parallel JDBC reads
Perform an ETL job on a streaming data source, Parameterize a code base and manage task dependencies, Submit and monitor jobs using the REST API or Command Line Interface
Mange Delta lake using Data bricks to use the interactive Databricks notebook environment, Create, append and upsert data into a data lake.

Skills and Experience

-
2-4 years of development experience in cloud platform includes Azure Databricks, Azure SQL, Apache Spark (Scala/Python) etc
:

Primary Responsibilities

Develop ETL processing and Data extraction using Azur Data Factory/ Data Flow/ Azure Databricks.
Develop Apache Spark SQL using Scala/Python to examine and query datasets.
Develop Dataframes for ETL transformation and loads
Write scripts for automated testing of data in the target facts and dimensions.
Capture audit information during all phases of the ETL transformation process.
Write and maintain documentation of the ETL processes via process flow diagrams.
Collaborate with business users, support team members, and other developers throughout the organization to help everyone understand issues that affect the data warehouse