Data Engineer Jobs

Looking for a skilled data engineer with at least 4 years of experience, who has worked with Snowflake before. The project involves (goal of project not provided) and requires someone who can efficiently manage and process a large amount of data. Ideal candidates should have experience with data migration, performance optimization, and building data pipelines.

1) There are 3 tables – master table (employee) and two child tables (employeeDept, employeestatus) 2) Data is stored in Master Kafka Topics (here kafka topics are used as a DB more than streaming, data of all employees stored in these topics), so we have 3 master kafka topics. 3) There are 3 Change Data Capture(CDC) kafka topics. 4) There are 1 Kafka sink topic Requirement: when there is a change in data, CDC kafka topic(point3) will trigger message to flink, and flink needs to send consolidate information of all data for particular row to kafka sink topic(point4) Employee table : Name, ID, Address, DeptId,statusID columns employeeDept : DeptId,DeptName columns employeestatus: StatusID,StatusName columns John(Employee Table) had an address change, employee CDC kafka topic triggers...

Add history functionality to existing ETL process in pyspark. Need to account for racing conditions on primary key.

Project Description: I am seeking a skilled freelancer who can assist me with repairing a corrupted Fastqc Gzip file. Unfortunately, I don't have a backup of the original file, but I do have a copy of the corrupted file. The purpose of this project is to recover the data from the corrupted file. Required Skills and Experience: Proficiency in Gzip compression Experience with data recovery Strong attention to detail Good communication skills Ability to work efficiently and effectively Project Details: Verify file integrity: Confirm the corruption of the Fastqc Gzip file. Attempt file decompression: Use Gzip decompression tools to extract data from the file. Data recovery: If the file cannot be successfully decompressed, employ specialized techniques to recover the data....

Need someone who can build AI Code Converter which converts SCALA Scripts to SQL Scripts. It should be purely develop in python and use AI for the conversion. It is straight forward where user will provide script and ai converter will convert it to SQL. No need of any UI, we can pass folder path or file path to program.