Career Center

Senior Data Engineer

Location: Remote
Posted On: 01/13/2023
Requirement Code: 62323
Requirement Detail
Required :
Establish fully functional object store compatible with S3 access APIs which will:
a. Provide ability to define fine grain access control lists on HCP objects
b. Support multiple storage tiers in the view of cost optimization and allows moving objects between storage tiers
c. Support event notifications over objects, allowing a subscription model to build data pipelines
d. Record all the object level operations, and expose access logs for audit
e. Support multi-part uploads to move handle large objects into the store
f. Support replication of objects between different object stores
g. Provide comprehensive monitoring and observability metrics, generate alerts on configured thresholds
h. Create appropriate documentation for the activities above Integrate Iceberg with HCP, Spark, and Trino which will:
a. Provide a central metastore or catalog pluggable with multiple compute engines
b. Support time travel, automated maintenance like purging, compression
c. Support native reads on Iceberg, which can be leveraged by compute engines
d. Create appropriate documentation for the activities above
Spark
S3
Apache Iceberg (Nice to have) or similar
Can search and investigate on tools
PoC approach implementation experience
Delivery Ownership
Autonomous


Apply Now
SAICON Ranked #142
  • SAICON Ranked #142

  • America‚Äôs Fastest Growing Companies