Thanks Hims. I have taken exact approach as you mentioned. the S3 bucket is mounted using s3Fuse on EC2 and use SFTP connection to browse files. But, getting lost when it comes to mask 100 of GBs of Json and other formatted files. Couple of challenges to list down:
- JSON is not supported
- Not easy way to browse files from different folder structures and apply masking rules
- Files are grouped by defining the patterns, but jobs doesn't execute successfully all the time
- It doesn't load more than 200 files at a time.
Trying to explore how other customers are handling File masking (JSON, XML) of S3 bucket.
Thanks,
Santosh