Masking In-Memory DBs

  • 0
  • 1
  • Question
  • Updated 5 months ago
Delphix does not support In-Memory DBs. We have an In-Memory DB called as Mnesia which is written in Erlang, do we have any past experiences supporting that ? Or any approaches that were developed ?
Photo of Mayank Ahluwalia

Mayank Ahluwalia

  • 678 Points 500 badge 2x thumb

Posted 5 months ago

  • 0
  • 1
Photo of Gary Hallam

Gary Hallam, Official Rep

  • 1,702 Points 1k badge 2x thumb
Hi Mayank,
Our general approach to masking unsupported data sources is to extract, mask and then reload.  So you use whatever extract and load tool is available for your data source and extract to a supported file system - such as a csv file.  Mask the file and then load the masked data back.  We have used this technique with unsupported mainframe type files (Non-stop Tandem, Base24 for instance).  You can combine this with the virtualisation capability to provide storage savings and instant provisioning of masked files.
Regards,
Gary
Photo of Mayank Ahluwalia

Mayank Ahluwalia

  • 678 Points 500 badge 2x thumb
Thanks Gary. The challenge here is that we get a CSV file with JSON's inside. There are about 1200 unique JSON structures and most of them are highly nested with the nesting going up to 100 in certain cases. Can Delphix handle JSON data inside a CSV ? Also, is there a challenge that we should expect with parsing by Delphix given the high amount of nesting ?
Photo of Gary Hallam

Gary Hallam, Official Rep

  • 1,702 Points 1k badge 2x thumb
Hi Mayank,
I imagine that JSON inside a CSV might be fraught with challenges.  We do not support JSON generally at the moment but it is something we are investigating for inclusion onto the roadmap.
I would look at ways of pre-processing the JSON to get into a supported format.  Of course, it is worth testing to see if the CSV idea works.
Regards,
Gary
Photo of Mayank Ahluwalia

Mayank Ahluwalia

  • 678 Points 500 badge 2x thumb
Thanks Gary. We are also looking at translating it into XML files, where each unique JSON structure becomes a XML file. How about high nesting within XML ? You foresee any challenges in handling of complex XMLs ?
Photo of Gary Hallam

Gary Hallam, Official Rep

  • 1,702 Points 1k badge 2x thumb
Hi Mayank,
It's probably worth testing this.  Are you working with anyone in Delphix on this particular project?  We could take this off-line and get someone to look at your file structure.  I'm not aware of any challenges with Delphix XML masking; there are certainly a lot of tools out there that do XML-JSON mapping but I haven't used these with complex structures.  Let me know if you'd like me to connect you with one of our partner managers to take this forwards.
Regards,
Gary
Photo of Mayank Ahluwalia

Mayank Ahluwalia

  • 678 Points 500 badge 2x thumb
Thanks Gary, we have just started our implementation and for now have been talking to Roy during the sales and pre-sales and now Peter Majeed.  Will explore this further and see