I agree 10GB-25GB memory should be more than enough for your masking jobs to run. If you are running multiple masking jobs with similar min/max values, please check that you are not exceeding the total memory on the masking engine. For your reference please see our KB article on this topic.
There may also be a table(s) setting causing this issue, I would suggest isolating the table(s) to help determine the cause. Also, our Delphix Support team can help you with this specific issue. Thanks Jack