Old archive logs are consuming a lot of space on my Engine. Can I get rid of these?

  • 0
  • 1
  • Problem
  • Updated 3 years ago
  • Solved
I've noticed that logs are taking up several TB of space.  Is there a way to get rid of them?
Photo of Thom T

Thom T

  • 838 Points 500 badge 2x thumb

Posted 3 years ago

  • 0
  • 1
Photo of Ranzo Taylor

Ranzo Taylor, Employee

  • 1,552 Points 1k badge 2x thumb
Official Response
Archive logs are used in 2 situations, to provide point-in-time recovery and to make snapshots consistent for provisioning.  This is identical to their use in typical recovery operations which use incremental backups.

You can control the retention of logs with your retention policy, and this doesn't have to be as long as your snapshot retention policy.  You can retain snapshots for 30 days, but archive logs for only 1 day for example.  In this case, you'd still be able to provision from 30 days ago using the snapshot, but you'd only have "point-in-time" provisioning for the last day.  This can save a lot of space.

The first step is always to adjust your log retention policy and wait until the purge routine has run at night.

Note though that Delphix will ALWAYS keep logs required to make snapshots consistent and provisionable, even when those logs are old.  Typically, this is only a small number of logs, because snapshots are very fast, so Delphix only needs a little logging to make it consistent.  However, in the case where BCT is disabled (or not working properly), snapshots can take many hours, and Delphix will retain a large number of logs to make them consistent/provisionable.  

Many users don't realize that having BCT off means higher disk space usage for logs, and longer provisioning times as those logs are applied.

The best approach is to enable BCT!  But while you work towards that, Delphix provides a new PurgeLogs routine which can remove old logs.  This should be viewed as an emergency procedure only, since it will make existing snapshots un-provisionable, and it's impossible to predict whether you might need those in the future.  If you are looking to use this regularly, it's a sign that you need to reevaluate your retention policy and/or get more disk space (and enable BCT!).

This routine is described here: https://support.delphix.com/hc/en-us/articles/203804386-How-to-Use-the-PurgeLogs-Operation.