14 December 2013: Saturday

Backups:

  • Yep – that NFS mounted ZFS pool went to 97% full.  df -h reported 25T of 25T used and nagios had a heart attack.  I’m clubbing everything in sight including the weekend’s full saves that were using that volume and the cloning process (which I’ll have to fix).  Finally, the clone that had been running since forever finished and the staging script could get to the disk (it had been waiting since 8:30 last night!) and start to clear data off.  I updated the device configuration to tell Networker to say the device is full at 95% utilized.  Hopefully, that will work going forward and avoid this death spiral of trying to write those last few compressed bytes into the thing.
    • Left it alone for several hours and it is moving data off to tape, so I have started the two groups that do need to run and Networker is waiting for the disk utilization to fall below 95%.

Tags:

Leave a Reply

You must be logged in to post a comment.