• A-Z
  • Directory
  • myUVM
  • Loading search...

Frank's Activity Log

16 April 2014: Wednesday

Posted: April 16th, 2014 by fcs


  • Yet another SSN with dashes showed up.  Oh, and another former student … This one came over without enough info… pushing back.  I was convinced to go ahead and add it…


  • Recovery performed for Alum, delivered to same, and thanks received.  All good.  Tapes can go back to the vault.


  • First round of interviews…

15 April 2014: Tuesday

Posted: April 15th, 2014 by fcs


  • One Issue – an updated to the SSN of a BANNER feed matched it with an almost gone corrupted entry.  Cleaned up the mess.
  • Monthly automated account purge… 10 errors.  All of them trying to add email bounce rules that already exist.  I like those, really easy to deal with… Yep, ignore!


  • Two groups failed.
    • Two systems in one group had their Networker client updated last night.  Cleaned that up by installing compat-libstdc++-33 on these systems…
    • Structured Biology’s group also failed.  Will discuss that with the admin.
  • Windows file server: due to issues – forcing a full.
  • Clones:
    • Started CoM/IS and Disk.

14 April 2014: Monday

Posted: April 14th, 2014 by fcs


  • Three (3) PeopleSoft failures on Friday evening’s update:
    • All three are “STU” class employees.  Two of them match on name, birthday, and local address.  Those two were merged.  The third, matches on name and birthday, but has no matching addresses.  That one, I sent an email about.  Payroll confirmed the possible matching student id was the correct one, and I performed the merge.


  • SR 62269034 – EMC recommended blasting the PEER INFORMATION to fix the GSS-API failure messages.  I had asked them what the error message meant.  Apparently, since that seems to have fixed the problem, it means that the Networker client has corrupted its own PEER INFORMATION…  I have updated the issue and asked them to clarify.
  • RHEL3, Networker 7.1 client backups failing.  Suddenly the backups on the old RHEL3 email system (please don’t ask – it should be killed) are failing.  The error message is the save syntax dump, indicating that the savefs command is invoking the save command with bad arguments.  Unfortunately, the Networker 7.1 client is no longer supported.  Fortunately, this is just the / and /boot partitions on that old system (the email is actually protected in a different way).
  • Clones:
    • tape clones started and completed.
  • VEEAM webex…  Waiting for answers to questions
  • PHD – doing web research/reading
  • Vault recovery – retrieved 3 tapes, running scanner to see what is there.

11 April 2014: Friday

Posted: April 11th, 2014 by fcs


  • No errors in Wednesday night update.  One error in last night’s – another former student.  did the dance!


  • Mike forwarded a message from a former student looking for files, after I left on Wednesday.  Researching where the desired files were in 2012.
    • Contacted the person.  Vetted their identity.  Sent them an email to get the name of the file they want recovered.  Told them it would be next week before the tapes are back from the vault to recover the file for them.
  • Qualstar has requested that I reboot that tape drive and see if it will continue to work.
    • Rebooted the drive and put it back into service.  Bets on how long it will run before it code 4′s again?
  • Final NFS Fileserver merged save group activated, allowing the older sharing single server per weekend groups all disabled now.
  • Clones:
    • Tape: Completed at 4:51 am this morning.
    • Disk: 15/220.914 running, 110/769.017 left – finished at 11:21 am.
    • CoM/IS: 1/6,233.374 running, 0/0 left – finished at 9:13 am.
  • Cleaning tape expired.  Replaced with fresh one.
  • Installed (and configured, and tested) the Networker client on the Print and Mail center’s new system.  Forced a FULL save – this is an OS upgrade.

9 April 2014: Wednesday

Posted: April 9th, 2014 by fcs


  • New posixGroup department groups went in without issues.
  • One update issue:
    • A Former Student – SURPRISE!  We danced.


  • clones started.
  • No other backup system issues today.
  • [EDIT] at 5pm, drive 1310119697 (newly installed yesterday to replace 9311243881) reported a code 4.  A code 4 is a “Microcode or Tape Drive problem. Tape drive determined that a microcode or tape drive hardware failure occurred.” (according to the IBM documentation about the SCD code meanings on Ultrium Drives.  I took a dump with itdt and sent that and the error (emailed from the XLS) to Qualstar.


  • A person complains that they can’t print to a specific printer via repgen…  The Repgen folks got the person sorted out.


  • Account services couldn’t find a person to update their calendar info.  Pointed out that the person’s name was not what they thought it was…

8 April 2014: Tuesday

Posted: April 8th, 2014 by fcs


  • Yeah, there were some issues.
    • Four former students showed up.  We danced.
    • One new student employee. No SSN, yet Banner (if I found the right entry) has one.  Emailed payroll to see if they have the SSN.  Will hold that one until I hear back.
  • Pushed out the posix group changes into production – will hit ldap.uvm.edu tonight.
  • Mopped up another (perceived) duplicate netid.
  • Recycled ldap servers for issues.
  • [After Hours] New SSL Certs on all LDAP servers.


  • 128 Tapes brought back from the Vaults and placed in the library.  Relabeling of the tapes into the Scratch pool completed.
  • Replaced the failed tape drive on stornode10 (CoM/IS).  Failed drive handed over to FedEx for deliver back to the vendor.

7 April 2014: Monday

Posted: April 7th, 2014 by fcs


  • Update issues on Friday night:  A duplicate SSN from Banner and a Former Student.  Email and Dance.
  • [EDIT] Oops!  The dancing this morning duplicated some dancing I did on Saturday.  Thankfully, the AUDIT blared the pager at 5pm and I was able to fix it before tonight’s update went KaBoom.
  • [UPDATE] Account Services found a two accounts one person issue – I cleaned that up for them.


  • The first BIG (one of the three had two filesystems) weekend full of the NFS file servers, completed at 6:55am today (later than we really want).  But, nobody will believe me that the file systems are too large.
  • The email backup screwed up on Saturday.  Opened issue with EMC about their undocumented error message.  How kind of them to provide an error message guide that doesn’t include the error messages that their code spits out.
  • Tape moving.
    • [Update] Moved 128 tapes to the vaults.  Rearranged all the tapes in the vaults so they are in numeric order instead of the jumble they had been.  It took all day, but it got done.

4 April 2014: Friday

Posted: April 4th, 2014 by fcs


  • No errors in the LDAP update.  Uhoh!  That’s wrong.  There were two errors, but they didn’t get caught.  The SIS feed included two new students but had put dashes into their SSN’s.
    • Updating the SIS feed pre-process code to gripe about this (and remove them) as this causes too many issues when it happens (even though it doesn’t happen very often).


  • Tape Drive: stornode10: 3500507631205aca8/9311243881 code A
    • The pause of the CoM/IS clones seems to have worked.  The CoM/IS groups have all completed, and removing the pause started the clones back up.
  • NFS Fileservers {Full,Incremental} 1 groups enabled.  The older NFS Filserver {1,2,3} groups updated to remove the pieces of the file sytems that are in the Full/Incremental groups that were enabled.
  • PeopleSoft developers need a tape from the vault for a backup that was made the end of February because their script marked the backup as to be purged after 7 days at the time of creation.  Doh!

3 April 2014: Thursday

Posted: April 3rd, 2014 by fcs


  • Four issues in last night’s update:
    • A new one – a uvmSIName attribute with too many values… hmmm
      • This appears to be someone messed up causing the SIS extract to drop a student.  Question sent off to registrar’s office.
    • Three former students newly arrived… danced.
  • Development system dealt with the contrived tests I threw at it and had the same issues with the update.
    • Discovered some logging issues – a little bit of code tweaking will fix that.
  • Discovered that there was a PeopleSoft feed error on 3/20 that I missed.  It was a new student employee that didn’t find a programmed match in LDAP.  I found them and merged the records today (after PeopleSoft rep reached out to me directly today and I went researching).


  • Clones:
    • CoM/IS:
      • Remaining: 2; 1,997.899
      • Running: 2; 4,781.427
      • Completed: 273; 12,035.540
    • Disk:
      • Completed: 309; 19,415.026
      • Finished on 2 April at 21:46
    • Tape:
      • Completed: 140; 924.692
      • Finished on 31 March at 15:26
  • EMC Documentation:  Retrieved new Hardware and Software compatibility documents.
  • Staging rules updated to leave an additional 5TB on the two “UVM Fullsave” AFTDs – in the hopes of getting closer to their 50TB ZFS quota and a minimum of two weeks on the disk before being moved to tape.
  • Moved another thirty-three tapes into the firesafe.  Now have 143 tapes in there headed for the Vault.
  • Tape Drive: stornode10: 3500507631205aca8/9311243881 code A
    • itdt dump taken and sent off to Qualstar.
    • [After Hours] Qualstar says the drive has failed.  They are sending a new one.  Unfortunately, it won’t arrive until Monday.  I have issued a pause to the CoM/IS clone process.  Hoping it saves their backups for tonight.

2 April 2014: Wednesday

Posted: April 2nd, 2014 by fcs


  • No issues in the update last night.
  • To test out the posixGroup changes, I’m letting the development system free-wheel for a bit (not resetting to the production systems database every morning).  Found and clubbed one bug to death.  I wonder how many more are in there.


  • No failures overnight.
  • Clones:
    • CoM/IS:
      • Remaining: 4; 6,779.327
      • Running: 4; 9,352.897
      • Completed: 269; 2,682.643
    • Disk:
      • Remaining: 149; 4,031.878
      • Running: 8; 1,581.898
      • Completed: 152; 13,801.249
  • EMC SR 61542494 – the start up code will now check that the /var/run/nsrexecd. file not only that the is running, but that it is actually nsrexecd – and will start properly if it is not nsrexecd.  Gave them permission to close the issue.
Contact Us ©2010 The University of Vermont – Burlington, VT 05405 – (802) 656-3131