• A-Z
  • Directory
  • myUVM
  • Loading search...

Frank's Activity Log

28 August 2014: Thursday

Posted: August 28th, 2014 by fcs

LDAP:

  • Issues last night:
    • Former Student – did the dance.
    • BCA – Found and merged.
    • Singleton with wrong last name – found enough that the code doesn’t look at to be comfortable doing the merge.
    • Graduate Student Employee – found the graduate student record from Banner and merged.
  • LDAP -> AD failed last night.  The extract from AD was not delivered to be updated.
    • Process re-ran between 8:20 and 9:10 this morning.

BACKUPS:

  • CoM/IS and Disk clones continue to run.  CoM/IS has 8TB to go, Disk  12TB.
  • Looking at a client performance working through the testing to see where it needs to be tweaked.

FootPrints:

  • Added a disk to the production system to hold the database backups to stop the nightly warnings about file system space.

SecurID:

  • Getting the 8.1 documentation.
  • Working to figure out how big a project this migration from 6.1 to 8.1 is going to be.

27 August 2014: Wednesday

Posted: August 27th, 2014 by fcs

LDAP:

  • Issues feeding into Active Directory last night.  The Domain Controller that we usually apply the updates to was not responding.  Needed to adjust the control file to give the updates to one of the surviving controllers.
  • Two issues in the update:
    • A former student – did the dance.
    • A student employee who’s last name does not match their student record.  Sent a question off to HR to see if they need to change or if I need to talk with the Registrar’s office about getting their student record changed, or if it is two completely different people who happen to share several pieces of information that are not normally shared.
  • Dealing with a claim that some person has two accounts – not certain of if the claim is right or wrong.  HR has proven to me that they have the correct info in the system (as correct as the person was in filling out the paperwork).  Therefore, it appears that we either have two people, or a single person who has reported two different id numbers to us.

BACKUPS:

  • Clones were started yesterday afternoon.  The Tape clones finished overnight (having just half a terabyte to clone).  Disk and CoM/IS clones are still running with 21 TB and 16 TB total to be done, respectively.
  • Increased pelican cpu count to 3, I hope that is enough.  It might need more than 2GB of memory too – but it won’t let me go past 3GB.
  • Researching the new “probe based” group type – how does that compare to a savepnpc based group?  – it is basically, a savepnpc that keeps running the probe (savepnpc pre-script) until it is successful [and there is no post-script], within the defined window.

Other:

  • Doing Maestro cycling for DBAs – and gathering logs/files for IBM.

26 August 2014: Tuesday

Posted: August 26th, 2014 by fcs

LDAP:

  • One former student in last night’s update.  The dance was done.
  • Working on network services feed.  Have sent them their first sample of the new format.

BACKUPS:

  • Qualstar issued RMA 49117 for drive SN 1310286459 – the one that seems to have the serial interface that intermittently disconnects.  I expect the new drive to arrive today, and am delaying the start of the cloning process until after I replace the drive as that tends to cause issues.
    • Multiple issues today… a drive that was in use is now reported as missing as is the robot – until the next time I cycle NetWorker.  I really could use that issue being fixed by EMC.
  • Discussing the possibility of a savepnpc process to keep an Incremental group from running if the Full group has not yet completed.

SecurID:

  • Assisted with the SSLVPN 6am update by flipping the “Node Secret has been sent” bit.

Other Stuff:

  • Figured out the semi-daemon magic for the user_trickle script.

25 August 2014: Monday

Posted: August 25th, 2014 by fcs

LDAP:

  • Issues:
    • Last Friday night’s update:  PeopleSoft added a SSN now matches a pre-existing entry!  merged.
    • OSP notified Account Services of duplications.  One was the above one from PeopleSoft.  The second is not confirmed by the facts I see to be a duplication.
  • Active Directory update failure.  The compromised account disabled during the update window problem struck again last night.  Took a while to sort it out.

BACKUPS:

  • I messed up and forgot to update the file server client definitions for the weekend full saves, then when I did remember and update it, I caused the incremental storage to overflow.  Then, the recovery of that has caused one of the full save storage disks to fill up.  Yes, this is going to be an interesting week.

22 August 2014: Friday

Posted: August 22nd, 2014 by fcs

LDAP:

  • No issues last night!  Hallelujah!

BACKUPS:

  • The stornode5 is penguin-backup is stornode5 and neither is configured correctly reared its ugly head yesterday and continues today.  It was working perfectly for two weeks and then I had to go and restart the backup server and it changed something – which should not happen, but I’ve learned to expect and accept that NetWorker does the unexpected and seldom works as expected.
    • So… the easiest and quickest and gets the recovermail and recoverinbox commands working again… is to shoot stornode5 and rename that system penguin-backup (as a storage node).
    • Kent found instructions for doing split access routing in Linux.  It is working… YAY…
  • Looking at savepnpc to keep the Incremental and the Full saves from stepping on each other on the fileserver systems.

21 August 2014: Thursday

Posted: August 21st, 2014 by fcs

LDAP:

  • Issues:
    • Former Student – dance done.
    • New “BCA” – no match found – asking the Registrar’s office for the EmplID of this employee.
      • Got the emplid, found the record.  Discovered that I was searching in the where’s this student manner and that’s not right for a BCA.  DOH!  merged.
    • The “BCA” from yesterday turns out to be a typo on the part of HR, and it is the right record.  Merged.

BACKUPS:

  • Disk clones completed – that makes them all.
  • Tape Drive at 40006 appears to have failed its serial interface – it is not talking to the XLS, which thinks someone has removed it.  Seems for a few cents more, they could have put a sensor in there to determine that a drive housing was installed but it was not talking.  Disabled the drive in NetWorker so it will stop trying to put a tape into it – and requested a replacement from Qualstar.
    • reseat of drive happened at just the wrong moment and the library lost track of where a tape went.  Had to put it in physical and open a door, then let it inventory and put it back in logical mode to fix the problem.
  • email backups are a mess – thanks to stornode5′s alias problems earlier in the week!

Projects:

  • Got patches installed on 7914 and jbod connected… now to make it purr again…

20 August 2014: Wednesday

Posted: August 20th, 2014 by fcs

LDAP:

  • Issues:
    • New “Faculty” (aka Banner Course Assignment) with an SSN that does not quite match the Faculty (PeopleSoft) that I think this is.  Email off to Registrar’s office…
    • New Graduate Student employee.  Found and merged.

BACKUPS:

  • CoM/IS clone finished overnight, Disk is close.
  • Oh joy!  Account Services deleted a departmental account per request and suddenly the customer determined that they needed that account after all.  Recovery time.

Hardware:

  • Failed drives in MANNSAN1 and STORNODE10 replaced…
  • 7914 (veeam-repos0) fibre card replaced with M5120/1GB/battery SAS RAID card.
    • RAID6 license obtained and installed.

19 August 2014: Tuesday

Posted: August 19th, 2014 by fcs

LDAP:

  • Issues:
    • Multiple CatCards: Notified CatCard office of the issue.
    • Two PeopleSoft new employees without SSNs:
      • found and matched them both.
  • VPN groups: Three people added to a VPN by request of the owner.
  • Special run for CatCard office – to give them some new data.

BACKUPS:

  • stornode5 and spogiprod both were “not configured properly” – deleted their aliases and let NetWorker create them again… bingo — they work.  I hate whatever that corruption issue is.
  • Tape clones finished.  Disk and CoM/IS clones running.

Other:

  • Account Services system has been up for a year… Oops… rectified.
  • Multiple rounds of cycling the Maestro GUI service.

18 August 2014: Monday

Posted: August 18th, 2014 by fcs

LDAP:

  • Issues:
    • Grrr – PeopleSoft again with the entry with no SSN later gets an SSN that matches an existing record and causes me to have to merge things.  Merged the records, disabled the duplicate account.
    • Three Student/Graduate Fellowships without SSNs.  Found them all and merged them in.

BACKUPS:

  • Disk failure on storage node 10 – except the Nagios check failed to alert.  Well, the Nagios (nrpe actually) check script was flawed.  Repaired, updated on the rest of the NetWorker systems so this won’t happen again…
    • Stole a disk out of veeam-bs0 (powered off, not in use) to get storage node 10 protected again, and reported the failed disk to our maintenance company – should have a new one on Tuesday.
  • Disk clones from last week still running – sigh – aborted them.
  • storage node 1 (stornode1) has been removed as a storage node.  Left the machine up, just in case.

SecurID:

  • Update to add new VPN end point.

15 August 2014: Friday

Posted: August 15th, 2014 by fcs

LDAP:

  • Registrar rolled semester forward from 201406 to 201409 yesterday.  Caused 18,515 updates (all good!)
  • Issues:
    • Multiple Barcodes: Notified CatCard office they were delivering a duplicate entry to LDAP.
    • Singleton Match with different last name:  OSP had notified me yesterday that this was going to happen.  Fixed.
  • Meeting with the CatCard office about their feed, how it works, and changes that can be made.
  • LDAP Account purge ran – 14 ERRORS – trying to add bounce rules for accounts that already had them.
  • LDAP Management password updated.

BACKUPS:

  • New email storage node is running with a higher load than expected, but not anything that is terrible – adjusting the monitoring points so it will stop doing the “WARNING” reports that cause others to become agitated.
  • Clones: CoM/IS clones should finish up today (4TB in process/19GB left), Disk clones probably not (30TB left).
  • Sharding update for the email backups (changed directive of safety net client from “Penguin backup directives” to “Penguin directives” and disabled the shard client to move back to not using shards at all)
  • Parallel stream choices adjusted for file server backups that will do full saves this weekend.
  • defined clients for two new file servers that will be grown in the coming week(s).
  • Putting plan (checklist) together for Monday’s work.

Code:

  • Attempting to grok what Ben was trying to do in user_trickle with the $SIG{INT} definition – since system is used to call the move user process, there is really no way to get a sigint to the parent process while the child is off doing its thing…
Contact Us ©2010 The University of Vermont – Burlington, VT 05405 – (802) 656-3131