UVM Blogs

LDAP:

Backend DB stats script updated to give improved stats.
AD feed update to handle displayName better.  When will it go production???
Privacy flags… probably a great time to re-write and install […]

LDAP:

PeopleSoft upgrade will prevent the input of invalid SSN’s (so much for our special 9* entries to flag special kinds of employees).  I have to work out the code changes to get around this.
One privacy […]

LDAP:

Still working on moving the database that holds the backend from Oracle’s Berkley DB (the old Sleepycat DB, or BDB) to Howard Chu’s MDB (Memory Data Base) that he has developed in the past year.  It is […]

Oh my, it has been two weeks since I last updated this. That’s never good. It means I’ve been running around like a be-headed chicken!

Backup things, LDAP things, meetings and dealing with vendors. Just not a […]

Backups:

Update kerberos server definition so it should take a full tonight
Manually force a full on skink since it was upgraded from RHEL4 to RHEL6 yesterday
Storage Array Maintenance

Replace a failed […]

Backups:

This was backup maintenance day.

OS upgrades
Disk maintenance (fsck’s)
The completely unwanted manual editing of a configuration database because the Linux startup processes are not […]

LDAP: URI formatting questions answered No privacy flags!  A minor miracle! A few update matching failures that required some digging More work on a sanitize script to obfuscate enough data to make the LDAP database safe to release to the vendor. Backup: Do the EMC SR dance – where the heck is 8.0.0.2 (aka NW 8.0 Build 172)?  Oh, not quite…[Read more]

Today is the 11th anniversary of the terrorist attack that collapsed the World Trade Towers in New York City and killed over 3000 people.  Let’s have a moment of silence. Thank you, now on with our daily report… LDAP: Privacy flags, flags and even more flags HYC wants to work with me to find the source of the [...]

LDAP: WAF hit because I got up and left my pager by the bed and then the 6:30 ldap update on the test server (where I’m working out how the mdb backend will perform) blows up and pages her out of bed.  Not a good way to start the weekend. The test system has 8GB of memory and [...]

LDAP: Privacy flags – bumper crop today PeopleSoft manual merges of two entries mdb test worked on development primary server, now to test it on the development replica ruhroh!  mini-httpd is not working correctly on RHEL6!  Must fix that.  The pidofproc function is not working on RHEL6 like it does on RHEL5.  The straight pidof binary command…[Read more]

NetWorker

Testing NW 8.0.0.2:

It uses /bin/logger which doesn’t exist. Opened SR 49973418.

client backup failures

LDAP

Privacy Flags

LDAP: Multiple Privacy flags Scratch one out of the look-aside Build 2.4.32 for RHEL6 testing (mdb backend development) RAVE queries vs daily feed Backup: Approving that NW142341 fixed the issue with 7.6sp4.1 so SR can be closed Attempting to get NW141877 to fix the yum breakage of 8.0sp0 One fake bad tape cleansed [...]

LDAP

ldif_diff reports a problem in the database, thanks to the update made to the code yesterday

Backup
Put together checklist for Monday & Tuesday’s work

General
Xi-Text: Duplexing issue with one of the printers

LDAP: Privacy flags Update Banner entries in purgatory waiting for PeopleSoft entries to appear Update local C code to deal with current openLDAP bugs (blank dn’s in slapcat output) – on development server only! NetWorker: Full tape removal Worn out tape drive replacement More blank tapes put into the library Printers Three (3) printers…[Read more]

Backup: Clones – Recloning save sets from CEMS and COMIS that had to be purged due to tape issues Tapes – Adding more tapes to the scratch pool Tape Drives – Received replacement drive for worn out one – replace it tomorrow LDAP: Monthly account purging ran into problems – missing a perl module. Got it installed and restarted [...]

Backup SR 46157224 : Call in and attempt to get some form of feedback on this nsrd crash and refusal to start from 1am Sunday morning. This is getting very frustrating that nsrd (the NetWorker brain process) randomly crashes and EMC is apparently unable (or just unwilling) to work on the issue. Backup Maintenance Day : OS Upgrades on all [...]

Backups

update second saturday fileserver client definition

LDAP

Create recipe for Saturday’s upgrade work
Lay groundwork for Saturday’s upgrade work

VacationWoohoo! A day off!

LDAP Working on the replacement for the ldaprw system Updating the uvm-ldap-schema rpm package Testing out the creation of a new PeopleSoft attribute Backups Create a team sendmail alias Replace a worn out cleaning tape Force a start of one large backup group which finally finished its monthly full after 4+ days [...]

Backup Not properly configured clients: Again with the aliases that are not quite what NetWorker wants today, but they were fine last week. Huh??? Short tapes: There were another eight tapes written short. However, I’m not going to do anything about them until I get the response from IBM whether the drive failed due to defect or contamination. [...]

Backup
a tape drive that was brand new on September 23 is now broken and does not work.

LDAP
proceeding on ldaprw replacement

You have ssh’d to a server and have tunneled X back, but you have to do work in an X application as a service account and you have to use sudo to get to that account. $ echo $DISPLAY | sed "s,localhost,`hostname`/unix," | xargs xauth list bumrap.example.com/unix:10 MIT-MAGIC-COOKIE-1 _hex_string_ $ sudo su - service_account $ [...]

TCSH conversion Responding to the people who were confused by the message that went out to the last 1,980 accounts still configured to use tcsh as their loginShell. Backup Discussion with Ben concerning the new fileservers Replace failed 750GB drive in SATABeast1 LDAP Match an employee with the required student record Rolling restarts to…[Read more]

Project from H*** The New Net::LDAP installed on the production ldap server yesterday caused issues overnight. The update reported a lot of new errors and didn’t update Active Directory very well. I have disabled that code until I can make the time to udpate all the scripts so that they can deal with the Net::LDAP::Entry attributes method [...]

Admin LDAP Nightly errors — handle three Banner Course Assignment errors, one deletion, one addition, and one modification of entries that have never matched with the HR feed Backup NetApp Filer backups — the full from 25. Dezember was trashed because stornode8 crashed that night. It appeared that the backup was not happening, but checking…[Read more]

Admin Email 3800 email messages in the INBOX after 2 weeks off… yikes! LDAP kerberos account for ldap entry with no posixAccount objectclass. Turns out there was a failure to remove the kerberos principal in April 2007. Solution: remove the kerberos principal and update the password expiration code to flag kerberos principal with ldap entry…[Read more]

Admin LDAP Two more “faculty” from Banner teaching courses that start next month and nothing in PeopleSoft Alumni email-for-life addition VPN Groups testing Backup More email exchanges with EMC about almost everything Weekly cloning started One AFTD at 98% full — it should survive until later in the week when the disk gets staged …[Read more]

Admin

LDAP

Privacy Flags

BACKUP

MED9300 cluster something was wrong in the definition of med9302, added shortname alias and re-tried, that seems to have worked — must be another hostname doesn’t match DNS issue.
Index: bad again today
Windows DR: nothing new

Admin Backup It has now been over three months that I have not gone a single day without talking with EMC about some problem in NetWorker. This has reached the point where it is a serious drain on my time, energy and well-being. I really need either for EMC to magically fix their code problems and just work [...]

Admin LDAP Join errors — 1 of 2 fixed. Banner being down I think is preventing the second from being delivered to LDAP. Privacy flags Backup Examine save group failures Recycle expired tapes Footprints 129287: email notifications of new requests being sent to former agents. Resolved — Footprints stores this list in the MRprojects file…[Read more]

Admin LDAP Fix join errors from this morning Rant at openldap developers for calling 2.4 stable with all the issues still floating around on 2.4.18. Backup Oops… too many clients defined… who was supposed to keep count this week…. hmm, looks like it was my turn. Darn! EMC 31051592 — Yes, I am really running 7.4sp5 and I really do…[Read more]

Admin LDAP Fix join failures from yesterday Privacy flag requests Fix join failures from today Backup EMC 31359116 — savegrp core — message back from the SME indicates that they got the core without the binary to examine. That’s strange because I packaged the core and the savegrp binary in the zip file that I uploaded. I have verified…[Read more]

Admin Backup EMC 31192910 — Continuing to work on the sev 1 issue — Windows server 2008 DR doesn’t work. EMC 31051592 — Reopen — get LGTsc30901 facts — created June 27th, hot fix available August 4th — but EMC support people I dealt with didn’t know of it, couldn’t find it when I called in the issue on [...]

Admin Backup EMC 31051592: no nsrck’s hung this morning. EMC 31099632: no word from EMC — have to refresh their memory tomorrow. Scouting for word on whether 7.4sp5 is safe or not — appears it is good Tapes and Disks picked up by SecurShred for destruction. Start to lay out design for scripted staging. Investigations into why backups on…[Read more]

Admin Backup EMC 31051592: Forced to level 1 to get call back, two hours on the phone with support, re-defined 42 clients that had been removed over the course of the last 15 years, nsrim -X now runs without causing nsrck to hang. Acid test will be whether there is a hung nsrck or not tomorrow morning. EMC 31099632: [...]

Admin Backup Manually clean tape drive, EMC 23189374… still broken since 14. April 2008! EMC 31051592: kill off the nsrck that started and hung at midnight. Discover the “undef” volume problem is a side effect of this daily process not working. Continue manual nsrck -L6′s against all clients in /nsr/index EMC 31099632 — open, sev 2 –…[Read more]

Admin Backup EMC 31051592: worked for an hour on the phone with Wallace on Friday and ran commands on Sunday afternoon. The “nsrck -m” ran without error. However, the “nsrim -X” appears to have hung up trying to work on a client that was removed in March of 2008. Will spend more time working with EMC this morning. [...]

Admin Backup 31051592: Sev 2 — nightly nsrim/nsrck pair starts up (nsrim forks and starts nsrck) but nsrck never finishes… one was running for 17 days. Does not use any cpu time, just sits there…. EMC is trying to force me to upgrade to the brand new 7.4.5 release. Volumes going “undef” instead of “expired” — 5 more this [...]

Admin Backup EMC SR 30744500: Discovered man nsrck -MXv (long string of client names) running yesterday when I shutdown networker for monthly maintenance. Could those (some running from as far back as August 1, be related to the ongoing corruption problems in daemon.raw? Found more tapes that have volretent value of “undef” instead of…[Read more]

Admin Backup Monthly Backup Maintenance Qualstar XLS: Upgraded firmware from version 2 to version 3. Process took 7 hours. AFTD: fsck’d /disk1 on storage nodes 5, 7, and 8. LTO4 drives: Upgraded firmware to 94D0 to address a code “A” issue with rewinds. Rebooted all storage nodes and NSR Server. LDAP Privacy flags Accounts Automated Purge…[Read more]

Admin Calendar OS Maintenance applied and Calendar database maintenance run Backup rebooted gnu – read-only file system error — more fallout from the Day of Disaster! Changed nsrports range on rh4-build. Discovered the real problem in NW7.5 is that some piece of nsrexecd is hard coded to open port 7937 — so if 7937 is in the range of ports…[Read more]

Admin Disaster Day The postmortem remains to be done and will require the assistance of IBM and EMC (VMWare), but we’ve had a fun day of it. It started on Thursday (while I was out) when working with IBM about an issue that had caused the replication connection between two SAN systems to break. IBM determined that there [...]

Admin

LDAP:

Update delEntry routine to remember the PeopleSoft emplid value

Admin Backup: Monthly maintenance day: Qualstar XLS832700 firmware update — FAILED X-Y board is original and needs to be replaced to be certain it will work with the new firmware. Qualstar is researching replacement options (maintenance agreement coverage) NetWorker updated to 7.4.4.6 to fix several open issues OS Maintenance applied to…[Read more]

Admin Backup Today is the start of the server migration from Solaris to Linux. This is a move that is not officially supported by EMC. Their stance is that you just start over and if you want the tapes from the previous operating system you run the scanner command to import them into the new server. I should [...]

Admin

Calendar
Quarterly DB Maintenance
Backup
Several AFTD’s were getting very full, one reached 99%. Manually staged savesets off the disks (AFTD’s) to tape volumes.

Admin

FootPrints
Upgraded to v9.0.3. One fallout, the old group.html files did not work with this version. Took several hours on Thursday to fix them.

Admin Backup Licensing for Linux migration: still confusion and EMC rep is out on vacation today. New tapes relocated to room Tape Library is in from my office. LDAP Nightly update failed in an infinite loop that did not set off any monitoring alerts. Three hours of work to design, implement, test, and deploy the repaired logic. I really…[Read more]

Admin Backup Adjusting management scripts for differences between Linux and Solaris paths More work chasing the license issues Requested an update on the status of EMC SR 29279000 Opened EMC SR 29683538 (ibis1 index messup) as a Sev 1 LDAP Setting privacy flags Audit sync of replicas and primary server databases (zero errors found) [...]

Admin Backup Getting licensing worked on for the server replacement in two weeks. It’s absolutely amazing that none of the licenses that were purchased before EMC bought Legato are to be found, but we pay maintenance on these same licenses year after year. I begin to think the licensing is going to be the biggest challenge of this [...]