Posted: August 1st, 2014 by fcs
- Dealing with issues from Tuesday night on…
- OSP – former student, new name… verifying.
- Confirmed. Merged, and account created.
- Two former students… created.
- Employee issues:
- Duplicate employee – wrong SSN entered – verifying.
- Confirmed. Tossed it aside (will get a removal failure later).
- Two Student Employees – found and updated.
- Mangled SSN – I think it was entered incorrectly – verifying.
- Confirmed and merged with the student.
- One botched SSN discovered, now cleared up duplicate account.
- Updated a VPN group per footprints request from Business Process folks.
- Removing Veeam
- Waiting on a new evaluation license so I can do the work safely.
- Obtained – but I still had to manually remove the vcenter bits. Doh!
- Why are the new file servers failing to back up?
- Ah – things were changed and they had another rolling outage round yesterday while I was out.
- Backups for failed systems manually started back up this morning (from the start).
- Updated the nagios checks so if the filesystems fill up over the weekend, only email will be sent, no pages will go out. That means that I’ll need to be more hands on this weekend, as much as I can be since I’ll be traveling both days.
Posted: July 30th, 2014 by fcs
- Multiple Issues…
- Account Scripts updated to work with the new FS Servers farm
- VPN group bouncing for Network Services testing.
- The horse didn’t choke! Backups still running at over 1000 MB/s to stornode3 and stornode4 from the 14 4TB file systems.
- Put the 184.108.40.206 client and manual pages RPMs into the RHEL5/32, RHEL5/64, and RHEL6/64 distribution channels to force that darned 220.127.116.11 mess off the cfengine managed systems.
Posted: July 29th, 2014 by fcs
- One former student. added.
- Complaint from PeopleSoft land, people are adding phone numbers with codes that nobody told me to expect and are now upset that LDAP doesn’t deal with those codes. Surprise!?!?!
- Some work on the account scripts… will they survive tonight’s changes???
- New NFS file server systems did not have the client installed before they attempted backup. client code was installed, and backups were run.
- Something went wonky with one of the drupal servers. It seems to have a new name or something. No – it was a configuration issue that only comes to light because we turned on IPv6 on the backup servers yesterday! Fixed by forcing the routing to use the correct source address on that system when it is talking to machines on the IPv6 address.
Posted: July 28th, 2014 by fcs
- Nothing to write home about
- Updating the VMware hardware versions to maintain compatibility.
- Clones all finished last week (at some point!)
- One of the windows clients attempted to back up the “B:\” disk again (that’s really the Windows Recovery Partition and should not be getting backed up) – this is the first time this error has happened since NetWorker 8 was installed on the servers. This client is still running the NetWorker 7.6 (according to what I can see in NetWorker’s records).
- IPv6 addresses assigned and put into production. Fingers crossed that NetWorker doesn’t explode.
- Looking at SureBackup Recovery – can it solve my Linux FLR issue?
- Dealing with the fallout from Friday’s upgrade:
- Attachments: The path that FootPrints uses for where they are stored changed to a better default place, and I didn’t notice. Had to shutdown the service (Friday evening) and move the old Attachments to the new place.
- Inability to enter issues… Getting reports from one office that they are having trouble. Trouble that sounds like Workstation and Browser issues – and not FootPrints issues, but with so much having changed, I still have to check it out.
Posted: July 25th, 2014 by fcs
- One issue – found the match, merged the new information in.
- Something was changed in the way the filesystems are managed and the account creation/removal scripts all went “KaBoom” this morning. Why does it always have to happen on a busy day?
- Something is horribly wrong – no failures last night.
- SSL Cert for new system.
- Nagios config for new system.
- mysql configuration done.
- Nekkid FootPrints 9.5.4 (64-bit) installed.
- Filesystem juggling done.
- Ready for noon, at 11:02… (I think!)
- FP 9.5.4 running on RHEL6 64-bit @ 13:36
- snapshot completed, FP upgrade to 11.6.02 started @ 14:00
- upgrade done – customization begins at 14:30
- And complete at 15:10! YAY
Posted: July 24th, 2014 by fcs
- Two PeopleSoft feeds that now have SSNs that duplicate existing entries.
- Clones finished up.
- Answer from Veeam tells me that what I thought I could do will not work. Back to the drawing board.
- Egad! The mailing list is discovered to be terribly out of date. Less than 24 hours before the upgrade. Ugh! Hoop jumping commences.
Posted: July 22nd, 2014 by fcs
- Multiple clients are not properly defined today – thank you EMC for that little bugger – added them to the script and re-run the jobs.
- Figuring out how to quote for new backups… yeah… sure… trying to figure out what new backups will really look like!
- And… figuring out that I haven’t got a ghost of a chance of ever doing a successful recover again!
- Tape clones complete, CoM/IS clones close to done, Disk clones still chugging.
- 10G networking cables installed on repository and stornode5 systems.
Posted: July 21st, 2014 by fcs
- No issues over the weekend.
- NetWorker: maintenance day – 18.104.22.168 successfully deployed, SR 64273236 work done.
- Backup Issues from over the weekend:
- One windows system failed because of low disk space, not enough space left to take VSS snapshots.
- The big Windows Fileserver again started backing up the S:\ disk that we have told it not to.
- The webmail development system failed because of a read-only file system.
- Veeam webex for some double checking.
Posted: July 18th, 2014 by fcs
- Two SSN/surname issues. One is a simple typo on entry (will merge and let PeopleSoft clean up its mess). The other is a likely marriage but need to confirm – waiting on response from Payroll.
- Disk clones finished yesterday (Tape clones were finished earlier in the week). CoM/IS clones finished this afternoon.
- Another tape drive with a code 6, and makes a tape go “unlabeled” which an nsrjb -II fixes — but Damn, NetWorker… come on!
- Figured out the Linux FLR hoops with Veeam.
- Figuring out the web server part of Veeam.
- Oh – Veeam delegation is not going to work without the system file indexing working :/
- Install the authentication manager on the new computers for Account Services.
Posted: July 17th, 2014 by fcs
- Another SSN only wrong surname issue. Found enough to feel comfortable doing the merge on my own.
- Updated the cn on three departmental accounts to fix a problem during the creation (wrong information entered).
- Two systems failed to backup. One is being researched (A CoM/IS Windows system that has had a lot of issues lately) and the other was powered off yesterday.
- More learning and trying out things in VEEAM.
- SR 64273236: Intermittent ignoring of directives
- There is a problem of a configuration file that was clobbered (or never created) that is supposed to be fixed in 22.214.171.124. They want me to upgrade to that. Well, Monday is the maintenance day – I can do it on that day. I will also recover the lost Config.xml file on that day (it was erased when 126.96.36.199 was removed and 188.8.131.52 installed on April 21).
- SR 64026734: scanning using persistent names loses the library
- Suggestion to exclude the scsi address of the controller. No change (what format)?
- dodo.uvm.edu deployed (VEEAM restore of test server to new location) to become the new footprints.uvm.edu.