Posted: November 21st, 2014 by fcs
- Former Student: did the dance.
- Invalid SSN on new employee: found the foreign student entry and merged them.
- Invalid SSN on new employee: found the foreign student entry and merged them.
- Account Problems:
- The email account creations failed this morning…
- It was a timing issue. The email server the accounts get created on was messed up at 3am. It got fixed at 4:30am.
- Clones: completed last night.
- Oh damn – tape 1310292963 threw a code A last night. ITDT dump taken and sent to Quaestor for analysis.
- Hmm, one of the CoM/IS groups failed as well – looks like possibly another error with their disk volume??? Either that or a bad package removal.
- EMC closes issues 66992740 (recover errors) and 66904082 (inactivity timeout affecting jobs that are waiting for space) as Nope, could not have happened. Darn it… well, fine. I will worry about them later.
- EMC wants EMCGRAB run on all three servers for 66847046 (DD compression path differences). Zoiks… there goes my day.
Posted: November 20th, 2014 by fcs
- A former student: dance done.
- A one item match with the wrong last name: Issue was a difference in the way the name was entered. Merged and did the rest of the magic to get this account set up.
- Tape and Disk clones are finished, CoM/IS is chugging along.
- SR 66904082 – inactivity timeout applying on full AFTD issue: per phone conversation, I attached the five (5) failure emails from the FS-Gen3-* groups to the SR. EMC is attempting to replicate the situation and see if they can reproduce the problem…
- I was going to take the snapshot from the production SecurID primary server – but I remembered at the last instant registration. Doh… I’ll have to muck with that this evening. Performed the database and log dumps at 7pm.
Posted: November 19th, 2014 by fcs
- CoM/IS clients all died last night – It is believed that a controller failed (or failed over) and Windows continued to send all traffic to the failed controller – which obviously did not work. Things have been stabilized (or fixed) and the backup jobs are being restarted slowly.
- I replaced the failed tape drive from Monday after hours last night.
- Clones started at 8:45am.
- Finally downloaded and looked at the FormMail.pl tool that the portal problem submission form was using. They have their email address variable named “realname” but the tool expects the email address to be in “email” and the person’s name to be in “realname” – and hence the parentheses that get added around the email address. David has updated the forms to use the correct variable name and things appear to be much better now.
- Checking documentation – making recipe for snapshot of 6.1 and installation of 8.1
Posted: November 18th, 2014 by fcs
- Issues: None.
- Two accounts one person – merged as requested.
- There were no improperly defined client errors last night. I think that is a first after a maintenance day.
- Info sent to Commvault for a price/design.
- Another myuvm portal help ticket created from an invalid address that blows up Footprints. Why can’t the footprints code check for mismatched parentheses in the email address and not barf? Oh… once I “Check Submission” and get the green light – I can then modify it all I want before I actually Submit it, and I do not have the Check it again. Oops!
- IP Address keyword search issue: BMC is pretty adamant that we should turn off the full text indexing for the entire footprints system. I am really not comfortable with that recommendation.
- Finally asking the question about the bulk removal of expired/disabled tokens. There are no bulk tools (without spending money) – but there are ways to remove the specific tokens we are interested in… through the tokens menu item.
Posted: November 17th, 2014 by fcs
- Issues: none over the weekend.
- Backup Maintenance Day…
- Data Domain upgrade to 126.96.36.199-453843
- NetWorker upgrade to 188.8.131.52.Build.360
- XLS Drive 1310119697, SCD 5 on reboot (OUCH).
- Qualstar RMA # 49284 issued for the drive.
Posted: November 14th, 2014 by fcs
- No Issues in the past two night’s updates.
- The Foreign Exchange Student issue seems to have resolved itself. YAY!
- Registrar’s office has questions about the BCA process. Sorting that out and answering.
- re-created five (5) CoM/IS clients that had been deleted on Wednesday – because they are needed for their resources to be backed up.
- Tape and CoM/IS clones completed. Disk clones have 22TB left to go.
- NMC does not work with Java 1.8u25 – needed to download and install the Java 1.6 from Apple. Sigh.
- Worked through the portal help request with the developer so it stops generating emails that cause trouble for FootPrints.
Posted: November 12th, 2014 by fcs
- No Issues in last night’s update.
- Another foreign exchange student asks for help as his account was nuked on 11/7.
- Banner Extract programmer has found the error and is testing a fix.
- Another DD security release. Same fix (184.108.40.206), but this time to fix seven OpenSSL vulnerabilities. Just five months from vulnerability announcement to fix release.
- RAID6 upgrade for 5120 in bujbod1 purchase authorized.
- Second round of labeling the mass of tapes that returned from the Vault on Monday.
- Removal of fifty-nine (59) clients that have been gone for so long that there is no data left about them (routine maintenance).
- Gave up and started clones.
- Rolled out 8.1sp2.1 on test server – wound up having to rebuild the mhvtl software because it doesn’t work with the current kernel.
- Working on the Server Error (BMC Issue=327325) – the Luminis form is taking the garbage input from the user and placing it in the From: header of the email message. FootPrints then falls over dead.
Posted: November 11th, 2014 by fcs
- No issues last night.
- Got a person in “Thou shalt not VPN” land who should not be there. Thou art being expelled from here, Trespasser! I found that one, and 188 others. So, there is obviously some path from the land of the departed back to the land of the present that does not remove this VPN block. I just need to figure out where it is.
- Stupid NW claims we have 279 physical clients. Oops – time for cleanup!
Posted: November 10th, 2014 by fcs
Upgraded the macbook pro to Yosemite over the weekend. WiFi does not work at home and perl does not work at all (which means nothing that is based on perl – like mosh works). What a wonderful way to start the week.
- One Issue over the weekend: Student Employee that did not have an SSN. Found by name/birthday/address and merged.
- One person, two Accounts discovered by Account Services. Merged the two accounts together.
- Cleaned up last week’s aborted clones.
- Generated the November Vault Tape Retrieval list.
- Generated Full Tape List – there are 199 full tapes to go to the vault, and 336 brought back.
- Account Services wants to know if there is a way to deal with the removal of unassigned and expired tokens in bulk.
Posted: November 7th, 2014 by fcs
- No Issues in last night’s update (even with the decreased number from Banner).
- The numbers that Judi had warned me of were practically exact – and the entire process ran for a grand total of two more minutes than it did the night before.
- The email snapshots failed (as they have been every other day) because the backup was still running. I sure hope we get this ironed out sooner rather than later.
- Waiting for the usage on the DD to go down so I can swing the remainder of the data back over to that.
- Clones: Tape will finish up soon (doing the last 800GB) – Disk has 20TB left to go – CoM/IS is done.
- Emailed EMC about the recover errors and that starting nscd had removed them – still asking for the answer of what those messages mean (as in – why does it have problems when having to wait for nslcd to supply the answer that nscd would have buffered/cached).
- Phone Call and Email after I left yesterday concerning the failure of searching for keywords that are numbers. The recommended course is to disable the full-text indexes so the searches are forced to go through every record in the tables being searched. The concern is what that would do to the throughput and if it would cause us to have different performance issues. I have asked if it is possible to disable the full-text index at the workspace level instead of at the system level.
- BMC wants steps followed to gather up the Luminis process (I was not aware that all of these were Luminis submissions, Travis confirmed that he thinks they all are). Reaching out to Travis to gather the information BMC wants.
- Ran the 6.1 to 8.1 migration assessment tool – found mostly green and a couple of yellow areas. Forwarded the report to Account Services for comment.