Posted: October 30th, 2014 by fcs
- No issues – so leave me alone already!
- Setting up a new VPN group.
- Tape and CoM/IS clones finished, Disk chugs along (13+ TB left).
- The “Incremental” volume has been filling up daily – what are they throwing at me now???
- The fsXX systems seem to be confused about what is and is not a change.
- Could “restorecon” command have modified all of the files?
- EMC continues to ignore my SR’s.
- BMC actually wants me to work on the issues I have open with them???
- Providing updates to BMC on issues 327325 (Server Error), 333937 (IP Search), 332683 (Root owned files).
- OH – now BMC tells me they have an issue with numerical keyword searches on Linux/MySQL systems (333937). Great. Thanks.
Posted: October 28th, 2014 by fcs
- Cycle for new schema for that two year old project.
- Deal with issues in last night’s update:
- Blank line – Hello Account Services…
- Wrong surname with one datum match: Appears that OSP put the name in reverse of what the Registrar’s office did – now to determine which one is wrong. Determined, and repaired.
- DD compression even with the MTU 8000 is still at 5.8x instead of the 11.3x of history. Moving the backups to tape again. Grrrr
- Posted to the EMC SR, and emailed the tech. Still in need of a response.
- Stopped the mail folders from going to the dd.
- Left the local systems (old email servers) going to dd through the new hardware.
- moved the inbox backup group to go through stornode9 as it did briefly.
- Re-Wrote the audit.pl (post processing the nsrjb -C output) because the darned column placement moved again.
- Set the ft_min_word_len = 3 in the /etc/my.cnf on the test server.
- rebooted said test server.
- identified (fulltext.sql) the tables that have full text indexes.
- repaired said tables (fulltext.repair.sql) on the aforementioned test server.
Posted: October 27th, 2014 by fcs
- Doh! No issues over the weekend. Why can’t NetWorker be more like LDAP???
- 1,430 BCA only affiliations removed after the change went into production last week. YAY!
- List of removals sent to Registrar’s Office for corroboration.
- And a new object class to facilitate the completion of an old old project.
- stornode4 still battling the full disk problem. stage stage stage stage…
- stornode3 sitting there asking if it is ok to delete the stopstage file that I told it to delete, but did not use the “-f” parameter of the rm command. Doh!
- Making a second attempt at coding a work around for the EMC nsrstage changes the clretent value. This time, I want it to work, and be efficient.
- DD reports that it got 4.9x compression in the last 24 hours. Not good.
Posted: October 24th, 2014 by fcs
- Two former students from Banner – did the dance.
- Added monitoring for the temporary insecure idiotic ldap server that Oracle will be allowed to use until they can actually do something that is worth doing.
- email backups continue to fail every other day because of the time it takes to move all that data to tape.
- The new staging code (to deal with the clretent becoming the ssretent value) has not moved data off the AFTDs fast enough and one of them has filled up
- Plan is to deploy the 8.1 appliance at the permanent home – need to resolve the Backup Space issue first.
- Another round of “what’s broken” questions from BMC concerning the root owned files per issue 332683.
- Opening another issue – searching fails to find ip addresses now (. is a separator and ip address bytes are three characters => skip!). Issue # 333977 opened.
Posted: October 23rd, 2014 by fcs
- One issue last night – a “modifying the WRONG entry” error:
- Employee record gained an SSN and now it matches a former student LDAP entry. Merged the Banner info into the new HR entry (because the HR entry has the posixAccount) and purged the former student entry.
- Rolled out updates to network services feed and banner feed processes.
- Weekly Clones have finished! YAY.
- 596453: Surprise! I called “Mark” back (he left a VM after I left yesterday) and he was in the office at 6:45AM! Anyway… the answer is that as long as the 8.1 servers are using different names and addresses there is no reason that we cannot run the 6.1 and 8.1 servers in parallel during the cutover. The issue will be token related (pin changes on the 6.1 servers will not be reflected on the 8.1 servers).
- Next Steps: Pick a famous couple (names) and a network (addresses) for the new 8.1 servers.
- BMC responded to my 332683 issue that if I want the files to not be owned by root, I should move the crontab entry to the FootPrints user crontab. To which I called BULLSHIT! The root crontab is created by the FootPrints install process, it is not anything that I have done. And when we ran the 9.5 version of FootPrints it either did not generate files that were owned by root or the FootPrints code at least dealt with them without generating errors for the customer/agent to see.
Posted: October 22nd, 2014 by fcs
- No issues in last night’s update.
- POODLE recycles.
- POODLE (CVE-2014-3566) recycles – take two.
- Network Services feed – updated.
- CoM/IS clones have about 8.5TB working and left.
- Disk clones have about 4.3TB working and left.
- A Windows client (big file server) hung its nsrexecd process – after it was recycled, the saves that had all listed as queued started running.
- Opened case 596453 to confirm that the 8.1 appliances and the 6.1 servers can run concurrently to minimize the downtime required for moving host agents from the 6.1 servers to the 8.1 appliances.
- BMC concedes in issue 332683 that it is the root cronjob that is causing the files to be owned by root. And then they had the gall to ask if there was anything else they could do. Uhm… yeah… fix it!
Posted: October 21st, 2014 by fcs
- No Issues last night.
- I think I have found the solution… TLSProtocolMin 3.1
- Tape clones finished, the rest of them run on.
- MED23 B continues to perform the full from the weekend.
- Finally got back to the manual … have made a proposal for migration.
- No response from BMC on either of the issues I have open with them. Darn it.
Posted: October 20th, 2014 by fcs
- One issue over the weekend.
- A Visiting Graduate Student who didn’t match up. Found the match and merged.
- Opened SR 66570742 to deal with the fact that NetWorker is not properly keeping track of what data is really stored on AFTDs. There are some AFTDs where it is correct, but most of them are wrong (and the ones that are correct are the ones that are actually using compression)… HUH????
- Backup System Maintenance cancelled.
- Clones started.
- I see that one of the two issues I have open (327325) got a response from BMC folks that I never saw, so I answered it on their footprints web site. The other one (332683), I have requested that I get an update on. I’ll call them tomorrow if nothing else happens today.
Posted: October 17th, 2014 by fcs
- No issues in last night’s update.
- Disk clones have 13T left (at 7:41AM)
- NetWorker frozen in place. Apparently, the “only use X% of a disk” option is too stupid to be used. NetWorker appears to calculate the % used of a disk based on the media database and not on any actual real checking of the disk space usage. Is it because the programmers don’t know how?
- Live Chat with EMC about this issue: jobkill – kill them off one by one.
- Killed them all off – jobkill does not see them any more. However, NMC still sees them, and they are still listed by my sessions.sh.
- draining stage and clone commands.
- doing another live chat with EMC – I’ve got nothing left to lose (and time to kill).
- EMC confirms – recycle (and kill -9 any stragglers) is only way forward.
- NetWorker put into maintenance mode – announcement sent out of impending recycle.
- Recycle started at 2pm, back in business at 2:30.
- Reading the 6.1 -> 8.1 migration guide. I will have to touch every single Agent Host whether I change the IP address/Name of the SecurID servers or not. So, why not go ahead and do it the right way???
Posted: October 16th, 2014 by fcs
- Two one-item match, wrong surname issues in last night’s update:
- Both PeopleSoft:
- One is a former student – asking for confirmation of details. HR confirms, merged and account created.
- One is an OSP feed – problem is a space that was or was not included… merged.
- Disk clones still chugging, 20.8TB left to go.
- email backups are running, so stopped the staging of the data off the data domain.
- Patrick called on the 00591816 (firewall) case at 4 pm. I called him back at 6:40 am – not surprisingly, he was not available. He called back later. The internal firewall is there, we can change it, RSA does not recommend changing it and may close a case if they ask us to revert our changes and we refuse. I need to experiment. Yes, I can update the /etc/sysconfig/iptables files and it stays put and doesn’t seem to cause problems. Now to read up on how to migrate from 6.1 to 8.1
- Issue 332683: I suspect it is root’s crontab entry that runs FOOTPRINTS/cgi/MRRunScheduledScripts.pl constantly that is doing this. Asking for confirmation.