I messed with it this week, rewrote some of the core code. Some much better SQL.
On Aug 19, 2014, at 7:00 AM, Paul Bierman wrote:
Well, my 5 minute drive gave NO errors the searches come back fast…so good news, thanks!
Geology Landscape Change project exploded Chapter 6
Server hosting BigBlueButton exploded
On Aug 6, 2014, at 10:45 AM, Frank Swasey wrote:
Wes,
Yesterday morning /etc/cron.daily/apt applied a libc update to morpho. While that shouldn’t have caused a problem, it might have. NetWorker is not working (it throws an invalid pointer error in munmap_chunk() which is in glibc). I have upgraded the NetWorker client to 8.1.1.7 (the latest and greatest) but it still is failing. EMC would like us to reboot the system because it might resolve the issue.
Is there any particular time that a reboot would be better or worse? Are you ok with me (or someone from SAA) rebooting the system?
—
Frank Swasey
On 8/6/14, 10:56 AM, Wesley A. Wright wrote:
Check https://www.uvm.edu/ctl/apps/bbb/?c=bbb&m=listall
If nothing is running, ok to reboot
On Aug 6, 2014, at 11:27 AM, Jim Lawson wrote:
Hi Wes,
I took a look at morpho and it appears that the system is somewhat
broken. A glibc upgrade has occurred which seems to break ABI, but
there are some packages which are held back (I presume they were copied
and installed by hand) which are preventing the system from finishing
its install.There are various suggestions that aptitude is making to resolve the
package state: removing libreoffice, removing openoffice, or removing
bbb (followed by an install of a more updated version?) might get the
system into a better state.
On 8/6/14, 11:53 AM, Wesley A. Wright wrote:
Why does all this happen while I’m on vacation? I’m on a farm in southern canada armed with an iPad and < 200mb remaining on my at&t international data plan. Why do I keep reading email?
Most recent bigbluebutton is .81. That's what is installed. Libreoffice is a bbb dependency. I don't think OpenOffice is installed, but it might be left over from a previous bbb install, in any case it has been superseded by libreoffice and is no longer required.
I suppose bbb can be removed and reinstalled, although there are a handful of undocumented site specific configuration files that might get overwritten, thus rendering bbb useless.
So what to do? Remove all XOffice, reinstall libreoffice , don't mess with bbb unless we really have to? Regress to earlier glibc? Wait until I get back (aug. 18)?
On 8/6/14, 12:08 PM, Jim Lawson wrote:
It looks like bbb is still working, which is very surprising to me,
since the glibc breakage is so bad that we can’t run scp or iptables.Maybe it can wait until the 18th, but it probably shouldn’t wait any
longer because you might have to recover from backups to fix this
particular problem. (That’s a bare-metal restore, BTW, since ‘recover’
on morpho is also badly broken.)
On Aug 6, 2014, at 4:19 PM, Jim Lawson wrote:
Mike and I tested it and, astonishingly, BBB is still working (we shared
a 2-person videoconf.) If you are comfortable waiting until you are
back on the 18th to look at it, that’s up to you. Mike and I are both
on vacation the week of the 18th, so assistance from SAA that week may
not be as timely.
On 8/7/14, 2:07 PM, Wesley A. Wright wrote:
Found this:
https://bugs.launchpad.net/ubuntu/+bug/1352769
Seemed to work. apt-get working now, scp too.
Included with latest upgrade was bbb-client update
https://groups.google.com/forum/m/#!topic/bigbluebutton-setup/db8VxAn_iaY
Which should not break anything,but I don’t have the equipment to test.
How many points do I win for fixing this?
Sent from my iPad
Geology Landscape Change project exploded Chapter 5
On Aug 1, 2014, at 10:56 AM, Jim Lawson wrote:
The session data stored for this webapp is … large. We’ve moved it
out of the database and onto the local filesystem, which has helped the
database a lot, but now we’re just filling the local disk (and probably
thrashing I/O besides)Currently there are 3126 session data files in /tmp/glcp. 46 of them
are over 100MB in size! A gross way we could handle this is a cron job
that runs (say once an hour):find /tmp/glcp -type f -size +5M -exec rm -f {} ;
Really, this app should not store so much data in the session! But you
asked for “what SAA can do about this”.Any other suggestions?
I’ve been mulling this over for a week now and getting nowhere. Perform a search -> store results ‘set’ in session -> read from session to display results -> retain last MAX_SETS Results is the central,design philosophy of the application ( as designed by long gone CS grad student). Search for trees, get 2000+ results! what should we do with them?
The related problem: start_session when session > 100mb results in php out of memory error. Can we crank this up, too?
Couple of things I’ll try: change max_sets from current 7 to 2 or 3. Remove browse all 75000 images option, too unwieldy. Expire sessions more freqently.
Clearly this app does not scale well. Good mews is much code assumes a 5 digit ID number, so we are limited to 99999 images.
Geology Landscape Change project exploded Chapter 4
On Jul 26, 2014, at 6:28 PM, Paul Bierman wrote:
I got in a dozen times with no connection warnings…
searched for dog and horse…all good.
but cow, well that got me the attached warning.
p
Session cache getting too big, need to delete something somewhere, but not sure what or where quite yet. Leaving for Canada in 30 minutes, may have to wait until Aug 10 or so before I can revisit
short term client fix: delete uvm.edu cookies in browser
Posted in Uncategorized, Wes
Tagged Boffins, database, geology landscape change, php
Leave a comment
Geology Landscape Change project exploded Chapter 3
I spent a lot of hours on this today. I think I’ve nailed it down. Altered general form of Sql to
SELECT ls, sequence, county, town, title, PHOTOGRAPHER, images.FILENAME, MEDIA, PHOTOYEAR, PHOTOSOURCE, description, keywords, format_extent, DATUM, longitude, latitude, utm_northing, utm_easting, ELEVATION, format_creation, contributor_collector, contributor_scientific_advisor, contributor_reviewer, coverage_periodname, coverage_placename, times_viewed, PUBLIC, (longitude”” or utm_northing””) has_location FROM images left join viewcounts on images.FILENAME=viewcounts.FILENAME where MATCH (images.FILENAME, OriginalFileName, ls, title, keywords, description, town, county, PHOTOSOURCE, PHOTODATE, PHOTOYEAR, PHOTOGRAPHER, MEDIA, identifier_citation, title_alternative, publisher) AGAINST (‘+dogs’ IN BOOLEAN MODE) AND (STATUS = ‘X’) ORDER BY title
Cache results using session variables
Extract start, limit subsets from session variables
PLEASE TEST, paint is still wet
WATCH OUR FOR out of space warnings. Session files getting pretty big
On Jul 26, 2014, at 1:58 PM, Jim Lawson wrote:
Yes, glcp ran out of space just now. Added 5gb of space to /. Weak network covg from here, so can’t do detailed troubleshooting …
Geology Landscape Change project exploded Chapter 2
On Jul 17, 2014, at 10:24 AM, Scott Dellinger wrote:
[Jim Lawson and I were discussing yesterday afternoon who to speak with about the landscape project’s maintenance, so I’ll let this serve tentatively as confirmation of who belongs on this thread. 🙂 ]
The landscape code for image searches performs some very inefficient searches against the database in ways that would not be so apparent with fewer images, but start quickly making queries pile up as the image count increases and multiple search engines start accessing your web pages simultaneously. Despite there only being 70 some thousand rows in your image tables in the database, these types of database queries in large numbers have recently been seriously impacting performance of WebDB services for other users as well as your own. Short of temporarily banning your user account from access, yesterday our best option in the short term was (further) restricting the number of simultaneous connections permitted by the “glcp” WebDB user, to 15. This is why you encountered an error about exceeding max_user_connections.
As a matter of scalability, 70 thousand rows is not a lot. Hundreds of millions of rows would be. However, database servers are not designed to efficiently handle queries such as “list all images without restriction and toss out all but one in the middle”, which is the main culprit type of query occurring here. This is very, very different from a conceptual “show me image number 5,002”, which can be extremely fast.
Because these queries are hurting performance on WebDB for other users at this point, we cannot raise the concurrent connection limit again unless these problems are resolved in some way. Obviously, the current state is not optimal for you or your users. I am not sure how this software is being regularly maintained at present, but here are the non-mutually-exclusive options I can think of, going forward:
• Move the landscape database onto a MySQL service on your existing dedicated web server. You would not have a connection limit this way and would not impact anyone else. This will not solve performance issues, which will continue getting worse over time.• Put a caching proxy server (probably Varnish) in front of your web server. For pages that are accessed frequently, this would query the database however many times one page requires, then not again for that page until the page expires from the cache. A basic Varnish setup for a web application that is not overly complicated can be easy to set up. This would not solve the database issues, but might put a big dent in their impact.• Rewrite the inefficient database queries to access data more efficiently. Selecting a single row at a time from a table should not involve querying half the table before picking out one row. There must either be some reasonable way to search by a specific identifier to retrieve one row in the first place, or at least have a periodic job that performs the regular searches and caches the results in a way that itself can be queried by an identifier.Who is currently maintaining the landscape project code? SAA would be happy to offer advice, but cannot do more than that at present.
So while on vacation…
- I added a nicer error message
- I edited http://www.uvm.edu/landscape/search such that it no longer performs a “select all” before offering to “Browse all”
- I disabled custom database session handling and restored php file based session handling
- will these changes help? I don’t know.
- I went back to vacation mode. Things are swell on my deck in Starksboro
mediamanager hiccup
Media Manager misbehaving again, and once again SSA was to blame
On Jul 11, 2014, at 6:58 AM, Benjamin Coddington wrote:
That corresponds with a change in request handling we made late Monday / early Tuesday. We’re routing incoming requests across a new set of proxy servers, and I found that their maximum request size was set to only 1048576 rather than the 209715200 we’ve set for filetransfer in the past. I’ve set it to match on the new systems.
Note for me: make systems unit test for max request length.
Geology Landscape Change project exploded Chapter 1
SAA reports that certain GLCP actions are creating giant log messages in mySQL servers and thus making their beepers beep. Problem associated with using mySQL for session handling and an ever increasing number of new images.
SAA made a couple of changes as a workaround, but not very satisfactory
mediamanager update
As a favor for Jing users, media manager now accepts and plays SWF files.
virtualbox
What is virtualbox?
VirtualBox is a powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use. Not only is VirtualBox an extremely feature rich, high performance product for enterprise customers, it is also the only professional solution that is freely available as Open Source Software under the terms of the GNU General Public License (GPL) version 2. See “About VirtualBox” for an introduction.
Presently, VirtualBox runs on Windows, Linux, Macintosh, and Solaris hosts and supports a large number of guest operating systems including but not limited to Windows (NT 4.0, 2000, XP, Server 2003, Vista, Windows 7, Windows 8), DOS/Windows 3.x, Linux (2.4, 2.6 and 3.x), Solaris and OpenSolaris, OS/2, and OpenBSD.
installed in a jiffy on my Mac. Converts various flavors (Parallels, VMWare) of virtual machines for import. I imported my Parallels Windows 8 VM with no trouble. Building that VM out for possible installation on CTL303 machines. Also built an Ubuntu (Linux) 14.04 system for another project. Goodbye Parallels?