26 March 2018 ~ 0 Comments

Solution: Out of Control MySQL Slow Log Files

I’ve been having an ongoing problem on one of my servers storage space, long story short the culprit ended up being that the mysql-slow.log file was 80GB.

It had gotten turned on for testing at one point and the threshold turned very low and I never thought to check it again over the years.

Anyway, I had a bit of a hassle figuring out how to resolve it so thought I would post it here for future reference.

How to Safely Clear Your Mysql-Slow.log File

This was actually pretty simple once I figured out the quirks.

The first thing to do is to disable slow query logging in mysql:

set global slow_query_logging=OFF

After it is disabled then it will no longer want to write to the file, if it is showing that the file is still in use then simply run:

flush slow logs

After it is disabled you can safely move, compress, or delete the old log file.  If you plan on deleting it then I would recommend at least renaming it first until everything is completed, just as a precaution if the file needs to be restored.

Once you have removed or deleted it, you can create a new one with the same name

>mysql-slow.log

Once it is created, you may need to adjust it’s rights.  This is where I got hung up on, the mysql service was unable to write to the file.

chmod 775 mysql-slow.log

After this is all done, you can turn slow queries back on

set global slow_query_logging=ON

And you will begin slow query logging again in your new file, you can now safely delete the old file if desired.

Hope this helps someone, it’s pretty straight forward but a lot of the other stuff I found when trying to research the problem did not mention having to change the access for the new file and it stumped me for a bit.

Continue Reading

15 September 2012 ~ 2 Comments

Liquid Web Hosting Reviewed – September 2012

If you’ve stumbled across this blog post from Google or another source and are not familiar with me, I’ll share some background with you so you can get an idea of my situation and hosting needs.  I am John Bell, and I operate a company called Insidmal Design LLC, I was  hosting 11 domains on Liquid Web dedicated servers, about 5 of which receive fairly heavy and steady traffic, and are heavily database driven sites.  I previously was hosting with GoDaddy but their lack of support and configuration options lead me to outgrow their hosting options, after much research I went with Liquid Web Hosting.

For the first while Liquid Web was an excellent company, I had superior support, fast service, everything was excellent, I reccomended them as a hosting company over and over again, as they were absolutely perfect for my needs, and their support was very helpful at discovering and resolving issues that were beyond my level of knowledge, but then everything went down hill.

Eventually I reached a place where one server was not enough to handle the capacity of my websites, and it was time to add a second server into the picture.  The idea was one server would handle the apache requests and another would handle the mysql requests.  The sales staff was very helpful at getting the perfect configuration for my needs, and that is when the positive experience ends.

I purchased the server and received an email from the tech support person who was to deploy it, after about 24 hours I received an email saying there was a problem with one of the packages and that the installation was corrupt and he would need more time to set it up, another day passed and I was told again that there were issues, again, and that it would have to be essentially started over from scratch.  It was setup and I was informed I would  need to change all my database connections to point to the other server, so I updated it will all the information provided, and then later was informed that because of the system technology I actually didn’t need to change it at all, as connections would be forwarded.  After this I proceeded to change them all back to how they were before on my 11 domains.  Later I received yet another email saying that due to some issue I actually would need to use different mysql login information for the domains, this being different then the initial data, so I yet again changed all the code in the scripts.  This seemed to do the trick on testing, so we decided to go live.  The server was live and not even 15 minutes after it was all set up, it crashed.  I waited 17 hours for a reply to my support ticket about the server being down, this means my sites were all inaccessible for an entire day, under their “Heroic Support” Sonar Monitoring blah blah blah bullshit.

I wasn’t aware as much about how servers worked as I am now (because of a lack of support) but when I finally called to complain, they finally decided to reply to my 20 hour old support ticket (30 minute response guaranteed?) .  They quickly discovered that apache had been grossly misconfigured so all the requests were going into Swap memory which was causing very slow response time and essentially the server overloading itself and crashing.  This issue was finally resolved, however then it was discovered that the initial tech who had to set up my server three times before he got it right, did not install the backup hard drive!  So I had to face even further downtime while they installed the hard disk.  Finally about 10 days later, things are running as they should.

 

We’ll fast forward about 3 months here – there is an issue on that same server and it crashes, I submit a support ticket because their 24×7 monitoring team somehow hasn’t noticed that my server has been down for half an hour, and I am informed there is an issue with Apache and the tech says he has to remove it and then rebuild it.  I said ok, do whatever you have to do to get my servers back online.  He does this, and from that moment on I was dominated with continuing server issues.  Before this point I was running with Keep Alive Off as the sites are very high traffic and keep alive on was filling all the server connections.  After this point, with Keep Alive off every page load resulted in tons of broken images, missing data, a complete failure in rendering the pages correctly.  I let them know, and they tell me that I just need to Turn Keep Alive on.  I inform them that this was a non-issue before the action, and that message goes literally completely ignored.  I turn Keep Alive on, and things run ok.  The next day the server crashes again, and again I have to submit a support ticket because their 24×7 Monitoring Team again goes completely unaware that my servers are completely inaccessible.  I am told this time that I do not have enough RAM to handle the connections I need to allow to not fill them with Keep Alive on.  I tell them, again, that before I had no problem with Keep Alive and now I can’t run with it off, that comment again goes ignored and I am suggested to upgrade the RAM.

So I upgrade the RAM, and everything seems ok, again until the following day.  Each day DAILY the server would crash at least once, it went at most two days without crashing.  I also had discovered that the firewall was blocking IP’s left and right for ‘Port Scanning’ even though they were IP’s of people who regularly used the sites as they were intened to be used, again a problem that did not exist before they “fixed it” the first time.  Each time I have to submit a support ticket because their monitoring team didn’t notice.  I spent an entire month of nights with little to no sleep because I was having to monitor my servers 24×7 because their monitoring team did not even once detect downtime.  I also am at this point receiving about 10-15 support tickets daily from various members complaining of latency on the server.

I had a support ticket open with LiquidWeb for 40 days for this issue, the cause of all these issues was NEVER determined, every time I would get a support tech, tell them what is going on, tell them how since this change they would reccomend the same simple solutions as far as configuring apache to reduce load, despite the load NEVER going up, it was all the same generic basic problem responses.  One support staff shift would end and the ticket would get passed on to the next, and the process would start again.  This went on for well over a month of time, still unable to determine the cause of the problems, I literally had no choice but to leave their company.

The “24×7 Sonar Monitoring” team not even once was able to prevent or even made aware of over 60 periods of server downtime.  Their Heroic Support would take as long as 6, 10, even 20 hours to reply.  Their Heroic Support was unable to resolve issues on the server at all, when it came to basic simple stuff they were very kind and helpful, however more complex issues like I had, it was a complete failure and breakdown of their support system.  I mentioned serveral times about how when a previous technician performed whatever they did to Apache, is when every one of the laundry list of issues began, and every single time I mentioned it, it was ignored and they would say the problem was something else, depending on the tech it would be a different cause, one tech went as far to tell me that it was from DDoS connections because an IP had 15 simultaneous connections to my server.  15 connections is not a DDoS attack by any stretch of the imagination.

It is simply unfathomable for me to see customer service run this way.  From the start when the intial tech had to set up the server 4 times before he could get it right and even then he failed to install a hard drive, to over a month of continued stability with absolutely no resolve and long response time waits.  I am unable to treat my customers with such nonchalance, as my support requests skyrocketed, I had the most difficult and stressful experience of my life trying to provide support to my members when I had no clue what the problem was, and even the company I was paying over $500 a month to do for me, was failing at it also.  I could provide no answers to my members about the cause of the problem or when it will be fixed, because of the failure of Liquid Web Support.

The very beginning of my experience with Liquid Web was one of the  best hosting experiences I had in my life, and it quickly turned in to the worst.  I really enjoyed the company and wanted to stay, but their complete lack of support and inability to monitor the servers with enough haste to justify the cost.  I was left literally no option but to leave the company because their lack of apparent interest in my purchases, was literally destroying my business.

Can LiquidWeb be a great hosting company? Absolutely.  Is it? Sure, so long as you don’t need to rely on their monitoring and support.

LiquidWeb Support was there when I didn’t need them, but when I needed them the most, they left me out to dry.  I know they get great reviews, and have great pricing compared to their managed hosting competitors, but please keep this in mind before making a purchase and supporting a company that is not willing to invest enough to properly train their employees beyond the most basic server operations.

 

</rant>

Continue Reading