Downtime

A MUST READ FORUM Place where new distillers can read many of the important documents that help introduce them to distilling. This is a read-only forum.

Moderator: Site Moderator

Downtime

Postby Uncle Jesse » Wed Nov 29, 2017 9:16 pm

Sorry folks, my fault.

HD has been running out of RAM recently so I arranged for 2 more gigs from our service provider. It costs more but they gave me a deal on it since I'm a long time customer.

Got the RAM and rebooted not realizing the server was not in shape to do it!

Had to spend hours doing what is known in FreeBSD land as "making world" which means rebuilding the entire system from source. Couldn't bring HD up while doing this.

Still finishing up with this upgrade. Went from FreeBSD 11.0 to 11.1 which includes a lot of good enhancements and security fixes but... sorry for the downtime!

Again, not quite finished.
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby cob » Wed Nov 29, 2017 9:22 pm

Uncle Jesse; I'm going to nominate you for sainthood, ya I know rule #5,

but your getting nominated regardless
be water my friend
cob
Master Distiller
 
Posts: 1613
Joined: Sun Jul 26, 2009 4:38 pm
Location: little puffs of dust where my feet used to be

Re: Downtime

Postby jon1163 » Wed Nov 29, 2017 10:03 pm

Thanks brother for all the work you put into the site so that we can all meet up and bull#&*-
User avatar
jon1163
Site Donor
Site Donor
 
Posts: 173
Joined: Wed Jul 05, 2017 8:57 pm

Re: Downtime

Postby Uncle Jesse » Wed Nov 29, 2017 11:27 pm

For those of you who speak tech geek, we were averaging 44-50% swap use. Now we aren't swapping. I've done some intensive MySQL operations and we're currently sitting at 683MB free RAM.

Much better.
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby Still Life » Thu Nov 30, 2017 1:03 am

Thought the reading finally sent me blind when the site went dark.
Glad the site is back up and healthier than ever.
You're the best, Uncle Jesse.
User avatar
Still Life
Site Donor
Site Donor
 
Posts: 1538
Joined: Fri Apr 22, 2016 4:24 pm
Location: Great State Of Missouri

Re: Downtime

Postby Uncle Jesse » Thu Nov 30, 2017 8:47 am

Sorry folks. Site went down again sometime last night - actually powered off! I'm not sure why. I've fixed it and will contact the ISP to see if they had a power out... but this one was strange. I'll be keeping an eye on it today.
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby nuntius01 » Thu Nov 30, 2017 8:47 am

great work :clap:
I'm just the bank and the mule

post your still pics here
viewtopic.php?f=16&t=66917
User avatar
nuntius01
Swill Maker
 
Posts: 431
Joined: Thu Jun 08, 2017 4:51 pm
Location: upper midwest

Re: Downtime

Postby Uncle Jesse » Thu Nov 30, 2017 9:24 pm

Still haven't swapped. Looking good!
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby Worm_Drippinz » Fri Dec 01, 2017 3:10 am

Thank you UJ. ...and to the rest of the lot fer puttin' up with me.
Worm_Drippinz
Site Donor
Site Donor
 
Posts: 319
Joined: Thu Jan 28, 2016 6:35 am

Re: Downtime

Postby Canuckwoods » Sat Dec 02, 2017 4:13 am

Thanks, Uncle Jesse
I don't think most of us realize how much time and effort goes into sites like this.
User avatar
Canuckwoods
Novice
 
Posts: 12
Joined: Mon Oct 23, 2017 2:18 am

Re: Downtime

Postby Uncle Jesse » Sat Dec 02, 2017 10:17 am

Second time in a few days the system was powered off. I'm in touch with the ISP and looking into causes.
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby RedwoodHillBilly » Sat Dec 02, 2017 11:22 am

I've found that "shutdown -h now" generally isn't a good idea in the daily cron job :P
John Barleycorn must die.
"and little Sir John in the nut brown bowl proved the strongest man at last.
The huntsman he can't hunt the fox, nor so loudly to blow his horn
and the tinker he can't mend kettle nor pots without a little barleycorn."
User avatar
RedwoodHillBilly
Site Donor
Site Donor
 
Posts: 756
Joined: Tue Sep 01, 2015 7:30 pm

Re: Downtime

Postby Uncle Jesse » Sun Dec 03, 2017 7:53 pm

Hopefully found the issue. Rebooted a bit ago which some of you may have noticed.

Keeping a close eye on the server.
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby Coyotey » Fri Dec 08, 2017 12:08 pm

"Bridge to Engineering"...."Scotty, status on the new reactor" ...."You'll have warp capability in a jiffy, running diagnostics now...stand by"

Thanks for your efforts and a great platform Uncle Jesse! Make sure you have enough time for stillin too though!

Skål!
By three methods we may learn wisdom: First, by reflection, which is the noblest; Second, by imitation, which is the easiest; and Third, by experience, which is the bitterest. - Confucius
User avatar
Coyotey
Novice
 
Posts: 53
Joined: Sun Nov 19, 2017 3:39 am
Location: Over the hills and far away.

Re: Downtime

Postby Chixter » Fri Dec 08, 2017 2:19 pm

Thanks for all your work and dedication.
When I was a boy, I prayed for a bicycle. I soon learned G~d doesn't work like this so I stole a bike and prayed for forgiveness.
User avatar
Chixter
Site Donor
Site Donor
 
Posts: 119
Joined: Sat Sep 09, 2017 1:39 pm

Re: Downtime

Postby fizzix » Fri Dec 08, 2017 2:23 pm

Like the others, I appreciate all you do Uncle Jesse, and thanks for allowing my sorry butt to join.
I thought maybe someone kicked the extension cord out of the socket this morning when the site was down.
I know you're on it though.
And when you lose control, you'll reap the harvest you have sown. -Dogs, Pink Floyd
User avatar
fizzix
Novice
 
Posts: 42
Joined: Tue Dec 05, 2017 4:08 pm

Re: Downtime

Postby Uncle Jesse » Fri Dec 08, 2017 11:14 pm

I've got the ISP scratching their heads. Meanwhile I have checked, double-checked, triple-checked everything on the server and I can't find any reason for it to power down every day. I don't get it. I'll keep trying until it is fixed for good.
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby fizzix » Fri Dec 08, 2017 11:28 pm

If it's any help, it's been going down around 4am thru 6am Central U.S. time.
Out at 6-ish CST this morning (Dec. 9th).
Anything power related at that time or alien abduction schedule?
And when you lose control, you'll reap the harvest you have sown. -Dogs, Pink Floyd
User avatar
fizzix
Novice
 
Posts: 42
Joined: Tue Dec 05, 2017 4:08 pm

Re: Downtime

Postby StillerBoy » Sat Dec 09, 2017 7:46 am

StillerBoy
Site Donor
Site Donor
 
Posts: 467
Joined: Thu Dec 01, 2016 6:27 pm
Location: Ontario

Re: Downtime

Postby Kareltje » Sat Dec 09, 2017 9:06 am

Maybe if we all donate our fores and heads to use as fuel? :twisted:
It feels desolated, if I can not reach HD.
But all my respect for your attempts, Uncle Jesse! Thank you! :thumbup: :clap:
User avatar
Kareltje
Distiller
 
Posts: 1202
Joined: Tue Feb 09, 2016 4:29 pm

Re: Downtime

Postby Uncle Jesse » Sun Dec 10, 2017 10:07 am

ISP dropped us back to 4 gigs of RAM and no power off weirdness last night.

Huh?
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby ExpatLad » Sun Dec 10, 2017 10:12 am

Uncle Jesse wrote:ISP dropped us back to 4 gigs of RAM and no power off weirdness last night.

Huh?


Sounds like the VM host is having issues, over allocated perhaps.

Side note, Thank you for all you do to keep the site online!
_____________________
EXPAT

"I do not aim with my hand; he who aims with his hand has forgotten the face of his father.I aim with my eye." - Roland of Gilead
_____________________
My current rig
User avatar
ExpatLad
Swill Maker
 
Posts: 333
Joined: Fri Feb 24, 2012 3:58 pm

Re: Downtime

Postby Uncle Jesse » Sun Dec 10, 2017 11:11 am

ExpatLad wrote:
Uncle Jesse wrote:ISP dropped us back to 4 gigs of RAM and no power off weirdness last night.

Huh?


Sounds like the VM host is having issues, over allocated perhaps.

Side note, Thank you for all you do to keep the site online!


Yet another reason FreeBSD jails are better than Xen or VMWare.
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby Uncle Jesse » Mon Dec 11, 2017 12:19 pm

Still running. I really believe the bump to 6 gigs of RAM from 4 gigs somehow caused those crashes.

I did a ton of work on the unix side trying to tighten things up and couldn't find anything which would cause the server to power down.

Unix servers don't normally power down. They core dump or panic or reboot.

Still keeping an eye on it.
If only the best birds sang, the woods would be silent.
User avatar
Uncle Jesse
Site Admin
 
Posts: 3213
Joined: Wed Apr 21, 2004 3:00 pm

Re: Downtime

Postby Single Malt Yinzer » Mon Dec 11, 2017 12:44 pm

Put SNMP on it & setup a monitoring server locally. Solarwinds, tons of other that you can run free for a small number of nodes. I doubt it will give you the root cause but it might point you in the right direction. Plus : STATS AND GRAPHS! Who can resist them?!?!?!

And thanks as always!
Keep up with everything faster: search.php?search_id=active_topics
Look for it in the HD Wiki: http://homedistiller.org/wiki
User avatar
Single Malt Yinzer
Site Donor
Site Donor
 
Posts: 247
Joined: Fri Sep 23, 2016 3:20 pm


Return to New distiller reading Lounge



Who is online

Users browsing this forum: No registered users and 1 guest