Back to semi-normal

I've got the main blog back up, so updates and info are over there. Sorry for the temporary detour.

Update

For the third time since last night, I'm starting this post to say that it looks like things are getting better (the last two times I started this post, something went wrong, almost immiedietly).

The search site should be up and running, I hope to have Groups up by the end of the day.

For those interested, the problem seems to have been related to Linux (specifically Fedora Core 2) and not disconnecting network/IO resources related to Java processes. While our code ran fine under Gentoo, something is different with the way Java process pipe to the console on Fedora. Tomcat would run for 10 to 15 minutes, then lock up, taking all networking stuff with it, to the point where I was unable to even ping the box. The only recourse was to reboot the box. As this seemed to be related to Java only, I went through our code and removed any references to System.out.println() or printStackTrace(), and things seemed to improve, although I'm still not 100% certain this has fixed things. I've seen the same strange behavior from other applications on the box, like MySQL, but not as frequently or as pernicious as what has happening with Tomcat. I've poured over the Fedora forums and haven't found anything related, so if anyone knows of a general fix or workaround for this type of problem, please let me know.

So what should have taken an hour or so went on for a couple of days. Ugh. The good news is that things should get progressievly better from here on out. The index we are running is a bit out of date, I'm still working on getting data off the old server, and will plug it in later. In the mean time, new entries will start to show up soon. Thanks!

Oh, and I'll try to put the regular Blogdigger Blog back up by the end of the day, so updates should (hopefully) start flowing through there.

Hello!

Hi, everybody. It's Greg, from Blogdigger. We're having traumatic server issues at the moment; two boxes are down, which leaves us up a proverbial creek. We're really working hard to get things back up, but it's going to take a bit of time. I'm starting this blog as a temporary means of keeping in touch, and try to give updates as to how things are going.

All I can say, is that I'm really sorry and that we're doing our best.

A quick rehash of the problems: our rack at Level 3 lost power sometime on Monday night. The main server refused to reboot, so we had to pull it and are currently doing data recovery on it (if anyone is an expert in getting data off a RAID 5 Gentoo box using JFS, email me, please). Our backup box went in last night, but it's now having configuration problems with Tomcat and networking. I'm tempted to install Resin and see how that works, but I think the problems lie at the networking level (this box is running Fedora Core 2). The good news is, we had a fairly complete backup of our data, and I'm hoping we'll be able to pull the little bit that's missing off the old box.

So that's the basic story. I'm working on getting dummy feeds set up for the media and search pages, so at the very least, your subscriptions won't return errors. To make matters somewhat worse, I'm scheduled to be out of town for Rosh Hashannah, starting tonight through Saturday (could this have happened at a worse time? It's possible, but I don't think so).

All I can say is that we will be back (down for one day, and I'm already getting emails from people interested in buying the code, which I guess is a good sign). Please bear with us, and thanks for your patience.

Words of consolation and encouragement can be left in the comments, and are much appreciated ;)