The night before Christmas, in a sysadmins shoes

Twas the night before Christmas, when all through the house
Not a creature was stirring, not even a mouse.
The data was all safe, protected with care,
In hopes that year-end bonuses soon would be there.

The servers were nestled all safe in their racks
Protected from malware and vile forms of hacks.
The data all encrypted, with a nice complex key,
Christmas OrnamentThe software came, of course, with a great big huge fee.

Policy was written, compliance was done,
That had all been the exact opposite of fun.
Systems were patched, and locked one and all,
The team had done well, and they had stood tall.

Their audit was done, validation complete.
Their systems all ready, documentation all neat.
They had made it through ‘nother holiday season,
And were ready to cruise, it all stood to reason.

When from intrusion detection arose such a clatter,
The Admin sprang from his browser, watching Lohan get fatter.
Away to the consoles, he flew in a dash,
Checking faults and tripwire, computing their hash.

The light from the flat screens just made the place glow
The Admin then cursed, it could not be so.
Systems were crashing, the network was down,
It seemed almost like he was ready to drown.

College was easy, he got As and Bs,
These attacks were relentless,from overseas.
They seemed to be everywhere, attacks all at once,
They made him feel just like a big dunce.

His training took over, he set a grim face,
He quietly looked forward, to his happy place.
Cutting off systems, segmenting them away,
He was rolling right now, no time to play.

The pager just beeped and phones they did ring,
Event responders the noise then did bring.
Another server was hit, and suddenly dropped
The Denial of Service attack was not stopped.

The admin felt helpless, and just needed a lift,
Like something from Claus, please one early gift.
When, what his wondering eyes they did spy,
An extra program running, and then he knew why.
That’s when he knew, he just knew, oh of course

This had all started with just one Trojan Horse.
Prolly came to a user through his e-mail,
Policy and training, to no avail.

He marched through the servers, saw them as bots
Chuckling now, thinking “that’s all you gots?”
Malware from servers, each one did he cull,
Sending them one at a time right to dev/null.

Now he was rolling, and almost chuckling with glee,
He looked up with dismay, what did he see?
The retail server was hit, and crashing, amok,
He SSHed in, but could only say “rats”.

He saw very quickly that they now had root,
And hoped that he would not get the boot.
As he sat back in his chair, and reached for a Dew,
He knew the primary site was all through.

He knew then and there it would be a long night,
So he switched everything to the failover site.
Laying his cursor on that one little app,
He clicked only once and prod1 took a nap.

He sprang for his phone, to his team gave a twitter,
Failover was done, and he had not been a quitter.
The net was a mess, the farm had been bought,
But in the end, it had not been for naught.

The systems had done for what they’d been built,
They’d been prodded and poked, and finally said “tilt”.
He was glad it had not been a resume generating event.
His last thought for the night was for the attackers to get bent

His relief came too little, and seemed way too late,
He had pulled eves alone, and faced his sole fate.
He briefed the next crew, with all he did know
And went outside, to fresh fallen snow.

He sprang to his Honda, and cranked up the tunes,
Wishing for beach, and wind tossed sand dunes
He said as he drove down the road to the night,
Merry Christmas to all, and to all a good-night.

Credits go to securityweek.com, via Tibi.

qmail is vulnerable… switch to postfix [HOWTO]

It just got to my ears that qmail contains a known vulnerability that might lead to massive abuse of the mail system and your server. What happens now is that usually mail transfer agents (MTA’s) require that you write a complete e-mail address when sending an e-mail, something like “george@wirelessisfun.com” but qmail may accept mails addressed to simple usernames like “administrator” or “george“. When trying to send those e-mails, of course it will realize that the  address is incorect, and it will send a non delivery report back to the originating user.

Now, do the math, and count the reports sent back for a spam frenzy of a few thousand SPAM mails sent by a malicious user. That will put quite a strain on your server, and possibly on other servers.

I only had qmail on a Parallels Plesk powered server, so below you can read how to switch from qmail to postfix, the other Plesk supported MTA.

Use an ssh client to connect to your server, make sure you have root privileges, and run the following command:

/usr/local/psa/admin/bin/mailmng –features | grep -i smtp_server

if the output of that command is:

$features[‘SMTP_Server’] = “Postfix”;
$features[‘SMTP_Server_package’] = “postfix”;

you are safe, and good to go, but if the output of the above command is:

$features[‘SMTP_Server’] = “QMail”;
$features[‘SMTP_Server_package’] = “psa-qmail”;

you should change to postfix. It’s quite simple, you just need to run the following command:

/usr/local/psa/admin/sbin/autoinstaller –select-release-current –install-component postfix

That should be it. The Parallels Plesk knowledge base has an MTA change article aswell, here: http://kb.parallels.com/5801

[Howto] cPanel default page on direct IP access

Taking care of cPanel hosting servers brought up a problem. Every time someone would type in the browser one of the server’s IP’s directly (except the main IP), the first domain registered on that IP would show up. Have you ever tried this on your server? You can imagine how that could be a problem when the owner of a site for a retirement home for example, would put in his IP and get a nude photographer’s website on his screen. If a domain does not have a wildcard subdomain added to it, if you mistyped a subdomain you would get again the nude photographer’s page. And that’s not cool, is it?

I’ve been looking around, searching and when I didn’t get any results to my queries I went the other way around and started poking with the templates used to generate the apache configuration files. The solution showed up pretty fast, the tests on the test machine went well, and we started deployment of the tweak to the production servers.

Basically, what you need to do is find the file:

/var/cpanel/templates/apache2/main.default

, back it up somewhere safe, then edit the original.

Look for the lines containing:

[%– FOREACH nvh IN namevirtualhosts –%]
NameVirtualHost [% nvh %]

place a comment in the file just below “NameVirtualHost” using # signs, something like:

### default cPanel page tweak

This will be visible as a comment later in your httpd.conf file.

Now look below for the first  VirtualHost definition block it looks like this on my x86_64 test machine:

ServerName [% servername %]
DocumentRoot [% serverroot %]/htdocs
ServerAdmin [% serveradmin %]
[%– IF supported.mod_suphp %]
 
suPHP_UserGroup nobody nobody
[%– END %]
[%– IF supported.mod_userdir && userdirprotect_enabled && defaultvhost.userdirprotect != ‘-1’ %]
UserDir disable
[%– IF defaultvhost.userdirprotect != %]
UserDir enabled [% defaultvhost.userdirprotect %]
[%– END –%]
[%– END %]

and like this on my i386 test machine:

ServerName [% servername %]
DocumentRoot [% serverroot %]/htdocs
ServerAdmin [% serveradmin %]
[%– IF supported.mod_suphp %]
 
suPHP_UserGroup nobody nobody
[%– END %]
[%– IF supported.mod_userdir && userdirprotect_enabled && defaultvhost.userdirprotect != ‘-1’ %]
UserDir disable
[%– IF defaultvhost.userdirprotect != %]
UserDir enabled [% defaultvhost.userdirprotect %]
[%– END –%]
[%– END %]

Make sure you know what architecture your server is running on if you are going to use the code on this page, but I suggest you use the code already in the file.

Copy the VirtualHost definition block just under the comment you added before, and change the [% vh %] to [% nvh %] on the first line of the VirtualHost definition block just like the NameVirtualHost parameter.

You can add a new line of # symbols under the text you pasted in the file, to have it stand out in the config file, and you should be about done with this file.

Back up your httpd.conf file located in

/etc/httpd/conf/httpd.conf

then run the following command:

/scripts/rebuildhttpdconf ; /etc/init.d/httpd restart

This will rebuild your httpd.conf, and then restart your apache server, with your brand new default cPanel webpage on all the server’s IP’s.

Plesk Bandwidth reporting error

I managed to bump my head into this situation a few times, and I think some of you have too, or will in the future.

The DomainsTraffic table in the Plesk psa database gets a weird value, usually quite huge, and the traffic stats for a certain domain will skyrocket overnight from values of a couple hundred MB’s usually to several GB’s. Domains get suspended, customers get pissed and the “techies”  got work to do.

It’s actually easy to find the problem and fix it:

Log into your Plesk server as root, enter mysql and find the domain in question using the psa database:

[root@nl-ams-sp1 ~]# mysql -uadmin -p`cat /etc/psa/.psa.shadow`
mysql> use psa;
mysql> select * from domains where name = "wirelessisfun.com";

Find the day where the records got corrupted and are causing the erroneous report:

mysql> select dom_id,date,http_in,http_out from DomainsTraffic where dom_id =(select id from domains where name = "wirelessisfun.com");

This will output quite some data, but the culprit line will be really obvious:

+——–+————+———+————–+
| dom_id | date | http_in | http_out |
+——–+————+———+————–+
[snip]
| 316 | 2010-08-02 | 0 | 472399336 |
| 316 | 2010-08-03 | 0 | 491239251 |
| 316 | 2010-08-04 | 0 | 470982351 |
| 316 | 2010-08-05 | 0 | 470829065 |
| 316 | 2010-08-06 | 0 | 493939844 |
| 316 | 2010-08-07 | 0 | 454701317 |
| 316 | 2010-08-08 | 0 | 100221521161 |
| 316 | 2010-08-09 | 0 | 144318797 |
+——–+————+———+————–+

Now, that you know the corrupted line, just update the http_out value with something similar to the days before, I used the exact value as the previous day.
Make sure you replace “wirelessisfun.com” with the actual domain name, and the date value with the date in question:

mysql> update DomainsTraffic set http_out = "454701317" where dom_id =(select id from domains where name = "wirelessisfun.com") AND date = "2010-08-08";

The MySQL output should be something like:

Query OK, 1 row affected (0.03 sec)
Rows matched: 1 Changed: 1 Warnings: 0

Now that the record for the domain traffic has been fixed, you can wait for the statistics script to run during the night, and the clients traffic will be updated automatically, or you can simply re-run the statistics for that domain from your bash prompt like this:

[root@nl-ams-sp1 ~]# /usr/local/psa/admin/sbin/statistics --calculate-one --domain-name=wirelessisfun.com

Simple enough, right? Use the above tip at your own risk, it worked for me everytime.

[Tutorial] Cum resetez parola de root?

Parola de root de pe un linux, e destul de importantă, și dacă o uităm, e destul de complicat să trăim fără ea. Nu de alta, dar mai trebuie adăugați useri, făcute update’uri si alte ghidușii specifice.

Cel mai simplu mod de a reseta o parolă de root, și care de obicei funcționează 99% este cu un Live CD (Ubuntu și Knoppix, sunt două distribuții care oferă astfel de LiveCD’uri). Se bagă discul în unitatea optică, se boot-ează de pe disc, se montează partiția pe care sălășluiește folderul /etc de pe sitemul linux căruia vrei să îi resetezi parola și suntem aproape de adevăr. Căutăm aici fișierul shadow

/etc/shadow

unde vom găsi mai multe linii de text. Linia care ne interesează pe noi ar trebui să fie prima, și arată ceva de genul:

root:$1$MEoQ7QU.S/5yQx0zDFANFZflNL3vN/:13291:0:99999:5:::

Bucata de text cu roșu este parola userului root, criptată. Important de ținut minte este că e incadrată de cate două puncte in ambele părți. E destul să stergi partea cu roșu, salvezi fișierul, rezultatul o să fie similar cu:

root::13291:0:99999:5:::

Acum poți să restartezi, scoți discul din unitatea optică înainte de boot, pornește sistemul, et voila, userul root nu mai are parola setată.

La promptul de login introduci “root”, iar când sistemul cere parola, un simplu enter ajunge. Simplu, nu?

După ce te-ai logat, tastează comanda passwd ca să ai totuși o parola de root setată.

Evident există si alte metode, cum ar fi “single user boot” dar despre asta într-un post viitor.

Sysadmin Appreciation Day

Azi, ultima zi din iulie, e Sysadmin Appreciation Day. http://www.sysadminday.com Si pana la urma, e firesc nu? Avem ziua copiilor, ziua mamei, ziua femeii, ziua piratilor, ziua minerilor, deci de ce sa nu existe si Sysadmin day?!

Sa beti o bere pentru sysadminul vostru, si daca il scoateti la plimbare azi, sa ii cumparati o inghetata 😉

Eu o sa sarbatoresc diseara, in Le General la AGWA/Coca Leaf Party.

picture-42

Geek joke

A c-string walks into a bar. It says to the bartender,
“Can I have a beer and some fadsh543%$%lkjfdslfj/bin/sh”.
The bartender turns and says:
Welcome root, You’ve got mail

Via Bibi, care a auzit-o de la Eti. Bibi, ai inteles-o?

Later Edit:

Si cum tocmai a aparut o simpatica vulnerabilitate de dd-wrt, si eu am dd-wrt acasa, am reusit sa il sparg…

Chestia se potriveste de minune cu gluma de mai sus din pacate…

marius-perijocs-macbook:ddwrt mariusperijoc$ nc 192.168.1.1 5555
id
uid=0(root) gid=0(root)
whoami
root
ps ax
BusyBox v1.11.1 (2008-07-27 16:09:08 CEST) multi-call binary

Usage: ps

ps
PID USER       VSZ STAT COMMAND
1 root      1172 S    /sbin/init noinitrd
2 root         0 SW   [keventd]
3 root         0 RWN  [ksoftirqd_CPU0]
4 root         0 SW   [kswapd]
5 root         0 SW   [bdflush]
6 root         0 SW   [kupdated]
10 root         0 SW   [mtdblockd]
14 root      1184 S    /sbin/watchdog
85 root      1164 S    resetbutton
129 root      1844 S    httpd -p 80
131 root       812 S    dnsmasq –conf-file=/tmp/dnsmasq.conf
230 root       700 S    cron
236 root      1532 S    dropbear -b /tmp/loginprompt -r /tmp/root/.ssh/ssh_ho
260 root      1164 S    process_monitor
380 root      1168 S    upnp -D -L br0 -W vlan1 -I 60 -A 180
482 root      1164 S    /sbin/wland
487 root      1168 S    ttraff
540 root         0 Z    [schedulerb.star]
544 root         0 Z    [wdswatchdog.sta]
759 root      1108 S    sh -c /tmp/exec.tmp>/tmp/shellout.asp
760 root      1104 S    /bin/sh /tmp/exec.tmp
762 root      1104 S    /bin/sh
768 root      1108 R    ps
^C
marius-perijocs-macbook:ddwrt mariusperijoc$

PS: L-am patchuit, deci nu va stresati, dar daca aveti prieteni cu dd-wrt’uri, anuntul oficial e undeva pe aici si fixul e aici.

Sysadmin manual

[Multumesc Trollii]

#include<hello.h>

SYSADMIN(1)                                           SYSADMIN(1) 

NAME 
       sysadmin  – responsible for everything imaginable that may 
       or may not have to do with the system you’re using.   Con- 
       traction of “system” and “administrator” 

SYNOPSIS 
       sysadmin [-ab] [-cd] [-ef] etc…… 

DESCRIPTION 
       sysadmin takes care of everything, is generally harangued, 
       must be supplied with coffee, chocolate,  and  alcohol  in 
       order  to  function  properly, cannot be exposed to direct 
       sunlight, and must not be allowed to have a life. 

       sysadmin is not intended  as  a  user  interface  routine; 
       other  programs provide user-friendly front ends; sysadmin 
       is used by everyone who can track him [her] down. 

       With no flags, sysadmin reads its standard input up to  an 
       EOF,  or  a  line which sysadmin wishes to parse, and then 
       proceeds to ignore it entirely and read news all day. When 
       invoked  with the -w option, sysadmin reads standard input 
       and responds according to terms of job description. 

OPTIONS 
       -bofh  Go into Bastard  Operator  From  Hell  mode.   This 
              option  causes  sysadmin to use tools stored in the 
              /usr/lib/bofh directory to parse the standard input 
              and route user tasks appropriately. 

       -cd    causes   sysadmin   to   become  caffeine-deprived, 
              resulting in system slowdowns. 

       -b     causes the sysadmin to function normally while aug- 
              menting  the  standard  input with beer(5).  Can be 
              used with the -t option  as  well,  depending  upon 
              which version of sysadmin you are running. 

       -t     causes  the  sysadmin  to  smoke tobacco, which can 
              result in significant performance improvement, pro- 
              vided you are running the correct version of sysad- 
              min. 

       -Cfile Specify an  alternate  configuration  file  (sysad- 
              min.cf is the standard). 

       -dX    set debuggin value to X. 

       -fFullname 
              Set the full name of the sysadmin

       -Bf    Create the sysadmin.cf configuration freeze file. 

       -lname Sets  the  name  of  the  “luser”  person (that is, 
              originator of a given request). -l can only be used 
              by “trusted” users (who are listed in sysadmin.cf). 

NOTES 
       The -t option should not be used with a version of  sysad- 
       min which is not capable of parsing tobacco input.  Though 
       the functionality of this command may seem similar to  the 
       -b  option,  it  should  not  be confused with that or the 
       related -c option.

And it’s a paaaarteeeee!!!!!

No, don’t think about clubs and pubs, don’t get you mind mixing drinks just yet, it’s the Ubuntu 9.04 Launch Party in Cluj-Napoca. 

The event will be hosted by the Romanian Ubuntu community (http://www.ubuntu.ro), and thanks to the Technical University of Cluj-Napoca the party will take place in Apmphitheatre no. 40, on Baritiu Street at 10AM. 

There is no entrance fee, and the participants will be provided with original Ubuntu install CD’s/DVD’s and Ubuntu stickers.

More details: here.