Linux records

From SusoSight

Records that I've set for myself (and maybe more) while using Linux.

Load

Highest 1-minute load average ever seen: 15358.87 (Really!) The load meter in Linux actually resets itself after 1024 back to 0. But I had a script that was logging the load on the machine every 5 minutes and it reset itself at least 14 times. The final load before rebooting was 1022.87, so 1024 * 14 + 1022.87 = 15358.87. I think this might be the world record or something. Needless to say, this machine had some disk io problems.

The second highest I've seen is 358 on one of Dreamhost's servers. Sheesh.

Uptime

I've had the Suso shared server up for 422 consecutive days. Later I had a Linux based router (for Suso) that was up for 701 days. Then I had to move it to our new network.

At my day job, we have a machine called sandbox that has been up since Tue Feb 6 08:47:22 EST 2007 (as of this writing, 1623 days). I left the job in 2012 and the host was still running. I know it was up for at least 2470 days. After that I no longer received the e-mails from the cronjob I setup before leaving. Just goes to show you that people don't follow sysadmin termination proceedures. :-(

UPDATE: So it turns out that they just unplugged the network cable on that box and left it run and finally shut it down at the 8 year mark with 2929 days 1 hr 14 min of uptime.

Files in directory

First I saw a directory with 933,008 files. Then I found another one with over 1.3 million files. This was at Cook. Later I ended up seeing over 13 million files on that partition in the same area. I think this is a filesystem record for me too. The only other place I've seen so many inodes being used is on a Usenet server. Later we ended up seeing a directory on the same server with 2,273,360 files. The directory table was 45MB and there are now almost 15 million files on the partition.

Disk space

This isn't really a record more than its just interesting. At Cook we had an ext3 /tmp filesystem that was misbehaving and reporting 101% full and that it had -64ZB used (yes, zottabytes, df actually can report this capacity). We were able to reset the counter by putting a 40 megabyte file in the partition. Not sure what negative disk space is though.

# df /tmp2
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/LocalVG/tmpbroken  5039616 -73786976294838173696   4815760 101% /tmp2
# df -h /tmp2
Filesystem            Size  Used Avail Use% Mounted on
/dev/LocalVG/tmpbroken  4.9G  -64Z  4.6G 101% /tmp2
# dd if=/dev/zero of=40mbfile bs=1M count=40
40+0 records in
40+0 records out
# df /tmp2
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/LocalVG/tmpbroken  5039616      8860   4774756   1% /tmp2

Log file size

I've seen web server logs grow to several GB easily in a month. The largest logs I've ever seen are when a PHP website is upgraded to a newer version of PHP and there is a lot of deprecated code. This happened to me in August of 2010 when I upgraded arvo to dorabella and there were a few sites using lots of deprecated function calls like ereg or whatever.

When I went to go help the customer out by trimming their error log, I removed the old and now 6.4GB error log and then restarted Apache. The new log grew to 2.5GB in just 5 minutes!!!! And this was on a Xen virtual machine too (for any doubters of the disk io performance of Xen, the load only was around 1.2 and other machines on that host weren't affected much). I ended up turning off error logging for that site temporarily.

Failed logins

"There were 320156 failed login attempts since the last successful login." "There were 118223 failed login attempts since the last successful login."

This isn't really something that would constitute a record because anyone could do it just by neglecting a host for a while. How this happened was that I setup a few VPSes in November 2014 for later setup and lockdown. I was greeted by these messages when I came back. Doing some stats from the logs the majority of these ssh connection attempts were to the root user and most were from China. Not a big surprise. One host tried over 10000 times, which tells me that they are trying everything on the 10k common password list.