Bugs, a failed walk, and photos


Since my rant about the state of bugs in Ubuntu, I’ve been going over my bugs, one by one, poking them and re-triaging them. I did this to one bug for gweled, 110268. One of the people who had experienced this one tagged it “bitesize” (easy to fix) and “packaging”. I reset it into “confirmed” state, assigned it to the MOTUs. (I tend to set them to “incomplete” whilst poking them to make sure they’re still an issue – that way if there is no response and I randomly disappear, launchpad will automatically mark them for expiration). As a result of my poking it, Effie Jayx was asked to it, and the bug is no-more in Hardy. Nice 😀

Thinking about it, I probably should have chucked that particular bug upstream earlier. The reporters had done all the work tracking down the bug… there we go, live and learn. I’m slowly poking all the bugs that haven’t been fixed, that I’ve triaged, making sure they all go to the right places…

A failed walk

‘How can a walk fail?’ I hear you ask. Quite simple, I was attempting to find a few geocaches, in a long walk around my local area. Geocaches are basically small hidden caches, which you find with a hand-held GPS.

So, I parked up with my recently repaired car (long story), and set off. However, I managed to go the wrong way (yes, even with a gps with topographical maps of the area), so I didn’t get anywhere close to finding any geocaches. I’ll probably try a completly different route next time…

On the way though, I did manage to take a couple of nice photos, so here are the best of them. As usual, all photos are under the Creative Commons share-alike attribution licence, click on the photos to see a bigger version on flickr.

This one is looking through a wire fence, fairly close to where my next one was taken. Sorry the background is blurry, should have put it into macro mode. The hill was quite steep here and my footing was tenuous so I wanted to move on quickly…

Hillside fence

Went past this wall whilst walking back to my car in the woods. Its falling to bits, but not being kept up, as it’s been replaced by wire fences. There are quite a lot of crumbling dry-stone walls in this particular wood.

Crumbling Wall


As usual, my water photography continues 😉
I took these two at a local park, which is very close to where I work. I go there to eat lunch sometimes, and there’s a artificial pond created by this weir/dam.

Dark Weir Weir

6 Random Photos

I decided it was about time I made another camera card-dump. I also picked a couple of pictures out of my archives and uploaded them. If you can guess the location of any of these, comment here or on the flickr photo page! (Virtual cookies for anyone whom gets it right!)
But no cheating, if you know me in person, and know where these are, don’t say anything 🙂

All photos are under the Creative Commons Attribution-Sharealike licence. Click on the photos to go to the flickr page. Enjoy!

Steam Railway

Went to a steam railway somewhere in Wales for the day (April 2007), with my cousins. There was a marquee with a wooden floor there.. on which somebody had at some point stuck an Ubuntu sticker! They certainly get around 🙂

This is the bottom of a hose used to refill the steam engines with water. It dribbles a bit all the time, whilst the railway is active. Was a really sunny day, light enough that the shutter time was quick: 0.004 sec (1/250th of a second).

Steam Engine Hose dribble

I then took a photo (second one) of the puddle where this water was going. Wind was blowing it about a bit, took a couple of shots to get a really nice one. Again, really sunny day helped, shutter time was 0.004 sec again

Steam Engine Puddle

Welsh Ponds

My grandparents came and visited me last year. (I have photos of them, but I’m not putting them on the ‘net.) We went and visited a fairly famous place in Wales. If you’re welsh, I would be VERY surprised if you haven’t visited here at least once. Not sure you’ll work out where this is from this photo though. Didn’t spend long there, just enough time to get this photo and have a bite to eat in the cafe.

Welsh Ponds

Christmas 2007

I’ve not many photos of Christmas just gone, because as mentioned, I managed to leave my camera behind when I left Wales for my family home. Anyway, I have a certain urge that fills me every Christmas. never acted on it, never will, but I want to find and use a pair of wire-cutters every time I see garden decorations. Don’t know why, but I do. This particular example has a very visible wire, and I saw it quite regularly over Christmas.

Christmas Sleigh

Night Shots

As you may have noticed, I quite like taking photos at night and in the rain. Well, both of these fill the “night” slot, and one of them also fills the “rain” slot. (It was more like torrential downpour, but there we go.. shame you can’t see it on the photo.)

Cold night, dark, walking home. The water on the road is runoff from the (large) hill about half a mile down the road. Sorry about the blurryness of this photo (the larger version is even worse.) Shutter time on this one was 0.3 sec (3/10ths of a second)

Empty Road

And last of all, this one I took tonight (3rd of Feb). During the torrential rain that stopped me walking to Church this evening. My camera went a little odd on this one, and didn’t take it in its full high-resolution, so there’s not a bigger version, I’m afraid. Shutter time was 0.125 sec (1/8th of a second). I overrode the camera to shorten the shutter time, else it would have been too long, and I’d have ended up with a blurred mess.

Dark Street in the Rain
Hope you liked all those. All comments & constructive critique would be appreciated 🙂

Heavy Rain

A couple of days ago, it was raining really heavily. So, out came the trusty digital camera, and here are the best three shots of the night. (I took about 25. I’m glad that digital film is free!)

Car splash
Falling rain, Running Water
Headlights running water

The really interesting thing about taking all the photos was holding the camera still whilst I was getting pelted with rain and wind. I had to tweak the shutter timings to try and get a good photo without the flash: the flash destroyed the image, the falling rain all reflected the light into the lense, and it ruined the photo. I’ve still got the images, if you want to see…

I’ll post about my car in the not-too distant future (and why I was walking through the rain instead of driving…)

For the person found this blog through google with the Ubuntu terminal command:

cd / && sudo rm -rf

as your search term, DO NOT RUN IT! It will destroy your computer! In fact, don’t run ANY terminal command with “rm” in unless you know exactly what its doing!

Notes / MySQL Transactions

Why should you not use your notepad as a coaster? This is why:

The notes did say this, before my lovely hot black coffee destroyed them:

“begin” == Start transaction
“commit” == End transaction, saving all changes
“rollback” == Undo all changes that you’ve made in the transaction

What are they? In mySQL 5, with a table storage type of “innodb” you can use transactions. This means, that you can make a set of changes together, like recording the removal of a credit, and the adding of an advert to a website, at one instance, even if it’s multiple mysql transactions within one connection.

You do that, by sending the command (literally “BEGIN;” or “begin;”) to start a transaction, run all the queries you need to run, and then run the commit if you want the all the changes to take, or rollback if you don’t. If any one of the sql queries fails after you’ve made a begin, and it shouldn’t have, you can rollback *all* the changes you’ve made to the database. Obviously, this needs one mySQL connection, but if you’re using more than one each run of the program you’re using, I’d be very surprised.

Xen Gutsy Links

A post full of links and unsorted info about using Xen on Ubuntu Gutsy Gibbon, so that I don’t forget. (This lot has taken ~2 days to collect…) May make a full howto sometime… probably not though, as my boss knows this stuff better than I. Xen is pretty cool though…

https://help.ubuntu.com/community/Xen (useless for gutsy…)

The Xen console doesn’t work (xm console <domU>), network works once, and then never again… Going to paste this here as its in a pastebin, not sure when it will expire… (Have tweaked slightly, snipped a couple of things out, added couple of things in)

>> OK, I have made a little progress on this.  I mounted my VM image on a
>> loopback and chroot'd to it.  I opened /etc/init.d/hwclock.sh
>> and /etc/init.d/hwclockfirst.sh and added the following to the top of each
>> file:
>>   exit 0

ALSO SEE: http://lists.cvsrepository.org/xen-tools/Jul07/0332.html

sudo xm create -c </path/to/config/file>


# Ever-increasing network device names?

# Just disable the correspondent udev rules by deleting/renaming your
/etc/udev/rules.d/<NUMBER>persistant-net.rules. (number was 70 for us...)

guest# mv /etc/udev/rules.d/{,.}z25_persistent-net.rules

# And then make sure your vif (in your instance config file) contains your mac
# address like:
vif = ['ip=,mac=<random valid mac addy>']

Just be sure to stop the instance (domU), and then restart (re-create) it again when needs be…

Also, when in the Xen console, try hitting enter before using the escape sequence (CTL +]) if it’s not working. Sometimes it enters into the console after the “login:” prompt has been printed, and doesn’t seem to pick up that its just had a new connection…
The ALSO SEE link: http://lists.cvsrepository.org/xen-tools/Jul07/0332.html

Using Serial console & xen console (serial console conflicts with xen console):

/boot/grub/menu.lst stuff, to enable us to use a serial console as well as xen:

serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=2 serial console

Goes in the top of your menu.lst, outside the automagic sections…

## additional options to use with the default boot option, but not with the
## alternatives (These options are not used with the Xen Kernel!)
# defoptions=quiet splash console=ttyS0,115200n8

For the default kernel, (NOT the xen kernel).
Goes inside the AUTOMAGIC section of your menu.lst, adding to the options already there.

## Xen Linux kernel options to use with the default Xen boot option
# xenkopt=console=tty0 console=ttyS0,115200n8 xencons=ttyS9

Note the xencons=ttyS9. Use a serial port (ttyS<No.>) that doesn’t already exist.

Your Xen kernel definition should look something like this, after you’ve run “update-grub”:

title           Xen 3.1 / Ubuntu 7.10, kernel 2.6.22-14-xen
root            (hd0,0)
kernel          /xen-3.1.gz
module          /vmlinuz-2.6.22-14-xen root=/dev/mapper/volume-root ro console=tty0 console=ttyS0,115200n8 xencons=ttyS9
module          /initrd.img-2.6.22-14-xen

The guys on irc ##xen @ freenode are really helpful…

Attacked by the Storm Botnet

Once upon a time there was no internet crime. Then humans came along…

Recently, we came under attack from the Storm / Nuwar Botnet. The post I made about it on the third of October: We had mis-identified it as a referral spam attempt. Close, but no cigar.

Now, I’ve always tried to keep my name & employer from becoming too widely spread on the interweb, although there is a couple of really, really easy ways you can find it, just from this website. (One of them being, ask me 😉 )

It appears, that as a result of the two posts I’ve made about the Storm Worm, someone decided to DDOS not this blog, but my employer’s un-related servers, attacking one of our customers’ managed servers, and then our webmail server. (This blog is hosted from servers in  the same rack as those servers.)

At its peak, the attack was drawing 8Mbps of data transfer. (About 1MB per second.)

Encryptec ddos attack

Graph is read from right to left. <<<<<<< Time Flows that way. <<<<<<<

You can see at 0930, when I got in work and started combating the attack. We only really stopped it the morning this graph just ends on…

Only problem, was that they were flooding our server with requests, literally using every available incoming connection on the server all the time.

For non-techies, a web site is hosted by a computer somewhere on the interweb,
that never gets turned off, connected to a really thick pipe to the internet.
Its configured to accept a certain number of new people visiting its website(s)
at once.

We’ve now completely mitigated this attack (to the point, where at most now its drawing 50kbps 1). Technically, we can mitigate (and sustain) a much more serious attack. This was basically a “Get Lost, and STOP POSTING ABOUT US” poke.

An expensive poke. A sustained 8Mbps transfer rate is expensive in bandwidth!

So far (*wanders off to check*) we’ve identified 23,265 ip addresses which have tried to attack us. That’s a lot of infected computers, but it could have been worse.

It appears the attack has been petering out, we are identifying one new bad ip (infected computer) once every 30-60 seconds. At its peak, we were picking up at least one new ip every second.

If we have another look at that graph of the attack:Encryptec ddos attack

The attack started at 1AM GMT, and ramped up to full power in about 20 minutes. That means that it takes the Nuwar / Storm botnet about 20 minutes for a command to filter down into its bots.

At the beginning of the attack, the pattern we were seeing was a bad request from one ip, then 3 different bad requests, then back to the first IP. Sometime during the attack, I think about 1400 or 1500 (2 – 3pm) they switched to hitting us repeatedly from one ip address, showing that someone was probably monitoring at least a small part of this attack, and had noticed that we’d started to block the attacks.
Now, this happened quite a while ago.

Encryptec DDOS attack - 4 week

So why haven’t I posted about it yet? Why has it taken me 2 weeks to blog about this?

Because, its only now that we feel that we are able to safely weather another attack, should the Zhelatin Gang decide to start poking us again. If they didn’t like me posting what I have, they’re not going to like me posting this.

A message to them: I do not like bullies. Go pick on someone your own size for a change.
Thanks to stopddos.org, for analysing the logs and identifying Storm as our attackers.

UPDATE 20/10/07: A little while ago I sent a part of our logs for geographical analysis to one of the nice guys at castlecops.com.
Here is the graph that resulted from that. This is the top 5 attackers from country, in a pie chart. As you can see, Germany (Country Code DE) was the biggest, closely followed by the US. If you want to see other attack graphs, go here: http://www.spamtrackers.eu/wiki/index.php?title=Botnet_hosting (ours is listed there as BB, moved around to match up with the others, and slightly tweaked.)

  1. 5KB per second

6th October 07 – Camera Dump #1

Ok, I’ve finally got round to dumping all my photos from my camera’s memory card onto my PC. So, after a bit of cropping, and (very slight tweaking) here are the first 4 good(ish) photos out of the 46 I’ve dumped off my card. All were taken at various dates, some going back quite a while into the midst of our (wierd) summer.

Click on each picture to go to it on flickr, more sizes / longer descriptions are available there.

Dancing Nats. Cool, but they can be really annoying.

Dancing nats

Sunset Skybeams

Sunset SkyBeams

Red Leaf. Here comes Autumn.

Red Leaf

Sparky Fire.

Sparky Fire

Storm Worm Analysis (Take 2)

I’ve read quite a lot in my search on information about the Storm worm.


Apparently, a better estimate of the Storm Worm Botnet’ current number of zombie machines is about 10 Million. As such, I’ve redone all my calculations (bottom of the article) with the updated numbers, and I’ve also spent some more time finding other numbers to remove some of the estimations from calculations.

I estimate, that the botnet currently has access to about 15,000 THz of CPU power. The fastest super computer currently in existence, Blue Gene L has 91.8 THz. So, this has fallen with the re-calculations.

I managed to find this report on the state of broadband in the US, which says that the average upload speed (all I’m interested in really) is about 371kb/s. So, I’ve recalculated all of my bandwidth calculations, working from that figure, as outside the US, e.g. canada, Japan, are likely to have much higher upload speeds. Also, Britain is starting to move to 448/812kb/s

About 442GB/s. Which, is equivalent to 339 Million emails per second, or 604 CDROMs worth of data every second.


So, what have the Zhelatin Gang (group of crackers behind the Storm Worm) been up to with all this data capacity?

This report says that they are currently selling distribution capacity, as well as as of the 13th of August, testing their DDoS capacity.

This report from spamnation.info estimates that they are currently attacking a number of Anti- Spam / Malware sites. In fact, a large number of malware sites have / are under attack, including 419eater, which was basically overloaded with about 450GBs an hour worth of traffic, taking it off-line. CastleCops.com, is currently weathering the same high-level of incoming traffic.

Here is a graph of the traffic hitting 419eater.com. The attack took 419eater offline for a number of days, and they’re only coming back online now. They are still under attack, but have moved hosts, to someone who can cope with a massive amount of data incoming.

419eater DOS attack graph

At 11:44, traffic stops, as the site is taken offline, because the guys who hosted their website could no-longer cope with the sheer amount of incoming traffic.

Self Defence

The storm worm is (unfortunately for us) quite clever. It detects when its being used on what is called a virtual machine, a tool that some security researchers use to keep their PC safe from the trojan/virus, whilst they are trying to disassemble it.

Also, the botnet will launch a DDoS attack at any computer that either:

  1. Downloads the virus too many times (Researcher)
  2. Scans an infected computer for the basic signs of infection

I hope all this information is useful. The storm worm has quite worried me recently, and the only real way to combat it now, would be for the ISP’s to take action. Which they are not going to anytime soon – it does not make economic sense to do so.

My calculations are below. If you have any more up-to-date information for me to base them on, I’d love to hear from you. Leave a comment, or send me an email. My address is in the “about” page, linked above.


All calculations are in computer-style notation, so * for multiplication, and / for division.

Processing Capacity (Zombies)

Assume 10 Million infected computers. 10,000,000.

Assume an average of 1.5Ghz processor in each computer. (Its probably more like 2.5Ghz, but safe side it.) 15,000,000 Gigahertz (Ghz)

15,000,000/1000 = 15,000 Terahertz (Thz)

Processing Capacity (Blue Gene L Super Computer)

Blue Gene L, has 131,072 Processors, each running at 0.7 GHz (700 Mhz).

131072 * 0.7 = 91750.4 Ghz

91750.4 / 1000 = 91.7504 Thz

Round to 1 decimal place = 91.8 Thz

Data Transfer (Zombies)

Assume 10 Million infected computers. 10,000,000.

Assume that each computer has about 371kb/s upload rate. (Probably a bit higher, but thats the average for the US, so safe-side it. 10 million is still a lot of computers…)

Get the 371 Kilobits into KiloBytes. 1 KiloByte = 8 KiloBits, so:

371/ 8 = 46.375KB/s per bot.

10000000 * 46.375 = 463,750,000KB/s transfer rate. Ok, that’s too mind-boggling. Lets get the numbers to be more sensible.

1 MegaByte = 1024 KiloBytes, so:
463750000 / 1024 = 452,880.859375MB/s. Not readable yet. Again.

1 GigaByte = 1024 MegaBytes, so:
452880.859375 / 1024 = 442.266464233GB/s

Err… I did do these sums right… *checks*. Wow.

Round to 0 decimal places = 442GB/s

Emails per second with 442GB/s bandwidth.

Assume an Average spam email size of 11.76 KB from This article, and rough confirmation from spamnation.info

From our bandwidth calculations above, there is 463,750,000 KB/s bandwidth available. So:

463750000 / 11.76 = 339434523.80952381 emails per second.

Round to 0 decimal places = 339,434,524 emails per second.

Round to 3 significant places = 339,000,000

CDs per second with 442GB/s bandwidth.

CD-ROM total size : 750MB.

From bandwidth calculations above, 452880.859375 MB/s.

452880.859375 / 750 = 603.841145833 CDROMS worth of data transfer per second.
Round to 0 decimal places = 604 CDROMS data per second.

Thanks to

Those on the CastleCops DDoS forum who helped provide data.. and the rest of the DDoS forum guys, for putting up with me whilst I find out more about the Storm / Nuwar botnet.

Useful Linux (Ubuntu) Tidbits

Here are just some morsels of information about the Linux Command line, and more specifically, Ubuntu Linux Command line / system.

  • Users can be added to a group with the command:
sudo adduser <username> <groupname>

In ubuntu, the default system shell is “Dash”. That does speed up your system boot, but it also introduces problems with those scripts which are designed to run in bash, but use /bin/sh to execute. (Which is a surprising amount… this has solved many problems for me. Especially with Asterisk and freePBX.)

  • To set Ubuntu back to using bash from dash, run the following command…
cd /bin && sudo rm sh && sudo ln -s /bin/bash /bin/sh
  • To add a user to the sudoer list (the list that controls who can use “sudo”) use the command
sudo visudo
  • Add a user underneath the “# User privalage specification” comment. If you want just a bog standard sudo user, able to do all on the system, add the line:
<username>    ALL=(ALL) ALL
  • ?Fun? tip: add “insults” to the end of the list of “Defaults” in visudo, so it will look like:
Defaults        !lecture,tty_tickets,!fqdn,insults
  • The system will insult you every time you enter your sudo password wrongly. For a random example, it just gave me this when I deliberately triggered it:
You speak an infinite deal of nothing
  • In Firefox, select the address bar quickly by hitting the “F6” key.
  • Type “pwd” to get the full path to your current directory. e.g.:
kirrus@asus:~$ pwd
  • Monit is a useful program, that gives you a good way of keeping an eye on your servers, making sure they don’t run out of harddisk space, or get a high CPU load. It can either perform some function (like stopping a program from running) during high CPU, or send you a warning email.

http://debianhelp.co.uk/monit.htm (useful howto/basic guide)

http://www.tildeslash.com/monit/ (main website)

Monit is in the ubuntu repositories:

aptitude install monit

Scary analysis of Storm Worm

The Storm Worm. A virus and set of malware that has been spreading across the internet since January 2007. According to this article, it is now estimated that it has turned up to 50 Million computers into bots (or Zombie Computers), and is more powerful than a supercomputer.

So, I thought, a fine time to do some number crunching, to see if we can see approximately how powerful this bot-net is. (See here for the full article, with calculations at the bottom, if you’re looking at this from the homepage.)
The bot-net will probably have a maximum of around 40,000 Terahertz (THz) at its disposal. To put that into perspective, the worlds fastest supercomputer (The Blue Gene L) has around 91.8 Terahertz.

Ok, that is a really, really impressive amount of processing power. But what use is that power without being able to get data (spam emails etc.) onto the internet?

61 GigaBytes a second.

You heard me right. The bot network, will have the estimated capacity to pump 61 Gigabytes of data onto the internet per second.

At 20KB an email, thats 3,200,000 (Three Million Two Hundred Thousand), emails per second.
Revised email calculations (see note below): at 11.73KB an email, thats 5,442,177 emails per second.

Or, at 750MB per CDROM, thats 83.3 discs per second.
I think the term “Houston, we have a problem” doesn’t even come close to showing the amount of pain these guys can cause. At that rate of data transfer, this cracker (“hacker” for mass-media) group will be able to take any website in the world off-line, with a Distributed Denial of Service attack. They may even be able to take the internet offline, with another Backbone attack.
Now, this has been a almost-worst-case scenario, but trust me when I say, this is not good. I think, something drastic may be in order.

Note: Checkout my math and assumptions by clicking the “Read More” (if you’re not already in the full post). If you see any problems, or have some more-up-to-date information, feel free to register, and add a comment. (Sorry about forcing you to register. Spammers have been causing problems ><.) Alternatively, my email address can be found on the “about” page above.”
Update: Looks like the bot-net is being brought into play to attack anti-spammer websites.

Update #2: The webmaster of spamnation.info has confirmed (roughly) the analysis of this article, which says the average spam email is 11.76KB. As such, I have revised my email calculations. I’ll probably post a follow-up to this, if I can get any more accurate data to go on, and I may re-do all of the calculations at some point, with a more conservative bot number estimate base.


All calculations are in computer-style notation, so * for multiplication, and / for division.

Processing Capacity (Zombies)

Assume 40 Million infected computers, (even though the article says 50,000,000, lets err more on the safe side…) 40,000,000.

Assume an average of 1Ghz processor in each computer. (Its probably more like 1.5, but safe side again.) 40,000,000 Gigahertz (GHz)

40000000/1000 = 40,000 Terahertz (THz)

Processing Capacity (Blue Gene L Super Computer)

Blue Gene L, has 131,072 Processors, each running at 0.7 GHz (700 Mhz).

131072 * 0.7 = 91750.4 Ghz

91750.4 / 1000 = 91.8 Thz (Rounded to 1 decimal place)

Data Transfer (Zombies)

Assume 40 Million infected computers. 40,000,000.

Assume that each computer has about 128kb/s upload rate. (Probably closer to at least 256kb/s, but lets err on the safe site. 40 million is a lot of computers…)

Get the 128 Kilobits into KiloBytes. 1 KiloByte = 8 KiloBits, so:

128 / 8 = 16KB/s per bot.

40000000 * 16 = 64,000,000KB/s transfer rate. Ok, that’s too mind-boggling. Lets get the numbers to be more sensible.

1 MegaByte = 1024 KiloBytes, so:
64000000 / 1024 = 62,500/s. Not readable yet. Again.

1 GigaByte = 1024 MegaBytes, so:
62500 / 1024 = 61GB/s (rounded to 0 decimal places). Err… I did do these sums right… *checks*. Wow.

Emails per second with 61GB/s bandwidth.

Assume an Average spam email size of 20 KiloBytes (This article says 11.76KB, but its out of date, and source is offline. Err on safe side.)

From our bandwidth calculations above, there is 64,000,000 KB/s bandwidth available. So:

64000000 / 20 = 3,200,000 per second.

Emails per second with 61GB/s bandwidth (Revised calculations).

The 11.76KB appears to be accurate, so lets revise these calculations to take that into account.

Bandwidth 64,000,000 KB/s.

64000000 / 11.76 = 5,442,177 emails per second (Rounded to 0 decimal places.)

CDs per second with 61GB/s bandwidth.

CD-ROM total size : 750MB.

From bandwidth calculations above, 62,500 MB/s.

62500 / 750 = 83.3 CDROMS worth of data transfer per second. (Rounded to one decimal point)