Updated Ajax Animator 31 December 2011
Well it’s been the first time in about two years, and it wasn’t a big change.
Offline Wiki Redux 30 December 2011
There’s just something incredibly alluring about the concept of holding the sum of human knowledge with you at all times. While near-ubiquitous connectivity alleviates this to a certain extent, the momentary lapses of networking are incredibly corrosive to an information dependent mentality. Wikipedia never ceases to amaze me and, while I’ve tried in the past to encapsulate part of its sheer awesomeness, this marks a much more significant attempt.
The differences start even before the data gets to the application. The preprocessing toolchain was entirely rewritten for a multitude of reasons. First of all, it compresses not the entireity, but rather the most popular subset of the English Wikipedia. Two dumps are distributed at time of writing, the top 1000 articles and the top 300,000 requiring approximately 10MB and 1GB, respectively. While ostensibly, the mere top 300k articles is far too narrow to delve deep into the long tail, the breadth of the meager 1/25th of articles consistently surprises me in its depth. The advantage is that at 1GB, it’s relatively easy to fit into any system. The algorithm which strips extraneous content has been made far more sophisticated than the original series of regular expressions. This enables greater compression and less accidentally omitted content.
On the application end, the application has switched from a GWT-compiled LZMA SDK to a speedy, pure javascript decoder. This makes page loads significantly speedier and allows greater compression ratios, for individual blocks can be made larger (256KB instead of 100KB). It also now uses WebGL Typed Arrays to further speed things up, such as sending data to and from the WebWorker thread.
The interface was redesigned with CSS media queries to dynamically transition between different modes in response to different viewing environments. The interface consists of two regions: the fixed position recessed left panel which holds the page title, a search bar, controls and the page outline. This collapses down to a toolbar header automatically when the screen estate is limited. It uses an Apple-esque noise texture background.
Downloads happen in little units called chunks (they’re half a megabyte for the dump file and about four kilobytes for the index). The local file can be built up out of order. While online, all storage operations check the virtual file, indexed db, or web sql database. If it’s not there, it transparently uses an XMLHttpRequest in order to fulfill the request and caches it to disk in the respective persistence mechanism. A bitset is used to keep track of which chunks are already downloaded and which need to be downloaded.
Pepper Spray Cop 30 November 2011
This is a hopefully more entertaining project which piggybacks on a meme which is probably already quite dated relative to the date of the publishing of this article. Nonetheless, that meme and the entire occupy movement will likely hold some place in the annals of internet history. For those unexposed to this meme, UC Davis Police officer Lieutenant John Pike was photographed nonchalantly applying copious quantities of the practical food product (as Fox News phrased it) to the throats and faces of nonviolent protesters.
This app takes a picture and uses the Face.com face detection API in order to locate candidate faces. The image is drawn on canvas with the corresponding cop, and a orange mist is drawn by iteratively drawing random points with accordance to a certain probability distribution. The cop flips with a rather cool animation when you drag him around. http://metaception.com/pepper/
Surplus 4 29 October 2011
Recently Surplus stopped working. Well, it hasn’t been working for a lot of people for a long time already, but that’s besides the point. It stopped working entirely. Surplus has always been a rather fragile creature. It operates like a kid on a high speed scooter attempting to carry a house of cards between two strangers. That house of cards is part of a delicate system of frames inside frames inside frames inside frames that move between frames. Surplus is this fairly atrocious mess of frames.
Framing things works out fine until you discover that whatever you’re framing is trying to break out. Meet the X-Frame-Options header, the source Surplus’s recent predicament. It has well meaning motives: to prevent Google from suffering from evil attacks like Clickjacking, XSRF and other nasty things. Incidentally, security-wise, Surplus would probably belong closer to something of that nature than a legitimate application. It doesn’t use an API because applications generally wouldn’t find it useful.
Recently, all Google properties started including that X-Frame-Options header, and now can’t be embedded in frames. It wasn’t an absolutely unprecedented move, because just a few weeks earlier Google Video had started sending out the header (which led to an update which moved from a Google Video host frame). But now it was across all Google Sites, and there was no short term hack that could be done.
The solution was to take a random Google page which didn’t send out the header and mimic all the postMessage messages that are sent from the Google Plus notifications frame. Consequently, the entire frame signaling and attachment system had to be rewritten, and that system was so deeply tied into everything else that Surplus 4 ended up being almost an entire rewrite (the inner frame actions, the options page and the notifications parser did not change).
https://chrome.google.com/webstore/detail/pfphgaimeghgekhncbkfblhdhfaiaipf
Schedule Compare 02 September 2011
Posting this here is almost certainly useless. I assume very very few people who read this blog tend to be in the target 13-18 year old facebook-using high school student demographic. By the unlikely chance you are (if you aren’t, you can forward this to the nearest person who fits into this demographic, as I will add later, that I’m desperately looking for users).
It’s that time of year. The brief window where summer vacation isn’t technically over yet as school hasn’t started but you still know your classes for next year. You’re frantically attempting to complete those long procrastinated summer assignments, or like me, you’re desperately trying to avoid them by giving a yourself a false sense of productivity by building random apps.
My first foray into the realm of creating Facebook applications is fairly simple. It compares class schedules. In truth, the reason I made this was probably not the fact that I enjoy making useful tools, but more likely residual bitterness of rejection by a sci/tech high school over three years ago which has a school-specific schedule comparing app. Nonetheless, a neat side effect of this attempt is that it does happen to be quite cool.
This is also my first published app which is written in the CoffeeScript language. For those of you unaware, CoffeeScript is a language which is syntactically similar to Python but compiles into Javascript. It’s not a nasty GWT-esque compilation, but a relatively clean one (barring the underscores that result when you try doing comprehensions and the really cool stuff). I’ve always meant to write stuff in CoffeeScript, as it has quite a few awesome features. Most importantly is probably the ability to declare a function with two characters (->) rather than a massive “function(){}” and the array comprehensions.
Compiled (or should I say Transpiled?) languages have odd a few annoying properties, especially with debugging. The biggest issue was probably setting up everything: running a script which uses inotifywait to automatically compile your CoffeeScript once you hit “Save” on your editor of choice (gedit just because it works and comes with Ubuntu). Then when errors happen, your line numbers don’t match up and that’s also annoying.
The Facebook API is actually pretty good. My app reads Facebook schedules from your friends’ statuses. It’s not quite as easy as it should be. I could search the user’s news feed and that would be trivial except that it only gives me a subset of the statues that I want to be able to process. When using FQL (which I ended up pronouncing Feequel which sounds a bit like Fecal because it’s a SQL derivative, even though you’re not supposed to pronounce it “sequel”) it would only return/search the most recent status. I ended up doing a FQL request for each and every friend that the current user has, which is a on average a pretty big number. Fortunately it doesn’t seem like Faecbook has any API limits. Awesome.
For the longest time I was confused because my app inexplicably only worked for me. It turns out that my queries returned blank results for everyone else because I didn’t request the right permissions. That’s terrible. Absolutely terrible. First of all, the developer shouldn’t be entitled to have those magical privileges that the end users can’t have. It’s insanely confusing. And don’t just silently return no results and make the developer question his own sanity.
But it was a permissions issue - a one line fix in the end.
It’s also quite depressing that nobody’s using it. It’s pretty server intensive at the moment and it’s running on Google App Engine, which has that new pricing which means I should have my free quota expire after something like a meager 100 users. But I haven’t really come close to that. Why? I guess I have little influence over friends.