Skip to content


Offline Wiki Redux

There’s just something incredibly alluring about the concept of holding the sum of human knowledge with you at all times. While near-ubiquitous connectivity alleviates this to a certain extent, the momentary lapses of networking are incredibly corrosive to an information dependent mentality. Wikipedia never ceases to amaze me and, while I’ve tried in the past to encapsulate part of its sheer awesomeness, this marks a much more significant attempt.

The differences start even before the data gets to the application. The preprocessing toolchain was entirely rewritten for a multitude of reasons. First of all, it compresses not the entireity, but rather the most popular subset of the English Wikipedia. Two dumps are distributed at time of writing, the top 1000 articles and the top 300,000 requiring approximately 10MB and 1GB, respectively. While ostensibly, the mere top 300k articles is far too narrow to delve deep into the long tail, the breadth of the meager 1/25th of articles consistently surprises me in its depth. The advantage is that at 1GB, it’s relatively easy to fit into any system. The algorithm which strips extraneous content has been made far more sophisticated than the original series of regular expressions. This enables greater compression and less accidentally omitted content.

On the application end, the application has switched from a GWT-compiled LZMA SDK to a speedy, pure javascript decoder. This makes page loads significantly speedier and allows greater compression ratios, for individual blocks can be made larger (256KB instead of 100KB). It also now uses WebGL Typed Arrays to further speed things up, such as sending data to and from the WebWorker thread.

The interface was redesigned with CSS media queries to dynamically transition between different modes in response to different viewing environments. The interface consists of two regions: the fixed position recessed left panel which holds the page title, a search bar, controls and the page outline. This collapses down to a toolbar header automatically when the screen estate is limited. It uses an Apple-esque noise texture background.

Downloads happen in little units called chunks (they’re half a megabyte for the dump file and about four kilobytes for the index). The local file can be built up out of order. While online, all storage operations check the virtual file, indexed db, or web sql database. If it’s not there, it transparently uses an XMLHttpRequest in order to fulfill the request and caches it to disk in the respective persistence mechanism. A bitset is used to keep track of which chunks are already downloaded and which need to be downloaded.

http://offline-wiki.googlecode.com/git/app.html

Posted in Offline Wiki.

Tagged with , , , , , , , , , .


Pepper Spray Cop

This is a hopefully more entertaining project which piggybacks on a meme which is probably already quite dated relative to the date of the publishing of this article. Nonetheless, that meme and the entire occupy movement will likely hold some place in the annals of internet history. For those unexposed to this meme, UC Davis Police officer Lieutenant John Pike was photographed nonchalantly applying copious quantities of the practical food product (as Fox News phrased it) to the throats and faces of nonviolent protesters.

This app takes a picture and uses the Face.com face detection API in order to locate candidate faces. The image is drawn on canvas with the corresponding cop, and a orange mist is drawn by iteratively drawing random points with accordance to a certain probability distribution. The cop flips with a rather cool animation when you drag him around.  http://metaception.com/pepper/

Posted in Fun, Pepper Spray Cop.

Tagged with , , , , , , , , , , , .


Surplus 4

Recently Surplus stopped working. Well, it hasn’t been working for a lot of people for a long time already, but that’s besides the point. It stopped working entirely. Surplus has always been a rather fragile creature. It operates like a kid on a high speed scooter attempting to carry a house of cards between two strangers. That house of cards is part of a delicate system of frames inside frames inside frames inside frames that move between frames. Surplus is this fairly atrocious mess of frames.

Framing things works out fine until you discover that whatever you’re framing is trying to break out. Meet the X-Frame-Options header, the source Surplus’s recent predicament. It has well meaning motives: to prevent Google from suffering from evil attacks like Clickjacking, XSRF and other nasty things. Incidentally, security-wise, Surplus would probably belong closer to something of that nature than a legitimate application. It doesn’t use an API because applications generally wouldn’t find it useful.

Recently, all Google properties started including that X-Frame-Options header, and now can’t be embedded in frames. It wasn’t an absolutely unprecedented move, because just a few weeks earlier Google Video had started sending out the header (which led to an update which moved from a Google Video host frame). But now it was across all Google Sites, and there was no short term hack that could be done.

The solution was to take a random Google page which didn’t send out the header and mimic all the postMessage messages that are sent from the Google Plus notifications frame. Consequently, the entire frame signaling and attachment system had to be rewritten, and that system was so deeply tied into everything else that Surplus 4 ended up being almost an entire rewrite (the inner frame actions, the options page and the notifications parser did not change).

https://chrome.google.com/webstore/detail/pfphgaimeghgekhncbkfblhdhfaiaipf

Posted in Chrome Extensions, Google+, Surplus.

Tagged with , , , , , , , , , , , .


Schedule Compare

Posting this here is almost certainly useless. I assume very very few people who read this blog tend to be in the target 13-18 year old facebook-using high school student demographic. By the unlikely chance you are (if you aren’t, you can forward this to the nearest person who fits into this demographic, as I will add later, that I’m desperately looking for users).

It’s that time of year. The brief window where summer vacation isn’t technically over yet as school hasn’t started but you still know your classes for next year. You’re frantically attempting to complete those long procrastinated summer assignments, or like me, you’re desperately trying to avoid them by giving a yourself a false sense of productivity by building random apps.

My first foray into the realm of creating Facebook applications is fairly simple. It compares class schedules. In truth, the reason I made this was probably not the fact that I enjoy making useful tools, but more likely residual bitterness of rejection by a sci/tech high school over three years ago which has a school-specific schedule comparing app. Nonetheless, a neat side effect of this attempt is that it does happen to be quite cool.

This is also my first published app which is written in the CoffeeScript language. For those of you unaware, CoffeeScript is a language which is syntactically similar to Python but compiles into Javascript. It’s not a nasty GWT-esque compilation, but a relatively clean one (barring the underscores that result when you try doing comprehensions and the really cool stuff). I’ve always meant to write stuff in CoffeeScript, as it has quite a few awesome features. Most importantly is probably the ability to declare a function with two characters (->) rather than a massive “function(){}” and the array comprehensions.

Compiled (or should I say Transpiled?) languages have odd a few annoying properties, especially with debugging. The biggest issue was probably setting up everything: running a script which uses inotifywait to automatically compile your CoffeeScript once you hit “Save” on your editor of choice (gedit just because it works and comes with Ubuntu). Then when errors happen, your line numbers don’t match up and that’s also annoying.

The Facebook API is actually pretty good. My app reads Facebook schedules from your friends’ statuses. It’s not quite as easy as it should be. I could search the user’s news feed and that would be trivial except that it only gives me a subset of the statues that I want to be able to process. When using FQL (which I ended up pronouncing Feequel which sounds a bit like Fecal because it’s a SQL derivative, even though you’re not supposed to pronounce it “sequel”) it would only return/search the most recent status. I ended up doing a FQL request for each and every friend that the current user has, which is a on average a pretty big number. Fortunately it doesn’t seem like Faecbook has any API limits. Awesome.

For the longest time I was confused because my app inexplicably only worked for me. It turns out that my queries returned blank results for everyone else because I didn’t request the right permissions. That’s terrible. Absolutely terrible. First of all, the developer shouldn’t be entitled to have those magical privileges that the end users can’t have. It’s insanely confusing. And don’t just silently return no results and make the developer question his own sanity.

But it was a permissions issue – a one line fix in the end.

It’s also quite depressing that nobody’s using it. It’s pretty server intensive at the moment and it’s running on Google App Engine, which has that new pricing which means I should have my free quota expire after something like a meager 100 users. But I haven’t really come close to that. Why? I guess I have little influence over friends.

http://schedule-compare.appspot.com/

Posted in Education, Schedule Compare.

Tagged with , , , , , , , , .


Surplus

In a continuation of my rather unhelpful habit of documenting my activities on this blog long after you probably already know about it, I guess it’s time for me to discuss Surplus, my wildly popular (at time of writing) chrome extension which integrates Google+ notifications into Chrome.

Even more impressive, the name, which is a fairly common word is actually on the first page of a Google search for the word (around eighth result). It peaked at around 53,000 users and at one point made me the 329th most followed person on Google+.

 

Posted in Chrome Extensions, Google+, Surplus.

Tagged with , , , , , , , , , , .


Generating the iOS 5 Linen texture with Canvas



I guess the linen texture which is way too prevalent in Lion and iOS 5 looked pretty cool so I tried replicating the effect in canvas. It’s not instant but the texture is generated fairly quickly. It’s all done in around 20 lines of code. The basic idea is to first create a bunch of semi-transparent noise such as the stuff on the right (Though in the real one the opacity is only 3% and in the one on the right it’s been increased to 70%). To do that, we createImageData and set every fourth pixel to 6 if Math.random() < 0.1. That means approximately 10% of the canvas will be semi-transparent with the rest being totally transparent. I’m not clever enough to embed some steganographic message in the ostensible noise pattern, because I’m just way too lazy for that sort of stuff. But if you think that that last sentence was actually a decoy for my master plan, feel free to waste time decoding a message which probably isn’t there.

After that, the canvas is converted to a DataURL so it can be loaded as an image. After loading the image, we iterate 40 times and call drawImage on the original canvas with an offset to make every single point into a cross-shape. Demo.

Posted in Design.

Tagged with , , , , , , , , , , , , , .


Google+ Profile Suspended

My Google+ profile was suspended on grounds that were not wholly unexpected, but still shocking. I’m likely violating many of their conditions, as I’m 16 (under 18 at time of writing) though when my Google Account was registered, I probably lied and said that I was 30 (At that time, when I was 11, I tended to do that with all services to evade the nonsense COICA on Neopets). I’m not really the victim here (Though I’d like to pretend that I’m pretending that I am so I have an ostensible excuse for narcissism without admitting it by admitting it). I hardly ever share anything on Google+, so I lose nothing except for the ability to make an occasional uninsightful post and the ability to brag about my 3.4k+ followers.

I assume it’s the fault of every social network to obsess over its user base, to define a subset of humanity and designate those as members. To praise individualism, differences and variance yet rely so heavily on the homogenuity and uniformity of the users. A network can not be inclusive and exclusive at the same time, to assume that’s possible is the definition of insanity.

It’s a pity how a minor issue can inflate so much. A minor annoyance in the legalese which every average legally inept and time starved human that we all assumed Google would turn a blind eye to in order to focus on user adoption. “Use the name which people know you by”. It’s a startlingly obvious notion for social networks, if a software system is attempting to duplicate the nuanced structure of physical and digital relationships in a unified and coherent implementation, it makes sense to retain the basic handles which we rely upon. 

But rather than draw upon this logic, that sentence is followed upon by the possibly inconsistent idea to “Use your real name”. What if your “real name” isn’t what you’re known as? What if you don’t feel comfortable revealing this information? This attempt at recognizing nuance fails so blatantly at the most basic principle of consistency and blatantly disregards the considerations of a rather significant portion of web users.

Some say that this principle is good, that it filters out those who should not belong in the network anyway. That those who tend to use pseudonyms do not belong on a good faith network which depends on the cooperation and trust of every individuals. That pseudonyms are employed by criminals and those with malignant intent, and therefore all precautions must be taken to eliminate that risk.

A network has a different role in dealing with its users than a specific group does. There’s a rather interesting notion that is presented by Google’s terming of the rules as “Community Standards”: the idea that everyone who uses Google+ is a part of some sort of community. Not a loosely associated network of individuals through transparent circles, but a community as if every one had to occasionally encounter and take offense to each other.

Postel’s law, the idea that a software framework should be liberal in what it accepts and conservative in what it sends, should apply with the massive-social-network hopeful known as Google+. At the “web scale” which Google operates at, it should be clear that humans often have very few attributes in common. The slightest interpersonal misalignments turn into gaping canyons on the magnitude of billions.

Pseudonyms mean quite literally a fake name. Now, why would someone want a fake name? The simplest answer which by no means encompasses all or even most uses, is that someone has something to hide. As the evil voldemort-controlled ministry of magic in harry potter and the deathly hollows part 1 the movie leader said “You have nothing to be afraid of if you have nothing to hide”, which I found shockingly similar to Eric Schmidt’s “If you have something you dont want shared online, maybe you shouldn’t be doing it in the first place”. 

Does pseudonymity necessarily produce a culture incompatible with civilized society? I would say no, and even if the answer was yes, it doesn’t matter. The internet is a big and diverse place and it doesn’t matter if there are people who have questionable motives exist as they aren’t forcing their ideas on others. 

In Romeo and Juliet, Shakespeare (through Juliet) asks “What’s in a name?”, expressing the notion that the name is a meaningless and artificial convention which only serves to create an equally artifical and antagonistic gap between the protagonists. We can choose to subscribe to these existing and equally artificial distinctions and we should be able to choose not to. 

Posted in Google+.

Tagged with , , , , , , , , , , , , .


Inception

This post is going to be nothing like any other post, because it’s going to be about the video frame at the 650th second of the movie Inception.

With a bit of Googling, it seems the first part of this document is “Development of Highly Sensitive Portable Hydrocarbon Sensors” by  Dr. Michael Carpenter at the State University of New York.

Here follows the text of the document taken from http://www.utrc2.org/research/projects.php?viewid=155 marked up to correspond with the appearance of the document in Inception. The second part of the document in the frame does not appear to be from the same source and I’ve encountered difficulty in locating the source of that part of the document. The visible differences between the source document and the image from Inception are in bold while the redacted contents are in strikeout. The newlines are approximately where they are in the visible document.

January 2007 we have begun to develop and evaluate a cost

and space effective sensing assembly which will leverage our

past and future hydrocarbon sensor development efforts funded

by the NYS-DOT. The UTRC funded program is developing and

evaluating the proof of concept for a portable and cost effective

hydrocarbon sensor based on a compact optical spectrometer which

monitors the changes in photoluminescence of arrays of

hydrocarbon sensitive quantum dots (QDs). Through this

program we have recently demonstrated the ability to detect

xylenes at 50ppm concentration levels. While this is a very

promising result given the cost effective hardware that is

being used in this program, this detection limit needs to

be reduced to 100ppb levels for this device to have a range

of environmental applications. Therefore, we propose to

expand this program to include the following new

initiatives during the 2007-08 program:

1)Investigation of QD based materials with ppb level

sensitivity through the use of resonant energy

transfer sensing mechanisms,

2)Deposition of QD arrays to increase the sensing

dynamic range to ppb to % level concentrations,

3)Optimization of 6 element arrays for

increased selectivity, and

4)Implementation of the pattern recognition software

selected during the 2006-07 program.

And so the question is: Is there a specific reason that the secret document was of hydrocarbon detection? What does this have to do with the movie? Is there any significance of the redacted text?

Posted in Fun.

Tagged with , , , , , , , , , , , , .



Why the Chrome Web Store is Bad for the Web.

Note: I’ve changed a few things that will hopefully make my point a bit more clear

Apple got it right in 2007.

If you’ve read any of the other posts in this blog, you will probably come under the assumption that I’m a devout follower of the Church of Google. Thus it will probably be quite a surprise to read the headline, something which appears downright sacrilegious: it questions the infallibility of the great Google. But I try very hard to maintain some semblance of objectivity and rationality, and this post will be about why I think the Chrome Web Store is bad for the web.

The Chrome Web Store is the applications and extensions gallery for Chrome. It’s Google’s centralized repository and directory for discovering Chrome-related things. Just hearing the name of it, you can probably tell, it’s likely quite inspired by the iOS/iTunes/Mac App Store. It’s not because they aren’t able to innovate (or it might, but I won’t take that view), but it’s probably the result of the huge App Store boom. It’s not that even what Apple did was particularly innovative, but somehow it managed to secure billions of dollars for the company, and all it’s competitors quite rationally want a chunk of it. This however, isn’t about improving the state and future of the web, but rather the indulgence of buzzwords. This post isn’t only about the Web Store, but rather the entire Chrome Applications and Extensions systems. From distribution to installation and the user experience afterwards.

Introduction.

There are two types of installable web applications that exist in the Chrome Web Store: hosted apps, in other words “glorified bookmarks”, and packaged apps. Glorified bookmarks are relatively hard to create, expensive and have no real additional functionality. Packaged applications evade the standardized mechanisms for offline web properties and eliminates many of the advantages of web apps in the first place.

Chrome’s developer overview for creating installable web apps describes the system as a solution to one, rather insignificant, problem. It’s the problem of permissions escalation: some technical detail that hardly seems important. Put simply, it’s that users get annoyed when they’re asked to hit “Okay” to annoying permissions prompts. And so Google’s solution is to invent a certain class of web site which has different security properties, where all the permissions are put into a single prompt.

To users, however, the existence of a web app is a solution to a much different user experience problem: they want to hit nice large pretty icons to go to sites which they frequently visit. But somehow, the solution they opted for creating these large clickable bookmarks is quite terrible. The only user-facing purpose of installable applications is the ability to bookmark with a large icon, something that Apple got right with iPhone OS in 2007.

Apple got it right.

I love those four words in that order, it feels so sensationalist and rebellious. But before the Cult of Apple leaps on that statement, notice the wording “Apple got it right”. It doesn’t strictly mean that whatever Apple’s doing now is right, just that what it did is right. In fact, that’s exactly what happened. Apple got it right, then made it different, and Google made it wrong.

First, we need to recall the distant year of 2007. It was quite a while ago, and I won’t pretend that my memory is that great. But it was a long time ago, a full year before the first beta release of Google Chrome. The iPhone was released with it’s plethora of eight apps and no ability to install more. The App Store didn’t exist, and the closest semblance was the Installer.app for jailbroken devices (Cydia came later). A few months later, Apple released a series of updates, and Steve Jobs signaled what he believed to be the future of iPhone applications: The Web. It doesn’t come surprising that Apple’s Mobile Safari was and likely still is (more or less) the best browser for any mobile device.

The important aspect is the way these web applications were installed. You went to Mobile Safari, and browsed around. You found a web app, and you used the web app the way the web was intended. No installations, you just navigate to a URL and start using it. You find the app useful and/or awesome, and you “bookmark” it. But, instead of actually doing the browser “bookmark”, you hit the button right below: “Add to Home Screen”. It asks you for a name for that application, automatically prepopulated with the document page title. You hit “Add”, and you now have a nice, shiny icon on your home screen. You can hide the browser chrome and it becomes indistinguishable from the normal native application experience.

That app icon is just an image URL specified with a single meta tag. It’s totally decentralized in every way, and represents the openness and simplicity that simply makes sense for this platform. All a developer needs to do to enable their web site to turn into a fully fledged web application is to add a <link rel="apple-touch-icon" href="/customIcon.png"/> in the head section of the page.

Contrast that with what Google requires: creating a Google checkout account, entering credit card information, navigating to the Chrome Web Store page and clicking several links in the footer in order to navigate to the page where you have to pay $5 for creating an app, create several icons, copy the manifest.json template and editing some values pointing to the icon locations, going to chrome://extensions, enabling developer mode, adding the unpacked app to make sure that it works, then going back to the original directory, zipping it up, and uploading it to the Chrome Web store, where you have to write a description, add screenshots, reupload an icon, publish, wait ten minutes, and then spam the internet with that link and edit your site’s code to point to that page. It’s an awful much to go through in order to just create a bookmark.

Apple turns evil.

This subtitle is intentionally misleading. I don’t really think Apple’s evil, but that loaded four letter word is much more concise than the more appropriate phrase “Apple adopts a new platform and shifts ideologically to favor a system which is ultimately in conflict with and entirely inapplicable to the web in its current state or in the foreseeable future”.

Apple’s prescience of the power of the web was sadly a bit anachronistic. The web technologies that would enable their vision were not yet ready. The second browser wars haven’t really even begun, and the jailbreakers, despite handicaps, still managed to develop that platform more than the officially sanctioned web developers could. Browsers were too slow, hardware was to slow, there weren’t enough features, not enough could be done, and the paradigm was not well understood.

Apple followed the lead set by the jailbreak community and launched their own native application development and distribution system: The App Store. It was a hit, and soon became a super huge buzzword. It an all that it represented: centralized one click micropayment driven mobile advertising funded indie developer weekend novelty apps.

Google gets it wrong.

So there was an App Store craze, and everyone wanted one. So it logically follows that Google built an App Store. But the web had no notion of apps. There were web applications, but they weren’t rigidly defined as apps. This is where Google got it wrong. The Chrome Web Store needed to sell apps, and had to create a dichotomy out of the web in order to do so. It created a distinction between web apps and websites where none had existed and shouldn’t have ever existed.

The false dichotomy.

Steve Jobs said that on Mobile, people want Apps, not websites. Before blindly mimicking the concept of apps on another platform, one should probably explore why users like apps over websites. It’s because the mobile app offers a better mobile-optimized interface to whatever they’re doing.

Websites aren’t generally designed for mobile, they are often slower, and can’t make use of persistent user interface elements like a tab bar. Apps aren’t popular because of the existence of the App Store. It’s because there’s additional value provided in having those apps, that users use the App Store to get them.

However, web apps, just like websites are optimized for normal computers. Web apps are no better than web sites, and when web apps really have nothing to provide, their respective web stores are useless.

One purported reason for creating the distinction between apps and websites is to give developers the opportunity to charge for the application in the web store. But why should the ability for an author to receive money for his or her respective works be exclusive to web apps? Why not all web sites?

While it’s quite clear that if anything meant to supplant a desktop application and is built for the web can be considered a “web app”, nearly everything else exists in a sort of gray area. Facebook, Twitter and the other social networking are predominately content focused, but have some app-like characteristics, and so they could be considered “web apps” too, despite how there aren’t really desktop equivalents. But what about sites like the New York Times? Pure content sites would logically seem to be the farthest one can go from the concept of an “application”. It’s clear that any web site can be considered a web app.

Since anything can be considered a web app, the Web Store is a mere directory of a certain number of websites. It’s a limited subset of the internet with terrible discoverability properties restricted only to sites where the owner (or a particularly devout fan) is willing to pay $5 in order to allow a subset of users to bookmark the site. It’s proprietary, no other site can have quite the same properties as the web store because Google has the Web Store URLs hard coded into chrome somewhere. Searching in the web store really isn’t that great either, with no ability to search reviews, no pagerank, no search operators, no ability to search within the content of apps. You would figure that if Google were to clone a subset of the internet, at least they would get search right.

This closed, exclusive and excessively tedious process for creating mere bookmarks attacks some of the web’s traditional benefits and ideals.

Packaged applications.

The above sections dealt with how the “glorified bookmarks” are useless and downright harmful. There is a second class of applications which are similar to the former in that they also get a pretty large prominent and clickable icon, but different in that they actually provide functionality that is different from mere ordinary websites. Its virtues include that they tend to work offline and have the ability to do certain things that normal web apps can not do. However, it pretty much stops there.

Packaged apps work offline, but their mechanism evades the standardized system of HTML5. Rather than promoting the use of standards, they promote the use of a proprietary and nonstandard signed zip package.

As they’re “packaged”, they aren’t really “true” web apps, because they don’t actually operate in the scope of the web. They are much closer to desktop apps, practically. They have no URLs, and thus can’t be linked to, evading the very first two letters in HTML and HTTP: “HyperText”. One of the greatest things about web sites is that they can be linked to, and they almost always share that universal identifier to share with people. It’s universally accessible and one of the few things that actually enables intra-site interoperability.

Conclusion.

While the “glorified bookmark” class of applications, which make up the vast majority of the Chrome Web Store, can be quite easily fixed by implementing something akin to the iPhone OS home screen web apps, the “packaged applications” are a bit more interesting. They are the source of that problem which the applications system was meant to resolve: permissions. In the current state, there is no system for handling multiple permissions on the web, aside from flooding the screen with infobars, when even that only partially works. What the web needs is a user friendly, informative, and useful system for giving additional permissions to web sites.

Along with that, the Web Store handles the selling of applications. Accepting money is a two part process, consisting of authentication and payment. Browsers should handle user identity, since they have the resources to do it right, in a nuanced, secure, efficient and user friendly way. Once that’s done, payment would be a logical extension to that. A developer could drop in a Google Checkout widget to have one-click in-app-purchases by tying into the secure browser identity system.

The Chrome Web Store should be reduced to a community maintained directory of useful web applications, something like a wiki, and there shouldn’t be a $5 fee to add applications.

Some people have expressed the idea that the Chrome Web Store is useful because it allows Google, a trusted party, to take down dangerous or malicious applications quickly. And while this is true, note that the Web Store is not actually the sole means to install chrome applications, and a malicious party would most likely exploit those alternate channels, and the only way to combat those is to institute a sort of Kill Switch, much like the kind that iOS, Kindle, and Android already implement.

Posted in Chrome Extensions.

Tagged with , , , , , , , , , , , , , , , .