Blog » Distributed

Massive file sharing networks?

admin — Thu, 17 Sep 2009 19:39:09 +0000

With BitTorrent and everything else, logically everone just has a network around a single file everyone is interested in. It makes tit-for-tat really practical and quite easy. Almost like darwinian natural selection (well, evolution is like moore’s law, you can just about apply it to everything imaginable ignoring the massive pardigm tangents that result), the popular files are fast and the unpopular ones get slow and/or eventually die, totally lost.

Certainly this almost capitalistic approach is efficient for the massively popular files where the chances are that you will encounter many fast seeds. But for those less fortunate, the downloads may slow to a crawl as peers may be geographically distant.

What about something like freenet, where everyone has a cache of data that they may or may not be interested in. With such a massive netowrk, you no longer have to worry about trackers taking over as all of them are now on equal ground, finding trackers is useless as they are inhrently found. Leeching is solved by a global karma system, and all of the other problems are replaced with the single complicated problem of scalability (which totally makes the whole concept crap).

Somewhat better distributed pi calculator

admin — Wed, 27 May 2009 23:32:16 +0000

http://distributed-pi.appspot.com/static/beta.html

(Potential number of jobs is no longer theoretically finite)

Feynman Point

admin — Sat, 03 Jan 2009 23:29:44 +0000

http://en.wikipedia.org/wiki/Feynman_point

The pi calculator has reached the Feynman Point!

Calculate Pi!

admin — Tue, 30 Dec 2008 22:29:31 +0000

Press this button and calculate pi!

Please note that this is a distributed effort and there is no simple way to get the final value as of yet.

Distributed Computing Take III

admin — Tue, 30 Dec 2008 21:19:10 +0000

I donno why, but i’m revisiting this. I was trawling across Wikipedia one day, and I got to the article about Pi. I tried distributing Pi a while ago, actually, before I did the hashes. But I never ended up implementing it because it didn’t seem feasable, as all the algorithims I encountered (or tried porting) required lots of memory, something very hard to distribute for this scenario. But this time, I found these. Looking through them, and googling in the process, I found http://www.omegacoder.com/?p=91, and ported it over to Javascript. It was relatively slow compared to the SuperPi implementation in Javascript, but it was easily distributed.

One problem though, is that it gets slower every iteration (to find the net block of digits). Finding .141592653 will be roughly 20ms faster than the next 9 digits (it processes in blocks of 9). Not only would it take longer, but it occupied 100% of the CPU, and it would pop up that ever-annoying “This script may make your computer non responsive” window. So I implemented this pattern to make it not lock up any browser other than Chrome (and possibly WebKit Nightly).

Still, it would take up 100% of the CPU. I ran it overnight and got to digit 17,000.

Eventually, it would take about half an hour for a single iteration (at the 20000th digit). With web-based distributed computing, I can’t rely much more time than what Google Analytics reports to be 00:02:24 (my Average Time on Site). And that’s half an hour with a 3ghz Intel Core 2 Duo (it’s dual core, but the script, is single threaded).

I then split the function into smaller parts. the main function was split up, and the loops were divided across users. Now, it can scale easily. It uses virtually no visible CPU. and fits well into that 2 minute timeframe.

Try it out here, but don’t stay too long, because i only set there to be 500 “jobs”.