A three second mockup.
I don’t have an iPad, nor is it #1 on my wish list (It mostly means any tablet platform but since none of the other ones are really recognizable, I’m jumping on the 4-letter apple product bandwagon). But I am fascinated by touchscreens.
I started programming when I was 7 when I got my first Lego Mindstorms RIS/RCX 2.0 kit (and I loved the 13+ sticker on the box back then :P). So I’ve always had a fondness for the platform, it’s really great for getting kids into robotics and engineering. Kudos to Lego.
Recently, I’ve played around with the current rendition of the Mindstorms platform, the NXT. It’s an evolutionary advancement for the platform and maintains the original intuition of the system while catering to those who don’t really grow out of the original system.
The interface is, a very kid-friendly drag-and-drop block layout. I actually sort of like it, though it’s not something which a desktop application could easily be made in. It’s very procedural, and that’s well suited for telling a car to explode and magically arrange red and blue balls into designated corners.
But really, where drag and drop really shines, the place where it really is meant to be, is on a multitouch tablet. It just makes sense. On a large multitouch surface, coding using simple finger gestures and dragging just makes sense. Lego’s own Labview interface, called NXT-G has large icons and is built entirely on the dragging and dropping. Its something that just feels right on a touchscreen.
The gestures need to be tailored to the specific platform, I propose that two fingers, like on a Macbook, should be used to pan around the canvas of the code. Blocks are dragged from a list on the side onto an execution path. On a block already on the canvas, touching and dragging does the logical thing: it moves the position. Touching a block on the canvas without dragging makes a pie menu type display ooze out from the block. The list would be a bunch of output “pipes” which another finger can be used to drag and link onto other blocks which display another pie menu (though only showing inputs rather than outputs) where letting go would create the connection.
Implementation-wise, one could try porting NBC/NXC (which is written in Pascal and already has the makefiles for WinCE/ARM and FreePascal does seem to be able to compile to iPhone/iPod Touch and the iPad should be a virtually equivalent platform). Probably something made in SVG and/or <canvas> could be used to create the interface which can be loaded with a UIWebView or using the PhoneGap platform. Then it would convert the graphical representation into some NXC code, compile it, and use the built in Bluetooth 2.1 + EDR support in an iPad to send it to the Lego NXT brick and do magic.
I’ve been wanting to do the same thing since I started working with the iPhone and iPad. There are two problems with a LEGO Mindstorms NXT-G clone for the iPad. Firstly no access to the bluetooth stack, and using Apple private APIs to do so will get your app kicked out. Second, and Apple will kick you out for this too, no compilation to take place on-device! I though about using bluetooth to control the brick(s) from the iPhone/iPad to leap over the latter but still you need access to the bluetooth stack.
The only way if to make the project open for those with Mac’s, iPads, LEGO Mindstorms, and knowledge of how to reverse engineer the iOS framework libraries. Small target market and a lot of work. With the Android tables likely to come in droves in the next year it’s a much better choice of platform. I love iOS but Android is in fact more open to the problems mentioned above are non-issues.
It’s amazing how a few short months makes Android so much more of a viable platform. Not only Android, but the entire web platform. In fact, fairly recent Google App Inventor project is almost exactly the same thing, making apps for android (and it would be awesome if it were possible to make android apps on a nice android tablet, like the Galaxy Tab). Because the App Inventor/MIT Scratch/NXT-G style drag-drop block manipulation programming interfaces are natural and intuitive on a touchscreen.
Greetings and props on your technical accomplishments,
I sent the following letter to Apple at http://www.apple.com/feedback/ipad.html after a little fact finding from your blog. Hope others will make their wishes know to Apple too. Cheers, Doug
Dear Apple Employee,
I am interested in an iPad for my 5 year old for it’s educational apps like ABC magnetic letters, whiteboard lite, brushes, etc. I also was thinking about how it could be used to control Lego Mindstorms robots, but when I started looking it appears that this is not doable because of lack of access to the bluetooth stack. I think access to the Bluetooth stack or another solution to marry Lego Mindstorms with the iPad would be greeted with great enthusiasm by many robotics enthusiasts.
Sincerely, Doug
I guess the best way would be to make an app that was an extension of the one already running on your PC/Mac. You could build your program and then push it to your pc which would then bluetooth it to the NXT.
Any action on this? my classroom is slated to get iPads, so I would love it if my students could program and download (USB is good in classrooms with many NXT bricks) their programs from the iPad. Otherwise we have to keep some older macbooks around for the job.
The most recent version of iOS contains a Bluetooth API. The bigger hurdle remains, however — that Apple refuses to allow compilers into the App Store.
I have had some success using Pocket Cloud on my iPad to allow my son to log into his account on our iMac and do some programming, even while my wife was logged into her account and physically sitting at the iMac (using OSX Lion.) it works, but since the Mindstorms IDE was designed for a physical keyboard/mouse, it can be cumbersome.
Apple allows third-party software to generate/execute code. They changed their policy in Sep 2010.
Look here: http://www.apple.com/pr/library/2010/09/09statement.html
“…we are relaxing all restrictions on the development tools used to create iOS apps, as long as the resulting apps do not download any code. This should give developers the flexibility they want, while preserving the security we need.”
An important point to understand is that the restriction to “not download any code” is actually to not download code that’s executable/executed on the iOS device.
In the case of an alternate NXT-G-type app, a RBT file/code would be created/edited on iOS, and then forwarded to an NXT for execution. The iPad would be only editing “code” that it can’t execute. I don’t believe we want the iPad to execute RBT files. Apple’s license doesn’t prohibit any of this.
Hey, how to download that one?
Here is the link. NXT-G for iPad.
http://v.youku.com/v_show/id_XNTkyODg4ODU2.html
You can download it from apple app store or cydia now.
Good day! I could have sworn I’ve visited this website before but after
looking at many of the articles I realized it’s new to me.
Anyways, I’m definitely pleased I came across it and I’ll be book-marking it and checking back often!