Back in 2006, Grant Jordan and I theorized we could make a profit buying stocks that were being touted by email spammers.  We developed a method to visually fingerprint individual spammers and track their market returns over time.  The method worked for a few weeks and earned us modest gains, but eventually some unexplained macro effects permanently diminished the effectiveness of email spam and we had to give up.

This video is Grant’s presented on our findings at Defcon 17.  It’s long, but Grant’s style is quirky and entertaining.  Definitely take a look.

3d model of S&G 8500 safe cracker

Before building this robot, we built a machine that could crack an S&G 8400 series lock.  These locks have a butterfly knob in the center of the dial that must rotated to open the safe.  To learn more, check out my post about our original robot.

EDIT: Wow, it seems like this type of post is pretty popular right now. We just made a 10 CEO’s Under 30 List by Impact Lab.

However, I did hear from several people that they found the content of my talk very useful, so I suppose that redeems it.  Thanks to Michael Grinich for organizing this event and inviting me to speak!

Guts of the Justin.tv camera system

The second camera prototype was a marvel of RadioShack engineering.  It was a pack of 4 EVDO modems linked to our servers with a custom network protocol that could adjust the video quality based on available network bandwidth.  The box accepted any analog camera and used an Axis MPEG4 encoder to compress the video.  A small Linux-based computer read the compressed stream over ethernet and blasted the bits out USB ports to the bank of modems.

The project took three months, which doesn’t seem like much, but unfortunately it pushed the company’s launch date back by about two months.  I think the end result was worth it, though.  The video quality was good enough to make people question whether Justin’s head cam was actually live.  Once it was all ready, we gave our friend Nick Gonzalez over at Techcrunch the exclusive story on our launch.  His article launched a 3-month barrage of Justin.tv publicity ranging from the Today Show to the New York Times.

A few months after the launch, Brian Lam of Gizmodo did a phone interview with me and turned it into a colorful story about the creation of Justin’s camera.  Fortunately, since Gizmodo is a tech blog, he included the juicy technical details that over news sources omitted.

EDIT (1/10/10): Interestingly enough, a well-funded company called LiveU appears to have commercialized a strikingly similar idea.  Take a look at their backpack transmitters.

The truth is that I was sold on the opportunity to build cool hardware.  My due diligence on the team consisted of googling Justin and Emmett to make sure they weren’t business school predators or serial crackpots.  They sold their first company on Ebay for a quarter million bucks, which I thought was pretty legit, so I agreed to spec out a prototype camera after our first meeting.  They said I was competing against another “consultant”, so I went overboard and prepared a 20 page engineering document, including 3d renderings, video codec research, and production cost estimates.  It was worth a bit more than the $750 I was paid for it, but I believe in making good first impressions at any cost.

Well, it worked.  Justin and Emmett funded the construction of my first camera prototype, which was little more than a laser-cut box filled with rechargeable batteries, a wireless modem, and an MPEG2 video server.  Since it was all off-the-shelf stuff, I didn’t have to write a single line of code.  I shipped the box out to San Francisco, and Justin promptly strapped it to his back and streamed himself driving a go-kart around the city in a viking costume.  I wasn’t quite sure what to make of that.  Nor were my parents, who were looking the reason I was threatening to leave MIT for a few months.  Instead of a real explanation, they got a grainy go-kart video.

Regardless, I was convinced that even if the business flopped, we’d have a good time doing it.

Read Part 2 of Justin.tv History

I responded to a summer job offer to work at Panasonic Boston Laboratory doing robotics research.  It turns out that an engineer there just wanted to see if he could find an intern to whip up a cool robotic wheelchair to help disabled people navigate their homes.  He wanted it done over the course of a few months, which is the length of a typical summer internship at Panasonic.  I had my doubts about the timeline, but I was still happy to take him up on the offer.  We set a few long term goals, put together a budget, and starting buying parts.

The wheelchair started as a Jazzy Powerchair.  I stripped the heavy lead-acid car batteries in favor of some smaller gel cell batteries.  It had really powerful gearmotors, but I dropped the voltage down to 12v to extend the battery life and make things a bit safer.  The joystick interface looked fairly cryptic, so I ordered a Roboteq motor controller to wire up to the single board computer running Linux.  One of the project goals was to make every wheelchair modification easy to replicate.  I had a rudimentary picture of design for manufacture in mind. We had a steel frame welded at a local shop.  It contained the batteries, computer, motor controller, and power supplies.  I ended up with a nice, self-contained “power box” suitable for running anything from a wheelchair to a Battlebot.  The entire frame lowered directly into the space where the two large car batteries once lived.

Screenshot of object tracking code

One of my favorite sensors for use in robotics is the SICK Lidar, but that just didn’t make sense for a project like this.  Few people can afford an extra $6K for that kind of sensor, and it certainly wasn’t in the Panasonic budget.  I chose analog infrared sensors with a range of about one meter and sonar sensors with a range of several meters.  I ordered eight of each sensor and wired them to give close to 360 degree coverage of the perimeter of the wheelchair.

I also used this project as an excuse to dive into basic machine vision.  I played with stereoscopic cameras for a while when working on the DARPA Grand Challenge project, but in this case a simple webcam like the QuickCam Pro 5000 worked fine.  In fact, I enjoyed writing software for the webcam under linux so much that it spawned an a little robotics library.  Anyways, I didn’t get very far with the camera other than writing an library for it and doing a couple basic demos.  I did get the wheelchair to follow around a person wearing red, which basically killed productivity at Panasonic Boston Laboratory for the afternoon.

Layout of custom sensor board

As usual, I chose to build my own sensor processing electronics for this project instead of just buying something off the shelf.  In hindsight, it would have made more sense to just buy a USB data acquisition board, but I enjoy making these kinds of boards, and it’s at least a good learning experience.  This one used an ATMEGA128 to read the sensors and communicate with the Linux SBC. My supervisor also had some data-logging projects in mind, so he had me add a few connectors and write software for several other sensors.  It ended up supporting temperature sensors, thermocouples, flow meters, and some weird light sensor.  The resulting product was a pretty useful board, but it probably wasn’t worth the week it took to design the pcb and write software and firmware for everything.

At the end of the summer, Panasonic generously agreed to donate my robot wheelchair to the student-run robotics lab sponsored by MIT’s Edgerton Center.  I’ve been heavily involved in the group’s projects so far.  We’ve entered the DARPA Grand Challenge with an $80,000 robotic pickup truck and built a $30,000 robot to give tours to prospective MIT students.  Needless to say, I’m glad the wheelchair ended up in the hands of other robot enthusiasts.

I wrote this program because Ruby on Rails is hard to scale.  Or at least it was.  I don’t know if this is still the case.  Regardless, Twice does some really cool things and does many of them faster than Rails, so it’s worth checking out.

When my cofounder Emmett Shear and I sat down to brainstorm ways to make our website faster, we tried to imagine what kind of system could serve pages in just a few milliseconds.  The only think we could come up with to hit that kind of performance involved rendering as much of a page as possible directly out of memory – or even the whole page.  We ended up designing a system that caches about 99% of a fully rendered page and renders the other 1% on the fly.  The end result is that about 95% of Justin.tv’s half a billion monthly dynamic requests are served in under 10ms.

In-memory caching is certainly not a new concept, and Rails actually has some pretty good “partial” caching as well as full page caching.  But Justin.tv’s load profile didn’t mesh well with these caching strategies, so we had to try something else.  Since our visitors all show up at the exact same moment and create traffic spikes 10x higher than normal, there is no time to spend rendering complex templates or running database queries.  Yet somehow we still need to customize every page for logged in users and update pages to reflect changes as the content expires.  Twice was written to protect our application servers during these load spikes but also maintain a customized feel to each user .

Twice is a proxy server that sits between a hardened webserver like nginx or Apache and one or more application servers.  First, the webserver proxies incoming requests to Twice.  Twice inspects the HTTP headers and checks to see if the page has been rendered and is stored in memory.  If the page exists, Twice copies the page out of memory and customizes the rendered page for the user who requested it.  If the page isn’t found in memory, Twice proxies the request on to the application servers and caches the result if possible.

Here are some of the more interesting features of Twice:

• Cache backends supported: memcached (via pylibmc), memcached (via python), python dictionary
• Fetch a user session from a database, cache it, and render user attributes into a page
• Fetch memcache keys from a “backend” memcache and cache them in the Twice cache
• Request pileup prevention by using soft expiration
• Scales horizontally to any number of cache processes and memcached servers
• Explicit expiration across all cache processes without any IPC overhead
• OOM killer and file descriptor limit discovery
• Geoip, remote ip, memcache key, etc. can be rendered into a page or header
• Short-cache certain response codes
• Hash pages based on browser language header

Twice is open sourced under the MIT license.