Create your own sixthsense device

Hi friends you are all know that Pranav Mistry developed a Sixth sense technology, which is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information.

He reveals his Sixth Sense technology project to the world as opensource..

HARDWARE: 

Camera

The camera is the key input device of the SixthSense system. The camera acts as a digital eye of the system. It basically captures the scene the user is looking at. The video stream captured by the camera is passed to mobile computing device which does the appropriate computer vision computation. The major functions of the camera can be listed as:

• Captures user’s hand movements and gestures (used in reorganization of user gestures)
• Captures the scene in front and objects the user is interacting with (used in object reorganization and tracking)
• Takes a photo of the scene in front when the user performs a ‘framing’ gesture
• Captures the scene of projected interface (used to correct the alignment, placement and look and feel of the projected interface components)

Suggested Product: Logitech QuickCam Pro for notebooks

Projector

The projector is the key output device of the SixthSense system. The projector visually augments surfaces, walls and physical objects the user is interacting with by projecting digital information and graphical user interfaces. The mobile computing device provides the projector with the content to be projected. The projector unit used in prototype runs on a rechargeable battery. The major functions of the projector can be listed as:

• Projects graphical user interface of the selected application onto surfaces or walls in front
• Augments the physical objects the user interacting with by projecting just-in-time and related information from the Internet

Suggested Products: You can buy either laser (AAXA, Microvision) or L.E.D (3M MPro110) projectors.

Mirror

The mirror reflects the projection coming out from the projector and thus helps in projecting onto the desired locations on walls or surfaces. The user manually can change the tilt of the mirror to change the location of the projection. For example in application where the user wants the projection to go on the ground instead of the surface in front, he can change the tilt of the mirror to change the projection. Thus, the mirror in the SixthSense helps in overcoming the limitation of the limited projection space of the projector.

Suggested Product: Any 1”X1” first surface mirror

Microphone

The microphone is an optional component of the SixthSense. It is required when using a paper as a computing interface. When the user wants to use a sheet of paper as an interactive surface, he or she clips the microphone to the paper. The microphone attached this way captures the sound signals of user’s touching the paper. This data is passed to computing device for processing. Later, combined with the tracking information about user’s finger, the system is able to identify precise touch events on the paper. Here, the sound signal captured by the microphone provides time information whereas the camera performs tracking. The applications enabled by this technique are explained earlier.

Suggested Product: Microphone of Logitech QuickCam pro for notebooks

Mobile computing device

The SixthSense system uses a mobile computing device in user’s pocket as the processing device. The software program enabling all the features of the system runs on this computing device. This device can be a mobile phone or a small laptop computer. The camera, the projector and the microphone are connected to this device using wired or wireless connection. The detail of the software program that runs on this device is provided in next section. The mobile computing device is also connected to the Internet via 3G network or wireless connection.

Suggested Product: Any Windows computer

Now that you have all these pieces, you need a way to combine them. We recommend using Lego strips to form the base. The projector, camera, and mirror assembly can be directly put onto this base. You can also use Velcro to combine the products.

Suggested Hardware Components

These are the basic pieces that you should buy. You can choose any brand, and the following list are those items that worked well for us:

• Pico-Projector
• Laser
• AAXA ~$110
• Microvision ~$200
• L.E.D
• 3M MPro110 ~$270

• Camera
1. Logitech QuickCam Pro for notebooks ~$23

• Microphone
1. Microphone of Logitech QuickCam pro for notebooks ~$23

• A mirror assembly
1. Front faced mirror is best, 1”X1” first surface mirror (this can be purchased from anywhere, i.e. Ebay)

• Laptop Computer: Any Windows computer (this will act as the mobile computing device)

SOFTWARE:

How to run the Software Component of SixthSense

The prototype system runs on windows platform and majority of the code is written in C++ and C#. We will be uploading newer versions as it is being developed; this will also include a mobile version.

NOTE: We are moving the new code to Git this week. Meanwhile you can download it at the link below.

WUW v0.1 beta

Download

v0.1

or you can download latest release tag from: github

Instructions for running WUW_v0.1

WUW_v0.1 is the beta version of SixthSense code (WUW stands for Wear Ur World, a former name of the project). This version runs on Windows. Make sure you have Visual Studio and Direct X installed. Extract and copy the files from wuw_v0.1.zip to any location on your computer. Open WUW01.sln (which is present in the folder "CODE") then click on F5 to debug it. If you debug it successfully, a black screen appears with a little square box on the right just hover your mouse around it and you will find Option tabs such as Camera, Tokens and Apps.

Camera: you can choose which camera to use and configure the camera settings here. You may need to mirror the camera here using the camera settings. It is also recommended to use fixed/manual white balance and exposure such that the camera image is stable and light independent.

Tokens: This version of code uses colored fingertips as markers. One need 4 different colors in order to run most of the Apps. We recommend to use Red, Yellow, Blue and Green (or colors that are different enough). We have also noticed that the matt colored tape/objects works better as they have less reflective changes. When you click on ‘Add Markers’, the screen starts blinking and after 4-5 secs it stops with a still picture so that you can add those markers on your hand as new markers. You can simply drag-select the colored marker on the screen by mouse cursor. Once you have all four markers (O, P, M, and N) added the software will start tracking the four markers. You can set the thresholds to better the tracking. You can save those markers and can reuse in subsequent executions, or can add new markers every time.

Apps: there are three ways to launch any application. Apps tab can let you choose the apps you want to load and run. One can also load apps from ‘Menu’. ‘Menu’ is triggered using Namaste gesture. Once on Menu screen you can choose the App by hand-click (Hand Click gesture works by moving the O marker and hiding P marker.

(Some of the Apps require some particular settings. There are also several types of gestures that are supported as of now. Details on Applications and Gestures will be added soon here.)

You have to do....

• After downloading the code from Github, uncompressed the zip file to any location in your computer (Ex: F:\sixthsense-sixthsense-4e155fc)
• Now open MS Outlook for running the Mail app( without outlook you may get errors )
• Now click on WUW01. Fig.1
• A Window appears Fig.2. Now click F5 to debug it.(Make sure you're connected to the internet)
• A Black window appears as in Fig.3. There are certain options like camera,Tokens and apps

Camera:

• Here you can select the camera you want to use from the drop down options.
• If you are having multiple cameras you can select from those option or if you have single camera it will automatically start capturing.
• You can also adjust your camera settings (Brightness,contrast,..etc) by clicking on "Adjust Camera Properties".

Tokens:

• Click the "New Marker" button, you will see the camera flickering 3 times, during this period you will have to position you fingers (with the markers) in front of the camera and on the 3rd flicker you will have a still image.

• Now simply drag the mouse over the markers with small circles that should only cover the part of the marker,repeat this process four times for the four markers (M,N,O & P)
• Add the colour markers in the order : Red ,Yellow ,Blue & Green Fig.4

• You can also check this video for adding colour markers :

APPS:

Some apps are listed below:

Test:

• You can see whether the four markers are being tracked and moving while you are moving those particular markers that is on the fingers.Fig.5

Draw:

• The red marker (M) draws when the yellow marker (N) is hidden.When both the markers are visible you can move the mouse pointer to select different colours or eraser,etc Fig.6

Photo:

• Make a rectangle with your pointer fingers and thumbs and when the gesture is recognized,the countdown starts
• Remove you hands from the camera frame to capture the desired photo. (The countdown lasts for 3 secs) Fig.7

Gallery:

• Displays the photos that you have taken Fig.8

Mail:

• Displays a list of email from outlook (required that MS Outlook be running for this to work)

Weather:

• Displays a weather app that can be localised to your area.Fig.9

Stock:

• Displays 16 stocks values real time, use the slider at the bottom of the projected display to set the time scale.

Effects:

• Play with beautiful wavy lines and explore your imagination.Fig.10

Newspaper:

• It projects newspaper stories and related videos from Youtube or other web sites,directly onto the surface of the paper.Fig.11

Book:

• For book-recognition,the program gets activated with a gesture, and the system snaps a photo of the book, compares it with book-cover images it finds online, computes a match, and retrieves and projects the ratings.Fig.12

Clock:

• An Analog clock will be projected by doing a circle gesture. Fig.13

Menu:

• Displays different menu icons.Fig.14

(Note: Click on the Figs. to get the pictures.)

Configuring the code at Google (wuw_v0.1):

Same Requirements as written above.

• Download the code from wuw_v0.1 & follow the steps as written above.

How to add colour markers

• You can also follow the steps above to add the markers or check this video :

How to remove the Black box ?

This is an error where some part of the screen is covered with a black box.

• To remove the Blackbox follow the steps :

• Click on the WUW01.sln file ( In the folder named "Code" )

• Click on WUW.cs ,highlighted area & do as written in the Pic.1 like

• Right click the black box which is present in the extreme left.

• Click on the option send to back and it gets hidden.

• You can also check this video :