innovate – tweak, hack, and bend technology
DESCRIPTION
We will take a small device (available for $100 at Dallas Semiconductor), about the size of an iPod Shuffle, but capable of executing Java-Byte-Code, and connect it to a digital toy camera, (available for about $10 on eBay). Then we are going to write some Java code that we load on to the device and execute and even run some JUnit tests remotely. But be warned, we are targeting an 8-bit processor running at only 40 MHz and its 1MB SRAM serves as file-system, program memory, and runtime heap. Along the way we may add some more hardware and learn a few things about how CMOS sensors capture image data and how a Bayer-pattern can be used to decode the sensor data.TRANSCRIPT
Wolf Paulus
Wolf Paulus . com
Innovate: Tweak, Hack, and Bend Technology
© 2006 Wolf Paulus 2
Renaissance The period of European history at the close of the Middle Ages and the rise of the modern world; a cultural rebirth from the 14th through the middle of the 17th centuries.
Ren • ais • sance
© 2006 Wolf Paulus 3
© 2006 Wolf Paulus 4
Glass Gave Renaissance Paintings Their Glow
By looking beyond the limits of their usual practice and transforming materials from other trades, the great artists of the Renaissance created a palette that gave them an immediate and lasting reputation as brilliant colorists.
© 2006 Wolf Paulus 5
• To tweak, hack, and bend technology you need to know some of the Technology behind the Technology
• Look beyond the limits of your usual practice and transform materials and ideas from others.
Innovate -Tweak, Hack, and Bend Technology.
© 2006 Wolf Paulus 6
Hacking Nick Click, a digital toy camera.
© 2006 Wolf Paulus 7
Serial Port :
TXGroundRX
Power:9V Battery
Power
Shutter
© 2006 Wolf Paulus 8
9V Power
ShutterOn/Off
Serial Port :
TXGround
RX
© 2006 Wolf Paulus 9
Microcontroller vs. Microprocessor
• Microcontroller chips and most embedded devices embody the Harvard architecture, which has separate memories for programs and data.
• Microprocessor based machines usually have a von Neumann architecture, with a single memory for both.
© 2006 Wolf Paulus 10
Image Sensor, Key Features:
• Effective Resolution 160 x 120
• Color filters form a Bayer Color Pattern
• “Software is required to perform color processing.”
[Source: 50 pages ST VV5301 & VV6301 Mono and Color QSIF Digital Video CMOS Image Sensors]
© 2006 Wolf Paulus 11
Filter Array Sensor
• Only one color can be acquired per pixel.
• Most commonly used is the Bayer Pattern because of the eye’s higher sensitivity to green, 50% of the Bayer pattern consists of green information.
© 2006 Wolf Paulus 12
Decoding the Bayer Pattern
© 2006 Wolf Paulus 13
Shuffled Pixel Readout ..
Crosstalk: Parasitic capacitance between tracks can cause the fast edges of high-speed logic to propagate from one signal line to another.
© 2006 Wolf Paulus 14
A1 B1 C1 D1
E1 F1 G1 H1
A2 B2 C2 D2
E2 F2 G2 H2
8- 1x8 WDM
Silica-on-Silicon waveguides
A3 B3 C3 D3
E3 F3 G1 H3
A4 B4 C4 D4
E4 F4 G4 H4
A1
B1
C1 D3
E1 F1
G1
A2
B2
C2 D2
E2 F2
G2 H4
A3 B4
C3
E3 F3
G4 H1
A4 B3
C4 D1
E4 F4
G3
H3
D4
H2
Folded 128 x 128 Shuffle circuit
A 1
B 1
C 1
D 1
E 1
F 1
G 1
H 1
A 2
H 2
G 2
F 2
E 2
D 2
C 2
B 2
A 3
B 3
C 3
D 3
E 3
F 3
G 1
H 3
A 4
H 4
G 4
F 4
E 4
D 4
C 4
B 4
LiNbO3
64- 1x4switch array
A 1
B 1
C 1
D 1
E 1
F 1
G 1
H 1
A 2
H 2
G 2
F 2
E 2
D 2
C 2
B 2
A 3
B 3
C 3
D 3
E 3
F 3
G 1
H 3
A 4
H 4
G 4
F 4
E 4
D 4
C 4
B 4
Inputs
Outputs
.. leads to shorter Tracks
© 2006 Wolf Paulus 15
Artifacts due to color interpolation
© 2006 Wolf Paulus 16
The red, green, and blue photodetectors are located at different depths within the sensor.
Layered Image Sensor
© 2006 Wolf Paulus 17
For every pixel, three photodiodes are stacked like the three layers in color film. This arrangement utilizes the wavelength-dependent light absorption property of silicon to produce natural filters that use the incoming light to greatest advantage.
Layered Image Sensor
The red, green, and blue photodetectors are located at different depths within the sensor.
© 2006 Wolf Paulus 18
Filter Array vs. Layered Sensor
© 2006 Wolf Paulus 19
Communication Protocol
STX Command Data 1 ... Data n ETX
ACK STX Response Data 1 ... Data n ETX
• STX = ASCII character (02h)• Command = defined data byte• Response = defined response byte• Data = command specific parameter data• ETX = ASCII character (03h)• ACK = Acknowledge (06h)• NAK = (15h)
© 2006 Wolf Paulus 20
Example 1: Request Firmware Version
02h V 0 03h
06h 02h v 1 . 6 9 03h
Send Request via RS232 port at 57.6 kbps / 8bits / No Parity
Expected Response:
© 2006 Wolf Paulus 21
Example 2: Take a Picture
02h Cmd param 03h
06h 02h cmd error code response .. 03h
• Reset Image Counter to 0❷,A,0,❸ ⇒ ⇐ ❻,❷,a,0,❸
• Grab Image (Parameter defines delay and retries)❷,G,127,❸ ⇒ ⇐ ❻,❷,g,0,❸
• Upload Image (Parameter defines image index)❷,U,0,❸ ⇒ ⇐ ❻,❷,u,N1,..,N4, D1,..,Dn,❸
© 2006 Wolf Paulus 22
Java Code to drive the DigiCam
• cmd - array containing the command code, e.g.STX, CommandByte, ParameterByte(s), ETX
• size - number of data bytes expected on inputstream,excluding 4 bytes for ACK STX cmd ETX
• return - response bytes or null, if unsuccessful, e.g.ACK, STX, commandByte, DataBytes, ETX
public byte[] perform(byte[] cmd, int size);
© 2006 Wolf Paulus
private byte[] perform(final byte[] cmd, final int size) { boolean ok = false; boolean busy = false; int counter = 0; byte[] buffer = new byte[size + 4]; // data + ACK,STX,RESPONSE,ETX
try { // Flush the input stream while (in!=null && in.available() > 0) in.read(); } catch (IOException e) { // intented empty }
do { try { out.write(cmd); // Send Command out.flush(); busy = false;
int off = 0; // the number of bytes already read. int len = buffer.length; // the number of bytes still to read.
© 2006 Wolf Paulus
while (len > 0) { // Reads up to len-off bytes from the input stream into buffer. int k = in.read(buffer, off, len); if (k <= 0) // k is the number of bytes actually read. throw new IOException("Couldn't finish reading input file."); off += k; len -= k; }
if (buffer[0] != VV6301.ACK) // Expected: ACK,STX,cmd,..,ETX return null; ok = buffer[2] == (byte) (cmd[1] + RESPONSE_OFFEST); if (!ok) busy = buffer[2] == VV6301.RB_BUSY; } catch (IOException ioe) { System.err.println(ioe.getMessage()); } } while (!ok && busy && (++counter < VV6301.RETRIES)); return ok ? buffer : null; }
© 2006 Wolf Paulus 25
Serial Port, still a “must have”
• Keyspan High Speed USB-Serial Adapterhttp://www.keyspan.com/products/usb/USA19HS/
• Java communications libraries for OS Xhttp://prdownloads.sourceforge.net/jmri/JavaCommInstaller2.hqx
• Installation instructions for Mac OS Xhttp://jmri.sourceforge.net/install /MacOSX.html
© 2006 Wolf Paulus 26
Dallas Semiconductor: TINI board 390
© 2006 Wolf Paulus 27
TINI- Hardware
Ethernet Controller
512KFlash ROM
Two 512K SRAM
36 MHzCPU
Serial-IOController
384K Java Runtime 64K Shell
File Sys.Java Prog.Res.Files
© 2006 Wolf Paulus 28
• 8 Processes (incl. GC and Shell)
• 32 Threads per Process
• 24 Socket Connections
• A compiled class file cannot be larger than 32k
• 64k is the maximum size of any array
TINI’s Java Limitations
© 2006 Wolf Paulus 29
• A class is limited to 255 static and 255 instance member variables. (includes inherited members)
• A class method is limited to 63 local variables.
• Only UTF8 and ISO 8859_1 are supportedwith ByteToChar and CharToByte
• Only the default Locale is supported.
• There are a few more limitations that concern Classloading and Reflection.
more Java Limitations on the TINI
© 2006 Wolf Paulus 30
TINIConverter is a Java Program
Command line tool which outputs a *.tini file
StringBuffer sb = new StringBuffer(); sb.append(“this is a test”);
Create new object “java.lang.StringBuffer” Push reference to “this is a test” on the stackInvoke method “java.lang.StringBuffer.append(String)”
Create new object with class number 0x54 Push reference to “this is a test” on the stack Invoke method 0x12 of class 0x54
Reduces the constant pool and speeds up class loading
TINIConverterJa
vaP
seud
o B
yte
Cod
eTI
NI
File
© 2006 Wolf Paulus 31
Socket for the TINI board 390Null ModemSerial Port
5V Power10
-Bas
e-T
Eth
erne
t (+)
LEDR 330 Ω
(-)
zoom in on next slide
© 2006 Wolf Paulus 32
GREEN LED
330 Ω
VC
C =
5V
DC
Port 5
Bit 3 Por
t 5Bit
0
RED LED
330 Ω
© 2006 Wolf Paulus 33
Ohm’s law: U/R=I
3V / 330Ω = 9 mA
Tweaking Processor Port Pins
BitPort led = new BitPort(BitPort.Port5Bit0);led.clear(); // turns it onled.set(); // turns it off
• Driving an LED needs around 10 mA to make it clearly visible.
• A Port Pin can supply about 50 μA but can sink current up to 15 mA.
• LEDs have a voltage drop of about 2V. Therefore, the voltage across the Resistor is about 3V.
© 2006 Wolf Paulus 34
TINI in action ... (typical setup)
CAT5 Cable
TINI 390
Serial-Cable (3-wire)
Router
Nick Click
© 2006 Wolf Paulus 35
TINI in action ... (development setup)
CAT5 Cross Over Cable
5 VDC Power
Serial Cable TINI-390Nick Click
PowerBook
© 2006 Wolf Paulus
Demo
© 2006 Wolf Paulus
Demo
37
DS80C390: 192.168.200.6DS80C390: 192.168.200.5DS80C400: 192.168.200.4
Compact Wireless-G Broadband Router - WRT54GC Gateway 192.168.200.1 / 255.255.255.0
© 2006 Wolf Paulus 38
TiniNick -- Camera Driver & Image Server
Grabs Photos from Nick Click
Serial Port /Camera Driver Module
Access Properties Files
accepts client requests, adds new client sockets into Vector
streams new images to all Sockets in Vector
© 2006 Wolf Paulus 39
NickViewer -- Image Viewer Client
Converts 20KB bayer-encoded byte array in 60KB RGB-Bitmap.
Opens TCP Socket to TickNick Image Server.Updates Observer when new image arrives.
Builds Viewer GUI and observes SocketConnector for new images.
© 2006 Wolf Paulus 40
Shuffled Pixel Readout
© 2006 Wolf Paulus 41
Before Bayer Decoding
© 2006 Wolf Paulus 42
After Bayer Decoding
© 2006 Wolf Paulus 43
Image Resolutions Compared ...
Nick Click 160 x 120 1
Motorola V551 640 x 480 16
Canon PowerShot SD200
2048 x 1536 164
Canon EOS 350D / Digital Rebel XT
3456 x 2304 414
© 2006 Wolf Paulus
DS TINIm400 (DS80C400 MC)DS TINIs400 Socket Board
© 2006 Wolf Paulus
DS TINIm400 (DS80C400 MC)
© 2006 Wolf Paulus
DS TINIm400 (DS80C400 MC)
© 2006 Wolf Paulus
DS TINIm400 (DS80C400 MC)1 MByte
Flash Mem
DS80C400 MC with integrated Ethernet
Controller
© 2006 Wolf Paulus
DS TINIm400 (DS80C400 MC)2 x 512
KByte SRAM
© 2006 Wolf Paulus 49
•Hacking a $10 Digital Toy Camera.
•Tweaking Micro-Controller Ports
Hack, Tweak, and Bend Technology
© 2006 Wolf Paulus
Mont Ventoux (1,909 m; 6,263 ft) is by far the biggest mountain in the Provence, a region of southern France.
On April 26, 1336, Francesco Petrarca climbed to the top of Mont Ventoux just to see the view, which at the time was rather unusual.
This event is often cited as the start of the Renaissance.
50
© 2006 Wolf Paulus 51
Resources• Textbooks
• Don Loomis - TINI Specification and Developer's Guide - ISBN: 0201722186PDF version online: http://www.maxim-ic.com/products/tini/devguide.cfm
• Ken Arnold - Embedded Controller Hardware Design - ISBN: 1878707523• DeMuth, Eisenreich, Designing Embedded Internet Devices, ISBN: 1-878707-98-1
• Image Processing / Bayer Pattern• Tommy Olsen, Jo Steinar Strand, Agder University College Faculty of Engineering:
"An improved image processing chain for mobile terminals", May 2002http://student.grm.hia.no/master/ikt02/ikt6400/g23/Report.htm
• Diagnostic Instruments, Inc.(http://www.diaginc.com)Elements of Sensor Resolutionhttp://www.diaginc.com/downloads/public/pdf/3ShotColorMosaic.pdf
• Data sheets and Drivers• CPiA webcam driver for Linux (http://webcam.sourceforge.net)
RS232 command protocol specs (scicomms.doc, 10 pages, 82Kb).http://webcam.sourceforge.net/barbie/scicomms.doc
• VV6301 Mono and Colour QSIF Digital Video CMOS Image Sensorshttp://www.datasheetarchive.com/search.php?search=vv6301&sType=part
• Tini DS80C390 and Tini DS80C400http://www.maxim-ic.com/pl_list.cfm/filter/tini
• TINI, DS80C400 Evaluation Kit• http://www.maxim-ic.com/getds.cfm?qv_pk=4983• http://www.maxim-ic.com/TINIplatform.cfm
© 2006 Wolf Paulus
The End
© 2003-2006 Carlsbad Cubes© 2006 wolfpaulus.com
One more thing ...
TiffanyScreens
© 2003-2006 Carlsbad Cubes© 2006 wolfpaulus.com
Tiffany Screens
The term "democracy" was coined in ancient Athens in the 5th century BC. All Athenian citizens were free to speak in the Assembly.
“PowerPoint-ation”
• Lecture Style Presentation
• One guy presents everybody else is asleep
• Changing Presenters severely interrupts a meeting, requiring change of location, unplugging, etc.
• Presenter need to adjust Laptop’s screen resolution and refresh-rate.
• Since a picture is worth a 1000 words, it’s hard to win an argument against the presenter, even if you have your facts right.
0
25
50
75
100
20042005
20062007
TiffanyScreens
• Detects who is running the same application. aka your Peer Group
• One peer at a time broadcasts her screen onto other watching peers’ screens
• Seamlessly changing broadcasters
Group Dictionary
Toolbar
• Show your screen to your peers - exit [ESC]
• Watch the presenter’s screen - exit [ESC]
• Pick your Conference room
• Look who’s currently where, doing what
• Change your network settings etc. - if you really think you have to -
Statusbar
• Status LED• blue = idle• green = ready to receive• orange = receiving
• heartbeat - sharing your apps attributes• Confidentiality features• Number of peers, running this app.• Preferred watching window resolution
© 2003-2006 Carlsbad Cubes© 2006 wolfpaulus.com
TiffanyScreens.com
© 2003-2006 Carlsbad Cubes© 2006 wolfpaulus.com
Thanks for coming