debugging: rules & tools
DESCRIPTION
Finding and fixing bugs is a major chunk of any developers time. This talk describes the basic rules for effective debugging in any language, but shows how the tools available in PHP can be used to find and fix even the most elusive errorTRANSCRIPT
![Page 1: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/1.jpg)
DEBUGGINGR U L E S A N D T O O L Sian barber - http://phpir.com - @ianbarber
![Page 2: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/2.jpg)
![Page 3: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/3.jpg)
I did XI wanted YInstead I got Z
![Page 4: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/4.jpg)
http://www.debuggingrules.com
![Page 5: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/5.jpg)
UNDERSTAND THE SYSTEM
RULE 1
![Page 6: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/6.jpg)
RTFM
Framework
Wiki
PHPDoc
php.net
BugTracker
![Page 7: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/7.jpg)
$field = $document->getField($fieldName);
if ($field->storeTermVector) { /** * @todo term vector storing support */ require_once 'Zend/Search/Lucene/Exception.php'; throw new Zend_Search_Lucene_Exception ('Store term vector functionality is not supported yet.');}
![Page 8: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/8.jpg)
TOOLS
Komodo
Eclipse Zend Studio
NetBeans
PHPStorm Text
Mate
![Page 9: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/9.jpg)
MAKE IT FAIL
RULE 2
![Page 10: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/10.jpg)
Recreate And AutomatePHP UnitSimpleTestSeleniumPHP SlimJMeterAb
![Page 11: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/11.jpg)
CONTROL VARIABLES
LoadTime
DBStateSession
Request
ServerEnv
User
Client
Timing
![Page 12: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/12.jpg)
class Test_User extends PHPUnit_Framework_TestCase {
public function testUpdate() { $r = ("a", "b"); try { $u = User::fetch($this->newName); $result = $u->update($r); } catch (Id_Exception $e) { $this->fail($e->getMessage()); } $this->assertEquals($r, $u->getRoles()); } }
[root@localhost ~]# pear channel-discover pear.phpunit.de[root@localhost ~]# pear install phpunit/PHPUnit
![Page 14: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/14.jpg)
class Example extends PHPUnit_Extensions_SeleniumTestCase { protected function setUp() { $this->setBrowser("*chrome"); $this->setBrowserUrl("http://general.dev/"); }
public function testMyTestCase() { $this->open("/"); $this->select("locale", "label=de"); $this->waitForPageToLoad("30000"); $this->assertTrue($this->isTextPresent( "Datenbank-Einrichtung")); }}
![Page 15: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/15.jpg)
http://jakarta.apache.org/jmeter/
![Page 16: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/16.jpg)
QUIT THINKINGAND LOOK
RULE 3
![Page 17: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/17.jpg)
if($this->hasRoles) { $this->deleteRoles(); }
try { foreach($submittedRoles as $role) { $this->addRole($role); } } catch(Identity_Exception $e) { $this->log->warn('Role add failed.'); throw new Service_Exception_BadRequest( 'Cannot add user role'); }
![Page 18: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/18.jpg)
Instrumentation:Inspect or Report
![Page 19: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/19.jpg)
Xdebughttp://www.xdebug.org
[root@localhost ~]# pecl install xdebug
zend_extension=/usr/lib64/php/modules/xdebug.so
![Page 20: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/20.jpg)
xdebug.profiler_enable=1 xdebug.profiler_output_dir=/tmp
![Page 21: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/21.jpg)
xdebug.auto_trace=1 xdebug.collect_params=4
![Page 22: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/22.jpg)
Key
Key
Debugging Commands
Web Server
PHP Script
XDebugDebugging Client
![Page 23: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/23.jpg)
xdebug.remote_enable=1 xdebug.remote_port=9000 xdebug.remote_host=192.168.192.1 #xdebug.remote_connect_back=1
![Page 24: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/24.jpg)
![Page 25: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/25.jpg)
![Page 26: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/26.jpg)
[root@localhost ~]# lsof -p 4365 COMMAND USER FD TYPE DEVICE NAME httpd apache mem REG 253,0 /lib64/libssl.so.0.9.8e httpd apache mem REG 253,0 /usr/lib64/php/modules/xdebug.so httpd apache mem REG 253,0 /usr/lib64/libsqlite3.so.0.8.6
[root@localhost ~]# strace -cp 4365 Process 6910 attached - interrupt to quit Process 6910 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ------------- 36.91 0.037307 466 80 getdents 30.07 0.030393 950 32 7 access 17.04 0.017224 154 112 10 open 5.08 0.005136 12 444 9 lstat 3.87 0.003910 18 213 1 read 3.04 0.003071 29 107 close 2.57 0.002598 1299 2 writev 0.78 0.000785 1 631 stat
![Page 27: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/27.jpg)
[root@localhost ~]# strace -Tp 4365 Process 6910 attached - interrupt to quit epoll_wait(16, {{EPOLLIN, {u32=31389672, u64=47128707528680}}}, 2, 10000) = 1 <6.926140> accept(4, {sa_family=AF_INET6, sin6_port=htons(64930), inet_pton (AF_INET6, "::ffff:192.168.192.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [47128676139036]) = 17 <0.000094> fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000035> read(17, "POST / HTTP/1.1\r\nHost: general.d"..., 8000) = 522 <0.000044> stat("/mnt/hgfs/habari/htdocs/", {st_mode=S_IFDIR|0755, st_size=408, ...}) = 0 <0.000527> open("/mnt/hgfs/habari/htdocs/.htaccess", O_RDONLY) = 18 <0.000457> fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000037>
![Page 28: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/28.jpg)
[root@localhost ~]# strace -Tp 4365 Process 6910 attached - interrupt to quit epoll_wait(16, {{EPOLLIN, {u32=31389672, u64=47128707528680}}}, 2, 10000) = 1 <6.926140> accept(4, {sa_family=AF_INET6, sin6_port=htons(64930), inet_pton (AF_INET6, "::ffff:192.168.192.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [47128676139036]) = 17 <0.000094> fcntl(17, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000035>
read(17, "POST / HTTP/1.1\r\nHost: general.d"..., 8000) = 522 <0.000044> stat("/mnt/hgfs/habari/htdocs/", {st_mode=S_IFDIR|0755, st_size=408, ...}) = 0 <0.000527>
open("/mnt/hgfs/habari/htdocs/.htaccess", O_RDONLY) = 18 <0.000457>
![Page 29: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/29.jpg)
error_log("Message for the PHP log") error_log("Log to email", 1, "[email protected]"); error_log("Log to a file",3, "/tmp/m.log"); error_log("Message for the SAPI log", 4);
ZendLog
sfLogger
Drupalsyslog
KohanaLog
![Page 30: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/30.jpg)
LOGGING SYSTEM
SyslogMQ File
Database PipeGearman
![Page 31: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/31.jpg)
FirePHP
![Page 32: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/32.jpg)
[root@localhost ~]# pear channel-discover pear.firephp.org [root@localhost ~]# pear install firephp/FirePHPCore
require_once('FirePHPCore/FirePHP.class.php'); ob_start(); $firephp = FirePHP::getInstance(true); doStuff($firephp); $firephp->warn('Done Stuff'); $firephp->error('An error!'); $firephp->log($_GET); ob_end_flush();
function doStuff($firephp) { $firephp->trace('Stuff Backtrace'); echo "This page intentionally blank"; }
![Page 33: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/33.jpg)
DIVIDE ANDCONQUER
RULE 4
![Page 34: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/34.jpg)
<?xml version="1.0" encoding="UTF-8"?><phpunit bootstrap="./TestHelper.php" colors="true"> <testsuite name="Commonlib Tests"> <directory>./library/Cl/</directory> <directory>./library/Sso/</directory> </testsuite><filter> <whitelist> <directory suffix=".php">../Sso/</directory> <directory suffix=".php">../Cl/</directory></whitelist></filter></phpunit>
![Page 35: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/35.jpg)
Folder Root
SSOCL
Model UserAuth Client
PluginAdapter Exception Abstract
GearmanSFDCSSO
![Page 36: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/36.jpg)
<?php
$worker = new GearmanWorker(); $worker->addServer(); $worker->addFunction( "allow_access", "allowAccess"); $worker->addFunction( "deny_access", "denyAccess");
while ($worker->work());
![Page 37: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/37.jpg)
git bisect start git bisect bad git bisect good <known good rev>
git bisect ?
git bisect reset
Git Bisect
![Page 38: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/38.jpg)
Request
Front Controller
InputFilter
Controller
Model
Plugin
View1
2
3
good bad
![Page 39: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/39.jpg)
Firewall
Cache
Web Server
Database
![Page 40: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/40.jpg)
Firewall
Cache
Web Server
Database
Wireshark
Curl
Netcat
MySQLProxy
Tamper Data
Xdebug
![Page 41: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/41.jpg)
Tamper Datahttps://addons.mozilla.org/firefox/addon/966
![Page 43: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/43.jpg)
![Page 44: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/44.jpg)
[root@localhost ~]# { echo -ne "HTTP/1.0 200 OK\r\n \r\n"; cat test.html; } | nc -l 8080 GET / HTTP/1.1 User-Agent: curl/7.19.7 (i386-apple-darwin10.0.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 Host: general.dev:8080 Accept: */*
[root@localhost ~]# curl http://general.dev:8080 -v * About to connect() to general.dev port 8080 (#0) * Trying 192.168.192.129... connected * Connected to general.dev (192.168.192.129) port 8080 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.19.7 (i386-apple-darwin10.0.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3 > Host: general.dev:8080
![Page 45: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/45.jpg)
MySQL Proxy
local log_file = 'mysql.log' local fh = io.open(log_file, "a+")
function read_query( packet ) if string.byte(packet) == proxy.COM_QUERY then local query = string.sub(packet, 2) fh:write( string.format("%s\n",query)) fh:flush() end end
![Page 46: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/46.jpg)
CHANGE ONETHING ATA TIME
RULE 5
![Page 47: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/47.jpg)
Source Control
[root@localhost ~]#: svn stM index.php
[root@localhost ~]#: git stash saveSaved working directory and index state WIP on master: 0e22fdd Initial checkinHEAD is now at 0e22fdd Initial checkin
[root@localhost ~]#: git status# On branch masternothing to commit (working directory clean)
![Page 48: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/48.jpg)
Source Control
[root@localhost ~]#: svn log . ------------------------------------------- r4341 | rickc | 2010-09-24 03:01 | 3 lines
Don't test if Post::tags is empty before getting tags. There may be a performance hit for this. It can be dealt with if it is noticable. Fixes ticket #1235. Props to ilo for the patch.
Note that Post still needs to be updated to use real Tags, not an array of slug/label pairs as the tags member.--------------------------------------------
![Page 49: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/49.jpg)
Known Good
[root@localhost ~]# diff -yw web.conf web2.conf <VirtualHost *:80> <VirtualHost *:80> ServerAdmin [email protected] ServerAdmin [email protected] DocumentRoot /www/htdocs | DocumentRoot /www/hdocs ServerName general.dev ServerName general.dev ErrorLog logs/error_log ErrorLog logs/error_log</VirtualHost> </VirtualHost>
![Page 50: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/50.jpg)
KEEP ANAUDIT TRAIL
RULE 6
![Page 51: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/51.jpg)
Hot Key Loghttp://www.blacktree.com/
http://simplenoteapp.com/https://launchpad.net/mbhttp://do.davebsd.com/
![Page 53: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/53.jpg)
[root@localhost htdocs]# history 20 365 ls /mnt/hgfs/habari/ 366 ls 367 vi /etc/httpd/conf.d/web.conf 368 setenforce 0 369 /etc/init.d/httpd restart 370 vi /var/log/httpd/error_log 371 cd /mnt/hgfs/habari/ 372 ls 373 cd htdocs/ 374 ls 375 cd scripts/ 376 ls 377 cd .. 378 ls 379 vi /etc/php.ini
![Page 54: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/54.jpg)
CHECK THE PLUG
RULE 7
![Page 55: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/55.jpg)
[root@localhost ~]# apachectl -M Loaded Modules: core_module (static) mpm_prefork_module (static) http_module (static) so_module (static) auth_basic_module (shared) auth_digest_module (shared) authn_file_module (shared) authn_alias_module (shared) authn_anon_module (shared) authn_dbm_module (shared) authn_default_module (shared) authz_host_module (shared) authz_user_module (shared)
Sources of Plugs
[root@localhost ~]# whoami root [root@localhost ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:E8:03:F1 inet addr:192.168.192.129 Bcast:192.168.192.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fee8:3f1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:189820 errors:0 dropped:0 overruns:0 frame:0 TX packets:117261 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:83583857 (79.7 MiB) TX bytes:24261778 (23.1 MiB)
![Page 56: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/56.jpg)
GET A FRESHVIEW
RULE 8
![Page 57: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/57.jpg)
if(!is_string($i)) { throw new Xapian_Exception('Incorrect query type, expecting string or array of strings'); }
$q = $this->getParser()->parse_query($i);
![Page 58: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/58.jpg)
if(!is_string($i)) { throw new Xapian_Exception('Incorrect query type, expecting string or array of strings'); }
$q = $this->getParser()->parse_query($i, FLAG_SPELLING_CORRECTION);
![Page 59: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/59.jpg)
“Grovelling is not a substitute for doing your homework” - ESR
http://www.catb.org/esr/faqs/smart-questions.html
![Page 60: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/60.jpg)
![Page 61: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/61.jpg)
![Page 62: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/62.jpg)
IF YOU DIDN’T FIX IT -IT AIN’T FIXED
RULE 9
![Page 63: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/63.jpg)
Five Whys
The page is blank: Why?There is a fatal error : Why?
No mb_check_encoding(): Why?mbstring is not installed: Why?It isn’t part of the build: Why?
It was setup for another project.
![Page 64: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/64.jpg)
The Rules1. Understand The System2. Make It Fail3. Quit Thinking And Look4. Divide And Conquer5. Change One Thing At A Time6. Keep An Audit Trail7. Check The Plug8. Get A Fresh View9. If You Didn’t Fix It, It Ain’t Fixed
![Page 65: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/65.jpg)
?Questions
![Page 66: Debugging: Rules & Tools](https://reader034.vdocuments.site/reader034/viewer/2022042614/554f4c92b4c905b9508b49e9/html5/thumbnails/66.jpg)
When men were men and wore white coats - by Jitzehttp://www.flickr.com/photos/jitze1942/4292084185/Vacuum Tube Etch-A-Sketch - by Jurvetsonhttp://www.flickr.com/photos/jurvetson/197768962/Classic - by Oskayhttp://www.flickr.com/photos/oskay/1364147095Firmware Bug - by Oskayhttp://www.flickr.com/photos/oskay/1364148351Ready to hop - by Oskayhttp://www.flickr.com/photos/oskay/1364153441/Albino - by Oskayhttp://www.flickr.com/photos/oskay/1364143833Coming Through! - by Oskayhttp://www.flickr.com/photos/oskay/1364147807Tux the Pinguin - by Patrick van der VeldenFrom the photographer’s private collection
Image Credits