using python3 to build a cloud computing service for my superboard ii
Post on 10-May-2015
5.081 Views
Preview:
DESCRIPTION
TRANSCRIPT
Using Python 3 to Build a Cloud Computing Service for
my Superboard II
David Beazley(http://www.dabeaz.com)
Presented at PyCon 2011Atlanta
http://pycon.blip.tv/file/4878868/
Note: This talk involves a number of live demonstrations. You will probably enjoy
it more by watching the PyCon 2011 video presentation
(Drumroll...)
It Begins
Byte Magazine, January, 1979
Personal History
• Superboard II was my family's first computer
• What I first learned to program (~1979)
• It had everything that a kid could want
Easy Setup!
Peripherals!
Potentially Lethal Suggestions!
Schematics!
You could get an electric
shockYou could
look inside!
Pure Awesome!You could program
Pure Awesome!You could hack!
Note the encouragement
Backstory
• 1982-2010. Superboard sits in mom's basement
• July 2010. Eric Floehr mentions SB at Scipy
• August 2010. Brother brings SB to Chicago
• It still works! (to my amazement)
Question• What do you do with an old Superboard II
• 1 Mhz 6502 CPU
• 8K RAM
• 8K Microsoft Basic (vers. 1.0)
• 300 Baud Cassette Audio Interface
• 1K Video Ram (24x24 visible characters)
• Not a modern powerhouse
TO THE CLOUD!
HOW?(with Python 3 awesomeness of course)
(plus some ØMQ and Redis)
(and some 6502 assembler)
Getting to the CloudThis is the only I/O
A pair of RCA audio jacks
Used for cassette tape storage
Not too promising
A Possible Solution
Cassette Interface
• Behind the RCA jacks, sits a real serial "port"
• Motorola 6850 ACIA
• Ports are a 300 baud audio stream encoded via Kansas City Standard
• Might be able to hack it
Byte, Feb. 1976
Python Audio Processing
• pyaudio
• http://http://people.csail.mit.edu/hubert/pyaudio/
• A Python interface to the portaudio C library
• Allows real-time access to audio line in/out
• I ported it to Python 3
Reading Audio Inputimport pyaudiop = pyaudio.PyAudio()stream = p.open(format=pyaudio.paInt8, channels=1, rate=9600, chunksize=1024, input=True)while True: sample = stream.read(1024) process_sample(sample)
Building a Python Modem
pyaudiowriter
pyaudioreader
byte encoder
byte decoder
TCP socket
client
reader thread
writer thread
• A network socket bridged to two real-time audio encoding/decoding threads
KCS Audio Encoding
0 = 1 =
(4 cycles @ 1200 Hz) (8 cycles @ 2400 Hz)
'A' = 0x41 (01000001)
0 0 0 0 0 0 1 1110
start data stop
KCS Audio Decoding
89, 103, 117, 151, 194, 141, 99, 64, 89, 112, 141, 203, 152, 107, 88, ...
samples (8-bit)
000111000111000111000111000111111000000111111000000111111000000111111...
sign bits (0 = neg, 1= pos)
0001001001001001001001001001000001000001000001000001000001000001000001...
Idea: Each 1 bit represents a 0-crossing
sign changes (XOR adjacent bits)
Example Code
def generate_sign_change_bits(stream): previous = 0 while True: samples = stream.read(CHUNKSIZE)
# Emit a stream of sign-change bits for byte in samples: signbit = byte & 0x80 yield 1 if (signbit^previous) else 0 previous = signbit
KCS Bit Decoding
1001001001001001001001001001
idle (constant 1)
data bytestart stop
1 1 10 0 0 0 0
1001001001001001 discard
Sample buffer (size = audio samples for 1 bit)
Sign Change Bits push
~ 8 sign changes ~ 16 sign changes
0 1
Example Codefrom collections import deque
# Sample buffersample = deque(maxlen=SAMPLES_PER_BIT)
for b in generate_sign_change_bits(stream): sample.append(b) nchanges = sum(sample) if nchanges < 10: # Detected a start bit # Sample next 8 data bits ...
(The actual code is more optimized)
DemoMy Mac (linked via audio)
Interlude• It's Alive!
• Basic communication is just the start!
This is uploading machine code
~500 lines of 6502 assembler...getvar_copy_string:! ;; Move the variable address to INDIRECT where we can use it! LDA![0x95, Y]! ; Length! STA!%MEM_LENGTH! INY! LDA![0x95, Y]! ; address (low)! STA!%INDIRECT! INY! LDA![0x95, Y]! ; address (high)! STA!%INDIRECT+1! LDY!#0x00...
Wrote a 6502 assembler
~500 lines of Python 3
This is uploading machine code
~500 lines of 6502 assembler...getvar_copy_string:! ;; Move the variable address to INDIRECT where we can use it! LDA![0x95, Y]! ; Length! STA!%MEM_LENGTH! INY! LDA![0x95, Y]! ; address (low)! STA!%INDIRECT! INY! LDA![0x95, Y]! ; address (high)! STA!%INDIRECT+1! LDY!#0x00...
Wrote a 6502 assembler
~500 lines of Python 3
Under the covers
msgdrv.asm
asm6502.py
msgdrv.hex
.1C00/2005AEEE1E1EAD1E1E8D
pymodem
This is uploading machine code
~500 lines of 6502 assembler...getvar_copy_string:! ;; Move the variable address to INDIRECT where we can use it! LDA![0x95, Y]! ; Length! STA!%MEM_LENGTH! INY! LDA![0x95, Y]! ; address (low)! STA!%INDIRECT! INY! LDA![0x95, Y]! ; address (high)! STA!%INDIRECT+1! LDY!#0x00...
Wrote a 6502 assembler
~500 lines of Python 3
Under the covers
msgdrv.asm
asm6502.py
msgdrv.hex
.1C00/2005AEEE1E1EAD1E1E8D
pymodem
Messaging Driver
Superboard II Client
request
control
• Superboard issues requests
• Client responds and gets control
• Driver coexists with BASIC
message driver
BASIC Workspace7168 bytes
1024 bytes
This is uploading machine code
~500 lines of 6502 assembler...getvar_copy_string:! ;; Move the variable address to INDIRECT where we can use it! LDA![0x95, Y]! ; Length! STA!%MEM_LENGTH! INY! LDA![0x95, Y]! ; address (low)! STA!%INDIRECT! INY! LDA![0x95, Y]! ; address (high)! STA!%INDIRECT+1! LDY!#0x00...
Wrote a 6502 assembler
~500 lines of Python 3
Under the covers
msgdrv.asm
asm6502.py
msgdrv.hex
.1C00/2005AEEE1E1EAD1E1E8D
pymodem
Messaging Driver
Superboard II Client
request
control
• Superboard issues requests
• Client responds and gets control
• Driver coexists with BASIC
message driver
BASIC Workspace7168 bytes
1024 bytes
Example of making a request
10 A = 9620 B = 4230 C$ = "FOO"40 S = USR(37)
Client controlPEEK - Get a memory regionPOKE - Set a memory regionGET - Get a BASIC variableSET - Set a BASIC variableRETURN - Return to BASIC
Distributed shared memory!
Messaging Architecture• There are two parts (driver and a client)
superboardmessagedriver
BASIC
message client
Python 3
pymodemsocketaudio
• Uses a binary message protocol
USR(37) : \x20 \x06 \x01 \x25 \x00 \x02
Command Size Seq LRCData
• Protocol details not so interesting
Message Driver
• Interacts directly with Microsoft BASIC
• Uses "Undocumented" ROM routines
• Accesses BASIC interpreter memory
• For this, some resources online
• http://osiweb.org
Client Architecture• Client bridges Superboard II to the outside world
• Uses ØMQ (http://www.zeromq.com)
• And pyzmq (http://github.com/zeromq/pyzmq)
Message client
Python 3
pymodemsocketaudio
ØMQ
Services
Request Publishing• Requests are broadcast on a ØMQ PUB socket
• Retransmitted every few seconds if no response
Message client
Python 3
ØMQ PUBUSR(37) "37 1" "37 1" "37 1"To the
"Cloud"
• Requests include the USR code and a sequence #
...
Service Subscription• Services simply subscribe to the request feed
import zmqcontext = zmq.Context()requests = context.socket(zmq.SUB)requests.connect("tcp://msgclient:21001")requests.setsockopt(zmq.SUBSCRIBE,b"37 ")
• Now wait for the requests to arrivewhile True: request = requests.recv()
• Clients are separate programs--live anywhere
Request Response• Message client has a separate ØMQ REP socket
• Used by services to respond
Message client
Python 3
ØMQ PUBUSR(37) "37 1"
• Service initially acks by echoing request back
• On success, can issue more commands
ØMQ REP Service (subscribed to 37)commands
driver
Command Connection• Setting up the command socket
commands = context.socket(zmq.REQ)commands.connect("tcp://msgclient:21002")
• Complete request/response cycle
while True: request = requests.recv() commands.send(request) # Echo back resp = commands.recv() if resp[:2] == b'OK': # In control of Superboard # Do evil stuff ...
Commands• Commands are just simple byte strings
b"RETURN VALUE"b"PEEK ADDR SIZE"b"POKE ADDR DATA"b"GET VARNAME"b"SET VARNAME VALUE"
• Response codesb"OK DATA"b"FAIL MSG"b"IGNORE"b"BUSY"
Interactive Demo>>> request_sock.recv()b'37 1'>>> command_sock.send(b'37 1')>>> command_sock.recv()b'OK'>>> command_sock.send(b'GET A')>>> command_sock.recv()b'OK 96.0'>>> command_sock.send(b'GET B')>>> command_sock.recv()b'OK 42.0'>>> command_sock.send(b'SET Q 2')>>> command_sock.recv()b'OK'>>> command_sock.send(b'SET R 12')>>> command_sock.recv()b'OK'>>> command_sock.send(b'RET 0')>>> command_sock.recv()b'OK'>>>
Big PictureMessage client
ØMQ PUB
ØMQ REPUSR(N)
Up to 65536 Service IDs (N)
• Services can live anywhere
• Written in any language
• No real limit except those imposed by ØMQ
Big Iron
Superboard Emulation• I dumped the BASIC and system ROMS
• Loaded them into Py65
• Py65 : A 6502 Emulator Written in Python
https://github.com/mnaberez/py65
• Work of Mike Naberezny
• I ported it to Python 3
Emulation in 60 SecondsYou start with the
Superboard II memory map (provided)
Emulation in 60 Seconds
You identify hardware devices
Emulation in 60 SecondsYou read
(about keyboards)
Emulation in 60 SecondsYou read
(about video ram)
Emulation in 60 SecondsYou read
(about ACIA chips)
Emulation in 60 SecondsThen you just plug it into py65 (sic)def map_hardware(self,m): # Video RAM at 0xd000-xd400 m.subscribe_to_write(range(0xd000,0xd400), self.video_output)
# Monitor the polled keyboard port m.subscribe_to_read([0xdf00], self.keyboard_read) m.subscribe_to_write([0xdf00], self.keyboard_write)
# ACIA Interface m.subscribe_to_read([0xf000], self.acia_status) m.subscribe_to_read([0xf001], self.acia_read) m.subscribe_to_write([0xf001], self.acia_write)
# Bad memory address to force end to memory check m.subscribe_to_read([0x2000], lambda x: 0)
Interactive Demo
Question
• What do you do with an emulated Superboard?
A Superboard Cloud
StoredInstances(images of running
machines)
Program Storage
10 PRINT "I WILL THINK OF A"15 PRINT "NUMBER BETWEEN 1 AND 100"20 PRINT "TRY TO GUESS WHAT IT IS"25 N = 030 X = INT(RND(56)*99+1)35 PRINT40 PRINT "WHATS YOUR GUESS ";50 INPUT G
10 PRINT "HELLO WORLD"20 END
10 FOR I = 1 TO 100020 PRINT I30 NEXT I20 END
DatastoreCloudService
VirtualizedSuperboard CPUs
Building The Cloud
• I built it using Redis (http://redis.io)
• Ported py-redis to Python 3
• Redis is cool
• Can use it as a key-value store
• Has other data structures (sets, hashes, etc.)
• Queuing
• Atomic operations
Redis Example
import redisdb = redis.Redis()
# Key-value storedb.set('foo',data)data = db.get('foo')
# Queuingdb.lpush('queue',work)work = db.brpop('queue')
Superboard Cloud Features
• Remote program store
• Load/save programs
• Instance creation
• Creation
• Run with input
• Distributed shared memory
• It must be usable from the Superboard II
Program Load/Store
• BASIC program & workspace memory directly manipulated by the message driver
• Stored in Python object and pickled to Redis
settings
program
stringsBASIC
redis
msgdriver
cloudservice get
set
Instances
• Instances are a running Superboard
• 8K Program Memory
• 1K Video RAM
• Stored CPU context (registers, etc.)
• Stored in a Python object
• Pickled to Redis when inactive
Instance Execution• "Runner" programs watch a Redis queue
import redisr = redis.Redis()...while True: work = r.brpop("queue") # Wait for work ... inst = load_instance() # Get instance run_emulation(work) # Run emulation save_instance(inst) # Save instance
• Based on supplying keyboard input to SB
• Instance runs until no more input available
Instance Concurrency
• Can have arbitrary number of runners
Runners
Redis
• Asynchronous execution (w/ Superboard)
• Uses Redis setnx() for locking
import superboard as skynet
ØMQ PUB
ØMQ REP
Up to 65536 Service IDs (N)
Big Iron
pymodem
CloudService
StoredInstances
VirtualizedSuperboard CPUs
10 PRINT "I WILL THINK OF A"15 PRINT "NUMBER BETWEEN 1 AND 100"20 PRINT "TRY TO GUESS WHAT IT IS"25 N = 030 X = INT(RND(56)*99+1)35 PRINT40 PRINT "WHATS YOUR GUESS ";50 INPUT G
10 PRINT "HELLO WORLD"20 END
10 FOR I = 1 TO 100020 PRINT I30 NEXT I20 END
programs
redis
WHY?!
Non-Answer
• I don't actually want to use my Superboard II
• It's awful!
• It was painful even in 1980
A Better Answer• For fun
• Also to create a glorious mess!
• Everything a systems hacker could want!
• Hardware, device drivers, signal processing, protocols, networks, message passing, threads, synchronization, debugging, software design, testing, deployment, etc.
• Awesome!
Real Answer : Python 3
• Can Python 3 be used for anything real?
• I needed to check it out myself
• On a non-trivial project with many moving parts
• And with a variety of library dependencies
Lessons Learned
• You can use Python 3 to do real work
• However, it’s bleeding edge
• You really have to know what you’re doing
• Especially with respect to bytes/unicode
• Must be prepared to port and/or patch
Porting Experience
• py65 (easy, some bytes issues)
• pyredis (easy, bytes/unicode issues)
• pypng (hard, really messy I/O)
• pyaudio (hard, C extensions)
• pyzmq (worked, Cython patch for Py3.2)
Finding Python 3 Code
• Every single package I used was downloaded from development repositories (github, subversion, etc, etc)
• You can often find Python 3 compatible libraries in project forks or issue trackers if you look for it
My Thoughts
• Python 3 is cool
• It keeps getting better
>>> import numpy>>>
• It’s different and fun
• It might be a great language for distributed computing, messaging, and other applications
That’s All Folks!• Thanks for listening!
• Look for the “Python Cookbook, 3rd” edition in late 2011!
• More info on my blog
• http://www.dabeaz.com
top related