4053 - whispering gophers
TRANSCRIPT
-
7/21/2019 4053 - Whispering Gophers
1/46
Whispering GophersNetwork programming in Go
Andrew GerrandFrancesc Campoy
-
7/21/2019 4053 - Whispering Gophers
2/46
Introduction
This code lab demonstrates network programming with Go.
The final goal is to build a "whispernet": a peer to peer mesh network for transmitting
text messages.
-
7/21/2019 4053 - Whispering Gophers
3/46
Design overview
-
7/21/2019 4053 - Whispering Gophers
4/46
Pre-requisites
You must have these tools installed and working:
Go 1.1 (see http://golang.org/doc/install (http://golang.org/doc/install))
Mercurial (see http://mercurial.selenic.com(http://mercurial.selenic.com))
This is not an introducion to the Go Programming Language;
some experience writing Go code is required.
To learn Go:
Take A Tour of Go(http://tour.golang.org)
Check out the Go documentation(http://golang.org/doc)
http://golang.org/dochttp://golang.org/dochttp://golang.org/dochttp://tour.golang.org/http://mercurial.selenic.com/http://golang.org/doc/install -
7/21/2019 4053 - Whispering Gophers
5/46
Creating a workspace
To write Go code you need a workspace. Create one now if you have not already.
$ mkdir $HOME/gocode
$ export GOPATH=$HOME/gocode # or add this to ~/.profile
Build and install a helloexample program:
$ go get code.google.com/p/whispering-gophers/hello
This installs the hellobinary to $GOPATH/bin.
Check that this worked:
$ $GOPATH/bin/hello
It works!
-
7/21/2019 4053 - Whispering Gophers
6/46
The exercises
This code lab is divided into 8 exercises. Each exercise builds on the previous one.
The skeletondirectory in thewhispering-gophersrepository contains unfinished
programs that you should complete yourself.
There is also a solutiondirectory that contains the finished programsonly peek if
you really need to! :-)
-
7/21/2019 4053 - Whispering Gophers
7/46
Vital resources
Visit
code.google.com/p/whispering-gophers (http://code.google.com/p/whispering-gophers)
for code samples and support libraries.
These slides are available at:
whispering-gophers.appspot.com (http://whispering-gophers.appspot.com)
http://whispering-gophers.appspot.com/http://code.google.com/p/whispering-gophers -
7/21/2019 4053 - Whispering Gophers
8/46
Part 1: reading and writing
Write a program that
reads lines from standard input (os.Stdin)
encodes each line as a JSON object, written to standard output (os.Stdout)
This line of input:
Hello, world
should produce this output:
{"Body":"Hello, world"}
This is our system's basic message format.
-
7/21/2019 4053 - Whispering Gophers
9/46
Readers and Writers
The iopackage provides fundamental I/O interfaces that are used throughout most Go
code.
The most ubiquitous are the Readerand Writertypes, which describe streams of data.
package io
type Writer interface {
Write(p []byte) (n int, err error)
}
type Reader interface {
Read(p []byte) (n int, err error)
}
Readerand Writerimplementations include files, sockets, (de)compressors, image
and JSON codecs, and many more.
-
7/21/2019 4053 - Whispering Gophers
10/46
Chaining Readers
package main
import (
"compress/gzip" "encoding/base64"
"io"
"os"
"strings"
)
func main() {
var r io.Reader r = strings.NewReader(data)
r = base64.NewDecoder(base64.StdEncoding, r)
r, _ = gzip.NewReader(r)
io.Copy(os.Stdout, r)
}
const data = `
H4sIAAAJbogA/1SOO5KDQAxE8zlFZ5tQXGCjjfYIjoURoPKgcY0E57f4VZlQXf2e+r8yOYbMZJhoZWRxz3wkCVjeReETS0VHz5fBCzpxxg/PbfrT/gacCjbjeiRNOChaVkA9RAdR8eVEw4vxa0Dcs3Fe2ZqowpeqG79L995l3VaMBUV/02OS+B6kMWikwG
51c8n5GnEPr11F2/QJAAD//z9IppsHAQAA
` Run
-
7/21/2019 4053 - Whispering Gophers
11/46
Buffered I/O
The bufiopackage implements buffered I/O. Itsbufio.Scannertype wraps an
io.Readerand provides a means to consume it by line (or using a specified "split
funcion").
const input = `A haiku is more
Than just a collection of
Well-formed syllables
`
func main() {
s := bufio.NewScanner(strings.NewReader(input)) for s.Scan() {
fmt.Println(s.Text())
}
if err := s.Err(); err != nil {
log.Fatal(err)
}
} Run
-
7/21/2019 4053 - Whispering Gophers
12/46
Encoding JSON objects
The encoding/jsonpackage converts JSON-encoded data to and from native Go data
structures.
type Site struct {
Title string
URL string
}
var sites = []Site{
{"The Go Programming Language", "http://golang.org"},
{"Google", "http://google.com"},
}
func main() {
enc := json.NewEncoder(os.Stdout)
for _, s := range sites {
err := enc.Encode(s)
if err != nil {
log.Fatal(err)
}
}
} Run
-
7/21/2019 4053 - Whispering Gophers
13/46
The Message type
Messages are sent as JSON objects like this:
{"Body":"This is a message!"}
Which corresponds to a Go data structure like this:
type Message struct {
Body string
}
-
7/21/2019 4053 - Whispering Gophers
14/46
Error checking
Many functions in Go return an errorvalue.
These values are your friends; they will tell you where you went wrong.
Ignore them at your peril!
Use log.Println (http://golang.org/pkg/log/#Println)to print log messages, and log.Fatal
(http://golang.org/pkg/log/#Fatal)to print a message and exit the program printing a stack trace.
func main() {
log.Println("Opening gzip stream...")
_, err := gzip.NewReader(strings.NewReader("not a gzip stream!")) if err != nil {
log.Fatal(err)
}
log.Println("OK!")
} Run
http://golang.org/pkg/log/#Fatalhttp://golang.org/pkg/log/#Fatalhttp://golang.org/pkg/log/#Fatalhttp://golang.org/pkg/log/#Println -
7/21/2019 4053 - Whispering Gophers
15/46
Part 1: reading and writing (recap)
Write a program that
reads lines from standard input (os.Stdin)
encodes each line as a JSON object, written to standard output (os.Stdout)
This line of input:
Hello, world
should produce this output:
{"Body":"Hello, world"}
This is our system's basic message format.
-
7/21/2019 4053 - Whispering Gophers
16/46
Part 2: Send messages to a peer
Extend your program:
take an address string from the command line
make a TCP connection to the remote host
write the JSON-encoded messages to the connnection instead of standard output
-
7/21/2019 4053 - Whispering Gophers
17/46
Flag
The flagpackage provides a simple API for parsing command-line flags.
$ flag -message 'Hold on...' -delay 5m
package main
import (
"flag"
"fmt"
"time"
)
var (
message = flag.String("message", "Hello!", "what to say") delay = flag.Duration("delay", 2*time.Second, "how long to wait")
)
func main() {
flag.Parse()
fmt.Println(*message)
time.Sleep(*delay)
} Run
-
7/21/2019 4053 - Whispering Gophers
18/46
Making a network connection
The netpackage provides talk/code for network operations.
The net.Dial (http://golang.org/pkg/net/#Dial)function opens a nework connection and returns a
net.Conn (http://golang.org/pkg/net/#Conn), which implements io.Reader, io.Writer, and io.Closer
(or io.ReadWriteCloser).
(Usually you would use the net/httppackage to make an HTTP request; the
purpose of this example is to demonstrate the lower-level netpackage.)
func main() {
c, err := net.Dial("tcp", "www.google.com:80")
if err != nil {
log.Fatal(err) }
fmt.Fprintln(c, "GET /")
io.Copy(os.Stdout, c)
c.Close()
} Run
http://golang.org/pkg/net/#Connhttp://golang.org/pkg/net/#Dialhttp://golang.org/pkg/net/#Connhttp://golang.org/pkg/net/#Dial -
7/21/2019 4053 - Whispering Gophers
19/46
Part 2: Send messages to a peer (recap)
Extend your program:
take an address string from the command line
make a TCP connection to the remote host
write the JSON-encoded messages to the connnection instead of standard output
-
7/21/2019 4053 - Whispering Gophers
20/46
Part 3: Serving network connections
Write a new program:
listen on a TCP port,
accept incoming connections and launch a goroutine to handle each one,
decode JSON messages from the incoming connections,
print each message Bodyto standard output.
-
7/21/2019 4053 - Whispering Gophers
21/46
Listen/Accept/Serve (1/2)
The net.Listen (http://golang.org/pkg/net/#Listen)function binds to a socket and returns a
net.Listener (http://golang.org/pkg/net/#Listener).
The net.Listener (http://golang.org/pkg/net/#Listener)provides an Acceptmethod that blocks until aclient connects to the socket, and then returns anet.Conn (http://golang.org/pkg/net/#Conn).
This server reads data from a connection and echoes it back:
func main() {
l, err := net.Listen("tcp", "localhost:4000")
if err != nil {
log.Fatal(err)
}
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
}
fmt.Fprintln(c, "Welcome to the echo server!")
io.Copy(c, c)
}
} Run
http://golang.org/pkg/net/#Listenerhttp://golang.org/pkg/net/#Listenerhttp://golang.org/pkg/net/#Listenhttp://golang.org/pkg/net/#Connhttp://golang.org/pkg/net/#Listenerhttp://golang.org/pkg/net/#Listenerhttp://golang.org/pkg/net/#Listen -
7/21/2019 4053 - Whispering Gophers
22/46
Goroutines
Goroutines are lightweight threads that are managed by the Go runtime. To run a
function in a new goroutine, just put "go"before the function call.
package main
import (
"fmt"
"time"
)
func main() {
go say("let's go!", 3*time.Second)
go say("ho!", 2*time.Second)
go say("hey!", 1*time.Second)
time.Sleep(4 * time.Second)
}
func say(text string, duration time.Duration) {
time.Sleep(duration)
fmt.Println(text)
} Run
-
7/21/2019 4053 - Whispering Gophers
23/46
Listen/Accept/Serve (2/2)
To handle requests concurrently, serve each connection in its own goroutine:
func main() {
l, err := net.Listen("tcp", "localhost:4000") if err != nil {
log.Fatal(err)
}
for {
c, err := l.Accept()
if err != nil {
log.Fatal(err)
} go serve(c)
}
}
func serve(c net.Conn) {
fmt.Fprintln(c, "Welcome to the echo server!")
io.Copy(c, c)
} Run
-
7/21/2019 4053 - Whispering Gophers
24/46
Decoding JSON
Decoding JSON from an io.Readeris just like writing them to an io.Writer, but in
reverse.
type Site struct {
Title string
URL string
}
const stream = `
{"Title": "The Go Programming Language", "URL": "http://golang.org"}
{"Title": "Google", "URL": "http://google.com"}
`
func main() {
dec := json.NewDecoder(strings.NewReader(stream))
for {
var s Site
if err := dec.Decode(&s); err != nil {
log.Fatal(err)
}
fmt.Println(s.Title, s.URL)
}
} Run
-
7/21/2019 4053 - Whispering Gophers
25/46
Part 3: Serving network connections (recap)
Write a new program:
listen on a TCP port,
accept incoming connections and launch a goroutine to handle each one,
decode JSON messages from the incoming connections,
print each message Bodyto standard output.
-
7/21/2019 4053 - Whispering Gophers
26/46
Part 4: Listening and dialing
Combine parts 2 to part 3 (Listen andDial)
Test by connecting two instances
-
7/21/2019 4053 - Whispering Gophers
27/46
Part 5: distributing the listen address
Add an Addrfield to the Messagestruct,
When sending messages, put the listen address in the Addrfield.
-
7/21/2019 4053 - Whispering Gophers
28/46
Add an Addr field to Message
Add an Addrfield to the Messagetype:
type Message struct {
Addr string Body string
}
Now, when constructing Messagevalues, populate the Addrfield with the listen
address:
{"Addr":"192.168.1.200:23542,"Body":"This is a message!"}
-
7/21/2019 4053 - Whispering Gophers
29/46
Obtaining the listener address (1/2)
The net.Listenerinterface provides an Addrmethod that returns a net.Addr.
When listening on all interfaces, as specified by the empty hostname in the string
":4000"above, the net.Addrwon't be that of our public IP address.
To complete our program, we need to find that IP.
import (
"log" "net"
)
func main() {
l, err := net.Listen("tcp", ":4000")
if err != nil {
log.Fatal(err)
} log.Println("Listening on", l.Addr())
} Run
-
7/21/2019 4053 - Whispering Gophers
30/46
Obtaining the listener address (2/2)
The "code.google.com/p/whispering-gophers/util" (http://godoc.org/code.google.com/p/whispering-gophers/util)
package provides a Listenfunction that binds to a random port on the first available
public interface.
import (
"log"
"code.google.com/p/whispering-gophers/util"
)
func main() { l, err := util.Listen()
if err != nil {
log.Fatal(err)
}
log.Println("Listening on", l.Addr())
} Run
http://godoc.org/code.google.com/p/whispering-gophers/utilhttp://godoc.org/code.google.com/p/whispering-gophers/utilhttp://godoc.org/code.google.com/p/whispering-gophers/util -
7/21/2019 4053 - Whispering Gophers
31/46
Part 5: sending the listen address (recap)
Add an Addrfield to the Messagestruct,
Import"code.google.com/p/whispering-gophers/util" ,
Use util.Listeninstead of net.Listen,
Store the listen address string in a global variable named self,
When sending messages, put the listen address in the Addrfield.
-
7/21/2019 4053 - Whispering Gophers
32/46
Part 6: separate reading and writing
Separate reading from standard input and dialing into separate functions that run
in separate goroutines.
-
7/21/2019 4053 - Whispering Gophers
33/46
Channels
Goroutines communicate via channels. A channel is a typed conduit that may be
synchronous (unbuffered) or asynchronous (buffered).
package main
import "fmt"
func main() {
ch := make(chan int)
go fibs(ch)
for i := 0; i < 20; i++ {
fmt.Println(
-
7/21/2019 4053 - Whispering Gophers
34/46
Part 6: separate reading and writing (recap)
Separate reading from standard input and dialing into separate functions that run
in separate goroutines.
Pass messages from one function to another using a channel.
-
7/21/2019 4053 - Whispering Gophers
35/46
Part 7: connect to multiple peers
As we see new peer addresses, make connections those peers.
For each line read from standard input, broadcast that message to all connected
peers.
-
7/21/2019 4053 - Whispering Gophers
36/46
Sharing state
Mutexes are a simple means to protect shared state from concurrent access.
var (
count int mu sync.Mutex // protects count
)
for i := 0; i < 10; i++ {
go func() {
for {
mu.Lock()
count++
mu.Unlock() time.Sleep(5 * time.Millisecond)
}
}()
}
time.Sleep(time.Second)
mu.Lock()
fmt.Println(count)
mu.Unlock() Run
-
7/21/2019 4053 - Whispering Gophers
37/46
Tracking peer connections (1/2)
Each peer connection runs in its own goroutine.
Each goroutine has its own chan Message. It reads messages from the channel, and
writes them to the connection as JSON objects.
A central peer registry associates each peer address with its correspondingchan
Message.
type Peers struct {
m map[string]chan
-
7/21/2019 4053 - Whispering Gophers
38/46
Tracking peer connections (2/2)
Before making a peer connection, ask the peer registry for a channel for this address:
// Add creates and returns a new channel for the given address.
// If an address already exists in the registry, it returns nil.func (p *Peers) Add(addr string)
-
7/21/2019 4053 - Whispering Gophers
39/46
Sending without blocking
If a peer connection stalls or dies, we don't want to hold up the rest of our program. To
avoid this problem we should do a non-blocking send to each peer when broadcasting
messages. This means some messages may be dropped, but in our mesh network thisis okay.
ch := make(chan int)
select {
case ch
-
7/21/2019 4053 - Whispering Gophers
40/46
Part 7: connect to multiple peers (recap)
As we see new peer addresses, make connections to those peers.
For each line read from standard input, broadcast that message to all connected
peers.
-
7/21/2019 4053 - Whispering Gophers
41/46
Part 8: re-broadcast messages
Add IDfield to Message,
When creating a Message, populate the IDfield with a random string,
Track the IDof each received message; drop duplicates,
For each received Message, broadcast it to all connected peers.
-
7/21/2019 4053 - Whispering Gophers
42/46
Add ID to Message
Add an IDfield to the Messagetype:
type Message struct {
ID string Addr string
Body string
}
Now, when constructing Messagevalues, populate the IDfield with a random string:
{"ID":"a09d2abb1ad536ada",Addr":"192.168.1.200:23542,"Body":"This is a message!"}
-
7/21/2019 4053 - Whispering Gophers
43/46
Generating random strings
Use the util.RandomID (http://godoc.org/code.google.com/p/whispering-gophers/util/#RandomID)function to generate a
random string.
package main
import (
"fmt"
"code.google.com/p/whispering-gophers/util"
)
func main() {
id := util.RandomID()
fmt.Println(id)
} Run
http://godoc.org/code.google.com/p/whispering-gophers/util/#RandomID -
7/21/2019 4053 - Whispering Gophers
44/46
Tracking message IDs
To track messages IDs, use a map[string]bool. This works as a kind of set.
seen := make(map[string]bool)
To check if an id is in the map:
if seen[id] {
fmt.Println(id, "is in the map")
}
To put an id in the map:
seen[id] = true
You should implement a function named Seen`make sure it is thread safe!
// Seen returns true if the specified id has been seen before.
// If not, it returns false and marks the given id as "seen".func Seen(id string) bool
-
7/21/2019 4053 - Whispering Gophers
45/46
Part 8: re-broadcast messages (recap)
Add IDfield to Message
When a Message, populate the IDfield with a random string
Track the IDof each received message; drop duplicates
For each received Message, broadcast it to all connected peers
-
7/21/2019 4053 - Whispering Gophers
46/46
Thank you
Andrew Gerrand
Francesc Campoy