of food and computers - clark science center...mith college c omputer science dominique thiébaut...

mith College

Computer Science

Dominique Thiébaut [email protected]

Of Food and ComputersPresentation for

The Kahn InstituteNov 2, 2018

D. Thiebaut, Computer Science, Smith College

A Two-PartDegustation

https://encrypted-tbn0.gstatic.com/shopping?q=tbn:ANd9GcQeDV-F2E-QDSY2IpPVFG0o0sFqxDz1FYh3fQsSwmucJGpb34AQG1hUHZeObj0o-u1zDBFYLIgZrL3bdamwisMHNMibJerEAQ&usqp=CAYhttp://lh6.ggpht.com/6f4-GGlXjSPVddlQvIqDQNflex3BmSPIxvz3nXF-rfBqlarz76nLPsJITZacrg3_fEww-drM867dQZb6=s480


Examples of ML

https://www.youtube.com/watch?v=-RHG5DFAjp8

Google’s CEO, Sundar Pichai, demonstrated the capability on stage at the Shoreline Amphitheater during Google I/O


Examples of MLFully driverless cars are on public roads in Texas. Drive.ai is the second company to remove the safety driver from its autonomous vehicles

https://www.theverge.com/2018/5/17/17365188/drive-ai-driverless-self-driving-car-texas


Examples of ML

https://www.youtube.com/watch?v=rVlhMGQgDkY


Examples of ML

https://qz.com/877721/the-ai-master-bested-the-worlds-top-go-players-and-then-revealed-itself-as-googles-alphago-in-disguise/

In March 2016, DeepMind's AlphaGo beat Lee Sedol in a five-game match, the first time a computer Go program has beaten a 9-dan professional without handicaps.[5] Final score: 4 games to 1 in favor of AlphaGo.

https://en.wikipedia.org/wiki/Lee_Sedol

https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol

https://en.wikipedia.org/wiki/Go_ranks_and_ratings

https://en.wikipedia.org/wiki/Go_ranks_and_ratings

https://en.wikipedia.org/wiki/AlphaGo#cite_note-leesedolwin-5


Examples of ML


Examples of ML

https://www.youtube.com/watch?v=AmUC4m6w1wo

Researchers at the University of Washington have produced a photorealistic former US President Barack Obama. Artificial intelligence was used to precisely model how Mr Obama moves his mouth when he speaks. Their technique allows them to put any words into their synthetic Barack Obama’s mouth.


Neural Networks


Neural Nets


From https://docs.google.com/viewer?url=http%3A%2F%2Ftimes.cs.uiuc.edu%2Fcourse%2F510f17%2Fppt%2Fdeep-learning.pptx



100

60

20



100

60

20

Go

Stop

0.6

0.3

r

y

g



Go

Stop

r

y

g

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g



Go

Stop

r

y

g

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

y1 = h1*w21 + h2*w22 + h3*w23 + h4*w24

y1

y2



Go

Stop

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

y1 = h1*w21 + h2*w22 + h3*w23 + h4*w24

y1

y2

Train



Go

Stop

h1

h2

h3

h4

h1= r*w11 + y*w12 + g*w13

r

y

g

y1 = h1*w21 + h2*w22 + h3*w23 + h4*w24

y1

y2

Test


Model Error


Data VisualizationDaniel Smilkov and Shan Carter

Daniel Smilkov and Shan Carter (Google) Fernanda Viégas and Martin Wattenberg (IBM)

http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=gauss&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=5&networkShape=3,1&seed=0.45718&showTestData=false&discretize=false&percTrainData=80&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false


What can be "Learned?"

https://ml4a.github.io/ml4a/looking_inside_neural_nets/

http://techundred.com/how-snapchat-filter-work/

28

28



https://think-data.github.io/machine%20learning/python/2016/12/03/simple-neural-network-using-tensorflow.html


/ …….Training = Testing

From https://docs.google.com/viewer?url=https%3A%2F%2Fcs.brown.edu%2Fcourses%2Fcs143%2Flectures%2F17.ppt

Training Images Testing Images



Recognize bird songs Ask Alexa for the current weather Identify sounds in a video



the quick red fox jumped over the lazy brown sleeping dog

1: brown 2: dog 3: fox 4: jumped 5: lazy 6: over 7: quick 8: red 9: sleeping 10: the

brown dog: 1 1 0 0 0 0 0 0 0 0 red lazy: 0 0 0 0 1 0 0 1 0 0

quick over: 0 0 0 0 0 1 1 0 0 0 the sleeping fox: 0 0 1 0 0 0 0 0 1 1



1: eggs 2: flour 3: water 4: salt 5: parsley 6: cheese 7: milk 8: spinach 9: tomatoes 10: …

Recipe 1: 1 1 1 0 0 0 0 0 0 0… Recipe 2: 1 0 0 1 1 1 0 0 0 0… Recipe 3: 0 0 0 0 0 1 1 0 0 0… Recipe 4: 0 0 0 0 0 0 1 0 0 0…



egg

s fl

our

wat

er

sal

t p

arsle

y c

hees

e m

ilk

spi

nach

t

omat

oes

…

Recipe 1: 1 1 1 0 0 0 0 0 0 0… Recipe 2: 1 0 0 1 1 1 0 0 0 0… Recipe 3: 0 0 0 0 0 1 1 0 0 0… Recipe 4: 0 0 0 0 0 0 1 0 0 0…


#1

Dessert Time!

Photo credits: Zeng Mu


ML and Food


ML and Food

Computer VisionIEEEACM


Watson


Watson

• 2011, Jeopardy

• Ken Jennings & Brad Rutter

• About Watson: "It is no longer purely a question answering computing system designed from Q&A pairs but one that can now 'see', 'hear', 'read', 'talk', 'taste', 'interpret', 'learn' and 'recommend'."


Watson

• Applied to healthcare, investment management, insurance, diamond purchasing, oncology, and

food! • $1M for hardware, $1M for algorithm

• Currently, size of 3 stacked pizza boxes


Chef Watson!

• Started 2015, released to public 2016, retired 2018

• Based on the recipe archive of Bon Appétit

• Implements hedonic psychophysics & food pairing

• Web-based app

• Cookbook!


Cookbook


Instituteof CulinaryEducation


Roasted Tomato& Mozzarella Tart


In-HouseHigh-Flying Cook


https://www.slideshare.net/AndersQuitzauIbm/big-data-in-foods-chef-watson-agrofood-park


Watson can invent several dozen recipes that prominently feature prunes; it can satisfy a request for banana biscotti in a Creole or a Basque style; and it makes suggestions that no human would ever make, like adding milk chocolate to a clam linguine or mayonnaise to a Bloody Mary.

Alexandra Kleeman Cooking with IBM The New Yorker, Nov. 2016

https://www.newyorker.com/magazine/2016/11/28/cooking-with-chef-watson-ibms-artificial-intelligence-app


1 Million Recipes

#2Photo credits: Zeng Mu


http://pic2recipe.csail.mit.edu/


pairing of recipes

(instructions + ingredients) with

images


DemoUpload image, get recipe

http://pic2recipe.csail.mit.edu/demo

http://pic2recipe.csail.mit.edu/demo


Stats


Stats12 images/recipe


Stats

9 ingredients/recipe


Stats

9 ingredients/recipe

34 ingredients!


Stats

9 instructions/recipe


Stats

9 instructions/recipe

1 instruction!


Food Arithmetic

-

-

-

+

+

+

=

=

=


And now,for the

piece de resistance

…

#3Photo credits: Zeng Mu


A Survey of Dish IdentifyiersComparison of Image Recognition APIs on food imagesRohan Kulkarnibytes.grubhub.com

https://bytes.grubhub.com/https-medium-com-rohan-kulkarni-comparison-of-image-recognition-apis-on-food-images-cddc9105fc33

https://bytes.grubhub.com/@rohan_kulkarni?source=post_header_lockup

https://bytes.grubhub.com/https-medium-com-rohan-kulkarni-comparison-of-image-recognition-apis-on-food-images-cddc9105fc33


Nancy


Nancy

dish, food, cuisine, seafood, asian food, european food, paella, spanish cuisine, thai food, portuguese food


Noa


Noa dish, cuisine, food, leaf vegetable, vegetarian food, italian food, cookware and bakeware, european food, frittata, meal,


Tom


Tom

dish, cuisine, food, asian food, recipe, monjayaki, Okonomiyaki Monjayaki Recipe


Susie


Susie dish, curry, cuisine, food, gravy, stew, vegetarian food, indian cuisine, recipe, soup, taco soup


Dominique


Dominique dish, escargot, food, vegetarian food, vegetable, finger food, recipe, cuisine, Escargot Vegetarian cuisine


Amy


Amy

plant, flora, leaf, branch, tree, fruit, cherry, Leaf


Susan


Susan

vegetable, potato and tomato genus, fruit, food, vegetarian food, tomato, natural foods, nightshade family, Tomato Vegetarian cuisine

La Quinta Inns & Suites


Suleiman


Suleiman

dish, food, cuisine, vegetarian food, fried food, baked goods, side dish, Vegetarian cuisine Food Side dish


Pat


Pat

vegetable, potato and tomato genus, tomato, fruit, food, nightshade family, Tomato


Roberto


Roberto

still life photography, still life, gourd, cucurbita, Still life photography Photography Still life Gourd


Roberto(Second

Trial)


dish, meat, food, venison, animal source foods, steak, lamb and mutton, cuisine, Lamb and mutton Venison

Roberto(Second

Trial)


Lois


Lois

"A baked goodie - a fruit crisp I made in the summer, I think blueberry and peach"


Lois

chocolate brownie, baking, parkin, recipe, capirotada, stuffing, streusel, american food, bread pudding, chocolate brownie


Mona


Mona dish, food, meal, cuisine, salad, asian food, breakfast, leaf vegetable, southeast asian food, lunch,


Sara


Sara

dish, food, cuisine, meal, tableware, breakfast, recipe, vegetarian food, Vegetarian cuisine Breakfast


Kris


Kris

dish, cuisine, food, vegetarian food, rice, arroz con pollo, recipe, asian food, middle eastern food, breakfast, indian cuisine, full breakfast, american food, Ackee and saltfish


To Be

Continued…


FIN

of food and computers - clark science center...mith college c omputer science dominique thiébaut...

Documents