distributed development of cognitive systems at ibm research - brazil - cbsoft2016
TRANSCRIPT
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C. © 2016 IBM Corporation
Distributed Development of Cognitive Systems at IBM Research - Brazil
Alan Braz, MSc, Software [email protected], @alanbrazMaíra Gatti de Bayser ([email protected]), PhD, Research Staff Member, Master Inventor Claudio Pinhanez ([email protected]), PhD, Senior Manager
© 2016 IBM Corporation
IBM Research Labs IBM Research – Openings in 2011 IBM Research – Openings in 2012
China
WatsonAlmaden
Austin
TokyoZurich
India
Dublin
Australia Brazil
Africa▪ Next Gen Public Sector ▪Water & transportation▪ Human Capacity Development
▪ Natural Resources▪ Disaster management▪ Healthcare/Life Sciences
▪ Natural Resources▪ Social Data Analytics (FSS)▪ Visual Analytics & Insights▪ Industrial S&T
▪ Analytics & Intelligence▪ Systems &Software▪ Industry research
▪ Internet of Things▪ Big Data / Analytics▪ Enterprise Cloud Services▪ Energy, Commerce, Traffic
▪ Big Data Analytics▪ HW & SW Quality▪ Cloud▪ Mobile
Haifa
▪ Smarter Cities▪ Analytics
▪ Services ▪ Big Data Analytics▪ Front Office Digitization
▪ Semiconductors▪ Systems▪ Software▪ Services▪ Analytics
▪ Semiconductors▪ Processors
▪ Analytics▪ Storage ▪ Nanotech
Healthcare
▪ Nanotech▪ Security▪ Business
Analytics▪ Systems
Industry Solution Lab
IBM Research: 12 Labs Worldwide in 10 Countries - 3,000 employees
© 2016 IBM Corporation
IBM inventors received a record 7,355 U.S. patents in 2015, marking the 23th consecutive year it has topped the list of the world’s most inventive companies. IBM became the first company to be granted as many as 5,000 U.S. patents in a single year. It took IBM's inventors more than 50 years to receive their first 5,000 patents after the company was established in 1911. IBM's first patent: U.S. Patent #998,631 issued July 25, 1911.
23 Consecutive Years of IBM Patent Leadership
© 2016 IBM Corporation
IBM Research I Brasil
Mission: To be known for our science and technology and vital to IBM, Brazil, our partners in the region, and globally.
IBM @ Tutoia Street
São Paulo
IBM @ Avenue Pasteur
Rio de Janeiro
First IBM Research lab in the South America, founded in 2010
Strategic Focus AreasSmarter natural resources discovery and logistics with emphasis on NR analytics. Optimization, SW technologies
Cognitive Computing with focus in Social Data Analytics reinvent large-scale service systems and enterprises.
Industrial Technology & Science with emphasis in sensors and microfluidics for Health and Natural Resources
Visual analytics & comprehension: methods and technologies for expert knowledge capturing, modeling, and representation
© 2016 IBM Corporation
IBM Watson: From Jeopardy to Cognitive Solutions and ....
5
© 2016 IBM Corporation
... bringing Cognitive Computing to Everyone: Watson Developer Cloud
6
http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/
© 2016 IBM Corporation
Platform as a Service
7http://www.bluemix.net
© 2016 IBM Corporation
SOCIAL COGNITIVE TECHNOLOGIES
social knowledge mining
conversational cognitive agents
2016: Social Data Analytics Research Workstreams
social knowledge gatherer
social knowledge organizer
cognitive dialogue
framework
cog agent interaction
design
CognIA: self-service investment adviser
mission: to make Watson socially skilled
8
© 2016 IBM Corporation
Social Cognitive Technologies Research
9
Claudio PinhanezManager,Social Data AnalyticsPhD MIT
Maíra GattiPhD PUC-Rio
Ana AppelPhD USP-SC
Paulo CavalinPhD Quebec
Alan BrazMSc. Unicamp
Heloisa CandelloPhD Brighton
CapabilitiesKDD, HCI, Simulation, Machine Learning, Social Media, Agents, Design, Graph Mining, Computer Vision, Deep Learning, Large Databases, Agile.
+10 undergraduate/graduate interns
Marcia ItoPhD USP-Poli
Renan SouzaMsc UFRJ
Flavio FigueiredoPhD UFMG
Marisa VasconcelosPhD UFMG
©2015 IBM Corporation10 29 August 2016
Creating a new generation of self-service automated systems for investment and wealth management.
Investment Advisors
© 2015 IBM Corporation
Conversational Systems, the 4th Interface Paradigm
11
1. batch 2. command line 3. point & click 4. conversational
“There are more than 2 billion users of mobile apps” (The Wall Street Journal, December, 2015)
“Forget Apps, Now The Bots Take Over”(TechCrunch, Sep 29, 2015 by Beerud Sheth)https://techcrunch.com/2015/09/29/forget-apps-now-the-bots-take-over/
©2015 IBM Corporation12 IBM Research
Prototype: Direct, Simple, Conversational
● Natural Dialogue System○ Mixed initiative
● Dialogue Mediation○ Coordination control
● Investment Options○ Investment Concepts & Options – Q&A○ Investment Simulation – Task-oriented○ Investment Opinion – Hard to answer
● Portuguese● Brazilian Finance
○ Saving Accounts○ CD (Certificate of Deposit)
■ CDB (Certificado de Depósito Bancário)
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Challenges on Distributed Development
13
Picture source: https://www.linkedin.com/pulse/how-organize-efficient-work-distributed-development-team-soloshchuk
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Remote working… ?
14
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
How do distributed teams interact with each other…?
“Meetings-driven development”“Email-driven task management”
15
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Issues on Remote Working / Distributed Cognitive Teams
▪Communication style▪Expectations and establishing common ground▪Performance control▪ Interdependence
– Lacks share of information -> less creativity▪Communication patterns
– Remote working ≈ Closed door offices– Social interactions– Lack of social communication is associated with lower trust and cohesion
▪High volume of messages can lead to confusion▪Need skilled managers▪Balance onsite and remote working▪ Integrate social business technologies▪Software integration▪Privacy concerns
16
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Challenges on Cognitive Systems Development
17
Picture source: http://www.forbes.com/sites/ibm/2015/10/06/four-steps-to-get-started-with-cognitive-computing/
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Cognitive Computing
18
White paper: http://www.research.ibm.com/software/IBMResearch/multimedia/Computing_Cognition_WhitePaper.pdf
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Software Engineering & Cognitive Systems Development
▪Current well established SE practices:– Agile process– Extreme Programming (XP)– Continuous process– Design improvement– Shared understanding– Collective code ownership– Simple design
19
Waterfall Model(Royce, W.)
Prototyping Model (Brooks, F.)
V Model (Forsberg, K.)
Incremental Dev. (IBM)
Spiral Model (Boehm, B.)
1970 1975 1988 1991 2001
Agile Soft Dev. Kent Beck XP
Scrum
1996
FDD
RUP
Applicability Degree for Cognitive Systems Development
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Software Engineering & Cognitive Systems Development
▪New challenges:– Interdisciplinary areas– Innovation– Complexity– Algorithms evolution– Data management– Models management
20
Standard Waterfall Model
Spiral Model
eXtreme Programming
predictive adaptive
rigidmethodologies
iterativemethodologies
agilemethodologies
Extreme Programming. Kent Beck, 1996
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Complexity and Uncertainty
21
http://iq3group.blogspot.com.br/2012/12/solving-data-governance-by-scaling.html
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Challenges
22
White paper: http://www.research.ibm.com/software/IBMResearch/multimedia/Computing_Cognition_WhitePaper.pdf
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Methods and Tools for Managing Distributed Development of Cognitive Systems
23
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
DevOps
https://www.ibm.com/developerworks/br/devops/principles.html
24
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
DevOps
▪Enable collaboration, co-development and reuse (a.k.a. social coding).
▪Minimize the friction and fuss required to build and deploy your code. This will be a combination of automating what are often manual or custom processes, in combination with providing ready-to-go platforms and specific guidance on release requirements.
▪Provide ready-to-go tools, platforms and methods. Many of these services are currently available from various internal and external corporate offerings. We're working on providing many of these services automatically and without the friction of concerns about funding, signups, approvals, etc.
▪Make it easy (or at least, as easy as possible). This is all about providing ready and useful guidance on using tools, getting help, and learning what you need to know.
25
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Tools
26
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.27
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Managing tasks with GitHub issues
28
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
GitHub org structure
29
▪14 code repos– Java, Node.js, Python, Go, HTML
▪1 planning repo only with Issues– 81 Open – 325 Closed (since Jun 2015)
▪2 types of Infrastructure– Softlayer– Bluemix
▪2 Database paradigms– SQL (DB2 and DashDB)– NoSQL (Cloudant)
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Investment Advisor Team
30
Undergrad interns
Senior Manager
PhD
MSc
SoftwareEngineer
HCI, Machine Learning
Designer, UserExperience
Database,Graph Analytics
DataMining
Data Mining
Databases
Software Engineer,Multi-Agent Systems
Machine Learning
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Architecture - Several Micro-services
31
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
The Architecture
▪The Investment Advisor Prototype is built on top of SABIÁ - the Speech-Act-Based Intelligent Agents framework
▪SABIÁ is a conversational framework for multi-party chabots built on top of Akka middleware that enables mixed-initiative dialogues.
– Gatti de Bayser, M.; Cavalin, P.; Programming Conversational Agents with Akka Actors. Tutorial at 18th European Agent Systems Summer School (EASSS 2016), Sicily, Italy. Available to download.
▪Akka is a toolkit and runtime that implements the Actor model on the JVM and was inspired by Erlang in many ways.
▪Akka can be used to build highly concurrent, distributed, and resilient message-driven applications. A large ecosystem of more than 250 Github projects has grown on Akka, in programming languages like Scala, Java, Groovy or JRuby.
Akka.io. Akka is Open Source and available under the Apache 2 License.
32
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Actor Model
▪Originated in 1973
▪The actor model in computer science is a mathematical model of concurrent computation that treats "actors" as the universal primitives of concurrent computation.
▪In response to a message that it receives, an actor can:– make local decisions,– create more actors,
– send more messages, and
– determine how to respond to the next message received.
▪Actors may modify private state, but can only affect each other through messages (avoiding the need for any locks).
33
Carl Hewitt; Peter Bishop; Richard Steiger (1973). "A Universal Modular Actor Formalism for Artificial Intelligence". IJCAI.
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Cloud Deployment
34
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Main back-end project - Akka agents
35
▪Technologies– Java runtime– Akka toolkit (http://akka.io/)– Socket.io
▪11 team members contributed▪Average of 50 commits/member▪2 main developers
– 80+ commits▪Automated unit and integration tests▪Continuous deploy▪Performance profiling ▪Stress tests
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Main front-end project - Socket.io Web/mobile chat
36
▪Technologies– Node.js runtime– Ionic framework (Angular.js)– Socket.io
▪6 team members contributed▪Average of 29 commits/member▪2 main developers
– 60+ commits
▪Continuous deploy
▪Mobile and web interfaces
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Evolution
37
▪Sort of a 2-week Scrum Sprint called Step▪Review and Planning at the same meeting (2h)▪Weekly status meeting (like a “Daily Scrum”)▪Not fully self-organized team
Akk
a ag
ents
Cha
t
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Conclusions
38
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
Conclusions
▪It is not trivial to manage distributed teams on the development of cognitive applications
▪Several communication tools and methods are needed
▪Software engineering practices and DevOps can provide methods and tools to help tackling some challenges
▪There is still a need on improving the process so it can become more mature while with space to innovation and creativity.
39
© 2016 IBM CorporationGatti de Bayser, M.; Braz, A.; Pinhanez, C.
IBM Research – Brazilhttp://www.research.ibm.com/brazil/[email protected], @alanbraz