knime for the life sciences cambridge meetup · copyright © 2016 knime.com agcopyright © 2014...

Copyright © 2016 KNIME.com AG

KNIMEforthelifesciencesCambridgeMeetup

GregLandrum,Ph.D.

KNIME.comAG

12July2016

Copyright © 2014 KNIME.com AG 2Copyright©2016KNIME.com AG

• WhatisKNIME?

• Abitofmotivation:toolblending,datablending,documentation,automation,reproducibility

• Moreaboutthecompanyandthecommunity

• Somehighlightsfromrecentreleases


TheKNIME®AnalyticsPlatform


• NODESperformtasksondata

• NodesarecombinedtocreateWORKFLOWS

Status

VisualKNIMEWorkflows

Inputs OutputsNotConfigured

Idle

Executed

Error


Analysis &MiningStatisticsDataMiningMachineLearningWebAnalyticsTextMiningNetwork AnalysisSocialMedia AnalysisR,Weka, PythonCommunity /3rd

Data AccessMySQL,Oracle, ...SAS,SPSS, ...Excel, Flat, ...Hive, Impala, ...XML,JSON,PMMLText, Doc,Image, ...WebCrawlersIndustrySpecificCommunity /3rd

TransformationRow,ColumnMatrixText, ImageTime SeriesJavaPythonCommunity /3rd

VisualizationRJFreeChartJavaScriptCommunity /3rd

DeploymentviaBIRTPMMLXML,JSONDatabasesExcel, Flat,etc.Text, Doc,ImageIndustrySpecificCommunity /3rd

Over1000nativeandembeddednodesincluded:


Whyisthisimportant?

Realworlddataanalysis:

• Lotsofheterogeneousdatafrommultiplesources(datablending)

• Complexquestionstoaskofthedata

• Needtoapplymultipletools(toolblending)


Theproblemisthatwedon’thaveoneoftheseforworkingwithourdata


Theproblemisthatwedon’thaveoneoftheseforworkingwithourdata

It’samazinghowsimpleyoucanmakecomplexthingsifyoucontroltheentireprocess

Ifallofyourproblemslook

likethis: thenthisistheperfecttool:


Wetendtoneedabroaderassortmentoftoolsforourdata

https://www.flickr.com/photos/mtneer_man/5247813293

Ifwe’reluckytheyarethiswellorganized…


…butthisisalotmorecommon

10

https://www.flickr.com/photos/tilde-lifestyle-photog raphy/6906117081/


Whyisthisimportant?

Realworlddataanalysis:

• Lotsofheterogeneousdatafrommultiplesources(datablending)

• Complexquestionstoaskofthedata

• Needtoapplymultipletools(toolblending)• Thingsthatwouldbegreat:

– Wedidn’thavetospendhalfourtimeconvertingfileformats

– Wecouldfigureoutlaterwhatwedid,repeatit,andsharethatwithothers

ThisiswhereKNIMEcomesin

Copyright © 2016 KNIME.com AG 12

KNIME:thecompany


KNIME

• KNIME.comAGfoundedin2008• OfficesinZurich(HQ),Konstanz,Berlin,andSanFrancisco• 20+employees• MaintaineroftheOpenSourceKNIMEAnalyticsPlatform

– comprehensivedataloading,processing,analysis,modelingplatform– visualfrontend– open:toallsortsofdata,othertools(RandPython,a.o.),various

userpersonas– 20opensourcereleasessince2006– opensource.

• KNIMECommercialExtensionsforCollaboration,Productivity,Performance– 14commercialproductreleasessince2008


BroadRangeofKNIMEApplicationAreas&Customers

AdvancedAnalytics

Pharma

HealthCare

Finance

Retail

CustomerIntelligence

Manu-facturing


Happyusers!

Source:http://r4stats.com/2016/06/06/rexer-data-science-survey-satisfaction-results/


KNIMEAnalyticsPlatform:TryitNow!

1. Downloadfromwww.knime.com2. BrowsetheKNIMELearningHub

atwww.knime.com/learning-hub3. Downloadyourfreecopyof

the“KNIMEBeginner’sGuide”from:www.knime.com/knimepress(usecode:KNIME_Boston2016)

4. VisitushereoratourForum:www.knime.com/forum


TheKNIMEEcosystem


KNIMESoftware

KNIMEcommercialextensionstotheplatformforcollaboration,productivity,performance


KNIMEServer


KNIMEBigDataExtensions(commerciallicenserequired!)

• KNIMEBigDataConnectors– Packagerequireddrivers/librariesforspecificHDFS,Hive,Impalaaccess

– Hive(BigDataExtension)

– ClouderaImpala(BigDataExtension)

– Extendstheopensourcedatabaseintegration

• KNIMESparkExecutor– PackagerequireddriverstosubmitSparkjobs

– WrapsSparkDBmanipulationsandMLlib modules


BigDataConnectors

• SamemodeofoperationasthestandardKNIMEdatabaseconnectors

• Operationsareperformedwithinthedatabase


KNIMESparkExecutor

• BasedonSparkMLlib

• Scalablemachinelearninglibrary

• RunsonHadoop

• Algorithmsfor– Classification(decisiontree,naïvebayes,…)

– Regression(logisticregression,linearregression,…)

– Clustering(k-means)

– Collaborativefiltering(ALS)

– Dimensionalityreduction(SVD,PCA)


• Usagemodelanddialogssimilartoexistingnodes

• Nocodingrequired

FamiliarUsageModel


TheKNIMEcommunity


Opennessandthecommunity

• Veryactiveusercommunity(check theforums)• >250peopleatthe2016KNIMESummitinBerlin

• TheKNIMEAnalyticsPlatformisbothopensourceandanopenplatform.

• Technologypartners:provideandsupportnodesfortheir(usuallycommercial)softare.Someexamples:Schrodinger,ChemAxon/InfoCom,CCG,Cresset

• Weencouragethecommunitytoproducenodes(orsetsofnodes)andsharethemwitheachother.

• “TrustedCommunityExtensions”forcommunitycontributionsthatmeetacertainqualitylevel.


Someofthecommunitycontributions:

Thisisthesubsetmorerelevanttodrugdiscovery


HighlightsofrecentadditionsinKNIME3.1and3.2

Completelists:https://tech.knime.org/whats-new-in-knime-31https://tech.knime.org/whats-new-in-knime-32


• DefaultExecution

• StreamingExecution– Row-wise

– Process,pass&forget→FasterwithlessI/Ooverhead

– Concurrentexecution

Streaming


Streaming– Prosand Cons

Advantages

• LessI/Ooverhead(process,pass&forget)

• Parallelization

Disadvantages

• Nointermediateresults,nointeractiveexecution

• Notallnodescanbestreamed


Trees/Forest/Ensembles

• Randomforestnode(simplificationofthetree-ensemblenode)

• Supportofbinarysplitsfornominalattributes

• Missingvaluehandling

• Supportofbytevectordata(high-dimensioncountfingerprints)

• Codeoptimization– Runtime

– Memory


TreesandTreeEnsembles:Newnodes

• GradientBoosting– Alsobasedontreeensembles

– Boosting:Improvinganexistingmodelbyaddinganewmodel

– Shallowtrees

• RandomForestDistance– Distancemeasureinducedbyarandomforest

– Basedonproximity


FeatureSelection

• Automatedhelpfornarrowingdownthebestsetoffeaturesforamodel

• Supportsforwardandbackwardselection


Deeplearning4j– KNIMEIntegration

• Easynetworkarchitecturedesign

• Modular– Layerwise designofnetworks

• ModelImport/Export– Caffe Import

• Beginnerfriendly– Importpretrained networks

• Highlyconfigurable

• Supportsword2vecanddoc2vec


Deeplearning4j– KNIMEIntegration


ActiveLearning

• LabsExtension

• Involveusertoconstructtrainingdataset

• Workflowlooptoqueryandlabel‘interesting’datapoints

• Useduser-labeleddatasetonremainingdata


RIntegration

• Rewriteofinfrastructure– Significantlyfaster

– Concurrentexecution

• Nochangeofusagemodel


MongoDBandJSON(I)

• MongoDBisaNoSQLdatabasebasedonJSON• Specialsetofnodes

– duetolackofastandardSQLinterface


MongoDBandJSON(II)

• JSONnodesforworkingwithJSONdata– SimilartotheXMLnodes

• UsecombinationofMongoDBandJSONnodes


SemanticWeb/LinkedDataIntegration

• Accessandmanipulatesemanticwebresourcese.g.DBpedia

• ExecutesemanticqueriesviaSPARQL

• Usagemodelsimilartodatabaseintegration


Othercoolstuff

• Workflowcoach:suggestsnextnodestouse


Takehomes

• Openplatformbasedonopen-source softwarebackedbyacommercialentityprovidingenterpriseextensionsandsupport

• Strongfocusondatablending andtoolblending• Activeandengagedcommunity

• Greatsupportforlifesciences/chemistry fromthecommunity


Thanks!

Enjoytheothertalks.


14-16September,2016SanFranciscohttps://www.knime.org/fall-summit2016

knime for the life sciences cambridge meetup · copyright © 2016 knime.com agcopyright © 2014...

Documents