002 ugm2013 whats new final

What‘s new…

Bernd Wiswedel

KNIME.com AG, Zurich, Switzerland

Two feature releases last year: 2.6 & 2.7

Documented in Changelog, “What‘s new

summary” and as video on YouTube

What‘s new page on knime.org

KNIMETV Youtube Channel

Outline

Illustrative examples

• Swiss Survival Analysis

• KNIME Forum Analysis

• (Next Best Offer)

New Features in 2.6 & 2.7

Outline

Swiss Survival Analysis

• Survival Analysis / Actuarial Tables

• Using population and deaths data to predict

longevity

• Creating the tables

• Investigating the tables

• Creating customer tables for:

• Overall

• Personal

• Historical

• Forecasting

• Make it easy to use for the non-expert!

Outline

KNIME Forum Analysis

Learn something about the KNIME forum:

http://tech.knime.org/forum

Challenges:

• Get data into KNIME

• Extract simple statistics (how many posts,

response time, response length)

• Classify topics and detect topic shifts

• Identify content and users

Challenges:

Forum Analysis – Get Data

Two alternatives:

• Connect to underlying database, read

content

Doable but complicated:

7+ tables need to be read,

prepared and joined

Forum Analysis – Get Data

Two alternatives:

• Connect to underlying database, read

content

complicated and not generic

• Crawl the web page, parse html

• Use XML parser & Palladian’s html retriever

Forum Analysis – Structure of forum

Several Categories, “KNIME General”,

“KNIME Reporting”, “Palladian”, …

(~20 in total)

Discussion threads on several sub-pages

Each thread consists of an initial post

and a variable number of comments

Forum Analysis – Crawler Flow

Discussion threads on several sub-pages

Forum Analysis – Crawler Flow

Input for all subsequent workflows!

Challenges:

Forum Analysis – Simple Statistics

Input table from crawler workflow

Meta nodes perform simple

preprocessing, e.g. average number

of active users per month

Many different reporting nodes with

different statistics. Reporting

extension to generate PDF, DOC, …

Number of active users per year

An active user is an user with at

least one comment or one post in

that year.

Number of posts per year

Numbers are just posts (new

discussion threads), not comments

Number of posts per month and year

Big increase early 2011.

Coincidentally, Simon Richards

(richards99) joined

Who comments/answers on posts?

Response time

Number of comments per post

Challenges:

Forum Analysis – Classify Posts

• Use text mining to classify forum post into

categories such as ‘io’, ‘manipulation’,

‘mining’, …

• No training set available

(mis-)use KNIME node description

• See evolution of discussion topics over the

Want to classify forum post (only

first post, no comments)…

… using KNIME node description text

as labeled training set

Reads node descriptions from xml

dumps (generated with KNIME

command line tool)

Uses forum data input file and

prepares with text mining tools

Unzips an archive with all xml files

into temp location

XML files read with loop and

preprocessed (header and footer

removed)

Description is converted into KNIME

text document, from which

(stemmed) terms are extracted

Training data extracted. Learning

attributes are keyword

occurrences; target is document

category

Verify model by splitting data

into train/test.

Using random forest classifier to

address high dimensionality of

small (and sparse) data set

… continuing with main input branch

(Input table from crawler workflow)

Preprocessing similar to before,

extracting date, author, title, …

Extracting attribute table using the

keywords from the node description

(training) data.

Remainder of the workflow ranks

the prediction and prepares for the

report.

Hot topics have always been

manipulation and mining … tasks

that KNIME is very good at.

Note also increase of ‘flowcontrol’

over the years and low ‘r’ traffic

(separate forum category, not part

of this data set)

Challenges:

Forum Analysis – Content & Users

• Look at individual categories (KNIME

General, Developer, Reporting, …)

• Learn what is discussed

• See who is contributing

Input are all discussions

in one forum category…

Output is a multi page

report with tag cloud and

user connection graph

Combines KNIME’s text and

network mining extensions

Input table from crawler workflow

Main loop over all ~20 categories

General statistics per category

User network analysis

Text analytics

Text analysis: Forum posts converted

to documents and tagged (persons,

node names, node categories)

Terms fed into tag cloud, colors

represent persons (‘kilian’), nodes

(‘bow creator’), node categories

(‘xml’), …

Network analysis:

User connections

(content ignored)

Network analysis: Ignore topics, only

look at user relation ships. Network

nodes represent users, connections

represent (directed) relationships

between users

Network analysis: Very simple

user graph, visualized with

standard KNIME graph viewer

Data collected and send

to reporting extension

Multi page pdf output for

different forum categories

Text Mining forum category

RDKit (community

chemistry extension)

KNIME Users – not

dominated by any

particular users

Challenges:

Reviewing all workflows

• All workflows rely on the same input data

• Requires re-run of “Crawler” workflow and

updating parameters in analysis flow

What do all these flows have in common?

They all require the “Crawler” data

Reviewing all workflows

• All workflows rely on the same input data

• Requires re-run of “Crawler” workflow and

updating parameters in analysis flow

• Better: Use meta node and share it between

all instances

They all require the “Crawler” data

Now use it in all the

analysis flows

Nice … but now all workflows

fetch the data each time they

execute!

Let’s add a cache option.

Quickform Node defining a switch:

-Get data from web or

-use cached file (lives on server)

Meta Node Templates

• Meta nodes as isolated functional unit

• Shared on KNIME Server (or teamspace) for

use in other workflows or by other users

• Quickforms to expose relevant parameters

in meta node dialog or in wizard execution

• Can also be used on the KNIME server…

Knime Web Portal

Outline

NBO as a typical Project Collect training

data from

multiple sources:

- DB tables

- text files

- excel files

- SAS files

- binary tables

- map files

Define File Paths

and Parameters

Train and evaluate a number of

prediction algorithms to predict

variable Target

Retrieve old model

that has been decently working so

Compare

performances

and choose

best model

Recalculate

predictions

based on

best model

and save

current

NBO as an Example

Collect Training

Data from

multiple Sources

Select best

prediction model

Apply best

model to

score data

Select files and

define parameters

Build a

report

NBO Report

KNIME Server Training 109

Mean Error in %

e-mail

notification

Me.me@mycompany.com

Global

Variables

Quickform dialogs

Execution

Wizard File Upload

Quickforms

Selection

Quickform

Integer Input

Quickform „Workflow

Stopped“ light

Status

“Workflow

Running” icon

“Workflow

Running” light

KNIME User Training 115

Errors and Warnings

Report

Export report as

Results of past

Executions

Outline

New Features in 2.6 & 2.7 - Highlights

• Enhanced database functionality

• File Handling node collection

• More flexible R integration

• Streaming API

• Better (Java) scripting support

• Hypothesis testing nodes

• UI Changes

• Database update and delete

Enhanced DB functionality

• New type support: Boolean and Blobs

Enhanced DB functionality

• Set of nodes to read, (un)zip, copy, move,

convert, … files

• Add notion of unique resource identifier

(URI) and mime types

Used in 3rd party extensions

• Nodes to up and

download files:

ssh, http, ftp, …

File Handling Nodes

• Collection of Nodes to extract statistical measures

• Different t-tests

• Anova

• (Crosstab)

Hypotheses Testing Nodes

• Before KNIME 2.7:

• With KNIME 2.7:

Flexible R integration

• Enhanced functionality:

• define multiple outputs at once

• Script templates

• Better editor

• Syntax highlighting

• Auto completion

Scripting – Java Snippet & friends

Enhanced programming interface in KNIME

enabling nodes to be streamed and

distributed.

Streaming API

KNIME Explorer replaces “Workflow Projects”

KNIME UI Changes

Customizable Node repository

(getting from 1500+ nodes to <100)

KNIME UI Changes

Tons more …

Summary

Discussed KNIME Usage Examples

check “Examples” Server for even more

New functionality constantly added, thanks to

community, partners and customers

And more is coming…

002 ugm2013 whats new final

Documents

neuron-spectrum-am tm 002 final

calendario 2013 final 002

002 4th(final)

resultado final provision externa 002-2013

informe hidrologia final 002 enviar urgente

msf volte scn 002.final

hro bulletin template final 20191018 (002)

km 554e-20171010152144 - home - department of … · final...

pyatl dec 2015 color final (002)

final answers chap 002

whats left of me final

mega whats 2010 questions final

002 eco final

speaker slides 2016 final (002)

revit mep 2010 whats new training final

cuban missile crisis. april 1st do now: your final whats my...

whats on tap final presentation

sms-edital 002-2014 resultado final

002 final tech spec

whats shakn final