Active Folder : Integrating All Activities of Simulation on File System

Active Folder : Integrating All Activities of Simulation on File System

Suntae Hwang ( sthwang@kookmin.ac.kr )Daeyoung Heo ( dyheo@kookmin.ac.kr )

School of Computer ScienceKookmin University

Apr 10 2014

2

Active Folder: Integrating All Activities of Simula-tion on File System Active Folder: Integrating All Activities of Simula-tion on File System

⊙Active Folder – good for case comparative study●Tasks○Described as regular folders and files

●Products○Input or output of simulation ○Can be handled like regular file by using legacy software○Contains provenance information (meta data, task info,

etc)○Can be reproduced by the task which is extracted from the

provenance information●Apps(Computing Resource)○Computing server(Local, Grid, Cloud, what ever, …) is reg-

istered as regular folders and files○To submit a Job(task), just Drag&Drop the task folder to the

folder which represents computing server

ActiveFolder

3

Active Folder: Integrating All Activities of Simula-tion on File System Active Folder: Integrating All Activities of Simula-tion on File System

Storage(Cloud, Local, ..)

Producing

Sharing

Analizing

• Products• Tasks• Apps

User’s own environment

Computing(Cloud, Grid, Local, …)

Or Searching(DB, IR, …)

4

Formal Definition of Spaces in Active Folder Formal Definition of Spaces in Active Folder

Definition 1. Set P for Product

Definition 2. Set T for taskDefinition 3. Set A for producing/reproducingDefinition 4. Set M for monitoring.

5

Active Folder: Product and ReproducingActive Folder: Product and Reproducing

DATA

History

TaskProduct

Fall3D

Visualize & Analysis

Procedure

𝒍𝒆𝒕 𝒕=𝒕𝒂𝒔𝒌 (𝒑)𝒂𝒑𝒑(𝒕 ,𝑲 ,𝑷𝒌)

Differentparame-ters

Task can be ex-tracted from prod-ucts and used for producing similar results with differ-ent parameters

Product consists of data, history info and task(provenance)

6

Active Folder: Products Searching and ComparisonActive Folder: Products Searching and Comparison

⊙Similar or interested products can be searched by their provenance information and compared by var-ious legacy tools on conventional file system

7

Case Study : Volcano Eruption Simulation using Ac-tive FolderCase Study : Volcano Eruption Simulation using Ac-tive Folder

8

Short history of Active FolderShort history of Active Folder

⊙Active Folder on DropBox+EC2●Cost & Performance Problem with very large files

⊙Active Folder on NAS⊙NAS(Network Attached Storage)● Large Volume Storage● Network File System ( NFS,

SMB/CIFS, AFP … )● Most vendors support Cloud

solution like DropBox

at PRAGMA 25

at PRAGMA 24

http://www.youtube.com/watch?v=vXTh3tCHEPE

9

Spin

1000

Temperature273K

Planning

File & Folder

Everything is

Applications

Executio

nDrag & Drop

SSH

CHARMM

AMBER

NAMD

GAUSSIAN

ActiveFolder . . . ...

CHARM

M

SimulationScript

Activity

CHARMMT-SPACE

ssh.server.com

Server

A-SPACE

Active Folder Features 1:Computing by manipulating files or folders, which makes only files visible by hiding correlation of computing method, parameters, result files and computing re-sources

10

Spin

1000

Temperature273K

Activ

eFol

der . .

. ...

CHARM

M

Sim

ulat

ion

Script

How to create this product?

Products

P-SPACE

Product

Product

Data Provenance

Relationship

PROVIDED / USED

History Board

Created by CHARMM Activity

From ssh.server.com

At 2014.m.d

StageOut : 2014.m.d 12:30

StageOut : 2014.m.d 13:02

Active Folder Features 2:Keep the way of computing meth-ods and parameters with result files silently

11

Activ

eFol

der . .

. ...

CHARM

M

Sim

ulat

ion

Script

Temperature300K

Temperature293K

Searching

Data Provenance

Parameters

Para

mete

rs

Para

met

ers

Para

met

ers

Parameters

Para

mete

rs

Para

mete

rs

Para

mete

rs

Active Folder Features 3:Searching result files and other core-lated ones

12

Activ

eFol

der . .

. ...

CHARM

M

Sim

ulat

ion

Script

Reproduction

Using data provenance

Extracting

Activity

CHARMMT-SPACE

Parameters

Para

mete

rs

Para

mete

rs

Temperature

293K

ApplyingNew Methods/Parameters

Para

mete

rs

Para

mete

rs

Para

mete

rs

Para

mete

rs

Para

mete

rs

Para

mete

rs

Execution

Active Folder Features 4:Computing by manipulating files or folders, Re-generating either the same or similar ones by existing re-sult files

13

ssh.server.com

Server

A-SPACE

CHARMMSSH

Products

Charmm.dcd file created

StageOut : 2014.m.d 12:30

Charmm.dcd file modified

StageOut : 2014.m.d 1:13

stdout.afsx.log file created

StageOut : 2014.m.d 1:14

stdout.afsx.log file modified

StageOut : 2014.m.d 1:15

Tim

eli

ne

Data Synchronization

Between server and your active folder

• Automatic Stage-Out to ActiveFolder• Incremental update.

Active Folder Features 5:Returns computing results by progressive synchronization without connecting computing server

14

File Navigator

Products

P-SPACE

Activity

CHARMMT-SPACE

ssh.server.com

Server

A-SPACE

SAMBA

FTP

NFS

Data Delivery

Support Legacy File ServiceYour Desktop, Laptop or Mobile

Mic

rosoft

Win

dow

s

MacO

SX

Lin

ux

Smartphone

Clo

ud

Sto

rag

e

ActiveFolder

Apps Activities Products

Open this files by your favorite softwares

Active Folder Features 6:Support various file service such as NFS, Samba(CIFS) and FTP

15

Active Folder: Overall ViewActive Folder: Overall View

Products

P-SPACE

Activity

CHARMMT-SPACE

ssh.server.com

Servers

A-SPACE

Searching and Analysis

ActiveFolder

16

Active Folder: Software ArchitectureActive Folder: Software Architecture

17

ActiveFolder

ComputingEnvironment(Large scale

service)

ActiveFolder

Satellite Service

* Private Space - File Management - Private Archiving* For Power-User

Satellite Service

ActiveFolder

Satellite Service

Usage of Active FolderUsage of Active Folder

⊙Good for managing & sharing data in a small group of Lab

⊙For Power-users us-ing legacy SW in their own computing environment (with-out GUI sometimes)

18

Active Folder SiteActive Folder Site

⊙Active Folder is released

●http://www.activefolder.org

⊙Downloads ●Binary file for general installation(Synology(NAS) tested)●RPM for Redhat 6/CentOS 6

⊙Coming soon●Product searching interface●Synology packaging for NAS●Integrated using NAS and DropBox together

Attention! .org only, not .com

19

Thank you!

Questions?