r, paralelización, datos masivos y aplicaciones web...

R enBioinformática:paralelización y

Context

Parallelizingcode

Web applications

Large data setsandparallelization

R, C, andcompression onthe fly

Conclusions,future, et al.

Appendix

R, paralelización, datos masivos yaplicaciones web: ejemplos del uso de R

en bioinformática

Ramón Díaz-Uriarte

Dept. BioquímicaUniversidad Autónoma de Madrid

Madrid, Spainrdiaz02@gmail.com

http://ligarto.org/rdiaz

III Jornadas de Usuarios de R17-Noviembre-2011

(1 : 58)

Context

Parallelizingcode

Web applications

Appendix

License and copyright

This work is Copyright, c©, 2011, Ramón Díaz-Uriarte, andis licensed under the Creative Commons

Attribution-NonCommercial-ShareAlike License. To view acopy of this license, visit

http://creativecommons.org/licenses/by-nc-sa/3.0/ or send a letter to Creative Commons,559 Nathan Abbott Way, Stanford, California 94305, USA.

*****************************Please, respect the copyright. This material is provided freely, and if you use

it, I only ask that you use it according to the (very permissive) terms of the

license: attribution, non-comercial use, and a share alike license. If you have

any doubts, ask me. (2 : 58)

Context

Parallelizingcode

Web applications

Appendix

Outline

Context

Parallelizing code

Web applications

Large data sets and parallelization

R, C, and compression on the fly

Conclusions, future, et al.

Appendix

(3 : 58)

ContextBiological context

Computational context

Parallelizingcode

Web applications

Appendix

ContextBiological contextComputational context

Parallelizing code

Web applications

Appendix

(4 : 58)

Parallelizingcode

Web applications

Appendix

Chromosomes

From the Wikipedia; original sourcehttp://www.genome.gov/Pages/Hyperion//DIR/VIP/

Glossary/Illustration/karyotype.shtml(5 : 58)

Parallelizingcode

Web applications

Appendix

DNA→ protein

(From O. Rueda’s PhD Thesis)(6 : 58)

Parallelizingcode

Web applications

Appendix

Data from a microarray experiment

Slide from Gema Moreno Bueno, Department of Biochemistry, UAM(7 : 58)

Parallelizingcode

Web applications

Appendix

More microarray data

Modified from http://www2.warwick.ac.uk/fac/sci/moac/

students/peter_cock/r/heatmap/scaled_color_key.png

(8 : 58)

Parallelizingcode

Web applications

Appendix

DNA→ protein

(From O. Rueda’s PhD Thesis)(9 : 58)

Parallelizingcode

Web applications

Appendix

Chromosome

Olshen, 2005

Barrett et al., 2004

Arrays: a dot is a DNA fragment. Each array a sample. Each array all chromosomes. (For analysis, location in chromosome matters)

Hupe & Barillot, 2005

Calling gains and losses: hypothesistesting

Inferring number of copy gains/losses: estimation L

(10 : 58)

Parallelizingcode

Web applications

Appendix

Data, data, data (in Gigabytes)

Expression arrays (mRNA) > 40,000 probesCopy number with aCGH > 400,000 common;

some > 4 x 106

. . . . . .

(11 : 58)

Parallelizingcode

Web applications

Appendix

Multicores and computing clusters

Increases in CPU speed slowed down (< 20% peryear since 2002).Increase in the number of “cores”: 2, 4, 8. Next 10years?Inexpensive computing clusters with off-the-shelfcomponents.Must design our programs from the start: parallelprogramming

Image from http://faq.distributed.net/

(12 : 58)

Context

Parallelizingcode

Web applications

Appendix

Context

Parallelizing code

Web applications

Appendix

(13 : 58)

Context

Parallelizingcode

Web applications

Appendix

Standalone

(14 : 58)

Statistical Computingin Bioinformatics

Develop statistical methods

Implement existingapproaches

Implement for statisticians and bioinformaticians

Implement for wet lab users

- Parallel Computing

- Fault tolerance

Web apps:- User friendly

- No installation

- Statistical rigour - Best practices

Increased speed (40x - 60x)

Context

Parallelizingcode

Web applications

Appendix

R code

Code available for many procedures (but a few yearsago none parallelized!)Many computations embarrassingly parallelizable:

I bootstrapping and cross-validationI arrays (or samples)I arrays by chromosomesI parallel chains in MCMC

Figure production can be parallelized

(15 : 58)

Context

Parallelizingcode

Web applications

Appendix

Parallelizing R code

(Implement missing functionality: R/C)MPI: R packages Rmpi, papply, snow, snowfallLoad balancedWrappers over “mid level” functions in package: easeupdatingParallelize:

I Bootstrap samples/Cross-val. runs.I arraysI arrays by chromosomesI (or a combination of both)

(16 : 58)

Context

Parallelizingcode

Web applications

Appendix

Is it worth it?

Are speed improvements really worth the effort?Over what range of problems do see improvements?With what hardware can we see improvements?

(17 : 58)

Context

Parallelizingcode

Web applications

Appendix

What do we gain?

(18 : 58)

10 50 100 15020

● ● ●●

● ● ●

Sequential code

Parallelized code

10 50 100 150

●●

● ●

10 50 100 150

● ●●

●●

●BioHMM

10 50 100 150

●●

20,000 genes

Number of arrays (samples)

Context

Parallelizingcode

Web applications

Appendix

What do we gain?

Are speed improvements really worth the effort?Your effort: “R CMD INSTALL ADaCGH2”.

Over what range of problems do see improvements?10 to 103 arrays/samples;104 to 106 spots/genes.

With what hardware can we see improvements?2 cores to 120 cores.

Smaller clusters: more cost effectiveSingle node/multi-core: lesscommunication overhead

(19 : 58)

Context

Parallelizingcode

Web applications

Appendix

Where is this running?

varSelRF (CRAN)ADaCGH2 (BioConductor)SignS (launchpad:http://launchpad.net/signs)

(20 : 58)

Context

Parallelizingcode

Web applicationsWeb apps: how

Appendix

Context

Parallelizing code

Appendix

(21 : 58)

Context

Parallelizingcode

Appendix

Applications for wet lab researchers

Analyze data in a reasonably short time.User friendly access to methods that are statisticallyrigorous.

(22 : 58)

Context

Parallelizingcode

Appendix

Web-based applications

User-friendly interface.No hardware/software hassles for end users.Parallelization is transparent.Method selection can be partially transferred (to us).Short user wall time: use (hardware/software)resources rarely available to individual biomedicalresearchersJust type in a URL:http://www.some-application

Image modified from http://faq.distributed.net/

(23 : 58)

Context

Parallelizingcode

Appendix

Sometimes collaborations feel like . . .

(From http://www.bitacoradegalileo.com/2010/11/16/giordano-bruno-en-la-cara-oculta-de-la-luna/)

(24 : 58)

Context

Parallelizingcode

Appendix

Parallelization in web applications

(25 : 58)

- Fault tolerance

- No installation

Context

Parallelizingcode

Appendix

Main web-based applications

(26 : 58)

Dealing with raw data

Statistical analysis (sensu stricto)

Annotation and Interpretation

Remove artifactsfrom microarrays

- Missing data- Replicate spots

DNMAD preP

Differentiallyexpressed

Select genesfor

classification

Tnasas GeneSrFPomelo_II

Molecular signatures

survival data

SegmentaCGH

WaviCGH ADaCGH

Interpret results

IDClight PaLS

Context

Parallelizingcode

Appendix

What do we gain?

(27 : 58)

1 5 10 20

●●

CGHseg●

●●

HMM15000 genes, 40 arrays

Number of simultaneous users

Context

Parallelizingcode

Appendix

How it works: some key ideas

Each runI Parallelization (transparent for users)I Fault-tolerance (network problems, machine crashes,

bugs)I Check-pointing

Periodic tasks (keep system running 24h, 365 d)I Automatic monitorizationI Automated testing suite

(28 : 58)

Context

Parallelizingcode

Appendix

What happens

(29 : 58)

UserHead node (LVS):Send request to

one of the servers.

CGI: data checking,file upload

Execution: Python program

- Setting up LAM/MPI- Starting R

- Fault tolerance- Checking termination of R

- Checking run errors- Formatting output

R program

Autorefreshing HTMLuntil final results

Sequential code Parallelized code

Context

Parallelizingcode

Appendix

What happens: details

(30 : 58)

Head node (LVS)

Server 2

Server 1

Continue R execution till end

Apache

Server 3Server n

Read dataCreate MPI universe

Launch R, RmpiMonitor R execution

Maintain R process counters

(slave)

(Master)

(slave)(slave)Rmpi started

Halt MPI universe Produce and return results pages

Is R done?Yes

Return autorefreshing page

Stop execution Halt MPI universe

Return error

Not after K attempts

Context

Parallelizingcode

Appendix

MPI details

(31 : 58)

Sleep Can we run?(Count other lam daemons)No

Boot (new)LAM/MPI

Start R: continue from last checkpoint Sleep

Run outof time?

Are we done?R crashed (bugs)?

MPI universe:Servers 1 ... n

NFS sharedtemporary storage

NFS sharedstorage

Segmentation and Figures (over subjects and chrom.).

Rmpi crashed?LAM/MPI/nodes crashed?

Verify servers(modify LAM defs)

Context

Parallelizingcode

Appendix

http://signs.bioinfo.cnio.es

http://wavi.bioinfo.cnio.es

http://genesrf.bioinfo.cnio.es

(32 : 58)

Context

Parallelizingcode

Web applications

Appendix

Context

Parallelizing code

Web applications

Appendix

(33 : 58)

Context

Parallelizingcode

Web applications

Appendix

Chromosome

Olshen, 2005

(34 : 58)

Context

Parallelizingcode

Web applications

Appendix

Large data sets

Millions of spotsHundreds or thousands of subjects.No need to hold everything in RAM at once.

Package ff: “memory-efficient storage of large dataon disk and fast access functions.”

Combined with:I parallelizationI shared storage

(35 : 58)

Context

Parallelizingcode

Web applications

Appendix

ff and parallelization

ff stores the object on disk.Read that object from various R processes.Different R processes can write in different ff objects

(36 : 58)

Context

Parallelizingcode

Web applications

Appendix

ff and parallelization (I)

R1 R2 Rn

Common ff object

ff1 ff2 ffn

read only

Rmaster

(37 : 58)

Context

Parallelizingcode

Web applications

Appendix

R1 R2 Rn

Common ff object

ff1 ff2 ffn

read only

Rmaster

(37 : 58)

Context

Parallelizingcode

Web applications

Appendix

R1 R2 Rn

Common ff object

ff1 ff2 ffn

read only

Rmaster

(37 : 58)

Context

Parallelizingcode

Web applications

Appendix

ff and parallelization (II)

RmasterData

read only

(multicore)

R2 RnR1

ff1 ff2 ffn

R2 RnR1

ff1 ff2 ffn

Fig.1 Fig.2 Fig.n

ffout Results

(38 : 58)

Context

Parallelizingcode

Web applications

Appendix

RmasterData

read only (multicore)

R2 RnR1

ff1 ff2 ffn

R2 RnR1

ff1 ff2 ffn

Fig.1 Fig.2 Fig.n

ffout Results

(38 : 58)

Context

Parallelizingcode

Web applications

Appendix

RmasterData

read only

(multicore)

R2 RnR1

ff1 ff2 ffn

R2 RnR1

ff1 ff2 ffn

Fig.1 Fig.2 Fig.n

ffout Results

(38 : 58)

Context

Parallelizingcode

Web applications

Appendix

RmasterData

read only

(multicore)

R2 RnR1

ff1 ff2 ffn

R2 RnR1

ff1 ff2 ffn

Fig.1 Fig.2 Fig.n

ffout Results

(38 : 58)

Context

Parallelizingcode

Web applications

Appendix

RmasterData

read only

(multicore)

R2 RnR1

ff1 ff2 ffn

R2 RnR1

ff1 ff2 ffn

Fig.1 Fig.2 Fig.n

ffout Results

(38 : 58)

Context

Parallelizingcode

Web applications

Appendix

RmasterData

read only

(multicore)

R2 RnR1

ff1 ff2 ffn

R2 RnR1

ff1 ff2 ffn

Fig.1 Fig.2 Fig.n

ffout Results

(38 : 58)

Context

Parallelizingcode

Web applications

Appendix

RmasterData

read only

(multicore)

R2 RnR1

ff1 ff2 ffn

R2 RnR1

ff1 ff2 ffn

Fig.1 Fig.2 Fig.n

ffout Results

(38 : 58)

Context

Parallelizingcode

Web applications

Appendix

RmasterData

read only

(multicore)

R2 RnR1

ff1 ff2 ffn

R2 RnR1

ff1 ff2 ffn

Fig.1 Fig.2 Fig.n

ffout Results

(38 : 58)

Context

Parallelizingcode

Web applications

Appendix

ADaCGH2 (BioConductor package)Web-based applicationhttp://wavi.bioinfo.cnio.es.

(39 : 58)

Context

Parallelizingcode

Web applications

Appendix

Context

Parallelizing code

Web applications

Appendix

(40 : 58)

Context

Parallelizingcode

Web applications

Appendix

Chromosome

Olshen, 2005

(41 : 58)

Context

Parallelizingcode

Web applications

Appendix

Store and access (large) pre-computed results

HMM for aCGH data with Reversible Jump: ViterbiCommon regions: “count” on the Viterbi paths.

Fitting HMM/common regions: distinct operations.

C: number-crunching.R: wrapper and figures/tables.C: creates large amounts of data.

In package RJaCGH (CRAN).

(42 : 58)

Context

Parallelizingcode

Web applications

Appendix

Fit HMM

R C (HMM)

Store Viterbias gzipped file

return filenames

Find common regions

R C (common regions)pass filenames

ReadViterbi datareturn results

Figures, tables

(43 : 58)

Context

Parallelizingcode

Web applications

Appendix

Fit HMM

R C (HMM)

Store Viterbias gzipped filereturn filenames

Find common regions

Figures, tables

(43 : 58)

Context

Parallelizingcode

Web applications

Appendix

Fit HMM

R C (HMM)

Find common regions

Figures, tables

(43 : 58)

Context

Parallelizingcode

Web applications

Appendix

Fit HMM

R C (HMM)

Find common regions

Figures, tables

(43 : 58)

Context

Parallelizingcode

Web applications

Appendix

Context

Parallelizing code

Web applications

Appendix

(44 : 58)

Context

Parallelizingcode

Web applications

Appendix

Web-based: A few things we’ve learned

Configuration sucks (if you need to modify > 1 file)Too many languagesAdding test cases to the testing suites: web, RDocumentation: in the code, web pages, LATEX . . .

Too much R code to catch errorsUser interfaces: who designs them?

(45 : 58)

Context

Parallelizingcode

Web applications

Appendix

Fault tolerance and communicationManual check for errors (R ain’t Erlang)Too much network traffic

(46 : 58)

Boot (new)LAM/MPI

Start R: continue from last checkpoint

Run outof time?

Are we done?R crashed

(coding errors)?

MPI universe:Servers 1 ... n

NFS sharedtemporary storage

NFS sharedstorage

Rmpi crashed?LAM/MPI crashed?

(includes node crashes)

Verify servers(modify LAM defs)

Context

Parallelizingcode

Web applications

Appendix

Solutions?

Literate programming and org-modeAlternatives to MPI and/or use Erlang. . .Keep things as they are (only a few painful events ayear)

(47 : 58)

Context

Parallelizingcode

Web applications

Appendix

Single machine applications

Run applications in a single, multicore (e.g., 12)machineJust verify if the machine is up

(48 : 58)

Context

Parallelizingcode

Web applications

Appendix

Rethinking web-based applications

Users can get into trouble.

Sure, but we can do a good job . . .I Provide state-of-the art statistical and computational

approaches (R ;-)I Pedagogical examples and pipelinesI Minimize the chance of users getting into trouble

Web-based applications are here to stay

(49 : 58)

Context

Parallelizingcode

Web applications

Appendix

(49 : 58)

Context

Parallelizingcode

Web applications

Appendix

(49 : 58)

Context

Parallelizingcode

Web applications

Appendix

. . . so . . .

Forget about them: just write R (plus C, etc) codeGo for it

I R will do its job (which is only part of the job)I HPC availableI No problem with large data setsI But other tools and work necessary

(50 : 58)

Context

Parallelizingcode

Web applications

Appendix

Regardless of web-based applications . . .

Parallel computing can be used routinelyI (library(parallel) in R ≥ 2.14.0)

Large data sets with ff + parallelization.

(51 : 58)

Context

Parallelizingcode

Web applications

Appendix

Acknowledgements

O. M. Rueda, A. Alibés, A. Cañada, E. R. Morrissey,M. L. Neves, D. Rico.Funding: Fundación de Investigación Médica MutuaMadrileña, Project TIC2003-09331-C02-02 of theSpanish MEC and BIO2009-12458 of the SpanishMICINN. Ramón y Cajal Programme of the SpanishMinistry of Education and Science.CNIO (Spanish National Cancer Research Center).The R users and developers for a vibrant statisticalcomputing community and amazing platform.The organizing and scientific committees of the IIIJornadas de Usuarios de R.

(52 : 58)

Context

Parallelizingcode

Web applications

Appendix

Trying to fit it all

(53 : 58)

- Fault tolerance

- No installation

Informationintegration

Answer biological question

Context

Parallelizingcode

Web applications

Appendix

What do we gain?

(54 : 58)

Effect of number of arrays (number of genes = 7399)

Number of arrays (samples)

ase in s

peed (

tive to 1

20 40 80 100

1505 2476 4608 5664

620712048 23258 30102

29 51 95 118

Effect of number of genes (number of arrays = 160)

Number of genes

ase in s

peed (

tive to 1

1000 2000 4000 6000 12000 24000 48000

5804 6473 7544 8831 12355 20231 35072

106 116 144 179 268 442 788

Original SequentialNew sequential (1 CPU)Parall. 2 CPUsParall. 10 CPUsParall. 20 CPUsParall. 60 CPUs

Context

Parallelizingcode

Web applications

Appendix

What do we gain?

(55 : 58)

1 5 10 15

Breast data set (78 arrays x 4751 genes)

seconds)

1 5 10 15

DLBCL data set (160 arrays x 7399 genes)

seconds)

Context

Parallelizingcode

Web applications

Appendix

Tools. . .

(56 : 58)

Org mode R, Python, C/C++

Literate programming

Interactive debugging

ConfigFiles

Bazaar-NG

FunkLoad: - Whole system testing

- Regression testing - Stress testing

''Continuous Integration''

Commit hooks

Web app. frameworks

WebHelp

InstallScript Distributed computing

Parallel. (MPI, others) Grid computing Web services

Admin.

Context

Parallelizingcode

Web applications

Appendix

Too many languagesImpedance mismatch problem:“Building Web-based applications requires the mastering of anumber of languages/technologies (e.g. HTML, CSS, CGI, ASP,PHP, XML, etc..). Such languages and technologies werecreated to address different aspects on a by-need evolutionarymanner. The result is a plethora of tools that are fitted togetherin an ad hoc fashion.” El-Ansary, Grolaux, Van Roy, Rafea(2005) “Overcoming the Multiplicity of Languages andTechnologies for Web-Based Development Using aMulti-paradigm Approach”.

R and CHTML and Python: CGI, data entry, displayPython (and others): control and monitor MPIJavascript: AJAX and figures

(57 : 58)

Context

Parallelizingcode

Web applications

Appendix

r, paralelización, datos masivos y aplicaciones web...

Documents

paralelizaciÓn de algoritmos evolutivos como … ·...

presentación de powerpoint -...

estrategias de paralelización de heurísticas para el juego...

comparativa y analisis de variabilidad espacial entre las...

análisis de muestras complejas con r. la encuesta de...

corrigiendo errores en la paralelización con intel® ...

unidad de instrumentaciÓn sÍsmica - … · contenido de...

sisal: algoritmo de selección secuencial de variables...

intreggof un paquete para bondad de ajuste...

arquitecturas paralelas if - ehu arquitecturas paralelas 8....

clase 12 computación evolutiva técnicas avanzadas:...

libro de resumenes pdfauthor - comunidad r...

paralelización de algoritmos de optimización basados en...

contenido -...

paralelizaciÓn de algoritmo numÉrico...

esquema de paralelización h´ıbrida para máquinas de...

implementaciÓn y diseÑo de herramientas...

mapas temáticos a tiempo real - comunidad r...

oscar perpin~an lamigueiro - comunidad r...

universidad autÓnoma de chiapas facultad de ......1....