le risorse open sourceper la ricerca in ambito scientifico · le risorse open sourceper la ricerca...

1

Torino 6 Giugno 2007

Le risorse Open Source per la

ricerca in ambito scientifico

Claudio Gheller

CINECA, Consorzio Interuniversitario

[email protected]


Science and Computers: a close link

• Computers are fundamental in any aspect of research

• Development of new instruments

• Control and management of instruments

• Data acquisition, telemetry

• Data management and compression

• Data reduction and analysis

• Numerical simulations

• Comparison between numerical simulations and observations/experiments

FROM

OBSERVATIONSTO THEORY

2


Science and Computers: huge requests

Often scientific numerical applications requires enormous computationalresources

A typical high end run, today can need:

• 100 to 500 Gbyte of RAM memory (the memory of 100 to 500 pcs)

• More than 100000 computing hours (more or less 11.5 years)

• 10 to 100 Tbyte of resident disk space (thousand of times that of yourpc)

Special computing resources are needed!!!


HPC resources

Untill a decade ago, HPC for science was represented by:

• specialized pieces of hardware

• Specific (and propretary) management software (Operating Systems)

• Specific (and propretary) compilers and user libraries

Today the trend has completely changed.

HPC MEANS PARALLEL COMPUTING!!!

3


Hardware

With the rise of the Beowulfclusters, a new generation of supercomputing systemsbacame available, basedon OFF-THE-SHELF products(but be careful…)

Low cost HPC, but with some penalties…


BlueGene

4


Linux

Linux is now packaged for different uses in Linux distributions, which contain the kernel along with a variety of other software packages tailored to requirements.

A typical desktop Linux distribution comprises a Linux kernel, GNU tools and

libraries, additional software, documentation, a window system, window

manager, and a desktop environment. Most of the included software is freesoftware/open-source software which is distributed by its maintainers both as pre-compiled binaries and in source code form, allowing users to modify and compilethe original source code if they wish. Other software included with some distributions may be proprietary and may not be available in source code form.

Linux is a Unix-like computer operating system. Linux is one of the most prominent examples of free software and open source development; its underlying source code can be modified, used, and redistributed by anyone, freely.

The Linux kernel was first released to the public on 17 September 1991, for the Intel x86 PC architecture. The kernel was augmented with system utilities and

libraries from the GNU project to create a complete

and friendly operating system, which later led to the alternate term GNU/Linux.


Linux distributions

Slackware, one of the first Linux distributions, founded in 1993

Debian, a non-commercial distribution maintained by a volunteer developer

community with a strong commitment to free software principles

Ubuntu, a newly popular desktop distribution derived from Debian

Red Hat, maintained by the American company of the same name, which also

provides a community version in the form of Fedora

CentOS, a distribution derived from the same sources used by Red Hat,

Mandriva, a Red Hat derivative popular in France and Brazil,

openSUSE, originally derived from Slackware with the system management

software borrowed from Red Hat, maintained by the company Novell

Gentoo, a distribution targeted at power users, known for its FreeBSD Ports-like

automated system for compiling applications from source code

Knoppix, a LiveCD distribution that runs completely from removable media and

without installation to a hard disk

Linspire, a commercial desktop distribution based on Debian,

5


The GNU Project

The founding goal of the project was, in the words of its initial announcement, to

develop "a sufficient body of free software [...] to get along without any software

that is not free.“

Particularly interesting the GNU Compiler Collection, a fundamental instrumento

for code developers, that includes front ends for C, C++, Objective-C, Fortran,

Java, and Ada, as well as libraries for these languages (libstdc++, libgcj,...).

Many libraries like MPICH, LAM-MPI, GSL, PETSc, FFTW…

The GNU Project is a free software project, announced in

1983 by Richard Stallman. It initiated the GNU

operating system, software development for which

began in January 1984.


Simulation Software

AMRFLASH

Tree N-Body + SPH

GADGET

http://www.mpa-garching.mpe.de/galform/gadget

AMRCactusCodehttp://www.cactuscode.org/

Tree N-bodyFlyhttp://www.ct.astro.it/fly/

AMRPlutohttp://plutocode.to.astro.it/

AMREnzohttp://www.cosmos.ucsd.edu/enzo/

TypeSimulation packages

Some of the most ambitious packages have the goal of describing the behaviour

of complex systems… For astrophysics, many codes are available open

source:

6


The Millenium simulation

The Virgo consortium, an international group of astrophysicists from the UK,

Germany, Japan, Canada and the USA has realized the largest and most

realistic simulation ever of the growth of cosmic structure and the formation

of galaxies and clusters.

The "Millennium Simulation" was performed using the Gadget2 code and

employed more than 10 billion particles of matter to trace the evolution of

the matter distribution in a cubic region of the Universe over 2 billion light-

years on a side. It kept the principal supercomputer at the Max Planck

Society's Supercomputing Centre in Garching, Germany occupied for more

than a month. By applying sophisticated modelling techniques to the 25

Terabytes (25 million Megabytes) of stored output, Virgo scientists are able

to recreate evolutionary histories for the approximately 20 million galaxies

which populate this enormous volume


Data management tools

Several data management tools are available as free software:

MySQL is a multithreaded, multi-user SQL (Strucured Query Language) database

management system (DBMS) which has more than 10 million installations

PostgreSQL is a free software object-relational database management system

(ORDBMS), released under a BSD-style license.

HDF5 a standard and portable data format plus libraries and utilities

7


Data analysis and visualization tools

Plenty of data analysis and visualization tools and libraries are available as free

software:

E.g. Visualization Toolkit (VTK) is an open source, freely available software system

for 3D computer graphics, image processing, and visualization used by thousands

of researchers and developers around the world. VTK consists of a C++ class

library, and several interpreted interface layers including Tcl/Tk, Java, and Python.

Professional support and products for VTK are provided by Kitware, Inc. VTK

supports a wide variety of visualization algorithms including scalar, vector, tensor,

texture, and volumetric methods; and advanced modeling techniques such as

implicit modeling, polygon reduction, mesh smoothing, cutting, contouring, and

Delaunay triangulation. In addition, dozens of imaging algorithms have been

directly integrated to allow the user to mix 2D imaging / 3D graphics algorithms

and data.


Data analysis and visualization tools: VisIVO

VisIVO (which stands for Visualization Interface for the Virtual Observatory) is a visualization and analysis software for astrophysical data. VisIVO can handle bothobservational and theoretical data (i.e. produced by numerical simulations). Itcan be used both as a stand-alone application, that acts on local files, and as aninterface to the Virtual Observatory framework, from which it can retrieve the data.

The development of VisIVO is funded by the Italian Institute for Astronomy and Astrophysics (INAF) and by CINECA

See: http://visivo.cineca.it

8


Vantaggi del software libero

* essendo possibile modificare liberamente il software, è possibile personalizzarlo

ed adattarlo alla proprie esigenze

* il codice sorgente è sottoposto ad una revisione da parte di moltissime persone,

pertanto è più difficile che contenga bachi e malfunzionamenti. In ogni caso, è

sempre possibile per chiunque tenere un indice pubblico dei problemi, in modo

che gli utenti li conoscano

* se viene scoperto un baco o una falla di sicurezza, la sua correzione di solito è

molto rapida

* essendo il sorgente liberamente consultabile, è molto difficile inserire

intenzionalmente nel software backdoor, cavalli di Troia o spyware senza che

questi vengano prontamente scoperti ed eliminati, come invece è accaduto per

alcuni software commerciali

* non esistendo standard proprietari, le cui specifiche sono normalmente segrete,

è molto più facile costruire software interoperabile

* la complessità e le dimensioni di alcuni software liberi (ad esempio, dei sistemi

operativi) è tale che è necessario il supporto commerciale di un'azienda; il

software libero si presta a creare nuove opportunità di business nel campo della

formazione e del supporto, oltre che della eventuale personalizzazione del

software.


Limiti del software libero

* essendo un lavoro volontario, lo sviluppo del software libero sarebbe più lento

rispetto al software proprietario;

* alcune tipologie di software, soprattutto di nicchia, non sarebbero disponibili

come software libero; infatti i software di nicchia non avrebbero abbastanza

utenti per la creazione di una comunità che supporti lo sviluppo del software .

* lo sviluppo del software libero avrebbe una struttura anarchica, che porta a

risultati incoerenti e ad una mancanza di uniformità e consistenza;

* nonostante il codice sorgente sia liberamente disponibile, non tutti sono in grado

di apportarvi modifiche, inoltre l'usabilità spesso non è ottimale per un utente

inesperto;

* anche se spesso è disponibile molta documentazione, è sovente incompleta o

non aggiornata.

Alcune di queste critiche sono talvolta frutto di un'errata comprensione del

software libero. Molte persone tendono infatti a considerare il software libero

come prodotto esclusivamente da volontari, mentre sono molti i casi in cui è

semplicemente un modello di sviluppo adottato a livello aziendale.

9


Applicazioni commerciali del software libero

* Un software libero non deve necessariamente essere sviluppato a titolo gratuito

o a fondo perduto. Purché si rispettino i vincoli della licenza d'uso, è possibile

vendere un software libero;

* Vi sono alcune aziende che adottano il modello di sviluppo del software libero

per i propri prodotti commerciali. Il ritorno economico in questo caso può derivare

dalla fornitura di assistenza e di know-how.

* Lo stesso software può venire licenziato sia come proprietario, sia come software

libero. La versione libera talvolta dispone di meno funzionalità, o è limitata ad un

numero ristretto di piattaforme.

* Vi sono poi aziende che sono strutturate integralmente per la vendita e

l'assistenza di un determinato software libero: esempi classici sono alcune

distribuzioni di Linux, come Red Hat o SUSE. Queste aziende utilizzano come base il

software sviluppato dalla comunità, aggiungendo una serie di tool di

configurazione o sviluppo, curando gli aspetti più tecnici e dando agli utenti finali

un'assistenza mirata.

* Sfruttando le caratteristiche della licenza BSD, alcune aziende preferiscono

invece partire da un software libero per sviluppare un prodotto non libero.


Dynamical simulations:

The difficult story...

• We want to follow the evolution

of a physical system according

to some basic rules (gravity, fluid

dynamics, cooling...).

• Typically:

• Complex physics

• Plenty of data

• Complex algorithms

• Lots of communications

• Unbalanced behaviour

• Huge data files

10


Numerical simulations in astrophysics. Example 1

• The solar systemThe sun, birth and evolution of

the planets…

• CodeParticle based – SPH (Smoothed

Particle Hydrodynamics)

• PhysicsGravity, fluid dynamics, radiative

cooling

• ScalesSpatial = 109 km (100 a.u.)

Mass = 1031 kg (10 solar masses)



• Cluster of starsbirth and evolution of the stars,

cluster evolution



• PhysicsGravity, fluid dynamics,

radiative cooling

• ScalesSpatial = 1012 km (100000 a.u.)

Mass = 1033 kg (1000 solarmasses)

11



• Galaxy formationbirth and evolution of galaxies

(stars,blue + gas,red)



• PhysicsGravity, fluid dynamics,

radiative cooling

• ScalesSpatial = 1014 km (10 pc)




• Galaxy cluster formationbirth and evolution of a cluster

of galaxy. Gas behaviour.

• CodeGadget, N-Body + SPH

• PhysicsGravity, fluid dynamics

• ScalesSpatial = 1020 km (10 Mpc)


le risorse open sourceper la ricerca in ambito scientifico · le risorse open sourceper la ricerca...

Documents