introduction to irods

Introduction to iRODS

Christine StaigerSURFsara

10th Oct 2014

iRODS Concepts

The Basics

① Installation

② Tools for administration

③ Adding data and metadata

④ iRODS rules

Contents

3

iRODS Concepts

4

What is iRODS?

5

What does iRODS provide?• Storage virtualization of different disk and tape

storage systems

• A logical namespace across storage locations

• A rule engine to automate data management according to defined policies

• Various client interfaces

6

iRODS Components

7

In the background:iRODS Resources

• (Storage) Resource is a Software or Hardware system that stores data

• 3 Resource classes:

HighLatencyLow

Latency

ArchiveResource

Cache Resource

CompoundResource

POSIX like

IO calls

Resource Groups: Distributing files

8

LowLatency

Cache Resource

POSIX like

IO calls

Resource Group

LowLatency

Cache Resource

LowLatency

Cache Resource

@Server A @Server B @Server C

Resource Groups: Data replication

9

LowLatency

Cache Resource

POSIX like

IO calls

Resource Group

LowLatency

Cache Resource

LowLatency

Cache Resource

@Server A @Server B @Server C

Replicate

10

The users view:iRODS Data Collections

Collection0

DataObject

DataObject

DataObjectData

Object A

Collection1

DataObject

DataObjectData

Object X

…/Collection0/…/Collection0/DataObject A-B…/Collection0/Collection1/…/Collection0/Collection1/DataObject X-Z

Logical Name Space (iCAT) independent from physical resources

• Similar to Unix Directory and Files• ichmod to set user/group ACLs• Core.re: msiSetDefaultResc(Resource)• .irodsEnv: irodsDefResource=Resource

11

Hands on The Basics

Demo Data GridsVirtual machines: ssh user@IP

IP adress User Password

145.100.57.159 irods1 irods104

145.100.57.162 irods2 irods204

145.100.57.164 irods3 irods304

145.100.58.150 irods4 irods404

145.100.58.155 irods6 irods604

145.100.58.157 irods7 irods704

145.100.58.158 irods8 irods804

145.100.58.160 irods9 irods904

145.100.58.161 irods10 irods1004

Installation and configuration

• iRODS3: Install script ./irodssetup• iRODS4: (rpm, dmg) package and source code

• ~/iRODS/config/irods.config

# Database configuration$DATABASE_TYPE = 'postgres';…$DATABASE_HOST = 'localhost';$DATABASE_PORT = '5432';$DATABASE_ADMIN_PASSWORD = 'irods2';$DATABASE_ADMIN_NAME = ’alice';

# iRODS configurationIRODS_HOME = '/home/alice/iRODS';$IRODS_PORT = '1247’;…$IRODS_ADMIN_PASSWORD = ’alice';$IRODS_ICAT_HOST = '';

.irodsEnv file # iRODS personal configuration file.## This file was automatically created during iRODS installation.# Created Mon Jun 18 13:13:27 2012## iRODS server host name:irodsHost ’surfsara-cloud'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource 'demoResc'# Home directory in iRODS:irodsHome '/alicesZone/home/alice'# Current directory in iRODS:irodsCwd '/alicesZone/home/alice'# Account name:irodsUserName ’alice'# Zone:irodsZone ‘alicesZone'

The .irodsEnvdetermines the zone and the user for the icommands-client.You can only use onesuch file at a time.

iRODS i-commands

Documentation: https://wiki.irods.org/index.php/icommands

(we don’t provide a exhaustive list of i-commands)

16

iRODS control• Execute ./irodsctl <arg> in the folder iRODS

• Restarting, starting and stopping the iRODS and iCAT server with• <arg> = restart, start, stop

• Restarting, starting and stopping only the iRODS server with• <arg> = irestart, istart, istop

• Initialise an account and log in with iinit

ilsresc• ilsresc lists iRODS resources and resource-groups

• Usage: ilsresc [-lvVhA] [Name]

alice@sara-cloud:~$ ilsrescdemoResc

alice@sara-cloud:~$ ilsresc –l demoRescresource name: demoResc resc id: 10010 zone: alicesZonetype: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault …

User and Data administration• Change to admin mode: iadmin• Quit admin mode: q

• iadmin mkresc, iadmin rmresc

• Create new resource: mkdir Data (in your home, not in the iRODS environment) iadmin mkresc <RescName>“unix file system” archive you.rIP.add.ress “/home/<un>/Data”

• Make/delete a user: mkuser, rmuser, moduser (iadmin) mkuser user1 rodsuser (iadmin) moduser user1 password *****

• Parameters and their values: iadmin lt, iadmin lt user_type

Put and replicate data

• Add the file /home/login/put1.txt to resource iput -v put1.txt ils -l or ils -L (gives also physical location) iput -h

• Create a collection and add data imkdir /<yourZone>/archive iput put2.txt /<yourZone>/archive/put2.txt Check physical location of the data ils -L /<yourZone>/archive

• Replicate data across resources: irepl -R <YourResc> put1.txt• Check with ils -L• Trim number of copies: itrim -S <YourResc> -N 1 put1.txt

iRODS Metadata• Commands: imeta, iquest, idbo

• Add metadata: imeta add -d put1.txt “Date” “08.10.2013” imeta add -d put1.txt “AnotherDate” “04.04.2014” imeta ls -d put1.txt

• Remove metadata: imeta rm -d put1.txt “Date” “08.10.2013”

• Querying the iCAT: iquest "SELECT DATA_NAME,DATA_CHECKSUM WHERE

DATA_RESC_NAME like ’%Resc%’” DATA_NAME = put1.txt DATA_CHECKSUM = c16d7d3488677a3348b12eb82795d28c

iRODS Microservices and rules

• Micro-services are small, well-defined procedures/functions that perform a simple task

• Examples: “msiCollCreate”, “msiDataObjRepl”, “msiDataObjChksum”, …

• Convert any C procedure (createCollection) to a microservice by creating an interface routine (msiCreateCollection)

• Micro-services can be combined in rules to enforce policies• Rules can be executed on action, with time-delay or periodically

22

iRODS Rule Engine

Replicate to another resource

replicate{ msiWriteRodsLog("starting replication", *status); msiWriteRodsLog("source = *source", *status); msiDataObjRepl(*source, *destRescName, *CC);}INPUT *source=”/<yourZone>/home/<UN>/put1.txt”, *destRescName=“testResc”OUTPUT ruleExecOut

Save as replicate.r Execute irule -vF replicate.r Check with ils -L

Replicate to another collection• Useful for backup of a data collection• Changing the storage medium

• iput put2.txt

replicate{ msiWriteRodsLog("starting replication", *status); msiWriteRodsLog("source = *source", *status); msiDataObjRsync(*source, "IRODS_TO_COLLECTION", *destRescName, *destination, *rsyncStatus);}INPUT *source="/<yourZone>/home/<UN>/put2.txt", *destination=”/<yourZone>/coll1", *destRescName=“<RescName>"OUTPUT ruleExecOut

• ils -L /irodsXZone/coll1/home/irodsX

Thank you !

Appendix

Trouble shooting

ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = -9000 status = -9000 SYS EXCEED CONNECT CNT

Host configuration problem on the server.Solution: Edit iRODS/server/config/irodsHost

localhost you.rIP.XX.XX surfsara-cloudlocalhost 127.0.0.1

./irodsctl restart