Download - iRODS Tutorial II. Data Grid Administration
iRODS Tutorial
II. Data Grid Administration
Leesa Brieger
2
iRODS Tutorial Preview
I. iRODS Getting Started – Unix client (icommands)– Usage
II. iRODS Data Grid Administration– Installing server and iCAT– Changing default settings– Setting up users– Installing resource server and adding new resources
to a data grid/zone– Federating with other grids/zones, remote users– Microservices and rules for policy implementation
and enforcement
3
II. iRODS Data Grid Administration
4
iRODS Info• Main page: http://irods.org
• Github repositories: https://github.com/irods/ (Request a guest account to get access.)
• Github tickets (issues): https://github.com/irods/irods/issues?state=open
• Chat list: [email protected]
• The original iRODS wiki: https://wiki.irods.org
• The iRODS Manual: http://irods.org/dev/wp-content/uploads/2014/03/irods-manual-4.0.0.pdf
5
iRODS Books
Available from Amazon
• The integrated Rule-Oriented Data System (iRODS) Micro-service Workbookhttp://www.amazon.com/dp/1466469129
• An updated microservices reference book is coming out soon.
6
iRODS Download• Download link for binary packages from the iRODS main page
http://irods.org/download/
• Open source, BSD license
• Binary packages – rpm and deb – available for CentOS, Red Hat, Debian, Ubuntu, SUSE. Planned: Mac OSX, Windows– See http://irods.org/2014/03/irods-4-0-0-released/ – 4.0 tested platforms: CentOS 5 & 6, SuSE 11 & 12, Ubuntu 10 & 12
• iRODS source is available via GitHubhttps://github.com/irods/irods
• Other iRODS packages and plug-ins are available via GitHub athttps://github.com/irods
7
iRODS iCAT-Enabled Server (IES)
• Installation – we’ll do Ubuntu and CentOS.
• This is the heart of the data grid.
• Other resources, on other platforms can be added to the data grid.
• We will use PostgreSQL as the database for the iCAT for our installations; with database plug-ins, other DBs can also be used.
8
iRODS Data Grid Installation - Ubuntu 12.04
• First, set up PostgreSQL> sudo apt-get install postgresql-9.3
This creates a Unix service account “postgres”, the DB superuser.
• As user postgres, create a DB role/user “irods”, with rights to create DBs in the Postgres instance
> sudo su – postgres> createuser -d -e -l -P irods
Enter password for new role: ******* Enter it again: ******* CREATE ROLE irods PASSWORD
'md50061c043f3987cc21363299b9b1c0c3f' NOSUPERUSER CREATEDB NOCREATEROLE INHERIT LOGIN;
> sudo exit
continued…
Enter the “irods” DB pw here; this will be usedas the iCAT admin pw.
Installing the iCAT-enabled Server
9
• Once “irods” is a user/role for the database, then install iRODS• Download the iRODS packages
– irods-icat-4.0.0-64bit.deb – irods-database-plugin-postgres-1.0.deb
• Install the packages> sudo dpkg -i irods-icat-4.0.0-64bit.deb irods-database-plugin-postgres-1.0.deb> sudo apt-get -f install
• Open the ports for iRODS client and server contact and parallel transfers
> sudo iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 1247 –j ACCEPT> sudo iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 20000:20199 -j ACCEPT> sudo iptables -A INPUT -m state --state NEW -m udp -p udp –dport 20000:20199 –j ACCEPT
continued…
iRODS Installation - Ubuntu 12.04Installing the iCAT-enabled Server
This creates an “irods”service account.
10
• As user irods, run the setup script> sudo su – irods (Home directory for this user is /var/lib/irods)> ./packaging/setup_database.sh
This creates the iCAT database, using Postgresrole ”irods”, and starts
up the data grid, with iRODSadmin user “rods”
iRODS Installation - Ubuntu 12.04Installing the iCAT-enabled Server
11
Running the setup script setup_database.sh carries out these actions…• Prompts for config input
– hostname (localhost)– port #– DB user name and pw (user “irods” and pw as specified in the postgres setup)
• Creates an iRODS admin user “rods” with pw “rods” (This is an iRODS account, not Unix or postgres.)
• Creates configuration files in /etc/irods/
• Creates the iCAT and brings up the data grid• Creates the environment file for iRODS user “rods”
• Names the data grid (zone) and its resource with default names– tempZone– demoResc (and default physical path /var/lib/irods/iRODS/Vault)
iRODS Installation – Ubuntu 12.04Installing the iCAT-enabled Server
Change this password with
“ipasswd”
(/var/lib/irods/.irods/.irodsEnv)
12
# iRODS personal configuration file.# This file was automatically created during iRODS installation.# iRODS server host name:irodsHost 'tutorub.renci.org'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource 'demoResc'# Home directory in iRODS:irodsHome '/tempZone/home/rods'# Current directory in iRODS:irodsCwd '/tempZone/home/rods'# Account name:irodsUserName 'rods'# Zone:irodsZone 'tempZone’…
iRODS Environment File: ~/.irods/.irodsEnv
13
• This file contains the environment of the grid you want to contact.
• You can use multiple environment files to choose from among many grids (only one at a time has the name .irodsEnv) – This is only possible for Unix accounts that are not the “irods” service
account.
• NEVER change the .irodsEnv file in the “irods” service account running a grid
.irodsEnv
14
iRODS Data Grid Installation – CentOS 6
• Download the packages– RPM_INSTALLATION_HOWTO.txt– irods-icat-4.0.0-64bit-centos6.rpm– irods-database-plugin-postgres-1.0-centos6.rpm
• Get and install the packages that iRODS depends on> sudo yum install postgresql postgresql-server unixODBC
perl authd> sudo yum install postgresql-odbc
(This installs PostgreSQL that will be used for the iCAT.)
• Start the Postgres server and initialize database tables > sudo /sbin/service postgresql initdb
Initializing database: [ OK ]> sudo /sbin/service postgresql start
Starting postgresql service:
continued…
Installing the iCAT-enabled Server
Read this first
15
iRODS Data Grid Installation – CentOS 6
• As user postgres, create role/user irods with “create DB” privileges
> sudo su – postgres> createuser -d -e -l -P irods
Enter password for new role: ******* Enter it again: ******* CREATE ROLE irods PASSWORD
'md50061c043f3987cc21363299b9b1c0c3f' NOSUPERUSER CREATEDB NOCREATEROLE INHERIT LOGIN;
> exit
• Modify authd config file for xinetd.d (/etc/xinetd.d/auth) to remove the command line argument “-E”. Change [ server_args = -t60 --xerror --os -E ] to [ server_args = -t60 --xerror --os ]
continued…
Installing the iCAT-enabled Server
16
iRODS Data Grid Installation – CentOS 6
• Set the proper runlevel for authd: > sudo /sbin/chkconfig --level=3 auth on
• Restart xinetd > sudo /etc/init.d/xinetd restart
• Open your firewall, if necessary, to listen for iRODS and allow parallel transfers. Add the following to your /etc/sysconfig/iptables: -A INPUT -m state --state NEW -m tcp -p tcp --dport 1247 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 20000:20199 -j ACCEPT -A INPUT -m state --state NEW -m udp -p udp --dport 20000:20199 -j ACCEPT
• Restart the firewall: > sudo service iptables restart
continued…
Installing the iCAT-enabled Server
17
iRODS Data Grid Installation – CentOS 6Installing the iCAT-enabled Server
• Download the iRODS package and install> sudo rpm -i irods-icat-4.0.0-64bit-centos6.rpm
• As Unix user irods, run the database setup script:> sudo su – irods (Home directory for this user is /var/lib/irods)> ./packaging/setup_database.sh
• Respond to script promptsDB server's hostname: localhost (<--- "localhost" here for a local DB)port: 5432DB name: ICATusername: irodspw: *******
Enter your iCAT DBadmin pw here.
This creates Unixservice
account “irods”.
18
Running the setup script setup_database.sh carries out these actions…• Prompts for config input
– hostname (localhost)– port #– DB user name and pw (user “irods” and pw as specified in the postgres setup
• Creates an iRODS admin user “rods” with pw “rods” (This is an iRODS account, not Unix or postgres.)
• Creates configuration files in /etc/irods/
• Creates the iCAT and brings up the data grid• Creates the environment file for iRODS user “rods”
• Names the data grid (zone) and its resource with default names– tempZone– demoResc (and default physical path /var/lib/irods/iRODS/Vault)
iRODS Installation – CentOS 6Installing the iCAT-enabled Server
Change this password with
“ipasswd”
(/var/lib/irods/.irods/.irodsEnv)
19
Installing the iCAT-Enabled Server (IES) with a Remote iCAT PostgreSQL Database
On the database host machine:1) Update postgresql.conf2) Update pg_hba.conf3) Restart postgres4) Configure and restart iptables
On the iRODS server host machine:1) New setup script is necessary – will be supported in 4.0.32) During the configuration setup, give the hostname of the
remote database machine
20
Installing the IES with a Remote iCAT
• Location:– Ubuntu: /etc/postgresql/9.3/main– CentOS: /var/lib/pgsql/data
• Change required: – Update listen_addresses to contain the hostname (FQDN) or IP
address of the postgres host
1) postgresql.conf
21
2) pg_hba.conf• Location:
– Ubuntu: /etc/postgresql/9.3/main– CentOS: /var/lib/pgsql/data
• Change required: – Add a line to permit the ICAT server to connect using md5
authentication: host all all <hostname> md5
Installing the IES with a Remote iCAT
22
3) Restart postgres
• Ubuntu: sudo /etc/init.d/postgres restart
• CentOS: sudo /etc/init.d/postgres-9.3 restart
Installing the IES with a Remote iCAT
23
4) iptables• Ubuntu
– Install iptables-persistent: sudo apt-get install iptables-persistent
– Edit /etc/iptables/rules.v4– Add (before the COMMIT line): -A INPUT -m state --state NEW -m tcp -p tcp --dport 5432 -j ACCEPT– Restart iptables:
/etc/init.d/iptables restart
• CentOS– Edit /etc/sysconfig/iptables– Add this line:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5432 -j ACCEPT– Restart iptables:
service iptables restart
Installing the IES with a Remote iCAT
24
Server side: 1) Get the setup script that supports this (4.0.3)
2) During the configuration setup, give the hostname of the remote DB
Installing the IES with a Remote iCAT
25
# iRODS personal configuration file.# This file was automatically created during iRODS installation.# iRODS server host name:irodsHost 'tutoricat.renci.org'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource 'demoResc'# Home directory in iRODS:irodsHome '/tempZone/home/rods'# Current directory in iRODS:irodsCwd '/tempZone/home/rods'# Account name:irodsUserName 'rods'# Zone:irodsZone 'tempZone’…
iRODS Environment File: ~/.irods/.irodsEnv
26
iRODS Post-Install• Configuration parameters saved in /etc/irods/irods.config and
/etc/irods/server.config• Install logs are in /var/lib/irods/iRODS/installLogs/irods_setup.log
• Server logs are in directory /var/lib/irods/iRODS/server/log/
• Rule file: /etc/irods/core.re contains rules that are triggered by actions in the data grid
• Environment file /var/lib/irods/.irods/.irodsEnv is created automatically for service account “irods”
• Copy this file into ~/.irods/.irodsEnv wherever you want admin access to the data grid
• Copy this file and modify it for all other users’ access to the data grid
27
iadmin – administrative functions• h for help, “h <command>” for help with that command
• q to exit
• Add new users, modify passwords, add new resources, federate to remote zones, create resource groups,…– mkresc/rmresc– mkuser/rmuser, moduser (modify passwords)– mkzone/rmzone, modzone (for federation)
• Information on users, resources, tokens, etc– lt (el-tee), “lt <token type>” for listing of that token’s possible
values– lu, lr, lz,…
28
iRODS control
./irodsctl– start– stop– restart
./irodsctl –h gives documentation on using irodsctl
Other options have been deprecated – iRODS no longer controls starting and stopping the iCAT DB
Start/stop/restart the iRODS server
29
iRODS Installation – Change Defaults• Modify default zone name (tempZone) • Modify default resource path (/var/lib/irods/iRODS/Vault/)• Modify default resource name (demoResc)• Alternative: Leave demoResc alone (name and path) and
instead, introduce a separate, new resource on the same host; just stop using demoResc
Examples1. Change zone name from “tempZone” to “ubgrid”2. Change default resource path to “/opt/irodsVault/”3. Change default resource name from “demoResc” to “ubresc1”4. Introduce new resource “ubresc2” on this server
30
iRODS Installation – Change Defaults1. Modify zone name (data grid name tempZone)
– If you’re going to do this, it’s a good idea (though not absolutely necessary) to do it before you set up other users
> iadminiadmin> modzone tempZone name ubgrid
– Once done, must edit the ~/.irods/.irodsEnv files of all users and change the zone name wherever it appears in that file (3 places)
– Must also edit any rules (in /etc/irods/core.re) that might contain the zone name
– Edit /etc/irods/irods.config: $ZONE_NAME = 'tempZone'; |-> $ZONE_NAME = ’ubgrid’
– Restart the data grid:> ./irodsctl restart
31
iRODS Installation – Change Defaults2. Modify resource path (/var/lib/irods/iRODS/Vault)
– Again, best to do this before there are other users and much data in the data grid
> iadmin iadmin> lr demoResc
resc_id: 10010resc_name: demoResczone_name: ubgridresc_type_name: unixfilesystemresc_net: tutorub.renci.orgresc_def_path: /var/lib/irods/iRODS/Vault…
iadmin> modresc demoResc path /opt/irodsVaultLevel 0: Previous resource path: /var/lib/irods/iRODS/Vault
32
iRODS Installation – Change Defaults2. Modify resource path (continued…)
> iadmin iadmin> lr demoResc
resc_id: 10010resc_name: demoResczone_name: ubgridresc_type_name: unixfilesystemresc_net: tutorub.renci.orgresc_def_path: /opt/irodsVault…
If there are data objects in the resource and the physical path names need to be changed:
> iadmin iadmin> modrescdatapaths demoResc /var/lib/irods/iRODS/Vault/ /opt/irodsVault/
“iadmin> help modrescdatapaths” for help
33
iRODS Installation – Change Defaults
3. Modify resource name (demoResc)– Best to do this before there are other users on the data grid> iadmin iadmin> lr
bundleRescdemoResc
iadmin> modresc demoResc name ubresc1
iadmin> lrbundleRescubresc1
– Edit the .irodsEnv file to rename default resource if necessary: change irodsDefResource 'demoResc’
to irodsDefResource ’ubresc1'
34
iRODS Installation – Change Defaults
3. Modify resource name (continued)– Must change the resource name used in any /etc/irods/core.re rules
that use the default name– acSetRescSchemeForCreate
{msiSetDefaultResc("demoResc","null"); }– acSetRescSchemeForRepl
{msiSetDefaultResc("demoResc","null"); }
change to– acSetRescSchemeForCreate
{msiSetDefaultResc(”ubresc1","null"); }– acSetRescSchemeForRepl {msiSetDefaultResc(”ubresc1","null"); }
35
iRODS Installation – Change Defaults
4. Introduce new resource on the same host> iadmin iadmin> lrbundleRescubresc1
iadmin> mkresc ubresc2 unixfilesystem tutorub.renci.org:/opt/irodsVault2
Creating resource:Name: "ubresc2"Type: "unixfilesystem"Host: "tutorub.renci.org"Path: "/opt/irodsVault2"Context: "”
iadmin> lrbundleRescubresc1ubresc2
Create the physical pathfirst and make sure Unixuser “irods” has write
permission on it
“iadmin> lt” is useful in choosingparameters to the mkresc command
36
Changing Default Environment
> more .irodsEnv# iRODS server host name:irodsHost 'tutorub.renci.org'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource ’ubresc1'# Home directory in iRODS:irodsHome '/ubgrid/home/rods'# Current directory in iRODS:irodsCwd '/ubgrid/home/rods'# Account name:irodsUserName 'rods'# Zone:irodsZone 'ubgrid’ …
> ilsrescubresc1ubresc2
The Ubuntu Example
Default data grid name (zone name) was
changed to ubgrid;Default demoResc was
renamed to ubresc1 and its physical path was redefined;new resource ubresc2 on the same host was added (new
physical path).
Edit .irodsEnv file to accurately reflect
the changed environment.
37
Changing Default Environment
> more .irodsEnv# iRODS server host name:irodsHost 'tutoricat.renci.org'# iRODS server port number:irodsPort 1247# Default storage resource name:irodsDefResource ’resc1'# Home directory in iRODS:irodsHome '/centosgrid/home/rods'# Current directory in iRODS:irodsCwd '/centosgrid/home/rods'# Account name:irodsUserName 'rods'# Zone:irodsZone 'centosgrid’ …
> ilsrescresc1demoRescstateRescbundleResc
The CentOS Example
Default data grid name (zone name) was changed to centosgrid;default demoResc was
left as-is; new resources resc1 and stateResc on the
same host were added (new physical paths).bundleResc is createdwhen ibun is used.
Edit .irodsEnv file to accurately reflect
the changed environment.
38
Setting Up New Users
iadmin>lt user_type– rodsgroup – rodsadmin – rodsuser – domainadmin – groupadmin – storageadmin – rodscurators
• Use iadmin
• Two steps: mkuser and moduser (for a password)iadmin> mkuser user1 rodsuseriadmin> moduser user1 password *****
• Use iadmin to see what user types are possibleiadmin> lt
zone_typeuser_typedata_typeresc_typeaction_typerulexec_typeaccess_typeobject_typecoll_mapauth_scheme_type
iadmin> lt user_type
rodsgrouprodsadminrodsuserdomainadmingroupadminstorageadminrodscurators
Token List
Possible values
of token“user_type”
39
User Naming Convention
An iRODS user account is identified by the user name and the data grid/zone name:
– admin user on centosgrid: rods#centosgrid
– admin user on ubgrid: rods#ubgrid
– one of the rodsuser accounts on ubgrid: guest#ubgrid
40
Resource Server Installation on CentOS
• Install a non-IES iRODS server in order to add a resource to an existing data grid
• Host that will be running the iRODS server for this resource: tutorres.renci.org
1. Download irods-resource-4.0.0-64bit.deb from the download site2. Install the packages that iRODS depends on
> sudo yum install unixODBC perl authd> sudo yum install postgresql-odbc
3. PostgreSQL is set up on the IES, the iCAT-enabled server; don’t need it here on the non-IES resource server
continued …
Installing a non-IES Resource
41
Resource Server Installation on CentOS
4. Modify /etc/xinetd.d/auth locally to remove the -E from a command line argument for auth:
change [ server_args = -t60 --xerror --os -E ] to [ server_args = -t60 --xerror --os ]
5. Set the proper runlevel for authd> sudo /sbin/chkconfig --level=3 auth on
6. Restart xinetd > sudo /etc/init.d/xinetd restart
continued …
Installing a non-IES Resource
42
Resource Server Installation on CentOS
7. Open your firewall, if necessary– Add the following lines to your /etc/sysconfig/iptables
-A INPUT -m state --state NEW -m tcp -p tcp --dport 1247 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 20000:20199 -j ACCEPT -A INPUT -m state --state NEW -m udp -p udp --dport 20000:20199 -j ACCEPT
– Restart the firewall > sudo service iptables restart
8. Install the iRODS RPM > rpm -i irods-resource-4.0.0-64bit-centos6.rpm
9. In case of missing FUSE libraries, install those> sudo yum install fuse-libs and then install the RPM (repeat step 8)
continued …
Installing a non-IES Resource
43
Resource Server Installation on CentOS
10. As user irods, run the resource setup script> sudo su – irods> ./packaging/setup_resource.shAnswer the prompts for username and pw for iRODS admin user.> exit
Running this script will – create /etc/irods/server.config, /etc/irods/irods.config, and /etc/irods/core.re– connect to the iCAT-enabled server (the IES) running the data grid– create the /var/lib/irods/.irods/.irodsEnv file (for local service account irods)– bring up the iRODS resource server– add info to the log file /var/lib/irods/iRODS/server/log– add a resource to the existing data grid
continued …
Installing a non-IES Resource
44
Resource Server Installation on CentOS• Default resource name and location:
– <hostname>Resource– /var/lib/irods/iRODS/<hostname>ResourceVault
Example• Take a look at the new resource
> iadmin lr bundleResc resc1 stateResc demoResc tutorresResource
> iadmin lr tutorresResourceresc_id: 10020resc_name: tutorresResourcezone_name: centosgridresc_type_name: unixfilesystemresc_net: tutorres.renci.orgresc_def_path: /var/lib/irods/iRODS/tutorresResourceVault…
My resource host is
tutorres.renci.org,thus the defaultresource name
and path.See a listing of the data grid resources
See more info aboutthe new resource, in
particular the physical path
Installing a non-IES Resource
45
Resource Server Installation on CentOS
• Change resource name (before any data ingestion)> iadmin iadmin> modresc tutorresResource name resc2
• Change the physical path; my example path will be /opt/irodsVaultUnix: create the directory and set permissions for user “irods”– create the directory: > sudo mkdir /opt/irodsVault– set Unix user “irods” as owner: > sudo chown irods irodsVault– set group to “irods” for this directory: > sudo chgrp irods irodsVaultiRODS:– modify path: iadmin> modresc resc2 path /opt/irodsVaultcontinued …
Installing a non-IES Resource
46
Resource Server Installation on CentOS
Now take a look at the resources available on data grid centosgrid
> ilsresc bundleResc resc1 demoResc resc2 stateResc
> iadmin iadmin> lr resc2
resc_id: 10020resc_name: resc2zone_name: centosgridresc_type_name: unixfilesystemresc_net: tutorres.renci.orgresc_def_path: /opt/irodsVault …
Installing a non-IES Resource
From anywhere that you have your ~/.irods/.irodsEnv file and the icommands client, you have access to all the
resources of your data grid.
47
Resource Server Installation on Ubuntu
In this example, we’ll install another non-IES resource server on an Ubuntu host, to add yet another resource to centosgrid.
1. Download irods-resource-4.0.0-64bit.deb from the download site2. Go to the directory containing the .deb package and do dpkg...
> sudo dpkg -i irods-resource-4.0.0-64bit.deb
3. Use apt-get to install with all dependencies satisfied> sudo apt-get -f install
continued …
This creates theUnix account “irods”,
locally
Installing a non-IES Resource
48
Resource Server Installation on Ubuntu
4. As Unix user “irods” run the setup script> sudo su – irods> ./packaging/setup_resource.sh
5. Answer config questions about existing data grid this server is joining:
iCAT server's hostname or IP address: tutoricat.renci.orgiCAT server's port [1247]: iCAT server's ZoneName: centosgridiRODS admin username [rods]: rods
continued…
/var/lib/irods is the irods home directory
Installing a non-IES Resource
49
Resource Server Installation on Ubuntu
Running the setup script (./packaging/setup_resource.sh) will:– prompt for irods admin user and pw
– create /etc/irods/server.config
– connect to the iCAT-enabled server (the IES) running the data grid
– create the /var/lib/irods/.irods/.irodsEnv file (for Unix user irods)
– bring up the iRODS resource server
– add info to the log file /var/lib/irods/iRODS/server/log
continued…
Installing a non-IES Resource
50
Resource Server Installation on Ubuntu• Default resource name and location:
– <hostname>Resource– /var/lib/irods/iRODS/<hostname>ResourceVault
Example• Take a look at the new resource
> iadmin lr bundleResc resc1 stateResc demoResc resc2 tutorResource
> iadmin lr tutorResourceresc_id: 10020resc_name: tutorresResourcezone_name: centosgridresc_type_name: unixfilesystemresc_net: tutor.renci.orgresc_def_path: /var/lib/irods/iRODS/tutorResourceVault…
Resource host:tutor.renci.org
See a listing of the data grid resources
See more info aboutthe new resource, in
particular the physical path
Installing a non-IES Resource
bundleResc is created when ibun is used
(see the Getting Started
tutorial)
51
Resource Server Installation on Ubuntu
Change resource name (before any data ingestion)> iadmin iadmin> modresc tutorResource name resc3
> ilsresc bundleResc resc3 resc1 demoResc stateResc resc2
> iadmin iadmin> lr resc3
resc_id: 10023resc_name: resc3zone_name: centosgridresc_type_name: unixfilesystemresc_net: tutor.renci.orgresc_def_path:
/var/lib/irods/iRODS/tutorResourceVault
Installing a non-IES Resource
52
Resource Server Installation on Ubuntu
Change resource path(before any data ingestion)First create the directory and set permissions for Unix user “irods”
> sudo mkdir /opt/irodsVault> sudo chown irods irodsVault> sudo chgrp irods irodsVault> iadmin iadmin> modresc resc3 path /opt/irodsVault
iadmin> lr bundleResc demoResc stateResc resc1 resc2 resc3
iadmin>lr resc3 resc_id: 10023 resc_name: resc3 zone_name: centosgrid resc_type_name: unixfilesystem resc_net: tutor.renci.org resc_def_path: /opt/irodsVault …
Installing a non-IES Resource
53
Uninstalling the Resource Server
• To uninstall an iRODS resource server on Ubuntu> sudo dpkg --purge irods-resource
This deletes all files, including config files, and removes service account “irods”
• On CentOS> sudo rpm -e irods-resource
This will not remove the irods service account, so do> sudo /usr/sbin/userdel irods
54
Strict ACL Policy – acAclPolicy rule
• In /etc/irods/core.re, the rule acAclPolicy sets policy for strict access.
• Default setting is STRICT: acAclPolicy {msiAclPolicy("STRICT"); }- This disallows perusal of collections by users without read permission.
• By commenting out that rule and uncommenting the acAclPolicy rule that does NOT call the microservice to set the access control to strict, one can change the default behavior: acAclPolicy { } #acAclPolicy {msiAclPolicy("STRICT"); }
• This will allow free perusal of all collections by all users. Users will still not be able to view or download files to which they don’t have read permission
55
Federation Between Data GridsExample: Federate the centosgrid and ubgrid data grids
1. centosgrid acknowledges ubgrid> iadmin mkzone ubgrid remote tutorub.renci.org:1247
2. ubgrid acknowledges centosgrid> iadmin mkzone centosgrid remote tutoricat.renci.org:1247
3. Server-to-server authentication (the necessity for this can be disabled)In /etc/irods/server.config on each grid, define LocalZoneSID and then RemoteZoneSID.
continued…
56
Federation Between Data GridsExample: Federate the centosgrid and ubgrid data grids
3. Server-to-server authentication. Example: In /etc/irods/server.config
on centosgrid:LocalZoneSID terraRemoteZoneSID ubgrid-firma
In /etc/irods/server.configon ubgrid:LocalZoneSID firmaRemoteZoneSID centosgrid-terra
See the iRODS manual for info on scrambling these SIDs.
NB: These zone SIDs must now be set on ALL remote resourceson a zone to preserve server-to-server authentication
between the resources and their own data grids.
57
Federation Between Data Grids
Add remote users. First, see each grid’s usersubgrid:> iadmin lu rods#ubgrid
4. centosgrid adds remote user: > iadmin mkuser rods#ubgrid rodsuser
5. ubgrid adds remote user: > iadmin mkuser guest#centosgrid rodsuser
centosgrid:> iadmin lu
rods#centosgrid leesa#centosgrid guest#centosgrid
NB: rods#ubgrid is not
an admin accounton centosgrid
58
Federation Between Data GridsNow see each grid’s users…
ubgrid:> iadmin lu rods#ubgrid guest#centosgrid
centosgrid:> iadmin lu
rods#centosgrid leesa#centosgrid guest#centosgrid rods#ubgrid
59
Federation Between Data GridsUsers with accounts on remote, federated data grids: • authenticate to their “home” data grid and then inherit access to the
remote grid • have home collections on the remote data grid and ownership (write)
permissions there• can be granted read/write access to other collections just like any user• NB: the remote home collection is of the form:
/<remote_zone>/home/<user_name>#<local_zone>– So to do an “icd” or to write to your remote collection, remember to
append your local zone name to the collection name
– Example: User “guest” logs on to his home zone, centosgrid, and does an “iput” to his collection on ubgrid:
iput –R ubresc1 testfile /ubgrid/home/guest#centosgrid/testfile
60
Federation Between Data Grids
• To write to a remote federated data grid, users must know the resources available on the remote grid:
From centosgrid, to see the resources on ubgrid: > ilsresc –z ubgrid ubresc1 ubresc2
• User guest#centosgrid now wants to write to his home collection on ubgrid– User guest does a simple iinit and authenticates to his home data grid,
centosgrid> icd /ubgrid/home/guest#centosgrid> iput –R ubresc1 myfile> ils -L/ubgrid/home/guest#centosgrid: guest 0 ubresc1 1883 2014-05-13.21:31 & myfile ubresc1 generic /opt/irodsVault/home/guest#centosgrid/myfile
Writing across data grids
61
Removing/deleting Data or Resources
• “irm /zone/home/user/file1” moves file1 to /zone/trash/user/file1Not physically removed from disk
• “irm –f /zone/home/user/file1” physically deletes file1
• When administrator wants to remove a resource, it must be empty– If files are in the trash directory, resource is not empty
• To delete old users’ files for removing a resource– Admin user can use ichmod –M in admin mode, take ownership of
files, and delete them physically
62
Administrative Rights
• -M option for some icommands
• Some rules and queries are restricted to admin users
• Strict ACL exceptions for admins
63
ireg – the administrator’s side
Get data into iRODS without making an additional copy or moving it
Example: Directory /projects/state-data contains state LiDAR data that we now want in an iRODS repository… without moving or copying it
1. /projects/state-data is mounted on the iRODS server host
2. Data admin sets up existing directory as an iRODS resource:> iadmin mkresc stateResc unixfilesystem tutorres.renci.org:/projects/state-data
3. Register existing data into iRODS iCAT> ireg -C -R stateResc /projects/state-data /centosgrid/home/state-data
Register incoming files rigorously OR modify a directory only through iRODS once it has been registered to keep the iCAT consistent with the directory.
(-f option for picking up unregistered files)
64
S3 Resources – Cloud Management
See https://www.irods.org/index.php/S3_Resource
1. Set up an Amazon S3 resource– http://aws.amazon.com/s3/– You will need both the Access Key ID and the Secret Access Key
2. Download and build the libs3 library:
http://libs3.ischo.com.s3.amazonaws.com/index.html
65
S3 Resources – Cloud Management
3. Edit iRODS/config/config.mk– Uncomment the line: AMAZON_S3=1– Define the s3 libraries header directories, for example:
S3_LIB_DIR=/usr/libS3_HDR_DIR=/usr/include
4. Add path to the S3 library to the LD_LIBRARY_PATH environment variable:
set LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib
5. Rebuild the server./irodsctl istop./irodssetup (or gmake and then ./irodsctl istart)
66
6. Set up authentication to your Amazon resource– In server/config, use the file s3Auth.template as the template for the
s3Auth file– cp s3Auth.template s3Auth – Edit s3Auth as indicated in template file: add S3_ACCESS_KEY_ID
and S3_SECRET_ACCESS_KEY that you got from Amazon
7. Create an S3 compound resource> iadmin> mkresc amazonResc s3 compound irods00.lab.nsc.liu.se /snicVault> atrg s3Group s3Resc create resource group> atrg s3Group comp523Resc add resource of class “cache”
Configuring an Amazon S3 Resource
Path in s3storage
67
Cloud Resource• Admin creates the S3 resource - see the S3 resource and
group: iadmin>lr
bundleReschttpRescsnicRescamazonResc
iadmin>lrgs3GrouphttpGroup
• Any user can ingest and access data to the S3 resource(unless your own policy forbids it)> iput -f -K -R s3Resc irods-intro.pptx
• Cloud data is now managed by iRODS
> ilsreschttpRescsnicRescamazonRescs3Group (resource group)httpGroup (resource group)
68
Using the Cloud Resource
iput -R amazonResc iplant-file > ils
/snicZone/home/leesa: iplant-file C- /snicZone/home/leesa/slides
> ils -L/snicZone/home/leesa: leesa 0 snicResc 14 2012-09-24.19:49 & iplant-
file /home/irods7800/Vault/home/leesa/iplant-file s3Group leesa 1 amazonResc 14 2012-09-24.19:49 &
iplant-file /snicVault/home/leesa/iplant-file s3Group C- /snicZone/home/leesa/slides
69
Using the Cloud Resource
70
• Drivers support connections to external data
• These microservice plug-ins come with iRODS – two microservice drivers for each protocol (get & put)
• Instantiated through a compound resource
• Symbolic links implemented for http, slink, and Z39.50
• Admin users can implement new drivers: see How to Create a New MSO Type athttp://wiki.irods.org/index.php/How_to_Create_a_New_MSO_Type
MSO: Microservice ObjectsSupporting realizable objects
71
Symbolic Links to an http Source – the admin’s side
• Requires a compound coordinating resource with two child resources - one cache and one archive
• Set up the compound resourceiadmin> mkresc msoResc compound
• Set up the archive resource, of type msoiadmin> mkresc httpResc mso tutorub.renci.org:/fakeVault
• We will use an existing resource as the cache child resource, so now set up the parent-child relationships
iadmin> addchildtoresc msoResc ubresc2 cacheiadmin> addchildtoresc msoResc httpResc archive
Hostname is
necessary here;
physical path is
a dummy argument.
No Hostname for coordinating resource
72
Symbolic Links to an http Source – the administrator’s side• iadmin>lr msoResc• resc_id: 10035• resc_name: msoResc• zone_name: ubgrid• resc_type_name: compound• resc_net: EMPTY_RESC_HOST• resc_def_path: EMPTY_RESC_PATH• free_space: • free_space_ts 1969-12-31.19:00:00• resc_info: • r_comment: • resc_status: • create_ts 2014-05-22.16:58:35• modify_ts 2014-05-22.17:00:58• resc_children: ubresc2{cache};httpResc{archive}• resc_context: • resc_parent: • resc_objcount: 0
73
Symbolic Links to an http Source – the administrator’s side• By default, path checking is on, disallowing a rodsuser from putting
realizable objects into the mso resource. The user’s side:> ireg -D mso -R httpResc "//http://people.renci.org/~leesa/irods4.0/irods4.0-intro.pptx" /ubgrid/home/leesa/slides/irods4.0-intro.pptx
ERROR: regUtil: reg error for /ubgrid/home/leesa/slides/irods4.0-intro.pptx, status = -129000 status = -129000 PATH_REG_NOT_ALLOWED• By setting
• [ADD info here]
74
Symbolic Links to an http Source – the user’s side• User’s side:
> ireg -D mso -R httpResc "//http://people.renci.org/~leesa/irods4.0/irods4.0-intro.pptx" /ubgrid/home/leesa/slides/irods4.0-intro.pptx
ERROR: regUtil: reg error for /ubgrid/home/leesa/slides/irods4.0-intro.pptx, status = -129000 status = -129000 PATH_REG_NOT_ALLOWED• Admin side:
> ireg -D mso -R httpResc "//http://people.renci.org/~leesa/irods4.0/irods4.0-intro.pptx" /ubgrid/home/rods/slides/irods4.0-intro.pptx> ils slides/ubgrid/home/rods/slides: irods4.0-intro.pptx