national center for supercomputing applicationsnational computational science grid packaging...

National Center for Supercomputing Applications National Computational Science

National Computational Science

Grid Packaging TechnologyTechnical Talk

University of Wisconsin Condor/GPTMeeting

January 3,4 2002

Michael [email protected]

Specialist, Software DeveloperAdvanced Computational Environment

and Security, NCSA



GPT Purposes

Give developers an easy way to package software All packaging data is specified from a source

perspective. Packaging data is strongly typed.

Provides a means of building and installing collections of packages

Provides standard version negotiation method for managing updates to packages.



GPT and Globus

GPT was designed to re-implement the deployment requirements of Globus plus shared libraries. Globus most of Globus is libraries. Globus has deployment requirements for client tools,

service daemons, and development/build environments. Globus wants to release individual components

separately Globus wants to add outside software (ie. openssl,

openldap) to these deployments. Globus wants to work on multiple platforms.



Outline

Splitting up the source code Binary Package Types Dependencies Build Flavors Versioning Installations Bundling Setup The GPT Tools



Splitting up the Source Code

Source Package Types Source packages: contain source code and

documentation. Setup packages: contain configuration/localization

scripts and files. Test Packages: contain component test code.



Source Packages

Contains source code for programs, scripts, and libraries. All libraries are added to the link line.

Contains documentation and data files. All of these share the same version number.



Other Packages

Setup Packages Contains scripts and templates. These are assumed to have no build flavor. Generate files outside of the control of the packaging

system.

Test Packages Contains test harnesses and code. Installed in a special directory.



Outline




Binary Package Types

Type Description Flavored

programs and scripts. Yes No

Yes Yes

headers and static libraries. Yes Yes

documentation. No No

data non-localized data files. No No

Multi-flavored installation

pgm

rtlshared libraries and flavored scripts.

dev

doc



Package Transformations

TestPackage

SetupPackage

SourcePackage

pgm

data

doc pgm

data

devrtl

doc pgm

data

doc

Source Code

Installed Files localized files



Outline




Dependencies

A Dependency is a need relationship to another package RPM's "Requires:" does the same thing.

Different Dependency Types for Source an Binary Packages.

Types include both build and runtime dependencies.



Runtime Dependencies

Source Runtime dependencies are transferred to the specific binary package.

Runtime dependencies require a binary package type in the specification .



Build Dependencies

Are transferred to the binary dev packages Are used in the configure stage of the source

package build.



Source Runtime Dependencies

pgm_runtime Dependencies for the programs.

lib_runtime Dependencies for the libraries.

doc_runtime Dependencies for the documentation.

data_runtime Dependencies for the data files.



Source Build Dependencies

compile Requires header file from another package

pgm_link Programs require libraries from another package

lib_link Libraries depend on libraries from another package



Dependencies

HeaderFile

Library

Script

DoxygenPage

Library SourceCode

Documentation

Program SourceCode

Source Package

Package Foo

Package Fee

Package Foe

Package Fum



Binary Dependencies

Compile Only in dev packages

Build_Link Only in dev packages

Runtime_Link rtl and pgm packages

Regeneration Only in pgm packages

Runtime in any binary package



Source to Binary Dep Conversion

compile

lib_link

pgm_link

runtime

compile

runtime

build_link

runtime_link

regeneration

Static Linking

Dynamic Linking

Pass Thru

SourceDependencies

BinaryDependencies



Outline




Build Flavors

A Build Flavor is a set of compilation and linking options which have to be defined to produce binaries that can be linked with each other.

A development environment issue for older versions of Globus.

Now a runtime issue with GT 2.0 because of shared libraries and plugins.



Build Flavors

Build Flavor Choices Compiler Choice: gcc, vendor cc Size: 32 bit, 64 bit Debugging: Debug Symbols included or stripped Thread Package: pthreads, solaris, sproc, none message passing library: mpi, none

Standard Flavors in green Example

gcc32dbgpthr = gcc compiler, 32 bit, debugging symbols, pthreads



Outline




Versioning Purposes

Source code change identifier. Distribution identifier. Marketing tool. Interface compatibility identifier

for APIs. for protocols. for configuration files.



Version Negotiation.

Agreement on compatibility between package developer and package user. Package developer makes a compatibility

recommendation. Package user choose to accept or override the

recommendation.



Aging Version

Version scheme used by the package developer. Based on libtool and other shared library schemes.

4.1.3

major version numberminor version number - indicates a binary compatible change

age - indicatesbackward compatibility to themajor version number



Version Requirements

Simple Version Requirement. Package user specifies major version number. Trusts the developer compatibility recommendation.

Version Range Requirement. Package user specifies a range of major and minor

version numbers. Does not use the compatibility recommendation from the

package developer.

Combinations of these can be used for more complex requirements.



Versioning Example

Version for the package is 5.1.3

Version Requirement Is Compatible

3 Yes

1 No

4.2 to 5.4 Yes

4.2 to 5.0 No



Outline




Installation Objectives

Standard structure for both runtime and build installations.

Multiple build flavors can exist in the same location. Packaging data co-located with the installation.



Installation Structure

$GLOBUS_LOCATION

programs

libraries

include

etc

setup

test

bin

liblibexec

gcc32headers

gcc32pthrheaderspackagingmetadata

sbin



Multiple Installations

$GLOBUS_LOCATION

programs

libraries

include

etc

setup

test

bin

lib

libexec

gcc32headers

gcc32pthrheaders

packagingmetadata

sbin

Client Tools

$GLOBUS_LOCATION

programs

libraries

include

etc

setup

test

bin

lib

libexec

gcc32headers

gcc32pthrheaders

packagingmetadata

sbin

User Development

$GLOBUS_LOCATION

programs

libraries

include

etc

setup

test

bin

lib

libexec

gcc32headers

gcc32pthrheaders

packagingmetadata

sbin

Services



How to move an installation.

Copy Directory. Re-run setup scripts.



Outline




Bundling Objective

Be able to install a collection of packages or even an entire distribution with one command. Packages are not installed if compatible ones already

exist. All setups are consolidated into one post install script.

Be able to build a binary bundle with one command.



Bundling Data

Name and version of the package. Binary package types that are needed. Build options to be avoided ie. no threads. Build Flavors are chosen during builds for each

platform.



Outline




Setup Objectives

Allow installed packages to be configured with a script. Each script packaged in a setup package.

Script execution is recorded in the packaging data. Setup packages can be replaced by distributing

organizations to tailor a distribution.



Setup dependencies

Source package indicate a setup need by a special setup dependency.

Setup packages have two names and versions. One name/version for the specific package.

different setup packages have different name/version.

One name/version to fulfill the setup dependency. All setup packages will have the same name/version for this.



Setup Package Example

Regular PackageFoo

SetupPackageFum

SetupPackageFie

SetupPackageFoe



Outline




What does GPT offer?

An easy way for developers to package their software.

An easy way for people to build large sets of source packages.

Provides a package manager for those systems that don't have one.

Provides packaging metadata in an XML format. Compatible with other package managers



An easy way for developers to package their software

gpt_setup - Sets up Makefiles and autoconf to use packaging system.

gpt_edit - GUI to allow developers to edit the metadata.

patch-n-build - for those developers that want to keep their own build system.

globus-makefile-header - for those who want to build applications using Globus components.



An easy way for people to build large sets of source packages.

globus-build -swiss army knife for building packages, bundles, and development directories.globus-build [ \ -verbose \ -help \ -installdir=path_to_installation \ -builddir=path_to_build directory \ -log=logfile \ -srcfile=source_metadata_file \ -srcdir=source_directory \ -static=<build number> \ -install-only \ -force \ -std-flavors \ -all-flavors \ ] \[macro=value] [list of flavors to build] [list of source packages/bundles]



Provides a package manager for those systems that don't have one.

globus-install - Installs/uninstalls binary packages gpt_verify - Verifies that a Globus installation is

complete.



NMI Changes to GPT.

Changes for April NMI release. New schema/dtd for packaging data.

conversion script to update existing packages

Add package signatures and other installation verification mechanisms.

Add tools for installation management. Centralized file which records installations and installed bundles.

De-globus-ize the tools replace "globus" with something else for script names variables

etc. probably keep old globus names for compatibility purposes.

national center for supercomputing applicationsnational computational science grid packaging...

Documents