cce2008 why we chose hp-ux

8/8/2019 CCE2008 Why We Chose HP-UX

1/57

114 November2008

Why we chose HP-UX for ourMission Critical Application

Olivier S. MassSystems Administrator, Hydro Quebec Distribution

November 2008

[email protected]


2/57

214 November2008

What are we going to cover?

Ill spend most of my time on

this chapter: Part 1: Making a case for a

migration to HP-UX

Then Ill go quickly on thefollowing two:

Part 2: Easing the transition for

yourdevelopers Part 3: Smoothing things up for

yoursystem administrators


3/57

314 November2008

Part 1: Making a case for amigration to HP-UX

Description of our application

What factors motivated us to move

What were our criteria

Why we chose HP-UX

What technology was chosen

Tech Talk


4/57

Who are we? Hydro-Qubec Distribution is one of the largest power distributors

in North America and the biggest public electric utility in Canada.

Serving Quebec, a Canadian province spanning 1.6 squaremillion km, Hydro-Qubec delivers energy to 3.5 millioncustomers with an innovative distribution management solutionbased on Integrity servers running HP-UX.

A large portion of ourenergy is hydro-electric.

Quebec is a land ofclimatic extremes, andlarge distances separategeneration facilities fromload centres.


5/57

514 November2008

Overview of our project

In 2007, we performed successfully a migration from a timesharingOS to HP-UX

6 data centers scattered throughout the province (one on an isolated island!).

Up to 800km apart, around 50 servers total

Our migration went very well. This presentation will show you WhyWe Chose HP-UX on Integrity over other platforms.


6/57

614 November2008

What is our Application? Since the 1980s, when the use ofSCADA (Supervisory Control And Data

Acquisition) for distribution was rare, we pioneered tools and processes toimprove the efficiency of our network

Our network includes more than 100,000 km of overhead lines andunderground cables, 500,000 transformers and 190,000 switching devices.

Our SCADA application is a DMS (Distribution Management System).

It manages many tasks that are related to electrical distribution to the endcustomer. Outage and workforce management

Mobile communications with work crews

Construction projects Automation of the electrical network

Interfaces with corporate CRM systems


7/57

714 November2008

What are the components? The main app itself is a traditionnal client-server.

The clients run on Windows XP.

The workprocesses and databases are on HP-UX 11iv2:

Developed in mostly C/C++, a few bits of Java and legacy Fortran code as well

Oracle instances (Most at 10g, some at 9i)

We have an abundance of other tools and interfaces, all running on HP-UXas well:

SOAP Webservice and FTP/SFTP server for data exchange

Data warehouse

PHP web applications

Collateral servers: Stratum-2 NTP servers, SOCKS proxy, Subversion server

Mobile radio application (API with Motorola software)

Etc.


8/57

814 November2008

Example: State of the electrical network Each electrical line, and all its components, have a state.

Some states are entered manually, others are acquired real-time withremote interfaces.

Most of the management is done using the Windows client, with the HP-UX backend doing all the processing.


9/57

914 November2008

Example: Remote control

Manual cut-offs in the electricalnetwork are managed by automatedswitches such as this one

An operator managesremotely the state ofthe whole electricalline, communicatingwith the switch if

necessary

Inside the box is aremote-controlled

motor.


10/57

1014 November2008

Example: PHP Web Application Web-based PHP applications are catching on, as simple front-

ends can be deployed to end-users without the need to develop

and package a client. We have a few legacy ASP applications that are slowly being

rewritten in PHP as well.

Screenshot of our custom bugdatabase, used internally.


11/57


12/57

1214 November2008

Why did you move away from your

older platform? We already had a very reliable platform. Question is,

why bother?

The market share of our previous OS was slowly shrinking.

Our DMS is for sale through a partner and there were commercialinterests to port the application to benefit from an increasedmarketability.

System Administration and development expertise for the other platformwas becoming harder to find.

The CPU architecture we were using for years was going to bediscontinued

An effort was already required to recompile the code to a new CPUarchitecture, so there was an opportunity to migrate.

Oracle performance was not up to our expectations.


13/57

1314 November2008

What criteria were used to choose

HP-UX among others?

The usual reasons:

Performance

Reliability

TCO

And a few more: Security

Scalability

Support from the manufacturer

Manageability

Available applications

Ill detail these criteria in the next few slides


14/57

1414 November2008

Performance Performance is most important factor for us

During normal workloads, the application doesnt consume lots ofresources but we need a lot of headroom to be able to meet asudden demand.

The efficiency of our ground teams, and what the customer thinksof that efficiency, ultimately depend on the performance of ourservers.

If there is any major outage due to a natural disaster, the load will

go wayup. Case in point: A major ice storm in 1998 broke up large parts of our

electrical network and required intensive computing resources

Recent, less spectacular event: Two weeks ago, one region had anearly snowfall, with heavy flakes, and this interrupted electricity to morethan 50,000 customers. We have similar events many times each year.


15/57

1514 November2008

Reliability and TCO

Planned downtime is allowed, depending on the weatherbut any unplanneddowntime must be kept to a minimum

when teams are on the ground Like with slow performance, any downtime reduces operational

efficiency and this has a direct, visible impact to the customer.

TCO is a factor: Were not looking for the theoretical, 100%

uptime that some other platforms strive to offer with aheftier price tag

We can live with 99.995%: 26 minutes of downtime per year.

Clustering technology must be available

To minimize planned (and unplanned) downtime, and reachthat 99.995%

To implement a DR plan with an extended-distance cluster (e.g.Metrocluster)


16/57

1614 November2008

Security, Scalability, Support Increased security awareness requires a platform that has a

good security record

Regular audits are done within the company and we mustdocument our security process

The scalability of the platform is important

A natural disaster could require emergency additional capacity We have 6 data centers future consolidation is possible

The platform should support virtualization

Support from a top tier manufacturer is required Security patches, training and frontline support

Complete documentation on the OS and subsystems

Long support commitment and sales availability for OS releases


17/57

1714 November2008

Manageability and applications We must be able to perform trivial tasks online without disruption, e.g.:

Adding and reducing disk space

Especially thin provisioning in the future

Online addition and replacement of failed components

Dynamic reconfiguration of the OS kernel

Migration from one SAN to another

Minimal maintenance requirements on the OS must be needed

System Administrators must be able to go on vacation sometimes

A good ecosystem of applications must be available

We need to be able to run basic web services

Development tools must be good


18/57

1814 November2008

How does HP-UX measure up to our

criteria? (1/4)

The Itanium2 processor was, and still is, a leader interms ofperformance.

Especially with Oracle, which we needed

Sample benchmarks we wrote showed a potential 400%increase in performance

The fact that Integrity platform can run a variety ofOS reduced investment risks

We already had experience with other HP products,and going to Integrity was a logical migration path forus

All our Windows environments run on Proliant

Our SANs are based on StorageWorks products; HP-UX integrates well with them (especially withMetrocluster)


19/57

1914 November2008

We ordered a study to help us choose the new platformand OS

Concerning HP-UX, without any surprise, the first thing the report didwas quote Gartner:

HP-UX is an excellent operating system choice for the enterprise and it will be a solidmigration vehicle for the HP user base as well as developers looking for an evolvingplatform (Gartner, 2004)

They also wrote this concerning security: It is preferable to minimize security risks by favoring an operating system that has an

emphasis on performance, stability and ROI, rather than a mass technological choice.Most proprietary Unix operating systems fall in that category.

To get performance metrics, they extracted a few results from TPC(Transaction Processing Performance Council):

Integrity servers, either with HP-UX or Linux, were generally among leaders in Oracle9i performance.


criteria? (2/4)


20/57

2014 November2008

Linux on Integrity was considered since it was on parwith HP-UX for performance, but it had to be dropped in

the early stages We were looking for an outstanding support level

Our corporation was already deploying a major HP-UX

environment in another department

A SAP solution on HP-UX replaced the mainframe for ourCRM and invoicing (millions of customers). The projectwent in full production in early 2008.

Since were already an HP-UX customer, the learningcurve is reduced for any internal resource that could bebrought on to work on our environment.


criteria? (3/4)


21/57

2114 November2008

We needed applications to be able to run our PHP andSOAP web services:

We looked if HP-UX could get the job done, and it did:

Apache Server, Tomcat, PHP are bundled in HP-UX Web ServerSuite

Axis and XML libraries are available on Internet Express to run SOAPServices, but theyre unsupported

Increased performance was expected by migrating PHP applicationsfrom Windows 2003 to HP-UX

The bottom line is that we didnt need YAP (yet another platform) forour web services.

HP-UX Web Server suite and many of its modules areofficially supported by HP When we were running Apache + PHP on Windows, we had no

support.


criteria? (4/4)


22/57

2214 November2008

Building a roadmap

A roadmap was produced, covering everytask to follow:

2005: Design, studies and architecture.

2006: Porting effort and first phase ofequipment purchase for our test environments

2007: Second phase of acquisition and go live.

We referred to the roadmap all the waythroughout the migration, and this helped uskeep our vision.

Our older MA-based SANs were migrated toEVAs in 2005 and 2006 to have a head startbefore the migration to Integrity.


23/57


24/57


25/57

2514 November2008

Tech talk: What is a Metrocluster? A Metroclusteris a ServiceGuard

cluster that spans multiple sitesand disk arrays

It harnesses the power of yourXP or EVA-based CA datareplication

Its a turnkey solution that workswell and is well integrated

Drawback: three data centers arerequired, the third one hosting aquorum server


26/57

2614 November2008

Tech talk: Praise on blade servers and

Virtual Connect Ive been playing with a few bladesystem chassis since early 2008,

along with the Virtual Connect technology, and I find these productsto be of an outstanding value

Virtual Connect simplifies network connections and management

If your chassis is well planned, physical installation of a server is as quickand painless as it can be since you dont have to do any cabling

The architecture encourages you to double up LAN and FCconnections, and online firmware upgrades of all the componentscan be made online.

Good platform for Virtualization Allows a lot of density. Works wonders with ESX clusters, cant wait to

see whats cooking for Integrity VM 5.0


27/57

2714 November2008

Part 2: Easing the transitionfor your developers

Choosing the right development tools

Source control

Tips for the HP-UX developer

Open sources tools you cant live without


28/57

2814 November2008

Choosing the right compilers As far as C goes, you have two choices:

HPs own C/C++ compiler named aC++

Itanium optimized compiler

The ia64 version ofGCC (Gnu C Compiler)

an open-source and well-respected C/C++ compiler

The open source GCC can be used for simple programs, although

every major project should use aC++. We chose aC++. It is not expensive, and available for free to qualified ISVs

We also purchased the HP Fortran compiler.

Our few Java programs use the JDK handed out by HP, as well assome of their tools (such as jmeter)


29/57

2914 November2008

Hint to help reduce the porting effort

For years, every low-level function in the code of our application hasbeen using UTL, a homemade portable API.

#ifdefs were added throughout the code to be able to generateportable code between the older OS and HP-UX. New operatingsystems could also be added easily in the future.

Future porting was expected, so this preparation was started a fewyears ago.

We had a workable PA-RISC version for a while as a proof of concept

before recompiling on Itanium2.


30/57

3014 November2008

Debugging Our previous OS could not generate

core files in a timely manner, so theywere disabled

The core file can be used to playback and inspect the source codeline by line, as it was at the momentof the crash.

HP-UX, like all Unixes, generates acore file quicklywhen a processcrashes. Hint: kill 3 can be used on a hung process

to have it generate a core file.

The wdb debugger included withaC++ is a graphical front-end to gdb(GNU debugger), which is alreadywell known by many developers.

wdb


31/57

3114 November2008

Source control The open source Subversion (SVN) product is our

source control system

Its solid: absolutely no outage or problem since wemigrated all our server sources to SVN in 2006.

Running a complete SVN server on HP-UX requires acustom compilation, as there are no binaries available.

The basic svnserve server available from the portingcenter it too limited. We favored an Apache+SVNWebDAV server.

The developers liked SVN so much that they migrated

the sources of the client application from Microsoft VSSto SVN in 2007.


32/57

3214 November2008

Obtaining low-level HP-UX

documentation 1/2 There are few resources available for the programmer who

wants/needs to know how HP-UX works at the lowest level.

Quadrants, executable magic numbers, etc are not well explained,and documentation on them is hard to find.

It doesnt help that the available documentation is old, or out of print.

We decided to stay with a 32-bit application to reduce developmentefforts, but had to cope with HP-UXs memory management limitswith 32-bit apps.


33/57

3314 November2008

Obtaining low-level HP-UX

documentation 2/2 Ask your developers to subscribe to the cxx-dev and devtools mailing

lists.

Have them read the following whitepapers available on docs.hp.com: HP-UX memory management whitepaper

HP-UX process management whitepaper

Try to find the following out-of-print books: HP-UX internals by Chris Cooper (2004)

PA-RISC oriented, but can be useful to understand memory management

HP-UX performance and tuning by Sauers, Ruemmler, Weygant Also PA-RISC

The DSPP (developer and solution partner) site might have info, but itis hard to navigate; you also need to register as a company.

Look at the new knowledge on demand webcasts:http://www.hp.com/go/kod

The book Advanced Programming in the Unix environment is anexcellentbook strongly recommended if you need to develop low-level stuff.


34/57

3414 November2008

Glance and caliper It is recommended that you purchase glance,

either with the minimum of the VSE OE or

separately. Glance has been invaluable to ourdevelopers, who use it as a troubleshooting tool.

Caliper, a free profiler for Itanium, has also beenof tremendous help to increase software quality

One of our senior developers assisted to a lab atHPTF 2006 in Houston where he discoveredCaliper.

Ktracershould be investigated (I havent tried ityet)

Glance in action


35/57

3514 November2008

Quote from our developers

How did we make your applicationfaster than before? Caliper! Caliper!Caliper!

Caliper has helped us track bugs and

performance issues we would havecompletely missed otherwise


36/57

3614 November2008

The development environment Security restrictions prevent me from setting

up an IDE environment with a fancy setupsuch as the Eclipse IDE using a CIFS sharedmountpoint on HP-UX.

So developers write code directly on theserver through an SSH session, using emacs.

Emacs is complex, but extremelycustomizable

They built a lot of custom macros on top of it.

They also benefited from color syntaxhighlighting.

Emacs was taken from the HP-UX InternetExpress DVD

XEmacs with syntaxhighlighting


37/57

3714 November2008

Open-Source tools your developers

(and sysadmins) cant live without

lsof : Lists open files on the system, including

sockets tusc: traces user system calls, helps with

debugging

tcpdump orwireshark (formerly ethereal):Network sniffers

All are available from the HP-UX PortingCenterand/orHP-UX Internet Express

N.B. Perl 5.x is now officially included with theOS

Wireshark


38/57

3814 November2008

Part 3: Easing the transitionfor your system

administrators and usersChange ManagementTraining

Host Access Software

Customized environment


39/57

3914 November2008

Things Ive heard.. And the tips I suggest

Ill never move to HP-UX. Never. (long pause, breathing deeply)Im staying with what I have! YOU DIG, PAL?

Seek help with change management

Its hard to use! Im too old for this!

Give people proper training

Give them good host access software

Enhance your OS with helper tools

My current OS has been working for 20 years!

Thats a good point. So show that HP-UX does work, too.

Try to find an evangelist to help pushing your agenda

Dont make promises you cant keep


40/57

4014 November2008

Ill never move to HP-UX

Change management (1/2) If you feel like youre on a warpath with your

sysadmins and users, try to seek help from changemanagement professionals.

HR sent us someone who gave us a two-hourpresentation on change management.

He showed us how he predicted some would react to

our change of platform, from resistance to acceptance.Things happened exactlylike that.

This helped us prepare to ensure that the humanaspect of the migration would go as smoothly as

possible under the circumstances.

We also did an extensive risk analysis to ensurenothing was left off during the migration.


41/57

4114 November2008

A quick look on Wikipedia gives us the formula for change byRichard Beckhard and David Gleicher:

D x V x F > R

D = Dissatisfaction with how things are now;

V = Vision of what is possible;F = First, concrete steps that can be taken towards the vision

If the product of these three factors is greater thanR = Resistance

Then change is possible.

Ill never move to HP-UX

Change management (2/2)


42/57

4214 November2008

Its hard to use!

Plan training for your users and Sysadmins (1/2) Developers can read any general Unix book or follow the Unix

Fundamentals course.

Every System Adminstrator should follow:

Unix Fundamentals,

HP-UX System Administration I

HP-UX System Administration II

Other more advanced courses can be offered to super sysadmins based on your environment.

Logical Volume Manager ServiceGuard

Etc.


43/57

4314 November2008

HP can prepare custom training for you upon request.

I had them combine Sysadmin I and II in an accelerated 5-daycourse by removing concepts that were not relevant to ourenvironment such as printer configuration, NIS administration,etc.

This saved time, and costs.

Dontbe tempted by onsite training at your office

Attendees will have all sorts of reasons to be distracted

Its hard to use!

Plan training for your users and Sysadmins (2/2)


44/57

4414 November2008

Its hard to use!

Choose good host access software

As a transition to a new OS can be a hard seller to somepeople do your best to at least give them proper host access

tools.

Host access software will be the primary interface to yourservers. If it sucks, so will be the perception of your OS.

Take your time evaluating this kind of software, and you betterbe sure to pick software that works!An abundance of featuresis useless if the software is buggy, or hard to use.


45/57


46/57

4614 November2008

What Host access software do you use? 2/2 If you disallow FTP for security

reasons, youll need an SFTP client.

Putty has a command-line client butnobodywill want to use this onWindows

We use WinSCP, its free and easy touse but its not the best client in myopinion, it has too much features

Van Dykes SecureFX seems to begood.

An X Server can be useful, especially to run glance.

We purchased Starnet X-Win32

For those looking for a free product, try XMing

Logging to an X desktop is possible, but we dont do this. HP-UX comes bundledwith CDE (outdated) and MWM (outdated even more)

WinSCP


47/57

4714 November2008

Helper tools: Open-Source software to help your new

sysadmins and users

The developer tools mentioned earlier (lsof, tusc, tcpdump and wireshark) arenot included in the vanilla OS, but are very useful to the sysadmin.

Other suggested tools:

Bash shell: Users will be able to call back their history with the arrows.

Curl orwget: Lets you download data directly from an HTTP or FTP url. Its also useful if you need to write scripts for monitoring your web services

Rsync: File replication program, uses the rsync protocol

Gcc: If you dont want to purchase aC++, but plan on compiling C programs(open source or your own). I use gcc to compile most open source software, even if we purchased aC++ for our own application.

Install alternate text editors to complement vi: Nano is a good choice.

I did a trial ofMidnight Commanderbut it did not catch on, people still prefer agood old shell interface.


48/57

4814 November2008

Example: Curl in action Curl (or wget): These are two similar tools that let you download data

directly from an HTTP or FTP url.

I use curl often to quickly retrieve patches from the ITRC. Its quickerthan downloading them to my PC, then transferring them to theserver.


49/57

4914 November2008

Example: Adding colors (1/2) I added lots of colors and

custom shell prompts to

help system administratorsand support personnel intheir everyday use.

Its friendlier, and mostpeople appreciate the useof colors.

They can be easily

customized with Putty bychanging its ANSI colorsettings, or disabled ifnecessary.


50/57

5014 November2008

Colored shell prompts replace the blank $ or #

I aliased ls to be replaced by GNUs color-ls.

Vanilla HP-UXsession

System enhancedwith a custom

colored prompt andcolor-ls

Example: Adding colors (2/2)

M ld t h b ki f 20 !


51/57

5114 November2008

My older system has been working for 20 years!

Finding an evangelist Finding an experienced admin to act as an evangelist

will be of immense help.

A strong experience with any Unix is required. WithHP-UX, even better.

He will be able to answer common questions quickly toreassure your people, and show to even the mostdoubtful that the OS does work .

Dont let him promise something you cant deliver

Example: dont disclose performance metrics basedstrictly on advertised performance... Raw performance isnot the same as real-world performance.

Tech talk: Ignite-UX and Software


52/57

5214 November2008

Tech talk: Ignite UX and Software

Distributor I use Ignite-UX extensively to install my servers

The software is complex, but at least its well documented.Check the quick install guide.

Golden images can be burned on DVD and they help deployservers quickly our remote sites

Virtual Machines can be installed very quicklywhen usingIgnite-UX with a golden image: I counted 30 minutes.

SD Depots are also very useful to manage the environment

I have a reference SD depot (Golden Depot) that I use tokeep all the servers up to date

All software I use is stored in various Depots for future use; I

rarely have to search around for DVDs


53/57

5314 November2008

Conclusion


54/57

5414 November2008

Our migration went well End-user satisfaction is high

End-users perceived an almost 400% performance increase, as we expected from our earlybenchmarks

Better development and debugging tools increased overall software quality.

These kinds of things are noticed by management


55/57

5514 November2008

What about your support personnel? Wide acceptance from support personnel

was tough to gain in the beginning, but byplanning our migration correctly and

keeping our promises, we managed asmooth transition.

Their satisfaction is high as well:

System requires minimal maintenance

No unplanned downtime since we went inproduction; many were surprised with thereliability of the new Unix system.

The learning curve was not as steep assome expected


56/57

56 14 November2008

Any questions?


57/57

57 14 November2008

Thanks for attending!

Olivier S. [email protected]

cce2008 why we chose hp-ux

Documents