smug presentations q2 may2012

WelcomeSASUsersto

TheLifeCycleofaSASApplication

Wednesday23rd May2012

SMUG 23-May-2012 The Life Cycle of a SAS Application 2

Opening Slide

SAS Melbourne User GroupWednesday 23rd May 2012

The Life Cycle of a SAS Application

Flavio GiurcoManufacturing Systems Manager - Alcoa of Australia Limited

Sameer DasTechnical Architect - SAS Australia and New Zealand


Presentation Outline

Why SAS? How to set up SAS? What to Implement and show Value? Write the code or develop the processes Evolve with SAS Continuous Improvement Maintain and Upgrade

3SMUG 23-May-2012 The Life Cycle of a SAS Application

Why SAS?

Benefit of using SAS Value Business Need Evaluation/Decision


Where can SAS add Value?

e2e SAS Platform/Solutions

Successful with Dept. X already

Add value to all areas of business

Industry and Domain specific solution

offerings

Enterprise Analytical PlatformsCenters of Excellence

Solve complex business problems

Provide Industry leading Analytics

software and Skilled resources

Showing Value is critical

SAS Technology Platform

Start small and grow

Establish value for stakeholders

Engage SAS for best practice

implementations

SAS Business Solutions

Solve a specific business problem

Align with business and IT strategy of

the organisation

SAS platform is scalable to include

other solutions

Customer Size

P

r

o

d

u

c

t

/

S

o

l

u

t

i

o

n

M

a

t

u

r

i

t

y

Small/Medium Organisations Large Organisations

E

n

t

e

r

p

r

i

s

e

D

e

p

a

r

t

m

e

n

t

a

l


Business Need


Access to multiple data sources Integrated data, reporting and analytics platform From simple reporting and statistics To complex enterprise analytics platforms

Cost v/s Benefit discussion Framework to measure Value of using SAS

Why I chose SAS

It was already available on Corporate timesharing environment (Mainframe)

End user could do own thing! Did not require database access or expertise. Ease of adding variables and handling of missing values. High level procedures made job easier.

E.g. Proc Print would print all the variables without specifying their names and formats for them all.

As well as storing data, SAS catered for reporting, graphing and analysing.

Could handle large volumes of data.


What does the SAS platform look like?


SAS Technology Reference ModelScalability

R

e

l

i

a

b

i

l

i

t

y

Interoperability

Manageability

InfrastructureBlocks

Framework Services

UserInterfacesData Management

AnalyticsReporting

Integrated Platform Componentized High PerformanceBusiness Solutions

Managem

ent

Com

pute

Metadata

Security

Integration

Workflow

How to Set Up SAS?

Capture expected Usage Plan ahead Functional and Non-Functional requirements Data sources Output required Interfaces to other systems Infrastructure and architecture Installation and Configuration Setup security plan for the site Setup folder structure for users and system


How to Set Up SAS?


SAS deployments Which one?

Small Medium Large


Environments Physical/Virtual Servers Storage Operating Systems

SAS deployments Simple


SAS BI Client TierSAS Metadata, Application and Mid Tier

SAS Management ConsoleSAS Add-In for MS OfficeSAS Information Map StudioSAS Enterprise Guide

MS Windows XP/Windows 7MS Internet Explorer 7MS Office 2003

Foundation Services

Oracle JDK 1.6.0_24

Foundation SAS

Workspace Server

Stored Process Server

Framework Data Server

Authentication MechanismAD

Authentication MechanismHost (AD)

Java Application ServerJBoss Community Edition 4.2.3

GA

Web Report Studio

Metadata Server

SAS Content ServerData Store

SAS Content Server

Web Infrastructure Platform

SAS Shared Services

Oracle JRE, 1.6.0_24

Access To ODBC

HP DL380 ServerWindows 2008 Server R2 (64-bit) 2 Core (E5620), 12 GB RAM6X300GB Storage

Access To PC Files

SAS deployments Large and Complex


SAS Client Deployments

SAS Grid Control ServerSAS Metadata Server

SAS Middle Tier

Platform Process MgrTM

Platform LSF

Grid Management Service

Grid Node 2SAS Compute Server

Grid Node NSAS Compute Server

SAS Grid Control ServerSAS Metadata Server

SAS Middle Tier

Platform Process MgrTM

Platform LSF

Grid Management Service

Central Shared File SystemCFS

Active Passive Cluster

Platform Suite for SAS Platform Suite for SAS

SAS Workspace Server SAS Workspace Server

Grid Node 1SAS Compute Server

Platform Suite for SAS

SAS Workspace Server

SAS Client Deployment(Terminal Service)

SAS Administration Clients

SAS Clients

Windows Terminal Services

Grid Monitoring Server

Platform RTM

Apache Web Server

Network(LAN/WAN)

SAS Administration Clients

SAS Clients

Internet Explorer

Microsoft OfficeModelling Characteristics

Data Store

SAS/CONNECT SAS/CONNECTSAS/CONNECT

Foundation FoundationFoundation

What to implement and show value

Consider stakeholders: What are their needs? What is the business case for the application?

Without sponsorship for the application...... It will not fly OR it will wither and die.

Nothing succeeds like success: Look for quick runs on the board. Start small and easy Then build up in stages and complexity. Value to all make executives look good.

The Pareto principle: 80% of benefit from 20% of the effort


Alcoa Operations Within Australia


Alcoa Point Henry Smelter & Rolling Mill


Potroom - Smelting Cells/Pots


Individual pot measurements: pot voltage, temperature, metal level etc Potline measurements: line current, line voltage, power interruptions etc

The Opportunity

In a manufacturing production environment, equipment is often controlled by a dedicated controller that has limited storage capacity and reporting capabilities. To make better use of this data, there is a need to transfer it into an environment where the data can be stored indefinitely and analysed.

It was 1981 and the Modcomp mini-computer, used for controlling the 3 potlines, was limited to 36 days of daily data for 368 individual pots and 3 potlines.

Each week the Potroom process engineer printed the 7 day log for each pot for their line, reviewed the data and made corrections to pot resistance and/or feed set points. Then they filed the reports for future reference. There was no whole of line trending of performance.

The project scope: Transfer the pot daily records to the IBM mainframe for longer storage and reporting.


Initial Development Environment


Pot Control Computerat Pt Henry (VIC)

Sending Program

Daily data for 36 daysLine data x 3

Pot data x 368

Mainframe Computerat Booragoon (WA)

Receiving Program

Receiving Flat File

SAS Processing

Dial Up Modem

SAS development was by a single user, with code in single directory. The sending program read the latest line data for each line for each day and sent

to WA a fixed format ASCII record. The record was started with the letter L. Similarly Pot records where sent starting with the letter P. The receiving program appended both line and pot records to a common flat file. A SAS program ran automatically to read and reset the flat file and store the line

data and pot data in separate SAS datasets. SAS reporting programs were run manually.

Write the Code Read the Flat File

Use of OPTION Errorabend to catch any malfunctions prior to resetting flat file (MCOMP).


Intermediate Files General Guidelines

If you cannot access the data directly from the source device or database then use an intermediate transfer file / extract file for SAS to read the data.

Employee data often falls into this category due to confidentiality considerations and limiting what data is visible.

Decide on the file format based on what is easiest for the provider of the data. It is often a lot easier to write the SAS code than to change the report/extract. Common file format options:

ASCII text file - fixed format. Each variable is in a fixed column position.ASCII text file - CSV (comma separated variables) ASCII text file - TAB delimited ASCII text file XML formatMicrosoft Excel format

Avoid CSV format if the incoming data can contain commas. Need to devise a scheme to know what data needs to be processed. Use Proc Contents to see length of variables stored in SAS dataset for

opportunities to save space without losing precision.


The Finished Product

Bath Temperature 3 Potlines Side by Side (Using SAS/QC Proc Shewhart). Shows date and program name and version in footer


Source Code Who and When

SAS source code revision history and defining pgm program name


SAS code to capture run date time

SAS code to put run details into footnote

SAS/QC code fragment to create plot*-- define macros for use in the program ---;%let gph_dev=goptions ctitle=blue device=gif hsize = 7.9 vsize = 5.8 gsfname=gifout gsfmode=replace ; * GIF options;%let clip_factor=3; * level to clip charts;

%macro plot_it(new_name,meas,label,units,sas_file,grp,scale,min,max,step,format); *----- MACRO for create plot -----;symbol1 v=dot c=blue;symbol2 v=circle c=red;data clipped;

set &sas_file(keep=&meas line_id date_id date poweroff age);&meas = &meas / &scale;if &meas ne . then do;

if &meas > &max then &meas=&max; * limit values to bounds to avoid poor plots;else if &meas < &min then &meas=&min;end;

proc shewhart data=clipped(rename=(&meas=&new_name)); * ---- xs chart preparation by line to capture panel limits ---;by line_id;xchart &new_name*date_id=poweroff/stddevs zerostd nochart outlimits=limsline outhistory=histline;

title "&label";title2 "Last &ago Days In &days Day Blocks [Limited to &min to &max &units] (Square Points For Clipped Factor=&clip_factor)";filename gifout "'&publish.&grp._xs_&new_name..gif'";goptions reset; &gph_dev;proc shewhart graphics history=histline(rename=(line_id=_phase_)) limits=limsline(rename=(line_id=_index_));

xchart (&new_name)*date_id=poweroff/npanelpos=800 turnhlabel noxlabel phasebreak stddevs zerostdphaseref phaselegend nolegend nolimitlabel climits=green clipfactor = &clip_factor clipsymbol=squarereadindex=ALL readphases=ALL;

run;quit;filename gifout clear;run;%mend;


SAS Data Management

Data Quality

Enterprise Connectivity

ETL (Extract, Transform, Load)

Metadata Management

Data Migration & Synchronization

Data Federation

Master Data Management

Write the Code - Transferring the Data

Intuitive point-and-click design editor window. Quickly identify inputs and outputs and

create business rules in metadata. Push processing down to the database for

ELT execution. More than 300 predefined table and column-

level transformations.

Sophisticated metadata mapping technologies accelerates development

Impact Analysis/Data Lineage to see process and data relationships

Multi-user Collaboration with DEV/TEST/PROD promotion management

Powerful metadata reporting including runtime and performance analysis

Introducing SAS Code Importer and SAS Metadata

Imports existing SAS programs For each program makes a job in metadata Creates Inputs/outputs/mappings Registers libraries, tables into metadata

SAS Programs


SAS Metadata

Heart of the SAS N-Tier architecture is the SAS Metadata Setup SAS Metadata in a folder structure Metadata contains information from the entire platform

Users, Groups, Roles, ACLs Libraries, Tables, Columns, Information Maps Stored Processes, Reports, Models, Jobs, Flows Servers, Workspaces, Batch schedulers, Web Services

Metadata services connect all the business processes and artifacts in the platform together

Serves as a single version of the truth available to all Auditable, repeatable and secure environment from which to derive

intelligence


SAS Quality Lifecycle Analysis Monitor Workspace

Know the health of production: Check continuously new data

against Western Electric Rules or own rules.

Alert automatically responsible and interested stakeholders.

Initiate root cause analysis.

Engineer

More time is available for more awarding analysis and process & product improvement

Automatic monitoring ensures that no problems are overlooked

Seamless transition from monitoring to root cause analysis

Easy implementation of your own quality standards


Platform Evolution at Alcoa

Platform / operating system evolution:- IBM mainframe in WA- VAX server at Pt Henry and at Portland- Windows BI server servicing multiple sites

Current SAS configuration:- SAS BI Server located at Pt Henry with

Base SAS software, SAS/GRAPH, SAS/STAT, SAS/QC, SAS/ACCESS Interface to PC Files,SAS/ACCESS Interface to ODBC

- Standalone PCs (1 at Pt Henry and 1 at Portland)Software as above + SAS/ETS

- Enterprise Guide / Microsoft Office add-ins


Code Management Considerations

Store SAS production source code on shared network drive with modular components / applications.

Never have production code stored on personal drives. Adopt program naming convention or locations for Production and Test Separate production source code from ad hoc / once off code. Avoid use of spaces in output file names to avoid ambiguity with web / file links. Have include programs to define common system file assignments, library

names, custom formats etc. Consider having separate include programs based on logical groupings rather

than a single large program. The single large program may take additional execution time if the resources referenced are not local.

Store common routines in a library with use of Macros


Code Management - Modular Design

Modular design of source code and output (ownership)


Code Management - Source Code Archive

Adopt a naming convention and structure for the production, test and archives Example:

Line_Daily.sas =Current production versionLine_Daily_RFK_V212.sas =Test version by user(RFK) prior to productionLine_Daily_OLD_V212.sas =Archive of superseded production version V212


Code Management Include File Examples


Example of calling the SAS Include file from production programs

Code fragment from PTH_Production_Libnames.sas include file

Code fragment from PTH_Pot_Calcs_Library.sas include file

How has SAS evolved?

SAS started on the Mainframe, and moved to SMP arch SAS on the Desktop/Server to Enterprise deployments Support the business with Best-of-Breed SAS Software SAS 9 Platform Open, N-Tier Architecture


Enterprise Data and Systems

Operating Systems

z/OS AIX HP-UX Solaris Windows Linux

Enterprise Applications

SAP SiebelOracle Peoplesoft

Operational Data SybaseSQL ServerMySQLIMS-DL/1

JDBCODBCOLEDBPC Files

TeradataDB2Oracle

Enterprise ClientsCustom

ApplicationsWeb Based Application

Enterprise Portals

Standards Based Apps

Excel PowerPoint OutlookSharePoint Mobile PDF

SAS

NetezzaHP NeoviewAster DataGreenplum

Current SAS Platform Architecture


Storage Considerations: SAS or Database?

SAS datasets are generally updated making a copy of the original plus any changes (a father/son update). This requires room for 2 copies of the dataset. This is update method is not as efficient as specialist database programs.

As more data was added to the initial IBM implementation, the time to update files progressively increased and become unacceptable. So separate pot datasets were created for each line and year.

368 pots x 365 days x 137 variables = a lot of disk space! The line data having only 3 records per day was kept as a single dataset. Latest implementation has Microsoft SQL server as the repository for the pot

and line records which are populated directly by the pot controllers. Avoid duplication of data and aim for single version of truth. SAS is useful as a repository when the existing databases do not have the

required functionality. For example, the Alcoa Corporate Safety Incident system can provide incident and corrective action reports that are up to date as of today. So for reporting on week end and month end, SAS takes a daily snapshot of key data and stores it in a SAS dataset for subsequent reporting.

Review each application to determine the best repository for that application.


Storage Methodology: SAS or Database? Evolution at SAS


SASScoring

Data

Teradata EDW

Traditional Capabilities

Data

Teradata EDW

SAS In-DatabaseCapabilities

SAS

Analytic Modeling

SAS

Analytic Modeling

SASScoring

Data Modeling Preparation

Storage Methodology: SAS or Database? Evolution at SAS


Expanded SAS DI Studio support for MPP databases: Greenplum, AsterData nCluster, Sybase IQ

Enables more ELT pushdown and add supports for bulk load utilities In-Database Processing (ELT SQL Pushdown)

Optimized SQL Set Operator Transform Supports UNION, UNION AL, INTERSECT, EXCEPT, OUTER UNION

Easier to use and understand when combining data from multiple sources Results in better SQL performance in database

New Oracle Bulk Table Loader Supports Oracle specific load functions: Direct Path, Load to Partition, Oracle

Hints Generates passthru SQL Disable/renable constraints; Drop/Rebuild indices; Gather table statistics

Expression builder enhanced to list more native DBMS functions, including: Teradata 13, Oracle, DB2UDB 8.1, SQL Server, MySQL, and ODBC New wizard enables you to extend the function list with your own User Defined

Functions

Automation Current Evolution

Windows based Windows Tasks for scheduling Dedicated destinations for SAS log files Bat files for running programs Reports, plots and dashboards published on Intranet in

various formats:GIF (for plots), TXT, PDF, HTML, Microsoft Excel

System fully automated generating thousands of output files across several sites each day.


Automation - Folder Structure

Folders grouped by site and then common sub-foldersSAS_Log : SAS log filesTasks_Batch_Auto : Auto executed batch command filesTasks_Batch_Manual: Manually executed batch command filesTasks_Scheduler: Copy of Windows Scheduler Tasks


Automation - Command File


Windows Command File PTH_Daily_Pots.bat that runs several SAS programs sequentially

Command line syntax: C:\Program Files\SAS\SAS 9.1\sas.exe

Selects the executable version of SAS to be executed -NoTerminal

suppresses dialog boxes -SYSIN filespec.sas

submits the nominated file

Subsequent evolutions - Automation Slides

Use of integrated Scheduler Platform Suite for SAS Use Operating System scheduler Automating report generation using in-built schedulers Automated report distribution using Publish-Subscribe Automated source code version management using CVS,

Subversion etc. Automating job error notification using DI/code Command line driven job deployment delivers automated

job deployment


Maintenance and Upgrade

Monitor the system Remove bottlenecks Optimize code and usage Plan for regular updates

(http://support.sas.com/techsup/news/tsnews.html) Upgrade to latest (when feasible) but dont leave it too late


CPU(s) x Cores Brain

Memory x GB Fast local storage

Disk I/O Large storage Slow access

Network Connects distributed systems May connect disks

Identify bottlenecks after looking at all servers

Understand workload mix

SAS is I/O bound

Monitor system and Identify Bottlenecks


Migration to SAS 9.2/9.3 Examples

Choose from the following approaches Migrate content using the automated migration tools

as a part of the install processe.g. Simple Single machine install with no

customizations, no platform changes Install a new SAS 9.2/9.3 environment and promote

selected content from 9.1.3 e.g. Simple, Medium, Complex Selectively migrated

content in a large Govt. organization, Healthcare company, Top 4 bank

Or use a combination of these approaches Always discuss the approach in detail during the

migration assessment phase e.g. Complex Large Telco customer with a 4 year old

SAS Solution implementation with customizations


Q and A Opportunity

Please share your questionswith us. ?

Thank you


SAS Tips & TricksFormats

Matthew RodgerSenior Delivery [email protected]

We have all come across formats

We all had to handwrite something at some stage printed or cursive?

We have also had to chose a font in MS Word or an alignment? We have also formatted data in MS Excel These are all examples of formats, that is how you want your data

displayed. It is important to remember that formats change the way the data

is displayed, not the contents of the data

Types of SAS Formats

SAS Defined CharacterDate and Time ISO 8601 date, time and date time in ISO 8601 standardNumeric

User Defined Focus of this presentation

Basic Format Creation

PROC FORMAT LIBRARY=work;VALUE $mystate

'VIC' = 'Victoria''NSW' = 'New South Wales'other = 'Other'

;RUN;

Use VALUE to convert to text and INVALUE to convert to numbers.

Use $ when you are reading in text to be formatted. Dont use the $ when you are reading in numbers.

Using formats

Depends on the data type and what you intend to convert the data to.

From character to charactermy_new_var1=PUT(myvar1,$mformat.)

From numeric to charactermy_new_var2=PUT(myvar2,myformat.)

From character to numericmy_new_var3=INPUT(myvar3,$myformat.)

Using already created formats

If you have formats saved in a library, which is not your work library, you are able to use these

Just use options fmtsearch

LIBNAME mylib &dir;

OPTIONS fmtsearch=(mylib);

Creating a format from a dataset

DATA fmtdata(KEEP=start end label type hlo fmtname);SET mydata END=eof;start=myvar;end=myvar;label=myvar2;type='C';hlo=' ';fmtname='myformat';OUTPUT;IF eof THEN DO;

start="&default";end="&default";label="OTHER";type="C";hlo="O";fmtname='myformat';OUTPUT;

END;RUN;

PROC FORMAT LIBRARY=work CNTLIN=fmtdata;RUN;

Creating a format from a dataset

start and end are the start and end of a range. label is the numbers/characters required to be shown. type is the type of format. C for character to character, N for

number to character and I&J for the informats. hlo is for range boundaries. H for high, L for low and O for other.

A blank means that there is no upper or lower bounds or other. fmtname is the name of the format that will be invoked using SAS

code.

How to see the contents of a format

PROC FORMAT LIBRARY=work CNTLOUT=state;SELECT $mystate;

RUN;

If you dont use the SELECT statement, you will get all formats within a given library. Alternatively, you use the EXCLUDEstatement if you dont want to view a particular format.

Make sure you have a $ for a character to character format and an @ for an informat.

Formats and SAS 9.2

One of the great features of SAS 9.2 is the ability to write your own functions.

These functions can be used to create formats for reading and displaying data.

The End

Copyright 2011, SAS Institute Inc. All rights reserved.

James EnochEducation and Academic ManagerWednesday 23 May 2012

james_enochhttp://www.facebook.com/SASAustNZhttp://blogs.sas.com/content/academic/

Engaging with Academia in Victoria

63


SAS support for teaching and research

Teaching materials to support course development Train-the-trainer programs Graduate Research Fellowship Guest lectures from staff and customers Course projects, prizes, etc Work Placement links Certification programs

64


Where is SAS used for teaching in VIC?

65


Some examples of use

La Trobe University using SAS in their Masters of Business Information Management and Systems

RMIT University using SAS in their Bachelor of Science in Statistics

Swinburne University Using SAS in their Master of Science in Applied Statistics

University of Melbourne in Statistics courses Monash University in Finance courses NMIT in their Accounting degree

66


What SAS skills do students develop?

It varies between Universities and Faculties, however students pick up skills in:

Enterprise Guide Base SAS and additional modules Enterprise Miner Text Miner Soon - Forecast Server and JMP

67


Three ways for customers to engage with academia and SAS skilled graduates:

1. Guest Lectures

2. Student Projects

3. Work Placement Program

68


What is the SAS Work Placement Program? SAS run program that links students with SAS skills to

SAS customers for summer work placements.

The duration of a typical placement is between 6 to 12 weeks over the summer break.

Fantastic opportunity to link up with the right candidates. Great complement to your current recruitment program. Host employers interview and have final say on the

candidate.

Over 50% of placements lead to full-time employment.

69


Some of the value this provides to host employers as well as participants? Foot in the door that leads to further work at the host. Experience so they hit the ground running when

starting full-time.

Try before you buy. Enhances students academic performance by linking

theory to the real world.

Fresh ideas for an organisation. Develop professional relationships.

70


Contacting the Academic Program

Interest in hosting a Work Placement or other academic engagement?

Email me at [email protected] or the program coordinator [email protected] or come chat to me after this.

71


SAS Education what do we do? Who are we?

James Enoch, Education and Academic Manager Carol Yeomans, Education Account Manager

[email protected]

We run a wide variety of different training courses on SAS via different training modes

Keen to work with you to understand your needs and develop training plans to support your use of SAS: Carol has recently met with team managers and users to this end and to

structure training to maximise your investment To set a time to meet, email her or catch her after this presentation

72


Upcoming courses to point out:

We have a range of Melbourne based courses coming up that can be seen on our schedule http://bit.ly/JMbrGz

BKS courses run by Bart Baesens: Credit Risk Modelling Using SAS (30 July to 1 August) Analytics: Putting it all to work (2 August) Early bird 20% BKSES discount closes June 11 Reserve your seat sooner rather than later.

Mixed Model Analyses Using SAS in June


James EnochEducation and Academic Manager

[email protected]

Any questions?

ThankYouyoursignificantcontributiontotheSASUserGroup&theSMUGCommittee

FarewellandBestwishesMattRodger!!

smug presentations q2 may2012

Documents

sas applicationwhy sas

sas applicationpresentation

stakeholdersengage sas

sas applicationwhat

sas applicationwhere

sas applicationthe life

sas application access

sas continuous improvement