midwestpowernewslett

10
3 rd Quarter Midwest Newsletter 6/27/2022 Page 1 of 10 POWER Systems Tech University! Orlando, FL Sept 21-25 The IBM Power Systems Technical University featuring IBM AIX/Linux and IBM i is the premiere Power technical event. Don’t miss the opportunity to gain new skills, obtain an update on what’s new, network with colleagues, and meet with our subject matter experts in a professional networking collaboration of technical gurus and industry professionals like yourself. Choose from over 250 technical sessions that cover: New features, tips and best practices for IBM POWER6™, IBM AIX® V6.1 and IBM i v6.1 Workload consolidation delivering maximum ROI IBM PowerVM software to deliver industry-leading virtualization How to improve server utilization and share I/O resources for better total cost of ownership (TCO) Consolidation of Intel® and IBM i applications into a single chassis with the new blade servers and the IBM System i™ platform Best practices for a smooth migration to IBM POWER6 and IBM i v6.1 or IBM AIX V6.1 technology Dynamic reallocation tools and techniques to handle changing business cycles and surges in demand IBM Power Architecture® technology and the steps needed for a greener IT operation and financial advantage Enabling security, high performance and high In this Issue Power Systems Tech University New JS23/JS43 Blade Servers IBM Systems Director 6.1 update SLES 11 for Power POWER-i (i- series) corner VIOS Tips HMC Tips Active Memory Sharing SEA & Broadcast Storms Power5/ Power6 Firmware Update JS23 & JS43 Blades – New!! On April 28 th , 2009 IBM announced two new POWER6 blades – the JS23 and JS43. These latest editions to the Power Systems family pack a lot of punch. Both “J” blades run POWER6 4.2GHz processors and support the AIX 5.3, AIX 6.1, Linux, and IBM i (v6) operating systems. PowerVM Standard Edition is still included free of charge. Enterprise edition is also available, which adds Live Partition Mobility and Active Memory Sharing capabilities. The JS23 has 4 cores and supports up to 64GB of RAM (in eight DIMM slots). It supports one of the following disk drives: 73 / 146 / 300GB 10K RPM spinning drive or a 69GB solid state disk. Dual port gigabit Ethernet, 4 USB, and 1 serial over lan are integrated. Two PCI-E expansion cards are also available, CFFh and CIOv. The rPerf and CPW numbers are 36.28 and 14,400, respectively. Adding feature code 8446 converts the JS23 into the two-wide JS43 model. The blade is now double wide and has twice the number of features – 8 cores, 128GB of RAM (sixteen DIMM slots), two spinning or solid state disk, 2x integrated dual port gigabit Ethernet, and up to four PCI-E expansion cards (2x CFFh; 2x CIOv). The rPerf and CPW numbers are 68.20 and 24,050 , respectively. CFFh expansion cards: QLogic Ethernet & 4 Gb Fibre Channel (#8252) 4x InfiniBand DDR (8258) QLogic 8 Gb Fibre Channel (8271)

Upload: cameroon45

Post on 26-Jun-2015

346 views

Category:

Technology


2 download

TRANSCRIPT

Page 1: MidwestPOWERNewslett

b

3rd Quarter Midwest Newsletter 4/13/2023 Page 1 of 9

POWER Systems Tech University! Orlando, FL Sept 21-25The IBM Power Systems Technical University featuring IBM AIX/Linux and IBM i is the premiere Power technical event.

Don’t miss the opportunity to gain new skills, obtain an update on what’s new, network with colleagues, and meet with our subject matter experts in a professional networking collaboration of technical gurus and industry professionals like yourself.

Choose from over 250 technical sessions that cover:

New features, tips and best practices for IBM POWER6™, IBM AIX® V6.1 and IBM i v6.1

Workload consolidation delivering maximum ROI

IBM PowerVM software to deliver industry-leading virtualization

How to improve server utilization and share I/O resources for better total cost of ownership (TCO)

Consolidation of Intel® and IBM i applications into a single chassis with the new blade servers and the IBM System i™ platform

Best practices for a smooth migration to IBM POWER6 and IBM i v6.1 or IBM AIX V6.1 technology

Dynamic reallocation tools and techniques to handle changing business cycles and surges in demand

IBM Power Architecture® technology and the steps needed for a greener IT operation and financial advantage

Enabling security, high performance and high availability for 24x7 operations

Tools that simplify the life of the system administrator

Visit Power University for agenda, costs, and sign-up information. Looking forward to seeing you in Orlando, FL in September.

In this Issue

Power Systems Tech University

New JS23/JS43 Blade Servers

IBM Systems Director 6.1 update

SLES 11 for Power

POWER-i (i-series) corner

VIOS Tips

HMC Tips

Active Memory Sharing

SEA & Broadcast Storms

Power5/Power6 Firmware Update

JS23 & JS43 Blades – New!!On April 28th, 2009 IBM announced two new POWER6 blades – the JS23 and JS43. These latest editions to the Power Systems family pack a lot of punch. Both “J” blades run POWER6 4.2GHz processors and support the AIX 5.3, AIX 6.1, Linux, and IBM i (v6) operating systems.

PowerVM Standard Edition is still included free of charge. Enterprise edition is also available, which adds Live Partition Mobility and Active Memory Sharing capabilities.

The JS23 has 4 cores and supports up to 64GB of RAM (in eight DIMM slots). It supports one of the following disk drives: 73 / 146 / 300GB 10K RPM spinning drive or a 69GB solid state disk. Dual port gigabit Ethernet, 4 USB, and 1 serial over lan are integrated. Two PCI-E expansion cards are also available, CFFh and CIOv. The rPerf and CPW numbers are 36.28 and 14,400, respectively.

Adding feature code 8446 converts the JS23 into the two-wide JS43 model. The blade is now double wide and has twice the number of features – 8 cores, 128GB of RAM (sixteen DIMM slots), two spinning or solid state disk, 2x integrated dual port gigabit Ethernet, and up to four PCI-E expansion cards (2x CFFh; 2x CIOv). The rPerf and CPW numbers are 68.20 and 24,050 , respectively.

CFFh expansion cards:QLogic Ethernet & 4 Gb Fibre Channel (#8252)4x InfiniBand DDR (8258)QLogic 8 Gb Fibre Channel (8271)Voltaire 4x InfiniBand DDR (8298)

CIOv expansion cards:Emulux 8Gb Fibre Channel (8240)QLogic 4Gb Fibre Channel (8241)QLogic 8Gb Fibre Channel (8242)3Gb SAS Pass-through (8246)

Supported Operating System levels:AIX 5.3: TL07-SP9, TL08-SP7, TL09-SP4, &

TL10AIX 6.1: TL00-SP9, TL01-SP5, TL02-SP4, &

TL03SLES10 Service Pack 2 & SLES11RHEL 4.6, 4.7, 5.1, 5.2, & 5.3IBM i 6.1

Additional Information:JS23 / JS43 Home PageJS23 / JS43 Server Proven (Compatibility) web pageOr contact Rick Milton [email protected]

Page 2: MidwestPOWERNewslett

3rd Quarter Midwest Newsletter 4/13/2023 Page 2 of 9

Page 3: MidwestPOWERNewslett

3rd Quarter Midwest Newsletter 4/13/2023 Page 3 of 9

New Release of IBM Systems Director 6.1.1

To update Systems Director follow the steps below1. Logon into Systems Director2. Manage Tab3. View updates – upper right hand link4. Check for updates5. Select all and install updates.6. Recycle Systems Director services.

Systems Director 6.1 feature spotlight Update ManagerAcquires, distributes and installs required firmware, device drivers and operating system updates using predefined policies and workflows.

This includes compliance status to indicate what managed systems may require critical updates. The IBM Systems Director product itself is now updated via Update Manager and allows customers to see which updates or which configurations have been applied to a system

Download, Manage, and apply recommended Updates for: Power Systems - AIX, p, Linux, i5/OS, HMC and System FirmwareSystem x and BladeCenter - firmware and drivers

Updating AIX – uses a NIM server to upgrade AIX client partitions.

AIX updates are downloaded and staged on the Systems Director server “/opt/ibm/director/data/updatelib/” directory

Updates are then copied to the NIM server /exports directory, this needs to be a filesystem or else / will fill-up.

Systems Director will work with the NIM server to update and install the AIX update on the AIX client partition, and reboot the server if necessary.

#Note – Systems Director assumes that the NIM server was used to create the AIX client partition, meaning that the NIM server needs to know about the AIX client partition. If the NIM server does not know about the AIX client partition, then you need to create a NIM Client machine, and on the AIX client partition, run <niminit –a name=Client_Hostname –a master= NIMmaster_Hostname>

Updating Linux, HMC, firmware, etc… will use the Systems Director staging area to update the client system. Contributed by Doug Herman ([email protected]).

VIOS Tips: Two commands that might be of assistance in a VIOS environment (systat, and lsgcl). Sysstat lists current load on the system, similar to AIX version of uptime command, whilel lsgcl lists VIOS commands in datesetamped format.# sysstat 03:49PM up 80 days, 3:11, 2 users, load average: 0.00, 0.01, 0.08User tty login@ idle JCPU PCPU whatpadmin vty0 27Jun09 17days 20:54 20:54 -rksh# lsgcl | grep mkvdev (lists all VIOS commands with mkvdev in the command)Jun 23 2007, 11:22:17 padmin mkvdev -vdev djdata -vadapter vhost7 -devJul 3 2007, 14:17:14 padmin mkvdev -vdev aixgui -vadapter vhost0 -devJul 25 2007, 11:59:14 padmin mkvdev -vdev sandboxlv -vadapter vhost9 -devJul 25 2007, 12:08:29 padmin mkvdev -sea ent1 -vadapter ent2 -default ent2Jun 3 2008, 10:04:08 padmin mkvdev -vdev cd1 -vadapter vhost3Mar 4 2009, 16:14:31 padmin mkvdev -fbo -vadapter vhost5

Page 4: MidwestPOWERNewslett

3rd Quarter Midwest Newsletter 4/13/2023 Page 4 of 9

SLES 11 now Available for POWERSLES 11 delivers mission-critical support for Power to help improve service, reduce cost, and manage risk. For customers who have been waiting for Linux to approach AIX in terms of exploiting Power 6 capabilities, now may be the time to consider running Linux on Power. IBM controls development of AIX so Linux generally lags behind and catches up as the OpenSource community and respective distributions approve Power specific code. SLES 11 represents Novell's latest offering and represents a huge leap in many key areas. . If you are a Red Hat fan, be patient, as the distros tend to leapfrog each other with features at major releases. IBM is actively working with each distro to provide new, Power- specific code.

Demonstrating our commitment to Linux, the following contributions have been made by IBM to SLES 11 for Power DLPAR Memory Remove (this has been a long time coming) PowerVM Active Memory Sharing Dynamic Heterogeneous Multi-Path I/O N_Port ID Virtualization (NPIV) Virtual Tape IBM Installation Toolkit for Linux v3.1

http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/installtools/home.html IBM PowerVM Lx86 V1.3.1Note that Live Partition Mobility is not missing, it has been available since It was announced on Power 6 using SLES 10 sp1 and Red Hat 5.2.Contributed by Kevin McCombs [email protected]

HMC Tip – Using vtmenu to access your lpars

The HMC command vtmenu can be used to access any lpar within any managed frame within your environment.For those administrators who either don’t care for the time aspect of navigating through the HMC Gui or just preferthe command line in general, the vtmenu command provides this access.

Administrators can either telnet or ssh to the HMC and upon login, simply enter the ‘vtmenu’ command. If your HMC is controlling multiple servers or frames, you will be presented with a list of these frames. Once you select a frame, you then will be presented with a list of the partitions or lpars within that frame (see screen shot below).

Once you’ve selected an lpar, you’ll simply be presented with a console login screen.See IBM’s Infocenter webpages regarding vtmenu for further capabilities.

Contributed by Rick Beach [email protected]

iSeries Redbooks Website

http://www.redbooks.ibm.com/

Recent Releases IBM iSeries IP Networks: Dynamic! May 5, 2004 SG24-6718-00V5 TCP/IP Applications on the IBM iSeries May 11, 2004 SG24-6321-00Using IBM WebSphere Host Access Transformation Services V5 May 17, 2004 SG24-6099-00Lotus Domino 6 Multi-Versioning Support on the IBM iSeries May 21, 2004 SG24-6940-00IBM i5 and iSeries System Handbook: IBM i5/OS Version 5 Release 3 May 28, 2004 GA19-5486-25Domino 6 for iSeries Best Practices Guide June 18, 2004 SG24-6937-00

Page 5: MidwestPOWERNewslett

3rd Quarter Midwest Newsletter 4/13/2023 Page 5 of 9

Active Memory Sharing – New POWER Systems Feature

IBM announced Active Memory Sharing, a feature of PowerVM Virtualization technology that offers customers greater efficiency through intelligent management of memory allocation, and improve performance of the applications at the same time. Combined with the dynamic allocation of processors to LPARs, the dynamic allocation of memory depending on the workload characteristics improves flexibility and optimum resource utilization to lower the TCO.

PowerVM Active Memory Sharing is designed to increase memory utilization on systems that are running partitions with variable memory requirements. Instead of dedicating memory to partitions, Active Memory Sharing can automatically flow the memory between partitions as their memory demands change. For example, systems with partitions that serve different parts of the world or day and night workloads can have memory automatically moved from the partition that is winding down to the partition that is ramping up, improving memory utilization and system throughput.

A shared memory pool is similar to shared processor pool and all of the memory activities (memory allocation and de-allocation) takes place within the shared memory pool. The size of the shared memory pool can be changed dynamically and up to 128 logical partitions can be assigned to a memory pool. If the amount of memory required by the partitions in the shared memory pool exceeds the amount of memory available in the shared memory pool then paging will occur to the paging device owned by the VIOS.

This new virtualization feature is optionally configurable on a partition basis, enabling Power servers to support a combination of dedicated and shared memory partitions. Active Memory Sharing is only provided with PowerVM Enterprise Edition.

Minimum system requirements to use AMS are:* An IBM Power System server based on the POWER6 processor* Enterprise Edition of PowerVM* Firmware level 340_070* HMC V7R3.4.0 Service Pack 2* Virtual I/O Server Version 2.1.1 for both HMC and IVM managed systems* OS - AIX 6.1-TL03 or IBM i 6.1 or SUSE Linux Enterprise Server 11 on client LPARs* All resources on client LPARs should be virtualized (Processor and I/O devices)* paging devices on one or more Virtual IO Servers.

Partitions with dedicated Memory Partitions with shared Memory

References: 1. White paper on AMS http://www-03.ibm.com/systems/power/software/virtualization/whitepapers/2. Redpaper PowerVM Virtualization Active Memory Sharing REDP-4470-00 http://www.redbooks.ibm.com

Contributed by Ravi Singh ([email protected]).

Page 6: MidwestPOWERNewslett

3rd Quarter Midwest Newsletter 4/13/2023 Page 6 of 9

SEA Failover and Avoiding Broadcast StormsThe two common methods available to provide virtual I/O client partition network redundancy in dual Virtual I/O Servers configurations are Network Interface Backup (NIB) and Shared Ethernet Adapter (SEA) Failover. Network interface Backup is implemented at the client LPAR using two virtual ent devices. Shared Ethernet Adapter Failover was introduced at VIOS 1.2 Fix Pack 7. It has become the more popular method of choice. Shared Ethernet Adapter failover provides redundancy by configuring a backup Shared Ethernet Adapter on a different Virtual I/O Server logical partition that can be used if the primary Shared Ethernet Adapter fails. The client LPAR network connectivity continues without disruption. Typical SEA Failover Configuration

Shared Ethernet Adapter (SEA) Failover is implemented on the Virtual I/O Server using a bridging (layer-2) approach to access external networks. SEA Failover supports IEEE 802.1Q VLAN-tagging, unlike Network Interface Backup. With SEA Failover, two Virtual I/O Servers have the bridging function of the Shared Ethernet Adapter to automatically fail over if one Virtual I/O Server is unavailable or the Shared Ethernet Adapter is unable to access the external network through its physical Ethernet adapter.

Broadcast StormsA broadcast storm is a situation where one message that is broadcast across a network results in multiple responses. Each response generates more responses, causing excessive transmission of broadcast messages. Severe broadcast storms can block all other network traffic, but they can usually be prevented by carefully configuring a network to block illegal broadcast messages.

How to avoid a potential broadcast storm1. Upgrade your VIO code to current levels. The latest updates available from IBM Fix Central are VIOS 2.1.1.10 (Fix

Pack 21) and VIOS 1.5.2.1 (Fix Pack 11.1). See VIOS Support Site for VIOS fixes/levels

2. When setting up SEA Failover, be sure to create the standby SEA failover in one step using the mkvdev command. mkvdev -sea ent#_physical -vadapter ent#_data_veth -default ent#_data_veth -defaultid ent#_data_veth_vid -attr ha_mode=auto ctl_chan=ent#_ctl_chan

Do not create the standby SEA as a regular SEA and then chdev it to failover mode. If the SEA is configured in 2 steps, one to create the SEA adapters and the second one to run a chdev to add in the HA value, it can cause a broadcast storm.

3. Talk to your network engineers about implementing BPDU on the switch ports. The Shared Ethernet Adapter is designed to prevent network loops. However, as an additional precaution, you can enable Bridge Protocol Data Unit (BPDU) Guard on the switch ports connected to the physical adapters of the Shared Ethernet Adapter. BPDU Guard detects looped Spanning Tree Protocol BPDU packets and shuts down the port. This helps prevent broadcast storms on the network

Other related information can be found at:1) InfoCenter: Shared Ethernet Adapter Failover (SEA)2) Document from Cisco regarding Spanning Tree: Understanding Spanning Tree Protocol

Contributed by Tony Garone [email protected]

Page 7: MidwestPOWERNewslett

3rd Quarter Midwest Newsletter 4/13/2023 Page 7 of 9

IBM i-6.1 enhancements 2Q 2009 IBM i supports additional IBM Power6 hardware

o Virtual Tape support enables IBM i partitions to directly backup to PowerVM VIOS attached tape drive saving hardware costs and management time.

o IBM i supports additional options for customers looking to implement SAN solutions. Enhancements include smart fibre channel attachment of DS8000 extended to Power5 Systems, new attachment options for DS6800 and support for DS5000 with PowerVM.

o The i Edition Express for BladeCenter S configuration has been updated to increase the minimum memory from 2GB to 4GB on the PowerBlade JS12 Express blade and to include a new Intelligent Copper Pass-Thru Module.

o For more information: Announcement Letter 209-078

IBM i integration with Bladecenter and System x enhancementso i 6.1 supports the Microsoft software initiator

service with select models of Bladecenter and System x servers providing the same level of integration, while saving the expense of the iSCSI hardware adapter. Supported with Microsoft® Windows 2008 Server and Windows 2003 Server.

o For more information: IBM i Integration with BladeCenter and System x

Two new features join the DB2® Web Query for i family:o DB2 Web Query for i is enhanced with two new

features to enable integration between DB2 WebQuery reporting environment and Microsoft products. DB2 Web Query Adapter for Microsoft SQL Server provides connectivity from DB2 Web Query in IBM i to remote SQL Databases. DB2 Web Query Spreadsheet Client provides enhanced capabilities for user of Microsoft Excel 2002, or later

o For more information: Announcement Letter 290-090

DB2 Storage Engine for My SQL o IBM and MySQL are delivering a SQL storage

engine for MySQL on i5/OS. With a DB2 storage engine, the applications written to MySQL will run on i5/OS and the data will be stored in DB2. This will allow you to implement online and transactional MySQL applications while storing all of the data in a single, easy-to-manage DB2 Database.

o For more information: Using IBM DB2 for i as a Storage Engine of MySQL Redbook

The IBM Temporary Software License for io The IBM temporary Software License for i is

enhanced to offer a the temporary licensing of i5/OS and IBM i processors, users, and application servers.

o For more information: Announcement Letter 209-085

Contributed by John Bizon [email protected]

Solid State Device (SSD) announcement On April 28th, IBM announced the ability to support Solid State Disk (SSD) on IBM i 6.1. With no seek time or rotational delays, SSDs can deliver substantially better I/O performance than Hard Disk Drives (HDD), bridging the gap between memory and disk speeds. On many systems, HDD’s capacity utilization is held low to help ensure higher I/O performance and more consistent response time. Often, for performance sensitive workloads, this is less than 30-50% of capacity. SSD capacity utilization is not restricted and can run much closer to 100% without a performance impact.

Mixing SSD and HDD can provide a cost effective and highly efficient solution. It is typical for databases to have a large percentage of data which is infrequently used (“cold”) and a small percentage of data which is frequently used (“hot”). Only 10-20% of the data may be considered hot data, but it can be responsible for 80-90% of the disk activity. Since SSD offers the best performance, it should be focused on hot data. HDD offers a lower storage cost, so it should be focused on cold data. This can allow the use of larger HDD and/or use them at a higher percent of capacity, reducing the total quantity of drives needed.

To allow you to identify and move hot and cold data to the appropriate disk drives, IBM has included a “Trace and Balance” function as part of IBM i 6.1. It monitors an ASP (Aux Storage Pool) to determine hot and cold data. Upon request, it automatically moves hot data to SSD and cold data to HDD. You can monitor and rebalance an ASP at any time. A few key OS files are automatically placed on SSD. You can also specify specific database objects to be placed on SSDs as well.

Trans/min

Ap

pli

cati

on

Resp

on

se t

ime

72 HDD + 16 SSD No Balance

72 HDD + 16 SSD Data Balanced

Not only are SSDs capable of driving tens of thousands of I/O operations per second (IOPS), as opposed to hundreds for HDDs, they can also provide a more reliable system. Compare the chance of one SSD failing versus the chance of one out of several HDD failing (using the scenario of a few SSDs and a few larger HDDs replacing many smaller HDDs). SSD have no moving parts, which adds to its reliability. Using sophisticated management of flash memory, IBM can avoid over using storage locations (wear leveling). Finally, there is a large amount (80%) of spare storage in IBM SSDs on standby to extend its life. For more information refer to: Performance Value of SSD using IBM iContributed by Dean Woodke [email protected]

Page 8: MidwestPOWERNewslett

3rd Quarter Midwest Newsletter 4/13/2023 Page 8 of 9

Power5/Power6 Firmware Updates – Naming ConventionsIBM POWER systems firmware naming conventions:PPNNSSS_FFF_DDD· PP = package identifier;

a) If this value is 01, it is identifying server (system) firmwareb) On Power5 Systems, if this value is 02, it identifies power subsystem firmware (Bulk Power Code).

· NN = machine type/model groupThe server firmware for each machine type/model group is given its own unique 2-character code.In IBM Power Systems™, the server firmware for the individual machine type/model groups are identified by one of the following: EA, EH, EM, EL, ESIn Power6 Systems™ Bulk Power Code is identified by either: EB or EP

· SSS = Release Level Indicator (e.g., 310)· FFF = Service pack level within that release (this number is incremental and increases with each service pack)· DDD = Release or Service Pack level of the last disruptive level

Release Levels and Service Packs consist of a cover letter, an XML file and the firmwareRPM file (for example, 01EM310_001_001.xml and 01EM310_001_001.rpm).

Firmware packages also have Severity Definitions; here’s how they are classified:Symbol Meaning DescriptionHIPER High Impact/PERvasive Should be installed as soon as possible.

SPE SPEcial Attention Should be installed at earliest convenience. Fixes forlow potential high impact problems

ATT ATTention Should be installed at earliest convenience. Fixes forlow potential low to medium impact problems.

PE Programming Error Can install when convenient. Fixes minor problems.

When should you consider updating firmware ? There are some natural points at which firmware should be evaluated for potential updates:· When a subscription notice advises of a critical or HIPER (highly pervasive) fix.· When one of the twice-yearly Release Levels are released.· Whenever new hardware is introduced into the environment.· Anytime HMC firmware levels are adjusted.· Whenever an outage is scheduled for a system which otherwise has limited opportunity to update or upgrade.· When your system's firmware level is approaching end-of-service or “End of Service Pack Support”.· If other similar hardware systems are being upgraded and firmware consistency can be maximized by a more homogeneous firmware level.· On a “twice yearly” cycle if firmware has not been updated or upgraded within the last year.

Finally, be sure to check out the “Service and support best practices for POWER Systems” web page at:

http://www14.software.ibm.com/webapp/set2/sas/f/best/home.html

On this page is a link to firmware best practices labeled:

IBM Power Systems System Firmware (Microcode) Service Strategies and Best Practices

Contributed by Ross Coniglio ([email protected]).

ContributorsOH and No. KYCleveland, OH Kevin McCombs [email protected] Cincinnati, OH Ross Coniglio [email protected], OH Rick Beach [email protected]

IN PAIndianapolis, IN Brett Murphy [email protected] Pittsburgh, PA [email protected]

MIFlint, MI Dean Woodke [email protected] Detroit, MI John Bizon [email protected], MI Ravi Singh [email protected] Detroit, MI Rick Milton [email protected] Detroit, MI Doug Herman [email protected] Mgr, Brian Richmond [email protected]

Page 9: MidwestPOWERNewslett

3rd Quarter Midwest Newsletter 4/13/2023 Page 9 of 9