optimizing the efficiency of the ncar-wyoming supercomputing center facility a
DESCRIPTION
Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A Software Perspective. Theophile Nsengimana Collaborator: Ademola Olarinde Mentor: Aaron Andersen August 1, 2014. Project Goals. Work on Software Tools to Automate: Collection of NWSC Building Sensor Data - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/1.jpg)
1
Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility
ASoftware Perspective
Theophile Nsengimana Collaborator: Ademola Olarinde
Mentor: Aaron Andersen
August 1, 2014
![Page 2: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/2.jpg)
2
Project Goals• Work on Software Tools to Automate:
– Collection of NWSC Building Sensor Data
– Quality Control
– Visualization of Key Building Parameters
• Work with Ademola Olarinde:– Software Tools
– Requirements
– Data Exploration
![Page 3: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/3.jpg)
3
• Preplanned Method:– Store sensor-based raw data generated by Johnson Controls Inc. into a
data store.– sMAP (Simple Measurement and Actuation Profile) from Berkeley. [1]
• Current Method:– Restructuring raw data into CSV file(s).
Data Collection
![Page 4: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/4.jpg)
4
sMAP Overview• An open source modular software designed fundamentally to ease
the collection, storage and retrieval of time series data. • Time series source: archival and real-time from sensors.• Time series can be tagged with metadata
![Page 5: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/5.jpg)
5
sMAP Components
• sMAP sources– Connect to physical sensors to expose the data they generated to
sMAP archiver (repository) via http.– Real-time data.
![Page 6: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/6.jpg)
6
sMAP components
• sMAP archiver– A high-performance data store – Connects to both relational and time series databases. (Postgres for
metadata storage and Readingdb for time series storage)
• Applications– Make use of data: visualization, computing control optimal
strategies, etc.
![Page 7: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/7.jpg)
7
Challenges with sMAP
• General– Documentation– Community support
• Relatable to this project– Couldn’t load Archival data whose time was out of the range
[(time_now – 24 hours), time_now]
![Page 8: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/8.jpg)
8
Quality Control
• Proper Formatting– Hourly (or daily, monthly, yearly) Interval as opposed to 15-minutes
interval– Match time format across all generated csv files– Handle missing data– Eliminate irrelevant data– Merge properly formatted csv files into one csv file for faster access
![Page 9: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/9.jpg)
9
Quality Controldef day_interval(self): BY = ‘day’ init_dt, init_value = self.init_data.popitem() samedts = self.getSameDatetimes(self.init_data, init_dt, BY) #find datetimes with same day as init_dt value = init_value #the sum of values of ‘samedts’ missingCounter = 0 #missing data zeroCounter = 0 #how many times this sensor has been offline. for ndt in samedts: ch_value = self.init_data.pop(ndt) try:
if float(ch_values) != 0: value += float(ch_values) elif: zeroCounter += 1 except ValueError: missingCounter += 1 #calculate the average if missingCounter == len(samedts): value = ‘M’ elif (zeroCounter == len(samedts)) or (zeroCounter > 0 and (zeroCounter + missingCounter == len(samedts))): value = 0 else: value = value / ( len(samedts) – (missingCounter + zeroCounter)) self.fin_data[self.makedt(init_dt, BY)] = value
![Page 10: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/10.jpg)
10
Data Analysis & Visualization
• Python packages– Matplotlib – Numpy
• Main focus– Statistical correlation– Basic Control Charts– Plots
![Page 11: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/11.jpg)
11
Statistical Correlation• How a sensor’s performance is related to other various factors.• E.g., Cooling Towers vs Outside Air and Computer Load
Cooling Towers’ correlations against time, wet-bulb, wet-bulb depression, dry-bulb and computer load.
![Page 12: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/12.jpg)
12
Control Charts
• Whether a particular sensor’s performance is in a statistical control.
• E.g., evaluate the performance of Condenser Water pumps.
![Page 13: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/13.jpg)
13
Plots
• Visualize and understand the relationship between sensors’ performance and other factors (or sensors).
• E.g., Condenser Water Pumps vs time.
![Page 14: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/14.jpg)
14
Summary• Development of software tools to facilitate the process of collecting
data from NWSC, performing basic quality control, and analyzing as well as visualizing key parameters from data collected.
![Page 15: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/15.jpg)
15
Future Work
• Two possibilities:– Configuring sMAP properly and develop a high level custom
application on top of sMAP archiver for NWSC staff to monitor the facility.
![Page 16: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/16.jpg)
16
Future Work
• Two possibilities:– Stick with CSV files and take advantage of Google Fusion
Tables, an experimental yet powerful data visualization web application developed by Google. [2]
![Page 17: Optimizing the Efficiency of the NCAR-Wyoming Supercomputing Center Facility A](https://reader036.vdocuments.site/reader036/viewer/2022070406/568141cc550346895dada6e7/html5/thumbnails/17.jpg)
17
References
[1] sMAP 2.0 documentation http://www.cs.berkeley.edu/~stevedh/smap2/
[2] Google Fusion Table Help Center https://support.google.com/fusiontables/?hl=en