workload modeling and its effect on performance evaluation dror feitelson hebrew university
Post on 21-Dec-2015
221 views
TRANSCRIPT
![Page 1: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/1.jpg)
Workload Modelingand its Effect on
Performance Evaluation
Dror Feitelson
Hebrew University
![Page 2: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/2.jpg)
Performance Evaluation
• In system design– Selection of algorithms– Setting parameter values
• In procurement decisions– Value for money– Meet usage goals
• For capacity planing
![Page 3: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/3.jpg)
The Good Old Days…
• The skies were blue
• The simulation results were conclusive
• Our scheme was better than theirs
Feitelson & Jette, JSSPP 1997
![Page 4: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/4.jpg)
But in their papers,
Their scheme was better than ours!
![Page 5: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/5.jpg)
How could they be so wrong?
![Page 6: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/6.jpg)
• The system’s design(What we teach in algorithms and data structures)
• Its implementation(What we teach in programming courses)
• The workload to which it is subjected
• The metric used in the evaluation
• Interactions between these factors
Performance evaluation depends on:
![Page 7: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/7.jpg)
• The system’s design(What we teach in algorithms and data structures)
• Its implementation(What we teach in programming courses)
• The workload to which it is subjected
• The metric used in the evaluation
• Interactions between these factors
Performance evaluation depends on:
![Page 8: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/8.jpg)
Outline for Today
• Three examples of how workloads affect performance evaluation
• Workload modeling– Getting data– Fitting, correlations, stationarity…– Heavy tails, self similarity…
• Research agenda
In the context of parallel job scheduling
![Page 9: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/9.jpg)
Example #1
Gang Scheduling and
Job Size Distribution
![Page 10: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/10.jpg)
Gang What?!?
Time slicing parallel jobs with coordinated context switching
Ousterhoutmatrix
Ousterhout, ICDCS 1982
![Page 11: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/11.jpg)
Gang What?!?
Time slicing parallel jobs with coordinated context switching
Ousterhoutmatrix
Optimization:Alternativescheduling
Ousterhout, ICDCS 1982
![Page 12: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/12.jpg)
Packing Jobs
Use a buddy system for allocating processors
Feitelson & Rudolph, Computer 1990
![Page 13: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/13.jpg)
Packing Jobs
Use a buddy system for allocating processors
![Page 14: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/14.jpg)
Packing Jobs
Use a buddy system for allocating processors
![Page 15: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/15.jpg)
Packing Jobs
Use a buddy system for allocating processors
![Page 16: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/16.jpg)
Packing Jobs
Use a buddy system for allocating processors
![Page 17: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/17.jpg)
The Question:
• The buddy system leads to internal fragmentation
• But it also improves the chances of alternative scheduling, because processors are allocated in predefined groups
Which effect dominates the other?
![Page 18: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/18.jpg)
The Answer (part 1):
Feitelson & Rudolph, JPDC 1996
![Page 19: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/19.jpg)
The Answer (part 2):
![Page 20: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/20.jpg)
The Answer (part 2):
![Page 21: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/21.jpg)
The Answer (part 2):
![Page 22: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/22.jpg)
The Answer (part 2):
• Many small jobs
• Many sequential jobs
• Many power of two jobs
• Practically no jobs use full machine
Conclusion: buddy system should work well
![Page 23: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/23.jpg)
Verification
Feitelson, JSSPP 1996
![Page 24: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/24.jpg)
Example #2
Parallel Job Scheduling
and Job Scaling
![Page 25: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/25.jpg)
Variable Partitioning
• Each job gets a dedicated partition for the duration of its execution
• Resembles 2D bin packing
• Packing large jobs first should lead to better performance
• But what about correlation of size and runtime?
![Page 26: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/26.jpg)
Scaling Models
• Constant work– Parallelism for speedup: Amdahl’s Law– Large first SJF
• Constant time– Size and runtime are uncorrelated
• Memory bound– Large first LJF– Full-size jobs lead to blockout
Worley, SIAM JSSC 1990
![Page 27: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/27.jpg)
“Scan” Algorithm
• Keep jobs in separate queues according to size (sizes are powers of 2)
• Serve the queues Round Robin, scheduling all jobs from each queue (they pack perfectly)
• Assuming constant work model, large jobs only block the machine for a short time
• But the memory bound model would lead to excessive queueing of small jobs
Krueger et al., IEEE TPDS 1994
![Page 28: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/28.jpg)
The Data
![Page 29: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/29.jpg)
The Data
![Page 30: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/30.jpg)
The Data
![Page 31: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/31.jpg)
The Data
Data: SDSC Paragon, 1995/6
![Page 32: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/32.jpg)
The Data
Data: SDSC Paragon, 1995/6
![Page 33: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/33.jpg)
The Data
Data: SDSC Paragon, 1995/6
![Page 34: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/34.jpg)
Conclusion
• Parallelism used for better results, not for faster results
• Constant work model is unrealistic
• Memory bound model is reasonable
• Scan algorithm will probably not perform well in practice
![Page 35: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/35.jpg)
Example #3
Backfilling and
User Runtime Estimation
![Page 36: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/36.jpg)
Backfilling
• Variable partitioning can suffer from external fragmentation
• Backfilling optimization: move jobs forward to fill in holes in the schedule
• Requires knowledge of expected job runtimes
![Page 37: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/37.jpg)
Variants
•EASY backfilling
Make reservation for first queued job
•Conservative backfilling
Make reservation for all queued jobs
![Page 38: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/38.jpg)
User Runtime Estimates
• Lower estimates improve chance of backfilling and better response time
• Too low estimates run the risk of having the job killed
• So estimates should be accurate, right?
![Page 39: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/39.jpg)
They Aren’t
Mu’alem & Feitelson, IEEE TPDS 2001
![Page 40: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/40.jpg)
Surprising Consequences
• Inaccurate estimates actually lead to improved performance
• Performance evaluation results may depend on the accuracy of runtime estimates– Example: EASY vs. conservative– Using different workloads– And different metrics
![Page 41: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/41.jpg)
EASY vs. Conservative
Using CTC SP2 workload
![Page 42: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/42.jpg)
EASY vs. Conservative
Using Jann workload model
![Page 43: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/43.jpg)
EASY vs. Conservative
Using Feitelson workload model
![Page 44: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/44.jpg)
Conflicting Results Explained• Jann uses accurate runtime estimates
• This leads to a tighter schedule
• EASY is not affected too much
• Conservative manages less backfilling of long jobs, because respects more reservations
![Page 45: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/45.jpg)
Conservative is bad for the long jobsGood for short ones that are respected
Conservative
EASY
![Page 46: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/46.jpg)
Conflicting Results Explained
• Response time sensitive to long jobs, which favor EASY
• Slowdown sensitive to short jobs, which favor conservative
• All this does not happen at CTC, because estimates are so loose that backfill can occur even under conservative
![Page 47: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/47.jpg)
Verification
Run CTC workload with accurate estimates
![Page 48: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/48.jpg)
But What About My Model?
Simply does not have such small long jobs
![Page 49: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/49.jpg)
Workload Data Sources
![Page 50: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/50.jpg)
No Data
• Innovative unprecedented systems– Wireless– Hand-held
• Use an educated guess– Self similarity– Heavy tails– Zipf distribution
![Page 51: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/51.jpg)
Serendipitous Data
• Data may be collected for various reasons– Accounting logs– Audit logs– Debugging logs– Just-so logs
• Can lead to wealth of information
![Page 52: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/52.jpg)
NASA Ames iPSC/860 log42050 jobs from Oct-Dec 1993 user job nodes runtime date time
user4 cmd8 32 70 11/10/93 10:13:17
user4 cmd8 32 70 11/10/93 10:19:30
user42 nqs450 32 3300 11/10/93 10:22:07
user41 cmd342 4 54 11/10/93 10:22:37
sysadmin pwd 1 6 11/10/93 10:22:42
user4 cmd8 32 60 11/10/93 10:25:42
sysadmin pwd 1 3 11/10/93 10:30:43
user41 cmd342 4 126 11/10/93 10:31:32 Feitelson & Nitzberg, JSSPP 1995
![Page 53: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/53.jpg)
Distribution of Job Sizes
![Page 54: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/54.jpg)
Distribution of Job Sizes
![Page 55: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/55.jpg)
Distribution of Resource Use
![Page 56: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/56.jpg)
Distribution of Resource Use
![Page 57: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/57.jpg)
Degree of Multiprogramming
![Page 58: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/58.jpg)
System Utilization
![Page 59: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/59.jpg)
Job Arrivals
![Page 60: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/60.jpg)
Arriving Job Sizes
![Page 61: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/61.jpg)
Distribution of Interarrival Times
![Page 62: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/62.jpg)
Distribution of Runtimes
![Page 63: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/63.jpg)
User Activity
![Page 64: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/64.jpg)
Repeated Execution
![Page 65: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/65.jpg)
Application Moldability
![Page 66: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/66.jpg)
Distribution of Run Lengths
![Page 67: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/67.jpg)
Predictability in Repeated Runs
![Page 68: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/68.jpg)
Recurring Findings
• Many small and serial jobs
• Many power-of-two jobs
• Weak correlation of job size and duration
• Job runtimes are bounded but have CV>1
• Inaccurate user runtime estimates
• Non-stationary arrivals (daily/weekly cycle)
• Power-law user activity, run lengths
![Page 69: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/69.jpg)
Instrumentation
• Passive: snoop without interfering
• Active: modify the system– Collecting the data interferes with system
behavior– Saving or downloading the data causes
additional interference– Partial solution: model the interference
![Page 70: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/70.jpg)
Data Sanitation
• Strange things happen
• Leaving them in is “safe” and “faithful” to the real data
• But it risks situations in which a non-representative situation dominates the evaluation results
![Page 71: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/71.jpg)
Arrivals to SDSC SP2
![Page 72: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/72.jpg)
Arrivals to LANL CM-5
![Page 73: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/73.jpg)
Arrivals to CTC SP2
![Page 74: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/74.jpg)
Arrivals to SDSC Paragon
What are they doing at 3:30
AM?
![Page 75: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/75.jpg)
3:30 AM
• Nearly every day, a set of 16 jobs are run by the same user
• Most probably the same set, as they typically have a similar pattern of runtimes
• Most probably these are administrative jobs that are executed automatically
![Page 76: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/76.jpg)
Arrivals to CTC SP2
![Page 77: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/77.jpg)
Arrivals to SDSC SP2
![Page 78: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/78.jpg)
Arrivals to LANL CM-5
![Page 79: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/79.jpg)
Arrivals to SDSC Paragon
![Page 80: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/80.jpg)
Are These Outliers?
• These large activity outbreaks are easily distinguished from normal activity
• They last for several days to a few weeks
• They appear at intervals of several months to more than a year
• They are each caused by a single user!– Therefore easy to remove
![Page 81: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/81.jpg)
![Page 82: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/82.jpg)
Two Aspects
• In workload modeling, should you include this in the model?– In a general model, probably not– Conduct separate evaluation for special
conditions (e.g. DOS attack)
• In evaluations using raw workload data, there is a danger of bias due to unknown special circumstances
![Page 83: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/83.jpg)
Automation
• The idea:– Cluster daily data in based on various
workload attributes– Remove days that appear alone in a cluster– Repeat
• The problem:– Strange behavior often spans multiple days
n
Cirne &Berman, Wkshp Workload Charact. 2001
![Page 84: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/84.jpg)
Workload Modeling
![Page 85: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/85.jpg)
Statistical Modeling
• Identify attributes of the workload
• Create empirical distribution of each attribute
• Fit empirical distribution to create model
• Synthetic workload is created by sampling from the model distributions
![Page 86: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/86.jpg)
Fitting by Moments
• Calculate model parameters to fit moments of empirical data
• Problem: does not fit the shape of the distribution
![Page 87: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/87.jpg)
Jann et al, JSSPP 1997
![Page 88: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/88.jpg)
Fitting by Moments
• Calculate model parameters to fit moments of empirical data
• Problem: does not fit the shape of the distribution
• Problem: very sensitive to extreme data values
![Page 89: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/89.jpg)
Effect of Extreme Runtime Values
Change when top records omitted
omit mean CV
0.01% -2.1% -29%
0.02% -3.0% -35%
0.04% -3.7% -39%
0.08% -4.6% -39%
0.16% -5.7% -42%
0.31% -7.1% -42%Downey & Feitelson, PER 1999
![Page 90: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/90.jpg)
Alternative: Fit to Shape
• Maximum likelihood: what distribution parameters were most likely to lead to the given observations– Needs initial guess of functional form
• Phase type distributions– Construct the desired shape
• Goodness of fit– Kolmogorov-Smirnov: difference in CDFs– Anderson-Darling: added emphasis on tail– May need to sample observations
![Page 91: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/91.jpg)
Correlations
• Correlation can be measured by the correlation coefficient
• It can be modeled by a joint distribution function
• Both may not be very useful
![Page 92: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/92.jpg)
![Page 93: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/93.jpg)
Correlation Coefficient
system CC
CTC SP2 -0.029
KTH SP2 0.011
SDSC SP2 0.145
LANL CM-5 0.211
SDSCParagon 0.305
Gives low results for correlation of runtime and size in parallel systems
22yyxx
yyxx
ii
ii
![Page 94: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/94.jpg)
Distributions
A restricted version of a joint distribution
![Page 95: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/95.jpg)
Modeling Correlation
• Divide range of one attribute into sub-ranges
• Create a separate model of other attribute for each sub-range
• Models can be independent, or model parameter can depend on sub-range
![Page 96: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/96.jpg)
Stationarity
• Problem of daily/weekly activity cycle– Not important if unit of activity is very small
(network packet)– Very meaningful if unit of work is long
(parallel job)
![Page 97: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/97.jpg)
How to Modify the Load
• Multiply interarrivals or runtimes by a factor– Changes the effective length of the day
• Multiply machine size by a factor– Modifies packing properties
• Add users
![Page 98: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/98.jpg)
Stationarity
• Problem of daily/weekly activity cycle– Not important if unit of activity is very small
(network packet)– Very meaningful if unit of work is long
(parallel job)
• Problem of new/old system– Immature workload– Leftover workload
![Page 99: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/99.jpg)
Heavy Tails
![Page 100: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/100.jpg)
Tail Types
When a distribution has mean m, what is the distribution of samples that are larger than x?
• Light: expected to be smaller than x+m
• Memoryless: expected to be x+m
• Heavy: expected to be larger than x+m
![Page 101: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/101.jpg)
Formal Definition
Tail decays according to a power law
Test: log-log complementary distribution
20Pr axxXxF a
xaxF log)(log
![Page 102: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/102.jpg)
Consequences
• Large deviations from the mean are realistic
• Mass disparity– small fraction of samples responsible for large
part of total mass– Most samples together account for negligible
part of mass
Crovella, JSSPP 2001
![Page 103: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/103.jpg)
Unix File Sizes Survey, 1993
![Page 104: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/104.jpg)
Unix File Sizes LLCD
![Page 105: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/105.jpg)
Consequences
• Large deviations from the mean are realistic
• Mass disparity– small fraction of samples responsible for large
part of total mass– Most samples together account for negligible
part of mass
• Infinite moments– For mean is undefined– For variance is undefined
1a2a
Crovella, JSSPP 2001
![Page 106: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/106.jpg)
Pareto Distribution
With parameter the density is proportional to
The expectation is then
i.e. it grows with the number of samples
1a2x
xcdxx
cxxE ln1
][2
![Page 107: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/107.jpg)
Pareto Samples
![Page 108: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/108.jpg)
Pareto Samples
![Page 109: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/109.jpg)
Pareto Samples
![Page 110: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/110.jpg)
Effect of Samples from Tail
• In simulation:– A single sample may dominate results– Example: response times of processes
• In analysis:– Average long-term behavior may never happen
in practice
![Page 111: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/111.jpg)
Real Life
• Data samples are necessarily bounded
• The question is how to generalize to the model distribution– Arbitrary truncation– Lognormal or phase-type distributions– Something in between
![Page 112: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/112.jpg)
Solution 1: Truncation
• Postulate an upper bound on the distribution
• Question: where to put the upper bound
• Probably OK for qualitative analysis
• May be problematic for quantitative simulations
![Page 113: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/113.jpg)
Solution 2: Model the Sample
• Approximate the empirical distribution using a mixture of exponentials (e.g. phase-type distributions)
• In particular, exponential decay beyond highest sample
• In some cases, a lognormal distribution provides a good fit
• Good for mathematical analysis
![Page 114: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/114.jpg)
Solution 3: Dynamic
• Place an upper bound on the distribution
• Location of bound depends on total number of samples required
• Example:
Note: does not change during simulation
NFB 211
![Page 115: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/115.jpg)
Self Similarity
![Page 116: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/116.jpg)
The Phenomenon
• The whole has the same structure as certain parts
• Example: fractals
![Page 117: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/117.jpg)
![Page 118: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/118.jpg)
The Phenomenon
• The whole has the same structure as certain parts
• Example: fractals
• In workloads: burstiness at many different time scales
Note: relates to a time series
![Page 119: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/119.jpg)
Job Arrivals to SDSC Paragon
![Page 120: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/120.jpg)
Process Arrivals to SDSC Paragon
![Page 121: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/121.jpg)
Long-Range Correlation
• A burst of activity implies that values in the time series are correlated
• A burst covering a large time frame implies correlation over a long range
• This is contrary to assumptions about the independence of samples
![Page 122: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/122.jpg)
Aggregation
• Replace each subsequence of m consecutive values by their mean
• If self-similar, the new series will have statistical properties that are similar to the original (i.e. bursty)
• If independent, will tend to average out
![Page 123: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/123.jpg)
Poisson Arrivals
![Page 124: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/124.jpg)
Tests
• Essentially based on the burstiness-retaining nature of aggregation
• Rescaled range (R/s) metric: the range (sum) of n samples as a function of n
![Page 125: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/125.jpg)
R/s Metric
![Page 126: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/126.jpg)
Tests
• Essentially based on the burstiness-retaining nature of aggregation
• Rescaled range (R/s) metric: the range (sum) of n samples as a function of n
• Variance-time metric: the variance of an aggregated time series as a function of the aggregation level
![Page 127: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/127.jpg)
Variance Time Metric
![Page 128: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/128.jpg)
Modeling Self Similarity
• Generate workload by an on-off process– During on period, generate work at steady pace– During off period to nothing
• On and off period lengths are heavy tailed
• Multiplex many such sources
• Leads to long-range correlation
![Page 129: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/129.jpg)
Research Areas
![Page 130: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/130.jpg)
Effect of Users
• Workload is generated by users
• Human users do not behave like a random sampling process– Feedback based on system performance– Repetitive working patterns
![Page 131: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/131.jpg)
Feedback
• User population is finite• Users back off when performance is
inadequate
Negative feedbackBetter system stability
• Need to explicitly model this behavior
![Page 132: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/132.jpg)
Locality of Sampling
• Users display different levels of activity at different times
• At any given time, only a small subset of users is active
![Page 133: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/133.jpg)
Active Users
![Page 134: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/134.jpg)
Locality of Sampling
• Users display different levels of activity at different times
• At any given time, only a small subset of users is active
• These users repeatedly do the same thing
• Workload observed by system is not a random sample from long-term distribution
![Page 135: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/135.jpg)
SDSC Paragon Data
![Page 136: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/136.jpg)
SDSC Paragon Data
![Page 137: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/137.jpg)
Growing Variability
![Page 138: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/138.jpg)
SDSC Paragon Data
![Page 139: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/139.jpg)
SDSC Paragon Data
![Page 140: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/140.jpg)
Locality of Sampling
The questions:
• How does this effect the results of performance evaluation?
• Can this be exploited by the system, e.g. by a scheduler?
![Page 141: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/141.jpg)
Hierarchical Workload Models
• Model of user population– Modify load by adding/deleting users
• Model of a single user’s activity– Built-in self similarity using heavy-tailed on/off
times
• Model of application behavior and internal structure– Capture interaction with system attributes
![Page 142: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/142.jpg)
A Small Problem
• We don’t have data for these models
• Especially for user behavior such as feedback– Need interaction with cognitive scientists
• And for distribution of application types and their parameters– Need detailed instrumentation
![Page 143: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/143.jpg)
Final Words…
![Page 144: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/144.jpg)
We like to think that we design systems based on solid foundations…
![Page 145: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/145.jpg)
But beware:
the foundations might be unbased assumptions!
![Page 146: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/146.jpg)
We should have more “science” in computer science:
• Collect data rather than make assumptions
• Run experiments under different conditions
• Make measurements and observations
• Make predictions and verify them
• Share data and programs to promote good
practices and ensure comparability
Computer Systems are Complex
![Page 147: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/147.jpg)
Advice from the Experts
“Science if built of facts as a house if built of stones. But a collection of facts is no more a science than a heap of stones is a house”
-- Henri Poincaré
![Page 148: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/148.jpg)
Advice from the Experts
“Science if built of facts as a house if built of stones. But a collection of facts is no more a science than a heap of stones is a house”
-- Henri Poincaré
“Everything should be made as simple as possible, but not simpler”
-- Albert Einstein
![Page 149: Workload Modeling and its Effect on Performance Evaluation Dror Feitelson Hebrew University](https://reader038.vdocuments.site/reader038/viewer/2022110322/56649d5a5503460f94a3af1c/html5/thumbnails/149.jpg)
Acknowledgements
• Students: Ahuva Mu’alem, David Talby,
Uri Lublin
• Larry Rudolph / MIT
• Data in Parallel Workloads Archive– Joefon Jann / IBM
– Allen Downey / Welselley
– CTC SP2 log / Steven Hotovy
– SDSC Paragon log / Reagan Moore
– SDSC SP2 log / Victor Hazelwood
– LANL CM-5 log / Curt Canada
– NASA iPSC/860 log / Bill Nitzberg