SchedMark:                            
Evaluating Scheduler Performance

Eitan Frachtenberg

Modeling, Algorithms, and Informatics (CCS-3) Group
Computer and Computational Sciences (CCS) Division
      

A Plethora of Schedulers

Which Scheduler is Better?

• Better for what? movies? games? computation? parallel processing?
• Better where? uniprocessors? multiprocessors? clusters?
• Better how? response time? throughput? makespan? utilization?

The confusion will only grow as commodity architectures and workloads diversify upwards (faster), downwards (portables), and sideways (parallel)

Java phones, desktops, servers, SMTs, SMPs, CMPs, clusters...

SchedMark: Standardizing Scheduler Performance

• A benchmark suite to evaluate the impact of different schedulers
• No one-number-fits-all: context-sensitive metrics
• Incoporates a range of workloads and application classes


• Interactive, continuous-media, and batch applications


• Sequential and parallel programs


• desktop and server characteristics


• Static and dynamic workloads




• Results will allow an apples-to-apples comparison of schedulers and workloads

Challenges and solutions

• Portability: different schedulers run on different operating systems and architectures
• Workload is built around synthetic applications that emulate real-world scheduling requirements
• Minimal OS depedence using a portable multithreading library
• Applications are calibrated to take the same resources across architectures
• Parallel applications grow to fit number of processors
• Validity: different applications care about different metrics
• Applications measure their own metrics of interest (e.g., dropped frames, parallel speedup)
• Only relative metrics are used to facilitate comparison (e.g., slowdown instead of response time)
• Workload: feedback effects and interactions between applications in a mixed workload produce a different picture than applications in isolation
• Suite comprises static and dyamic workloads
• Stress the scheduler with increasing load

The Advantages

 
• Allows direct evaluation of scheduler effect on applications (current benchmarks focus on kernel metrics, e.g., context-switch latency).
• Provides insights such as "Scheduler X prefers interactive application as load increases" and "Scheduler Y doesn't handle parallel programs very well"
• Designed for growth to future architectures and workloads

Where We Are Now

 
• Early stages of implementation
• Characterization of workloads and application scheduling properties

For more information, contact: eitanf@lanl.gov
Or search for "SchedMark"