Dynamic Power Management Using Model-Free Reinforcement Learning and Bayesian Classification
To cope with the variations and uncertainties that emanate from hardware and application characteristics, dynamic power management (DPM) frameworks must be able to learn about the system inputs and environment and adjust the power management policy on the fly. We present an online adaptive DPM technique based on model-free reinforcement learning (RL), which is commonly used to control stochastic dynamical systems. In particular, we employ temporal difference learning for semi-Markov decision process (SMDP) for the model-free RL. In addition a novel workload predictor based on an online Bayes classifier is presented to provide effective estimates of the workload states for the RL algorithm. In this DPM framework, power and latency tradeoffs can be precisely controlled based on a user-defined parameter. Experiments show that amount of average power saving (without any increase in the latency) is up to 16.7% compared to a reference expert-based approach. Alternatively, the per-request latency reduction without any power consumption increase is up to 28.6% compared to the expert-based approach.
We have further extended the RL-based DPM framework to (i) hierarchical DPM framework which jointly perform component-level DPM with a CPU scheduler, and (ii) DPM of a power-managed system with battery-based power supply or hybrid (battery + supercapacitor) power supply.Related work:
- Y. Wang, Q. Xie, A. Ammari, and M. Pedram. “Deriving a near-optimal power management policy using model-free reinforcement learning and Bayesian classification,” Proc. of the 48th Design Automation Conf.,Jun. 2011.
- Y. Wang, M. Triki, X. Lin, A. Ammari, and M. Pedram. “Hierarchical dynamic power management using model-free reinforcement learning,” Proc. of Int’l Symposium on Quality Electronic Design,Mar. 2013.
Accurate Modeling of the Delay and Energy Overhead of Dynamic Voltage and Frequency Scaling in Modern Microprocessors
Dynamic voltage and frequency scaling (DVFS) has been studied for well over a decade. The state-of-the-art DVFS technologies and architectures are advanced enough such that they are employed in most commercial systems today. Nevertheless, existing DVFS transition overhead models suffer from significant inaccuracies, for example, by correctly accounting for the effect of DC-DC converters, frequency synthesizers, and voltage and frequency change policies on energy losses incurred during mode transitions. Incorrect and/or inaccurate DVFS transition overhead models prevent one from determining the precise break-even time and thus forfeit some of the energy saving that is ideally achievable. Through detailed analysis of modern DVFS setups and voltage and frequency change policies provided by commercial vendors, we introduce accurate DVFS transition overhead models for both energy consumption and delay. In particular, we identify new contributors to the DVFS transition overhead including the underclocking-related losses in a DVFS-enabled microprocessor, additional inductor IR losses, and power losses due to discontinuous-mode DC- DC conversion. We report the transition overheads for three representative processors: Intel Core2Duo E6850, ARM Cortex-A8, and TI MSP430. Finally, we present a compact, yet accurate, DVFS transition overhead macro model for use by high-level DVFS schedulers.Related work:
- S. Park, J. Park, D. Shin, Y. Wang, Q. Xie, N. Chang, and M. Pedram “Accurate Modeling of the Delay and Energy Overhead of Dynamic Voltage and Frequency Scaling in Modern Microprocessors,” IEEE Trans. on Computer Aided Design, 2013.