Model-Free Reinforcement Learning in Microgrid Control: A Review

Gurbuz F. B., Karaki A., Karaki A., DEMİRBAŞ Ş., BAYHAN S.

IEEE Access, cilt.13, ss.161762-161778, 2025 (SCI-Expanded) identifier

  • Yayın Türü: Makale / Derleme
  • Cilt numarası: 13
  • Basım Tarihi: 2025
  • Doi Numarası: 10.1109/access.2025.3609317
  • Dergi Adı: IEEE Access
  • Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Compendex, INSPEC, Directory of Open Access Journals
  • Sayfa Sayıları: ss.161762-161778
  • Anahtar Kelimeler: Hierarchical control, microgrids, model-free, Q-learning, reinforcement learning
  • Gazi Üniversitesi Adresli: Evet

Özet

The global shift toward distributed energy resources (DERs) has accelerated the deployment of microgrids (MGs), introducing unprecedented control challenges that traditional strategies often struggle to address. Model-free reinforcement learning (MFRL) has emerged as a promising paradigm for adaptive, intelligent control without the need for explicit system modeling. This paper presents a comprehensive review of MFRL applications in MG control, proposing a systematic taxonomy that classifies existing approaches by control hierarchy, architectural configuration, operational modes, and action spaces. We analyze critical design considerations - including reward function shaping, exploration strategies, and computational requirements - that influence practical deployment. Furthermore, we systematically evaluate key MFRL algorithms and map their suitability across primary, secondary, and tertiary control levels. By examining recent applications, we highlight that MFRL has reached considerable maturity across all control hierarchies, revealing clear trends: continuous-action methods excel in real-time primary control, distributed schemes enhance scalability in secondary coordination, and multi-agent frameworks enable complex tertiary-level optimization. Finally, the review identifies persistent implementation challenges and offers practical guidance for algorithm selection and deployment strategies in modern MG systems.This review aims to serve both researchers and practitioners seeking to deploy MFRL in modern MG systems.