## wpiea2023040-print-pdf

## Source details

**Canonical URL:** [wpiea2023040-print-pdf](https://www.imf.org/-/media/files/publications/wp/2023/english/wpiea2023040-print-pdf.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/wp/2023/english/wpiea2023040-print-pdf.pdf.md)
- [Structured JSON version](/-/media/files/publications/wp/2023/english/wpiea2023040-print-pdf.pdf.json)

---

### INTRODUCTION
- Purpose and scope
  - Build a relatively simple and extendable macroeconomic model based on reinforcement learning (RL) that can generate realistic macroeconomic dynamics comparable to models under the rational expectations assumption while not imposing perfect foresight on economic agents.
  - Use the resulting model as a prototype for future extensions in policy experiments or to customize it to better match the conditions, shocks or data of a particular or global economy.
  - Implement the deep deterministic policy gradient (DDPG) algorithm in a real business cycle (RBC) macroeconomic model to explore the feasibility of combining deep RL with standard macroeconomic models.
- Motivation: AI, ML, and RL for macroeconomic modeling
  - Macroeconomic modeling: construct models to describe behavior of a macroeconomic system for prediction, understanding variable relationships, or simulation.
  - Artificial intelligence (AI): design and development of intelligent computer systems capable of reasoning, learning, problem-solving, and acting autonomously.
  - Advantages of RL highlighted:
    - Better suited than previous AI models for imitating human behavior.
    - Require much less exogenous data because agents generate their own experience through interaction.
    - Can be efficient in computational resources in specific settings or algorithms.
- Why DDPG was chosen
  - Applicable to continuous action space problems, crucial for modeling macroeconomic variables.
  - Can handle high-dimensional state and action spaces typical in macroeconomic models.
  - Separation of policy and value functions allows independent analysis of components during learning.
  - Applicable to non-stationary problems and very long-time horizons.
  - Can be applied to partially observable Markov decision process (POMDP) problems or limited-observation-window settings in specific settings.
  - Known to perform well on a variety of challenging RL problems but can be unstable if not properly tuned.
- Key findings (as presented)
  - The RL-augmented RBC model performs similar to the RBC model under the rational expectations assumption once the learning representative agent has learned over many simulation periods.
  - The learning agent initially does not understand the economic structure, preferences, or dynamics; convergence requires significant training because the agent must generate its own experience.
  - To simulate realistic household behaviors that match empirical learning periods, further work is needed to calibrate parameters or transfer past experience to the learning agent.
- Limitations and caveats
  - Model structure is rudimentary but extendible; interpret results in that context.
  - Restricted scope: business cycle variations propagated only through an exogenous productivity shock; true technology shock magnitude likely smaller empirically than RBC models predict; unemployment explained via intertemporal substitution between labor and leisure.
  - RBC models used because they are core components of DSGE models, scalable, easy to build on, and well studied.
- Intended contribution and outlook
  - Encourage further research on AI and deep RL for macroeconomic problems.
  - Provide a base and extension for more advanced applications exploring deep RL for macroeconomic policy analysis at the Fund.
- Organization of the paper
  - Section I: literature review of AI and RL/deep RL applications in macroeconomic policy.
  - Section II: description of the RBC model.
  - Section III: DRL algorithm, environment, AI experiments, results, and issues during learning.
  - Section IV: conclusion.

### Reinforcement learning (RL) and Deep Reinforcement Learning (DRL)
- Overview
  - Reinforcement learning (RL): not given a set of training data; machine is given rules or objectives and learns via repeated interactions with the environment.
  - Main applications: control, robotics, optimization and gaming.
  - Deep reinforcement learning (DRL): combines RL with deep learning (DL) using a DNN to represent the RL agent.
- Literature and economics applications (selected points)
  - Surveys: Athey (2018), Cameron (2019), Nosratabadi et al., (2020), Hull (2021).
  - RL/DRL applications have focused mainly on microeconomic settings: dynamic pricing in electricity markets, auction theory, portfolio management and asset pricing.
  - Examples of research findings:
    - Feng et al. (2018): deep learning for automated design of optimal auctions; discover high-revenue auctions for multi-unit auctions with private budgets.
    - Zheng et al. (2020): RL for agent and social planner actions; AI-driven tax policies enhance equality-productivity trade-off over baseline.
    - Dütting et al. (2021): model auction as neural network; recover known analytical solutions and propose new mechanisms.
  - Macroeconomics literature:
    - DRL to find optimal policies or policy response functions (Hinterlang and Tänzer (2022); Covarrubias (2022)).
    - General equilibrium model solving with DRL (Curry et al (2022)).
    - Learnability of rational expectations solutions with multiple equilibria (Chen et al (2021)).
    - Modeling rationality and bounded rationality (Hill et al. (2021); Shi (2021)).
- Contribution of current paper
  - Tests a DRL algorithm in a macroeconomic model by implementing a representative DRL agent in an RBC model as a building block for New Keynesian DSGE models.

### II. A Real Business Cycle (RBC) Model — structure and optimization
- Baseline model structure
  - Identical and infinitely lived households and firms.
  - Business cycle fluctuations generated by real (technology) shocks.
  - Households own firms and rent out capital; firms issue debt (bonds) and equity (dividend).
- Household problem
  - Maximize expected utility: E0 ∑_{t=0}^{∞} β^{t} u(c_{t},1−h_{t}).
  - Budget and transition constraints:
    - x_{t} + c_{t} + b_{t+1} ≤ w_{t} h_{t} + r_{t} k_{t} + R_{t} b_{t} + Π_{t}
    - k_{t+1} ≤ (1−δ) k_{t} + x_{t}
    - k_{t} ≥ 0
    - k_{0} is given; transversality condition holds.
  - Variables: c_{t}, x_{t}, b_{t+1}, w_{t}, h_{t}, r_{t}, k_{t}, R_{t}, Π_{t}; h_{t} ∈ [0,1].
  - Consumer choices at time t: (x_{t} or k_{t+1}, c_{t}, b_{t+1}, h_{t}), given time t information and R_{t+1}.
- Optimization under Rational Expectations (representative household)
  - Lagrangian L = E0 ∑_{t=0}^{∞} β^{t} { u(c_{t},h_{t}) + λ_{t}( w_{t} h_{t} + r_{t} k_{t} + R_{t} b_{t} + Π_{t} − c_{t} − k_{t+1} + (1−δ) k_{t} − b_{t+1} ) }.
  - First order conditions:
    - u_{c}(c_{t},h_{t}) = λ_{t}
    - u_{h}(c_{t},h_{t}) = λ_{t} w_{t}
    - λ_{t} = β E_{t} λ_{t+1} { r_{t+1} + (1−δ) }
    - λ_{t} = β E_{t} λ_{t+1} ( R_{t+1} )
  - Combining conditions:
    - u_{c}(c_{t},h_{t}) = β E_{t} u_{c}(c_{t+1},h_{t+1}) ( r_{t+1} + 1 − δ )
    - u_{c}(c_{t},h_{t}) = β E_{t} u_{c}(c_{t+1},h_{t+1}) R_{t+1}
    - u_{h}(c_{t},h_{t}) = w_{t} u_{h}(c_{t},h_{t})
- Firms
  - Profit maximization: max_{K_{t},H_{t}} e^{z_{t}} F(K_{t},H_{t}) − w_{t} H_{t} − r_{t} K_{t}.
  - Production: neoclassical F (e.g., Cobb-Douglas); z_{t} follows AR(1): z_{t} = ρ z_{t−1} + ε_{t}, ε_{t} white noise.
  - First order conditions:
    - w_{t} = e^{z_{t}} F_{K}(K_{t},H_{t})
    - r_{t} = e^{z_{t}} F_{H}(K_{t},H_{t})
  - Firm debt indeterminate in this setup.

- Functional forms and baseline parameters (preserve values exactly)
  - Utility function parameters:
    - χ = 1 (logarithmic utility)
    - α = 0.64
    - u(c_{t},h_{t}) = ( c_{t}^{1−α} (1−h_{t})^{α} )^{1−χ} / (1−χ)
    - u(c_{t},h_{t}) = (1−α) ln c_{t} + α ln (1−h_{t})
  - Production function:
    - θ = 0.4
    - F(K,H) = K^{θ} H^{1−θ}
  - Discount rate – β: 0.99
  - Autoregressive parameter - ρ: 0.95
  - Standard deviation of ε_{t}, σ_{ε}: 0.007
  - Capital depreciation – δ: 0.012

- Deterministic steady state (assumptions and computed steady state values)
  - Assume z_{t} = 0, k_{t+1} = k_{t} = k^{*}, c_{t+1} = c_{t} = c^{*}.
  - Steady state conditions:
    - 1/β − 1 + δ = θ ( k^{*} / h^{*} )^{θ−1} ↔ k^{*} / h^{*} = ( (1/β − 1 + δ) / θ )^{1/(θ−1)} (=124.7)
    - y^{*} = ( k^{*} / h^{*} )^{θ} h^{*} ↔ y^{*} / h^{*} = ( k^{*} / h^{*} )^{θ} (=6.89)
    - i^{*} = δ k^{*} = δ ( k^{*} / h^{*} ) h^{*} ↔ i^{*} / h^{*} = δ ( k^{*} / h^{*} ) (=1.5)
  - Accounting identity for consumption:
    - c^{*} = y^{*} − i^{*} ↔ c^{*} / h^{*} = y^{*} / h^{*} − i^{*} / h^{*} (=5.39)
  - Ratio:
    - i^{*} / y^{*} = 1 − ( c^{*} / h^{*}  y^{*} / h^{*} ) = 0.28
  - Wage rate and capital rental rate at steady state:
    - w^{*} = (1−θ) ( k^{*} / h^{*} )^{θ}
    - r^{*} = θ ( k^{*} / h^{*} )^{θ−1} − δ

### III. AI Experiments — simulation design and DDPG implementation
- Simulation aims
  - Demonstrate learning behaviors of a representative RL agent and resulting economic dynamics.
  - Compare agent’s early learning decisions with decisions after many simulation periods.
  - Compare learning agent decisions with rational expectations agent in same environment.
  - Plot macroeconomic series to show qualitative similarity between RBC with RL agent and conventional RBC model.
- Environments
  - Two environments:
    - A dynamic and deterministic environment without shocks.
    - A stochastic environment with technology shocks.
  - Purpose: compare RL and rational expectations behavior in deterministic setup, then analyze RL performance in stochastic setup.
- DDPG implementation and algorithm mechanics
  - DDPG introduced by Lillicrap et al (2015), "Continuous Control with Deep Reinforcement Learning".
  - DDPG design:
    - Addresses continuous action spaces by approximating the action-value function with a DNN.
    - Extension of DPG to continuous action spaces, using DQN concepts to estimate the Q-function.
    - Actor-Critic structure: Actor = policy, Critic = Q-function.
    - Actor trained via Policy Gradient; Critic trained via Temporal Difference Learning.
    - Uses replay memory to break sample correlation.
    - Uses target networks for Actor and Critic updated via Polyak averaging.
    - Off-policy nature allows learning from suboptimal actions.
  - Advantages and disadvantages noted:
    - Advantages: can learn from suboptimal/wrong actions; applied successfully to robotic arms, drones, walking robots.
    - Disadvantages: can take long to converge due to exploration; can be unstable due to constant updates causing oscillations.
  - Experience and updates:
    - Agent collects experience per period: state, action, reward, next state.
    - Experience updates policy and value functions approximated by ANNs.
    - Policy network updates to maximize value function predictions; value function updates to minimize temporal difference errors.
  - Parameter tuning categories:
    1. Hyperparameters (exploration parameters): control learning rate, exploration rate, and related factors.
    2. Neural network parameters: control weights and biases of neural networks.
    3. Replay parameters (Memory parameters): size of replay buffer and batch size for training.

### Algorithm-related parameters and ANN architecture (baseline parameters from Table 2)
- Training parameters
  - Number of episodes: 200
  - Steps per episode: 2000
  - Updates per step: 5
  - Batch size: 128
- Network architecture
  - Hidden layer size: 16
  - Number of hidden layers: 2
  - Activation functions:
    - Actor: ReLU for each linear layer and Tanh for output layer
    - Critic: ReLU for all
- Learning rates
  - Actor: 1e-4
  - Critic: 1e-3
- Exploration
  - From 1.0 to 0.3, diminish by episode
- Architecture and modeling notes
  - Feedforward ANN with two hidden layers, each hidden layer has 16 nodes.
  - Described as simple compared to architectures for tasks such as the game of Go.
- Parameter roles and tuning guidance
  - Number of episodes: too few → insufficient data; too many → risk of overfitting.
  - Steps per episode: more timesteps indicate more difficult tasks or less efficient learning.
  - Updates per step: more updates speed learning but can cause instability or divergence if excessive.
  - Batch size: larger batch sizes converge faster but may increase overfitting and instability; smaller batch sizes yield finer updates but slower convergence.
  - Hidden layer size and depth: control network capacity; trade-off between underfitting and overfitting.
  - Learning rates govern update speed and how new information overrides old.
  - Activation functions: ReLU commonly used; Tanh for actor output to map to (−1,1).
  - Exploration schedule controls randomness in action selection.

- RL setup for the RBC model (state, actions, reward, transitions)
  - State (Period t state variables): k_t, z_t, k_{t-1}, h_{t-1}, λ_{t-1}^b, R_{t-1}
  - Action and action space:
    - h_t ∈ (0,1)
    - λ_t^i ∈ (0,1)
    - λ_t^b ∈ [1,2]
  - Reward function: Logarithmic utility: u(c_t,h_t) = (1−α) ln c_t + α ln (1−h_t)
  - Transition equations:
    - K_t = k_t
    - H_t = h_t
    - y_t = e^{z_t} K_t^{θ} H_t^{1−θ}
    - i_t = λ_t^i y_t
    - r_t = θ e^{z_t} K_t^{θ−1} H_t^{1−θ}
    - R_t = r_t + 1 − δ
    - Δ_{t+1}^b = b_{t+1}/b_t − R_t = λ_t^b − R_t
    - c_t = (y_t − i_t) − a Δ_{t+1}^b
    - a > 0 is a constant and assumed to be a = 10

### Experiment I: deterministic environment — setup and results
- Deterministic specification
  - e^{z_t} = 1 and z_t = 0 (no productivity shock)
  - Representative household is the learning agent; firms are rational.
  - Equilibrium condition: R_t = r_t + 1 − δ
- Key empirical observations
  - Training vs testing:
    - Training: exploratory behavior; policy and value functions updated.
    - Testing: learning halted; policy and value functions fixed; no exploration noise.
  - Labor hour learning:
    - Early training (first 10,000 steps): actions more random.
    - After 190 episodes (last 10,000 steps): labor choices concentrate toward deterministic steady state value of 0.33.
    - Labor hour series over 500 periods: testing series close to optimal deterministic steady state; training series more variable.
  - Distance metric and learning progress:
    - Distance per episode defined as d_t = (x̅_t − x_ss)^2 where x̅_t is the mean of variable in episode t and x_ss is steady state.
    - Labor hour and consumption per labor hour distances show gradual decline over episodes, but decline is not monotonic.
    - Non-monotonicity sources: changes in learning parameters (exploration level, neural network learning rate), aggregation effects; learning slow given uncalibrated parameters.

### Experiment II: stochastic environment — setup and results
- Stochastic specification
  - Productivity shock: z_t = ρ z_{t−1} + ε_t
  - Household is learning agent; firm is rational.
- Key empirical observations
  - Productivity shock series plotted for 500 simulation periods (Figure referenced).
  - After five episodes of learning, simulated consumption, output, investment per unit of labor and labor hour shown over 100 testing periods (Figure referenced).
  - Consumption per unit of labor and output per unit of labor move in same direction; investment today raises output tomorrow.
  - Labor hour learning: early choices (first 10,000 steps) more dispersed; last 10,000 steps concentrate towards optimal choice of 0.3.
  - Distance metric results:
    - Distances to deterministic steady states for consumption, output, investment per unit of labor decline gradually over 200 episodes, but not monotonically.
    - Volatility: distance series can be more volatile in stochastic environment; series after episode 125 appear more volatile than deterministic case.
    - Learning speed: RL agent appears to learn quicker in stochastic environment (small distance observed after episode 125) compared to deterministic environment (small distance only after around episode 165).
    - Interpretation: stochasticity exposes agent to a wider range of states early, aiding faster acquisition of effective policies.

### Issues during learning
- Instability of learning
  - Many DRL algorithms not proven to converge; instability may stem from exploration noise, incorrect learning rates, noisy gradient estimation, or insufficient network depth.
  - DDPG can be unstable even in trivial tasks.
  - Remedies suggested:
    - Use different noise process for exploration (e.g., Ornstein-Uhlenbeck noise instead of Gaussian noise).
    - Use different function approximators (e.g., LSTM instead of fully connected networks).
    - Modify learning process (different learning rates or optimization algorithms).
  - Example instability: after 200 episodes in deterministic environment, agent can become trapped in a suboptimal state and remain there despite further training (learning traps).
  - Frequency of traps unclear and case-dependent; may require restarting learning or hyperparameter/architecture changes.
- Parameter selection challenges
  - No general guidance for algorithm-related parameter selection or ANN architecture choice.
  - Rule of thumb: start with simple ANN architectures and increase complexity if needed.
  - Techniques to search parameters: grid search or evolutionary algorithms.
- Difficulty in reproducing results
  - High randomness in DRL makes reproducibility challenging.
  - To improve reproducibility: use the same random seed and record all hyperparameters and problem-specific details.

### Findings, network choices, and practical recommendations
- Findings and implications
  - In both deterministic and stochastic setups, a representative household RL agent learns consumption-saving and labor-leisure decisions and approximates behavior of a rational expectations agent over time.
  - Deterministic environment: decisions converge toward deterministic steady state (0.33 labor hour) but learning is slower and less explorative.
  - Stochastic environment: agent learns quicker and explores a wider state space early, but behavior and distance metrics can be more volatile.
- Network architecture choices
  - For the relatively simple RBC problem, ANN complexity reduced relative to common DDPG implementations (hidden layer nodes reduced to 16 per hidden layer).
- Practical recommendations
  - Tune Actor and Critic learning rates, soft update parameters, exploration schedule, number of episodes, and steps per episode according to research question.
  - Start with simpler ANNs (two hidden layers with 16 nodes) and increase complexity if model requires it.
  - Monitor instability and learning traps; adjust noise process, function approximator, learning rates, or restart training when needed.
  - Record seeds and hyperparameters to aid reproducibility.

### Annex I. DDPG algorithm
- Summary
  - The DDPG algorithm follows from Lillicrap et al (2019, page5).

*Source: wpiea2023040-print-pdf - INTRODUCTION; Reinforcement learning (RL); II. A Real Business Cycle (RBC) Model; III. AI Experiments; 4. Algorithm parameters; Annex I. DDPG algorithm*

### INTRODUCTION ...........................................................................................................

### INTRODUCTION

### Purpose and scope
- Build a relatively simple and extendable macroeconomic model based on reinforcement learning (RL) that can generate realistic macroeconomic dynamics comparable to models under the rational expectations assumption while not imposing perfect foresight on economic agents.
- Use the resulting model as a prototype for future extensions in policy experiments or to customize it to better match the conditions, shocks or data of a particular or global economy.
- Implement the deep deterministic policy gradient (DDPG) algorithm in a real business cycle (RBC) macroeconomic model to explore the feasibility of combining deep RL with standard macroeconomic models.

### Motivation: AI, ML, and RL for macroeconomic modeling
- Macroeconomic modeling: construct models to describe behavior of a macroeconomic system for prediction, understanding variable relationships, or simulation.
- Artificial intelligence (AI): design and development of intelligent computer systems capable of reasoning, learning, problem-solving, and acting autonomously.
- Reinforcement learning (RL) models are emphasized because:
  - They are better suited than previous AI models for imitating human behavior.
  - They require much less exogenous data to be trained because agents generate their own experience through interaction.
  - They can be efficient in terms of computational resources in specific settings or algorithms.

### Why DDPG was chosen
- DDPG is applicable to continuous action space problems, crucial for modeling macroeconomic variables.
- It can handle high-dimensional state and action spaces typical in macroeconomic models.
- The separation of policy and value functions allows independent analysis of components during learning.
- It can be applied to non-stationary problems common in macroeconomic modeling.
- It is suitable for problems with very long-time horizons, which may be important for macroeconomic modeling.
- It can be applied, in specific settings, to partially observable Markov decision process (POMDP) problems or limited-observation-window settings.
- DDPG has demonstrated good performance across a variety of challenging RL problems, though it is known to be unstable in some settings and can diverge if not properly tuned.

### Key findings from the experiments (as presented)
- The RL-augmented RBC model performs similar to the RBC model under the rational expectations assumption once the learning representative agent has learned over many simulation periods.
- The learning agent initially does not understand the economic structure, preferences, or dynamics; convergence to similar behavior requires significant training periods because the agent must generate its own experience to learn.
- To simulate realistic household behaviors that match empirical learning periods, further work is needed to:
  - Calibrate parameters, or
  - Transfer past experience to the learning agent as a starting point for learning.

### Limitations and caveats
- Model structure is rudimentary but extendible; results must be interpreted in that context.
- Restricted scope of the RBC model:
  - Business cycle variations are propagated only through an exogenous productivity shock.
  - The empirically implied magnitude of true technology shock is likely smaller than what RBC models predict.
  - Unemployment is explained in an overly simplified manner via intertemporal substitution between labor and leisure.
- RBC models are used here because they are core components of DSGE models widely applied in policy institutions and central banks, are scalable, easy to build on, and well studied—facilitating comparison of learning results with existing theory.

### Intended contribution and outlook
- Encourage further research on the application of AI and deep RL for macroeconomic problems.
- Provide a base and extension for more advanced applications that explore the use of deep RL for macroeconomic policy analysis at the Fund.

### Organization of the paper
- Section I: brief literature review of AI and RL/deep RL applications in macroeconomic policy.
- Section II: description of the RBC model.
- Section III: introduction to the DRL algorithm, the environment, AI experiments, results, and issues during learning.
- Section IV: conclusion.

*Source: wpiea2023040-print-pdf - INTRODUCTION*

### 3. Reinforcement learning (RL): It is different from both supervised and unsupervised learning in that it is

### 3. Reinforcement learning (RL): It is different from both supervised and unsupervised learning in that it is

### Overview of RL and Deep Reinforcement Learning (DRL)
- Reinforcement learning (RL) is not given a set of training data; the machine is given a set of rules or objectives and learns to best achieve these objectives through repeated interactions with the environment.
- Main applications of RL: control, robotics, optimization and gaming.
- Deep reinforcement learning (DRL) combines RL with deep learning (DL) by using a DNN to represent the RL agent (Li, 2017).
- DRL is used to solve problems too difficult for simple RL algorithms alone.
- For theory and algorithms in RL/DRL, see Atashbar and Shi (2022a).

### Literature and economics applications
- Surveys providing comprehensive reviews of ML and DL methods and use cases in economics: Athey (2018), Cameron (2019), Nosratabadi et al., (2020), Hull (2021).
- RL and DRL applications in economics are relatively recent and early-stage; literature mainly focuses on microeconomic settings.
- Example applications:
  - Dynamic pricing in electricity markets.
  - Auction theory.
  - Portfolio management and asset pricing.

- Specific research findings:
  - Feng et al. (2018): Model auction rules as a neural network and use deep learning for automated design of optimal auctions; discover new auctions with high revenue for multi-unit auctions with private budgets, including unit-demand bidders.
  - Zheng et al. (2020): Employ RL to examine actions of agents and a social planner in a gather-and-build environment; demonstrate AI-driven tax policies enhance the trade-off between equality and productivity over baseline policies.
  - Dütting et al. (2021): Model an auction as a multi-layer neural network, frame optimal auction design as a constrained learning problem, solve using standard ML pipelines; demonstrate generalization limits, recover known analytical solutions for multi-item settings, and propose new mechanisms where optimal mechanisms are unknown.

- Macroeconomics literature on RL/DRL:
  - DRL algorithms used to find optimal policies or policy response functions (Hinterlang and Tänzer (2022); Covarrubias (2022)).
  - General equilibrium model solving with DRL (Curry et al (2022)).
  - Learnability of rational expectations solutions with multiple equilibria (Chen et al (2021)): RL agent can locally converge to all stable states in a monetary model with numerous equilibria.
  - Modeling rationality and bounded rationality: Hill et al. (2021) solve rational expectations equilibrium models using discrete heterogeneous agents; Shi (2021) studies RL agents' consumption-saving behavior in a stochastic growth setting and effects of exploration levels on convergence.

- Current paper’s contribution:
  - Tests a DRL algorithm in a macroeconomic model by implementing a representative DRL agent in an RBC model, serving as a building block for New Keynesian DSGE models.

### II. A Real Business Cycle (RBC) Model — structure and optimization
- Baseline model:
  - Identical and infinitely lived households and firms.
  - Business cycle fluctuations generated by real (technology) shocks.
  - Households own firms and rent out capital; firms issue debt (bonds) and equity (dividend).

- Households:
  - Household maximizes expected utility:
    - E0 ∑_{t=0}^{∞} β^{t} u(c_{t},1−h_{t})
  - Subject to constraints:
    - x_{t} + c_{t} + b_{t+1} ≤ w_{t} h_{t} + r_{t} k_{t} + R_{t} b_{t} + Π_{t}
    - k_{t+1} ≤ (1−δ) k_{t} + x_{t}
    - k_{t} ≥ 0
    - k_{0} is given and the maximization satisfies the transversality condition.
  - Variable definitions:
    - c_{t} consumption, x_{t} investment, b_{t+1} bond holding, w_{t} hourly wage rate, h_{t} hours worked, r_{t} return on capital, k_{t} capital, R_{t} interest rate on bond holding, Π_{t} dividend payment.
  - h_{t} ∈ [0,1]; consumer receives utility from leisure.
  - Consumer choices at time t: (x_{t} or k_{t+1}, c_{t}, b_{t+1}, h_{t}), given time t information and R_{t+1}.

- Optimization under Rational Expectations (A.1):
  - Representative household’s Lagrangian:
    - L = E0 ∑_{t=0}^{∞} β^{t} { u(c_{t},h_{t}) + λ_{t}( w_{t} h_{t} + r_{t} k_{t} + R_{t} b_{t} + Π_{t} − c_{t} − k_{t+1} + (1−δ) k_{t} − b_{t+1} ) }
  - First order conditions:
    - ∂L/∂c_{t}=0 ↔ u_{c}(c_{t},h_{t}) = λ_{t}
    - ∂L/∂h_{t}=0 ↔ u_{h}(c_{t},h_{t}) = λ_{t} w_{t}
    - ∂L/∂k_{t+1}=0 ↔ λ_{t} = β E_{t} λ_{t+1} { r_{t+1} + (1−δ) }
    - ∂L/∂b_{t+1}=0 ↔ λ_{t} = β E_{t} λ_{t+1} ( R_{t+1} )
  - Combining conditions:
    - u_{c}(c_{t},h_{t}) = β E_{t} u_{c}(c_{t+1},h_{t+1}) ( r_{t+1} + 1 − δ )
    - u_{c}(c_{t},h_{t}) = β E_{t} u_{c}(c_{t+1},h_{t+1}) R_{t+1}
    - u_{h}(c_{t},h_{t}) = w_{t} u_{h}(c_{t},h_{t})

- Firms:
  - Profit maximization:
    - max_{K_{t},H_{t}} e^{z_{t}} F(K_{t},H_{t}) − w_{t} H_{t} − r_{t} K_{t}
  - K_{t} capital input, H_{t} labour input, F neoclassical production function (e.g., Cobb-Douglas), z_{t} follows AR(1):
    - z_{t} = ρ z_{t−1} + ε_{t}
    - ε_{t} sampled from white noise.
  - First order conditions (B.1):
    - w_{t} = e^{z_{t}} F_{K}(K_{t},H_{t})
    - r_{t} = e^{z_{t}} F_{H}(K_{t},H_{t})
  - The debt the firm issues is indeterminate in this setup.

### C. Functional forms and parameters (Table 1 baseline parameters — preserve values exactly)
- Utility function parameters:
  - χ = 1 (logarithmic utility)
  - α = 0.64
  - u(c_{t},h_{t}) = ( c_{t}^{1−α} (1−h_{t})^{α} )^{1−χ} / (1−χ)
  - u(c_{t},h_{t}) = (1−α) ln c_{t} + α ln (1−h_{t})

- Production function:
  - θ = 0.4
  - F(K,H) = K^{θ} H^{1−θ}

- Discount rate – β:
  - 0.99

- Autoregressive parameter - ρ:
  - 0.95

- Standard deviation of ε_{t}, σ_{ε}:
  - 0.007

- Capital depreciation – δ:
  - 0.012

### D. Deterministic steady state (assumptions and computed steady state values)
- Assume z_{t} = 0, k_{t+1} = k_{t} = k^{*}, c_{t+1} = c_{t} = c^{*}.
- Steady state conditions:
  - 1/β − 1 + δ = θ ( k^{*} / h^{*} )^{θ−1} ↔ k^{*} / h^{*} = ( (1/β − 1 + δ) / θ )^{1/(θ−1)} (=124.7)
  - y^{*} = ( k^{*} / h^{*} )^{θ} h^{*} ↔ y^{*} / h^{*} = ( k^{*} / h^{*} )^{θ} (=6.89)
  - i^{*} = δ k^{*} = δ ( k^{*} / h^{*} ) h^{*} ↔ i^{*} / h^{*} = δ ( k^{*} / h^{*} ) (=1.5)
- Accounting identity for consumption:
  - c^{*} = y^{*} − i^{*} ↔ c^{*} / h^{*} = y^{*} / h^{*} − i^{*} / h^{*} (=5.39)
- Ratio:
  - i^{*} / y^{*} = 1 − ( c^{*} / h^{*}  y^{*} / h^{*} ) = 0.28
- Wage rate and capital rental rate at steady state:
  - w^{*} = (1−θ) ( k^{*} / h^{*} )^{θ}
  - r^{*} = θ ( k^{*} / h^{*} )^{θ−1} − δ

### III. AI Experiments — simulation design and DDPG implementation
- Simulation aims:
  - Demonstrate learning behaviors of a representative RL agent and resulting economic dynamics.
  - Compare agent’s early learning decisions (e.g., labor hours) with decisions after many simulation periods.
  - Compare learning agent decisions with rational expectations agent in same environment.
  - Plot macroeconomic series to show qualitative similarity between RBC with RL agent and conventional RBC model.

- Environments:
  - Two environments set up:
    - A dynamic and deterministic environment without shocks.
    - A stochastic environment with technology shocks.
  - Purpose: compare RL and rational expectations behavior in deterministic setup, then analyze RL performance in stochastic setup.

- Implementation details:
  - DDPG algorithm implemented (introduced by Lillicrap et al (2015), "Continuous Control with Deep Reinforcement Learning").
  - DDPG design:
    - Addresses continuous action spaces by approximating the action-value function with a DNN.
    - Extension of DPG to continuous action spaces, using DQN concepts to estimate the Q-function.
    - Q-function (action-value function) maps state-action pair to expected cumulative rewards.
    - Actor-Critic structure: Actor = policy, Critic = Q-function.
    - Actor trained via Policy Gradient; Critic trained via Temporal Difference Learning.
    - Uses replay memory to break sample correlation.
    - Uses target networks for Actor and Critic updated via Polyak averaging.
    - Off-policy nature allows learning from suboptimal actions.
  - Noted advantages:
    - Can learn from suboptimal or wrong actions; useful when agent explores or lacks complete information.
    - Applied successfully to control tasks: robotic arms, flying drones, walking robots.
  - Noted disadvantages:
    - Can take a long time to converge to the optimal policy due to continued exploration.
    - Can be unstable due to constant updates causing oscillations.
  - Algorithm mechanics:
    - Agent collects experience by interacting with the environment; experience per period contains state, action, reward, next state.
    - Collected experience updates policy and value functions approximated by ANNs.
    - Policy network updates to maximize value function predictions; value function updates to minimize temporal difference errors.
  - Parameter tuning categories:
    1. Hyperparameters (exploration parameters): control learning rate, exploration rate, and related factors.
    2. Neural network parameters: control weights and biases of neural networks.
    3. Replay parameters (Memory parameters): size of replay buffer and batch size for training.

*Source: https://www.imf.org/-/media/files/publications/wp/2023/english/wpiea2023040-print-pdf.pdf*

### 4. Algorithm parameters: The parameters of the DDPG algorithm itself need to be tuned.  Actor and Critic

### 4. Algorithm parameters: The parameters of the DDPG algorithm itself need to be tuned.  Actor and Critic

### Algorithm-related parameters and ANN architecture
- Baseline parameters (from Table 2):
  - Number of episodes: 200
  - Steps per episode: 2000
  - Updates per step: 5
  - Batch size: 128
  - Hidden layer size: 16
  - Number of hidden layers: 2
  - Learning rate:
    - Actor: 1e-4
    - Critic: 1e-3
  - Activation functions:
    - Actor: ReLU for each linear layer and Tanh for output layer
    - Critic: ReLU for all
  - Exploration: From 1.0 to 0.3, diminish by episode
- Architecture and modeling notes:
  - Feedforward ANN with two hidden layers are used to approximate the policy and the value functions.
  - Each hidden layer has 16 nodes.
  - This is described as a simple ANN architecture compared to architectures used for tasks such as the game of Go.
- Parameter roles and tuning guidance:
  - Number of episodes: measure of simulation periods; too few → insufficient data to learn; too many → risk of overfitting.
  - Steps per episode: measure of task difficulty; more timesteps indicate more difficult tasks or less efficient learning.
  - Updates per step: more updates per step generally speed learning but can cause instability or divergence if excessive.
  - Batch size: larger batch size generally converges faster but may increase overfitting and instability; smaller batch size yields finer updates but slower convergence.
  - Hidden layer size and depth: control network capacity; too small → cannot learn complex patterns; too large/deep → overfitting and harder optimization.
  - Learning rates (Actor 1e-4; Critic 1e-3): govern how quickly the ANN updates and the extent new information overrides old.
  - Activation functions: ReLU commonly used (range 0 to 1 in description), Tanh used for actor output to map to (-1,1).
  - Exploration schedule: controls randomness in action selection; higher exploration → more randomness/exploration, lower → more exploitation.

### RL setup for the RBC model (Table 3)
- State (Period t state variables include): k_t, z_t, k_{t-1}, h_{t-1}, λ_{t-1}^b, R_{t-1}
- Action and action space (Period t action variables include):
  - h_t ∈ (0,1)
  - λ_t^i ∈ (0,1)
  - λ_t^b ∈ [1,2]
- Reward function:
  - Logarithmic utility: u(c_t,h_t) = (1−α) ln c_t + α ln (1−h_t)
- Transition equations (as presented):
  - K_t = k_t
  - H_t = h_t
  - y_t = e^{z_t} K_t^{θ} H_t^{1−θ}
  - i_t = λ_t^i y_t
  - r_t = θ e^{z_t} K_t^{θ−1} H_t^{1−θ}
  - R_t = r_t + 1 − δ
  - Δ_{t+1}^b = b_{t+1}/b_t − R_t = λ_t^b − R_t
  - c_t = (y_t − i_t) − a Δ_{t+1}^b
  - a > 0 is a constant and assumed to be a = 10

### Experiment I: deterministic environment — setup and results
- Deterministic specification:
  - e^{z_t} = 1 and z_t = 0 (no productivity shock)
  - Representative household is the learning agent; firms are rational.
  - Equilibrium condition: R_t = r_t + 1 − δ
- Key empirical observations:
  - Training vs testing:
    - Training: agent engages in exploratory behavior; policy and value functions are updated.
    - Testing: learning halted; policy and value functions fixed; no exploration noise.
  - Labor hour learning:
    - Early training (first 10,000 steps): actions more random.
    - After 190 episodes (last 10,000 steps): labor choices concentrate toward deterministic steady state value of 0.33.
  - Labor hour series over 500 periods: testing series close to optimal deterministic steady state; training series more variable.
- Distance metric and learning progress:
  - Distance per episode defined as d_t = (x̅_t − x_ss)^2 where x̅_t is the mean of variable in episode t and x_ss is steady state.
  - Both labor hour and consumption per labor hour distances show gradual decline over episodes, but decline is not monotonic.
  - Possible sources of non-monotonicity: changes in learning parameters (exploration level, neural network learning rate), aggregation effects; learning is slow given uncalibrated parameters.

### Experiment II: stochastic environment — setup and results
- Stochastic specification:
  - Productivity shock: z_t = ρ z_{t−1} + ε_t
  - Household is learning agent; firm is rational.
- Key empirical observations:
  - Productivity shock series plotted for 500 simulation periods (Figure 5 referenced).
  - After five episodes of learning, simulated consumption, output, investment per unit of labor and labor hour shown over 100 testing periods (Figure 6 referenced).
  - Consumption per unit of labor and output per unit of labor move in same direction; investment today raises output tomorrow.
  - Labor hour learning: frequency histogram shows early choices (first 10,000 steps) more dispersed; last 10,000 steps concentrate towards optimal choice of 0.3.
- Distance metric results:
  - Distances to deterministic steady states for consumption, output, investment per unit of labor decline gradually over 200 episodes, but not monotonically.
  - Volatility: distance series can be more volatile in stochastic environment; e.g., series after episode 125 in Figure 8 appear more volatile than deterministic case.
  - Learning speed: RL agent appears to learn quicker in the stochastic environment (small distance observed after episode 125) compared to deterministic environment (small distance only after around episode 165).
  - Interpretation: stochasticity exposes agent to a wider range of states early, aiding faster acquisition of effective policies.

### Issues during learning
- Instability of learning:
  - Many DRL algorithms are not proven to converge to optimal solutions; instability may stem from exploration noise, incorrect learning rates, noisy gradient estimation, or insufficient network depth.
  - DDPG in particular can be unstable even in trivial tasks.
  - Remedies suggested:
    - Use different noise process for exploration (e.g., Ornstein-Uhlenbeck noise instead of Gaussian noise).
    - Use different function approximators (e.g., LSTM instead of fully connected networks).
    - Modify learning process (different learning rates or optimization algorithms).
  - Example instability: after 200 episodes in deterministic environment, agent can become trapped in a suboptimal state and remain there despite further training (learning traps).
  - Frequency of traps unclear and case-dependent; may require restarting learning or exploring hyperparameter/architecture changes.
- Parameter selection challenges:
  - No general guidance for algorithm-related parameter selection or ANN architecture choice.
  - Rule of thumb: start with simple ANN architectures and increase complexity if needed.
  - Techniques to search parameters: grid search or evolutionary algorithms.
- Difficulty in reproducing results:
  - High randomness in DRL makes reproducibility challenging.
  - To improve reproducibility: use the same random seed and record all hyperparameters and problem-specific details.

### Findings and implications
- In both deterministic and stochastic setups, a representative household RL agent learns consumption-saving and labor-leisure decisions and approximates behavior of a rational expectations agent over time.
- Learning characteristics:
  - Deterministic environment: decisions converge toward deterministic steady state (0.33 labor hour) but learning is slower and less explorative.
  - Stochastic environment: agent learns quicker and explores a wider state space early, but behavior and distance metrics can be more volatile.
- Network architecture choices:
  - For the relatively simple RBC problem, ANN complexity reduced relative to common DDPG implementations (hidden layer nodes reduced to 16 per hidden layer).
- Practical recommendations:
  - Tune Actor and Critic learning rates, soft update parameters, exploration schedule, number of episodes, and steps per episode according to the underlying research question.
  - Start with simpler ANNs (two hidden layers with 16 nodes) and increase complexity if model requires it.
  - Monitor instability and learning traps; adjust noise process, function approximator, learning rates, or restart training when needed.
  - Record seeds and hyperparameters to aid reproducibility.

*Source: wpiea2023040-print-pdf - 4. Algorithm parameters: The parameters of the DDPG algorithm itself need to be tuned.  Actor and Critic*

### Annex I. DDPG algorithm

### Annex I. DDPG algorithm

### Summary
- The DDPG algorithm follows from Lillicrap et al (2019, page5).

*Source: wpiea2023040-print-pdf - Annex I. DDPG algorithm*

---


_Source: https://www.imf.org/-/media/files/publications/wp/2023/english/wpiea2023040-print-pdf.pdf_