Reinforcement Learning from Experience Feedback: Application to Economic Policy

IMF Working Papers, June 7, 2024

Source details

Canonical URL: Reinforcement Learning from Experience Feedback: Application to Economic Policy

Introduces Reinforcement Learning from Experience Feedback (RLXF), a procedure that tunes large language models (LLMs) based on lessons from past experiences.
RLXF integrates historical experiences into LLM training in two key ways:
- By training reward models on historical data.
- By using that knowledge to fine-tune the LLMs.
Case study: RLXF was applied to tune an LLM using the IMF's MONA database to generate historically-grounded policy suggestions.
Main implication: RLXF could enable more informed applications of LLMs for economic policy by equipping generative AI with a nuanced perspective informed by previous experiences.
Caution: Reliance on historical data may perpetuate biases and outdated assumptions.

Builds upon current methods in the application of Reinforcement Learning (RL) to LLMs.
Two-step integration of historical experiences:
- Train reward models on historical data.
- Fine-tune LLMs using the trained reward models.
Case study dataset: IMF's MONA database (used to generate historically-grounded policy suggestions).

Findings:
- RLXF demonstrates potential to provide LLMs with perspectives informed by past experiences.
- The procedure successfully produced historically-grounded policy suggestions in the MONA case study.
Risks and limitations:
- Heavy reliance on historical data could perpetuate biases.
- Historical assumptions embedded in training data may be outdated for current policy contexts.

Potential application:
- Use RLXF to inform economic policymaking by grounding generative AI outputs in historical experience.
Cautions for policymakers and practitioners:
- Assess and mitigate bias in historical datasets before training reward models.
- Complement RLXF outputs with expert judgment to avoid reliance on outdated assumptions.
- Monitor and validate model suggestions against current economic conditions and institutional knowledge.