## ftnea2023006

## Source details

**Canonical URL:** [ftnea2023006](https://www.imf.org/-/media/files/publications/ftn063/2023/english/ftnea2023006.pdf)

## Other formats

- [Markdown version](/-/media/files/publications/ftn063/2023/english/ftnea2023006.pdf.md)
- [Structured JSON version](/-/media/files/publications/ftn063/2023/english/ftnea2023006.pdf.json)

---

### Introduction — GenAI overview and adoption timeline
- Chat Generative Pre-Trained Transformer (ChatGPT) launched on November 30, 2022.
- Within a span of two months, the platform gained more than 100 million active users across the globe.
- By March 2023, several competitors had introduced their own iterations of what are now commonly referred to as generative AI systems (GenAI).
- GenAI is a subset of AI–machine learning (AI/ML) technologies, distinguished by their ability to create new content.
- Large language models (LLMs) are neural network–based models trained on massive amounts of data, including text and documents, and capable of producing understandable and meaningful text or human languages (Appendix 2).
- GenAI’s capabilities:
  - Process very large and diverse data sets.
  - Generate content in accessible and easily usable formats (including conversational).
  - Enhance efficiency and improve customer experience, risk mitigation, and compliance reporting for financial providers.

### Rapid AI adoption in financial services and sample applications
- Competitive pressures have fueled rapid adoption of AI/ML in financial services for efficiency, cost savings, reshaped client interfaces, forecasting accuracy, and improved risk management and compliance.
- GenAI is expected to accelerate AI adoption and has been explored for a broad range of applications.
- Illustrative industry uses:
  - Capital One and JPMorgan Chase: augment AI-powered fraud and suspicious activity detection, yielding reportedly significant reduction in false positives, better detection rate, reduced costs, and improved customer satisfaction.
  - Morgan Stanley Wealth Management: will use OpenAI’s technology to leverage its own data to assist financial advisors with insights into companies, sectors, asset classes, capital markets, and regions.
  - Wells Fargo: automating document processing, providing summary reports, scaling virtual assistant chatbots.
  - Goldman Sachs and Citadel: considering GenAI for internal software development and information analysis.

### Risk Considerations — overview of risk categories
- Identified risk categories:
  - Inherent technology risks: data privacy and embedded bias.
  - Performance risks: robustness, synthetic data, and explainability.
  - New cybersecurity threats posed by GenAI.
  - Potential broader risks to financial stability.
- Note on deployments: Distribution of risks between public and private GenAI applications may vary; risks are likely to be better managed in private/enterprise deployments.

### Data privacy
- AI/ML privacy concerns:
  - Data leakages from training data sets.
  - Capacity to unmask anonymized data through inferences.
  - AI/ML “remembering” information about individuals in the training data set after the data are used and discarded.
  - AI/ML outputs may leak sensitive data directly or by inference.
- GenAI-specific concerns:
  - Publicly available GenAI systems automatically “opt in” every user and continuously use inputs from users for training and fine-tuning, raising the possibility that sensitive financial data and personal information provided by financial institutions’ staff could leak out.
  - Several GenAI systems often explicitly state that they cannot ensure the security and confidentiality of the information and data provided by users.
  - Enterprise-level GenAI systems are being developed to address privacy concerns and could improve data security in principle, but residual concerns remain (for example, risks from GenAI’s capability to process and scrape broad data formats such as social media that may unintentionally collect and use personal information).
- Note on opting out: The “opting out” choice for user data collection and use needs to be explicitly exercised; however, opting out seems to limit, although it’s unclear to what extent, GenAI responses and thus possibly diminishes the technology’s utility.

### Embedded bias
- Definition: Embedded bias—computer systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others (Friedman and Nissenbaum 1996).
- Sources of bias:
  - Training data that are incomplete, unrepresentative, or underpinned by prevailing societal prejudices.
  - Algorithm design influenced by human biases.
  - Prompts directed to GenAI that themselves carry human biases.
- GenAI-specific amplification:
  - Models trained on broad online textual and other data formats that inherently carry human biases.
  - GenAI creates “new content” based on accuracy probability of each part of the answer rather than using training data solely for predictions.
  - SEO techniques may influence training datasets and skew GenAI outputs, introducing difficult-to-detect bias layers.
- Financial-sector implications:
  - Overreliance on GenAI-generated client profiles or transaction screening could lead to inaccurate or discriminatory client assessments, financial exclusion, and damaged public trust.
  - GenAI-based chatbots may misdirect client segments and do not absolve financial institutions from legal and regulatory obligations (see, for example, US Consumer Financial Protection Bureau 2023).

### Robustness
- Robustness concerns:
  - Accuracy of AI models’ output, especially in changing environments, and governance to safeguard against unethical use.
  - AI/ML models lose prediction accuracy when reliable signals become unreliable or behavioral correlations shift.
- GenAI-specific robustness risks:
  - Models can produce wrong but plausible-sounding answers and defend those responses confidently—“hallucination.”
  - Hallucination is particularly acute in conversational GenAI and can be amplified in dialog settings (Dziri and others 2022).
  - Contributing factors may include information misalignment or divergence between reference and source data in large datasets and aspects of model development and training (Parikh and others 2020).
  - Current mitigation efforts are narrowly focused on specific tasks (for example, abstractive summarization) rather than on broader hallucination causes (Ji and others 2023).
  - Enterprise-level GenAI could provide more focused, higher-quality, and more transparent training data, but hallucination risk will likely remain a concern in the foreseeable future.
- Financial services impact:
  - Hallucination can undermine robustness, raise safety and consumer protection concerns, and lead to wrong risk assessments, inappropriate advice, or unsuitable product offerings, exposing the financial system to significant risks and eroding public trust.

### Synthetic data — definition, rationale, and challenges
- Synthetic data:
  - Algorithm-created with a statistical distribution that mimics real data via deep learning model simulation.
  - Used primarily to train AI/ML and for testing model robustness.
- Rationale:
  - Alleviate privacy and confidentiality concerns.
  - Cost-effective alternative to real data.
- Challenges:
  - Data quality issues.
  - Potential replication of inherent real-world biases and gaps in generated data sets.
- Industry examples:
  - Apple uses synthetic data to enhance Siri's voice recognition capabilities.
  - Tesla deploys synthetic data in simulation of myriad driving conditions.
  - Retail firms adopt synthetic data to emulate consumer behavior patterns for actionable insights.

### Explainability
- Explainability needs:
  - Financial institutions must explain decisions and actions to internal and external stakeholders, including prudential supervisors; covers product development and marketing, risk management, regulatory compliance (such as anti–money laundering and combating the financing of terrorism), and consumer engagement.
- Explainability challenges:
  - AI algorithms have dense architecture, rely on numerous parameters, and are often ensembles of interacting models with input signals that may not be easily identifiable.
  - Trade-off between model accuracy and flexibility and its explainability. (Here, flexibility refers to algorithm’s capacity to approximate different functions and is directly related to the number of the model’s parameters.)
- GenAI exacerbation:
  - Breadth and diversity of data used make it exceedingly difficult to map output to inputs, including in the extreme case of hallucination.
  - Architecture and decision-making processes are highly opaque: multiple neural network layers and numerous parameters calculate probabilities for each part of outputs.
- Current state:
  - Research is ongoing (for example, Ullah and others 2020), but current techniques to improve GenAI explainability remain unsatisfactory.
- Implications for adoption:
  - GenAI output consists of texts (recommendations, advice, analysis) rather than decisions; human actors should make decisions and assume responsibility.
  - Financial institutions need to understand reasons for their actions; when based on GenAI outputs, institutions should be able to understand the generative process and its limitations.

### Cybersecurity
- New cybersecurity challenges posed by GenAI:
  - Generation of more sophisticated phishing messages and emails.
  - Facilitation of impersonation of individuals or organizations, increasing identity theft and fraud risks.
  - Proliferation of deepfakes (videos, audios, images) could inflict serious damage on organizations and individuals.
- Vulnerabilities:
  - Data poisoning attacks: influence AI models at training stage by adding special elements to training data to undermine accuracy or hide malicious actions awaiting special inputs.
  - Input attacks: influence AI models during operation.
  - Tools like SEO or GenAI-generated content could be used to manipulate the GenAI data environment for malicious purposes.
- Present risk context:
  - Current GenAI models are trained and operate on pre-2021 internet scraped data; risk may not be material now but could change rapidly with wider awareness and adoption.
  - Enterprise-level GenAI applications could be particularly vulnerable because they use focused data sets that could be targeted by purpose-built cyberhacking tools.
- “Jailbreaking” and prompt injection:
  - Increasingly successful “jailbreaking” attacks use carefully designed prompts to bypass GenAI rules and filters or insert malicious data/instructions (“prompt injection attack”).
  - Such attacks could corrupt GenAI operations or siphon out sensitive data.
- Uncertainty: Full scale of GenAI vulnerability to cyberattacks is not yet comprehensively understood; early signs indicate potentially substantial issues for large-scale adoption in sensitive and heavily regulated sectors such as finance.

### Financial stability
- AI/ML systemic risk channels (as highlighted in the IMF 2021 paper):
  - Greater homogeneity in risk assessments and credit decisions.
  - Out-of-sample risk, coupled with rising interconnectedness, could create conditions for systemic risk buildup.
  - AI/ML may automate and accelerate procyclicality of financial conditions (for example, automating risk assessments and credit underwriting decisions that are inherently procyclical).
  - In a tail risk event, AI/ML could quickly amplify and spread shocks and complicate policy responses.
- GenAI-specific systemic concerns:
  - Ease and cost-effectiveness of generating GenAI reports, and lack of effective regulatory regimes, could increase excessive reliance on GenAI and thus contagion and systemic risks.
  - Specific concerns:
    - Decisions based on GenAI-generated economic, market, or risk reports could be susceptible to herd mentality bias and mispricing risk if reports reflect public sentiments captured from the data sets used by the GenAI system, particularly at times of market euphoria. (GenAI applications could contribute to liquidity risk if their algorithms inadvertently promote among market participants herd behavior resulting in simultaneous buying or selling decisions; large-scale market dislocations could result.)
    - Hallucination could become systemically important if misleading information spreads in the financial system, aided by concentration of GenAI service providers and challenges in interpreting and identifying sources and counterparties. (Financial institutions' reliance on GenAI technologies from a small number of providers could result in concentration risk and lead to making those providers vulnerable to various operational risks or disruptions. The high dependence of financial business and the technological concentration could create a “too-big-to-fail” problem.)
    - GenAI could generate solvency and liquidity risks if AI-driven trades take higher-credit or market risks to maximize profit when models are not trained about risk management properly; herding behavior of GenAI investment advisors could affect market liquidity, and rumors propagated by GenAI could trigger bank runs.
    - Cybersecurity vulnerabilities and data-manipulation attacks by GenAI could create false and malicious content that generates public panic and, in financial services, could result in bank runs.

### Conclusion and regulatory recommendations
- Promise: GenAI could drive significant efficiency, improve customer experience, and strengthen risk management and compliance in the financial sector.
- Risks: Intrinsic GenAI risks could pose material threats to financial sector reputation and soundness and could undermine public trust.
- Enterprise-level GenAI:
  - Could help mitigate some risks inherent in public GenAIs.
  - May not be cost efficient for smaller financial institutions.
- Interim regulatory and supervisory actions needed:
  - GenAI use needs close human supervision commensurate with the risks that could materialize from employing the technology in financial institutions’ operations (for example, the use of AI for analysis or recommendations vs. implementation of AI systems that can make and execute decisions).
  - Prudential oversight authorities should strengthen institutional capacity and intensify monitoring and surveillance of the technology’s evolution, paying close attention to how it is applied in the financial sector.
  - Authorities should improve communication with public and private sector stakeholders and collaborate with jurisdictions at regional and international levels.

### Box 2 — Synthetic Data in AI: drivers, promise, and risks
- Drivers and promise:
  - Use driven by regulatory necessity and practical business needs.
  - Synthetic data cannot be attributed to any person or group, attractive where data privacy concerns exist—particularly in highly regulated sectors like financial and health services.
  - Can mitigate imbalances and biases in real data and help build more robust, explainable models that better meet regulatory requirements (Papenbrock and Ebert 2022).
  - Cost-efficient training-data alternative for businesses lacking access to proprietary real data, addressing lopsided data ownership favoring established incumbents.
  - GenAI could significantly expand synthetic data use in finance because it:
    - Is intrinsically geared toward generating new content and using more diverse data sources.
    - Can be used to code synthetic data–generator algorithms.
    - Better captures the complexity of real-world events.
  - Financial institutions can customize GenAI-generated synthetic data for specific functions (for example, fraud detection), product development and delivery, and compliance reporting.
- Risks transferred or introduced:
  - Unclear to what extent GenAI could impart some of its risks (for example, bias, accuracy) to generated synthetic data; if so, this undermines synthetic data quality and usefulness for training AI/ML systems.
  - The attractiveness of GenAI for generating synthetic data, coupled with complexity of generation, could blind financial institutions to risks embedded in training data.

### Appendix highlights — LLM comparisons (selected entries)
- BERT — Company: Google AI; Parameters: 340 million; Release Date: October 2018; GLUE score: 86.5; Main application: NLP, including question answering and natural language inference.
- Turing-NLG — Company: Microsoft; Parameters: 17 billion; Release Date: February 2020; GLUE score: 92.8; Main application: NLP, including question answering, natural language inference, and text summarization.
- Megatron-Turing NLG — Company: Google AI; Parameters: 530 billion; Release Date: October 2021; GLUE score: 92.6; Main application: NLP, including natural language understanding and natural language generation.
- LaMDA — Company: Google AI; Parameters: 137 billion; Release Date: May 2022; GLUE score: 93.4; Main application: Conversational AI, including question answering and generating different creative formats of text content.
- Blender — Company: Blender Institute, Netherlands; Parameters: 137 billion; Release Date: May 2022; GLUE score: 92.9; Main application: NLP, including question answering, natural language inference, and creative writing.
- Jurassic-1 Jumbo — Company: Google AI; Parameters: 1.75 trillion; Release Date: June 2022; GLUE score: 94; Main application: NLP, including natural language understanding and natural language generation.
- WuDao 2.0 — Company: Beijing Academy of Artificial Intelligence; Parameters: 1.75 trillion; Release Date: June 2022; GLUE score: 94.2; Main application: NLP, including machine translation and question answering.
- GPT-3 — Company: OpenAI; Parameters: 175 billion; Release Date: November 2022; GLUE score: 80.3; Main application: NLP, including machine translation, text summarization, and question answering.
- GPT-4 — Company: OpenAI; Parameters: 175 billion; Release Date: March 2023; GLUE score: 93.3; Note: Better than GPT-3 on complex tasks, though slower and more expensive; Main application: Similar to GPT-3.
- Claude — Company: Anthropic; Parameters: 137 billion; Release Date: March 2023; GLUE score: 92.2; Main application: Good on both broad and complex tasks, more balanced than GPT-4.
- Pi — Company: Inflection; Parameters: 137 billion; Release Date: June 2023; GLUE score: 92.8; Main application: Excellent on both broad and complex tasks, comparable with GPT-4.

### Appendix highlights — Transformer architecture and limitations
- Key transformer steps:
  1. Input embedding: tokens converted into vectors using an embedding layer, plus positional encoding.
  2. Self-attention mechanism: weighs relevance of each word to capture context.
  3. Layer normalization: aids faster and more stable training.
  4. Feed-forward neural network (FFNN): transforms contextualized vectors.
  5. Stacking of layers: repeat steps 2–4 multiple times; number of stacks is a model parameter.
  6. Output layer: final linear layer and softmax activation produce a probability distribution over the vocabulary.
- Limitations:
  1. Understanding context: models do not truly "understand" text as humans do; they are pattern-matching algorithms predicting next tokens.
  2. Lack of common sense: models do not possess innate human knowledge or common sense unless present in training data (example: may not inherently know that an elephant cannot fit inside a car unless seen in training data).

### Dependence on training data — key limitations and mitigation options
- Key limitations:
  - The quality and scope of the training data greatly affect the performance of generative AI models.
  - If the training data are biased, the model's output will likely also be biased.
  - If the training data lack certain information, the model won't be able to generate that information accurately.
  - Control and safety: challenging to control outputs; models might create inappropriate, offensive, or misleading content.
  - Resource intensive: training generative AI models requires significant computational resources and data, limiting accessibility for individual researchers or small organizations.
  - Inability to verify facts: models like GPT-3 can't access real-time information or verify truth beyond training cutoff; they draw on knowledge available up to their last training.
  - Hallucination: models may "make up" details not in input and not reflecting reality; major issue for factual tasks.
- Mitigation options and mechanisms:
  - Integration with knowledge graphs:
    - Knowledge graphs store structured information as entities and relationships for precise queries.
    - Complementarity proposed:
      - LLM interprets and generates natural language inquiries and responses.
      - Knowledge graph delivers factual and consistent information to inform responses.
    - Potential tasks improved: question answering, semantic search, information extraction.
    - Limitation: seamless integration of LLMs and knowledge graphs remains a complex challenge and active research area.
  - Fine-tuning:
    - Supplements broad pretraining with specialized or enterprise-specific data.
    - Produces more focused and accurate output, mitigating spurious or nonmeaningful text.
  - Prompt engineering:
    - Configuration of input structure and content to guide outputs.
    - Benefits: enhance precision, exercise some control over output, mitigate inherent bias.

*Source: FINTECH NOTES — Generative Artificial Intelligence in Finance: Risk Considerations (Introduction and Risk Considerations).*

### Introduction ...........................................................................................................

### Introduction

### GenAI overview and adoption timeline
- Chat Generative Pre-Trained Transformer (ChatGPT) launched on November 30, 2022.
- Within a span of two months, the platform gained more than 100 million active users across the globe.
- By March 2023, several competitors had introduced their own iterations of what are now commonly referred to as generative AI systems (GenAI).
- GenAI is a subset of AI–machine learning (AI/ML) technologies, distinguished by their ability to create new content.
- At the heart of GenAI are large language models (LLMs), which are neural network–based models trained on massive amounts of data, including text and documents, and capable of producing understandable and meaningful text or human languages (Appendix 2).
- GenAI’s ability to process very large and diverse data sets and to generate content in accessible and easily usable formats (including conversational) is proving useful in enhancing efficiency and improving customer experience, risk mitigation, and compliance reporting for financial providers.

### Rapid AI adoption in financial services
- Competitive pressures have fueled rapid adoption of AI/ML in the financial sector in recent years by facilitating gains in efficiency and cost savings, reshaping client interfaces, enhancing forecasting accuracy, and improving risk management and compliance.
- GenAI is expected to accelerate AI adoption in the financial sector and has been explored for a broad range of applications (see Box 1).

### Box 1 — Sample Generative AI Applications in the Financial Sector
- Capital One and JPMorgan Chase have leveraged GenAI to augment their AI-powered fraud and suspicious activity detection system, reportedly resulting in a significant reduction in false positives, a better detection rate, reduced costs, and improved customer satisfaction.
- Morgan Stanley Wealth Management will use OpenAI’s technology to leverage its own vast data sources to assist financial advisors with insights into companies, sectors, asset classes, capital markets, and regions around the world.
- Wells Fargo is building capabilities for automating document processing, including providing summary reports, and scaling up its virtual assistant chatbots.
- Goldman Sachs and Citadel are considering GenAI applications for internal software development and information analysis.

### Risk Considerations

### Overview of risk categories
- The note identifies risks that are:
  - inherent in the technology (data privacy and embedded bias);
  - related to performance (robustness, synthetic data, and explainability);
  - new cybersecurity threats posed by GenAI;
  - potential broader risks to financial stability.
- Distribution of risks between public and private GenAI applications may vary, and risks are likely to be better managed in private/enterprise deployments.

### Data privacy
- AI/ML privacy concerns include data leakages from the training data sets, the capacity to unmask anonymized data through inferences, and AI/ML “remembering” information about individuals in the training data set after the data are used and discarded; AI/ML’s output may leak sensitive data directly or by inference.
- GenAI raises similar privacy issues but also unique concerns:
  - Publicly available GenAI systems automatically “opt in” every user and continuously use inputs from users for training and for fine-tuning their responses, raising the possibility that sensitive financial data and personal information provided by financial institutions’ staff in their engagement with the GenAI could leak out.
  - Several GenAI systems often explicitly state that they cannot ensure the security and confidentiality of the information and data provided by users.
  - Enterprise-level GenAI systems are being developed to address privacy concerns, and in principle could improve data security for financial institutions, but residual concerns remain, including risks from GenAI’s capability to process and scrape broad data formats (for example, social media) that may unintentionally collect and use personal information that otherwise may have needed explicit consent.
- Note: The “opting out” choice for user data collection and use needs to be explicitly exercised; however, opting out seems to limit, although it’s unclear to what extent, GenAI responses and thus possibly diminishes the technology’s utility.

### Embedded bias
- Embedded bias: computer systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others (Friedman and Nissenbaum 1996).
- Sources of bias:
  - Training data that are incomplete, unrepresentative, or underpinned by prevailing societal prejudices.
  - Algorithm design influenced by human biases.
  - Prompts directed to GenAI that themselves carry human biases.
- GenAI-specific amplification of bias:
  - GenAI models are trained on broad online textual and other data formats that inherently carry real-life human biases.
  - GenAI uses its training data to create “new content” (textual answers) based on accuracy probability of each part of the answer, rather than using training data solely for predictions.
  - Search engine optimization (SEO) techniques may influence the training datasets and skew GenAI outputs, introducing difficult-to-detect bias layers.
- Financial-sector implications:
  - Overreliance on GenAI-generated client profiles or transaction screening could lead to inaccurate or discriminatory client assessments, financial exclusion, and damaged public trust.
  - GenAI-based chatbots addressing inquiries and complaints may misdirect client segments and do not absolve financial institutions from legal and regulatory obligations (see, for example, US Consumer Financial Protection Bureau 2023).

### Robustness
- Robustness concerns cover accuracy of AI models’ output, particularly in a changing environment, and governance to safeguard against unethical use.
- AI/ML models perform well in relatively stable data environments but lose prediction accuracy when reliable signals become unreliable or behavioral correlations shift.
- GenAI-specific robustness risks:
  - GenAI models can produce wrong but plausible-sounding answers and then defend those responses confidently — a phenomenon broadly referred to as “hallucination.”
  - Hallucination is particularly acute in conversational GenAI and can be amplified in dialog settings (Dziri and others 2022).
  - Proposed contributing factors include information misalignment or divergence between reference and source data in large datasets and aspects of model development and training (Parikh and others 2020).
  - Current mitigation efforts are narrowly focused on specific tasks (for example, abstractive summarization) rather than on broader hallucination causes (Ji and others 2023).
  - Enterprise-level GenAI could help by providing more focused, better quality, and more transparent training data, but hallucination risk will likely remain a concern in the foreseeable future.
- Financial services impact:
  - GenAI hallucination can undermine robustness, raise safety and consumer protection concerns, and lead to wrong risk assessments, inappropriate advice, or unsuitable product offerings, exposing the financial system to significant risks and eroding public trust.

### Synthetic data
- Definition and use:
  - Synthetic data are algorithm-created with a statistical distribution that mimics real data via deep learning model simulation.
  - Synthetic data are used primarily to train AI/ML and for testing model robustness.
- Rationale:
  - Synthetic data alleviate privacy and confidentiality concerns and are cost-effective alternatives to real data.
- Challenges:
  - Data quality issues and the potential for replication of inherent real-world biases and gaps in the generated data sets.
- Industry examples:
  - Apple uses synthetic data to enhance Siri's voice recognition capabilities.
  - Tesla deploys synthetic data in the simulation of myriad driving conditions.
  - Firms in the retail sector adopt synthetic data to emulate consumer behavior patterns for actionable insights.

*Source: FINTECH NOTES — Generative Artificial Intelligence in Finance: Risk Considerations (Introduction and Risk Considerations).*

### Box 2. Synthetic Data in AI

### Box 2. Synthetic Data in AI

### Synthetic data: drivers and promise
- Use is driven primarily by regulatory necessity and practical business needs.
- Synthetic data cannot be attributed to any person or group, making them attractive where data privacy concerns exist—particularly in highly regulated sectors like financial and health services.
- Synthetic data can mitigate imbalances and biases in real data and help build more robust, explainable models that better meet regulatory requirements (Papenbrock and Ebert 2022).
- Synthetic data provide a cost-efficient training data alternative for businesses lacking access to proprietary real data, addressing a lopsided data ownership environment favoring established incumbents.
- GenAI could significantly expand synthetic data use in finance because it:
  - Is intrinsically geared toward generating new content and using more diverse data sources.
  - Can be used to code synthetic data–generator algorithms.
  - Better captures the complexity of real-world events.
- Financial institutions can customize GenAI-generated synthetic data for specific functions (for example, fraud detection), product development and delivery, and compliance reporting.

### Risks transferred or introduced by GenAI-generated synthetic data
- It is not clear to what extent GenAI could impart some of its risks (for example, bias, accuracy) to the generated synthetic data; if so, this undermines synthetic data quality and their usefulness for training AI/ML systems.
- The attractiveness of GenAI for generating synthetic data, coupled with the complexity of generation, could blind financial institutions to risks embedded in training data.

### Explainability
- Financial institutions must explain decisions and actions to internal and external stakeholders, including prudential supervisors; this covers product development and marketing, risk management, regulatory compliance (such as anti–money laundering and combating the financing of terrorism), and consumer engagement.
- Ensuring explainability of AI-driven decisions is complex because:
  - AI algorithms have dense architecture, rely on numerous parameters, and are often ensembles of interacting models with input signals that may not be easily identifiable or known.
  - There is a general trade-off between model accuracy and flexibility, and its explainability. (Footnote: Here, the term flexibility refers to algorithm’s capacity to approximate different functions and is directly related to the number of the model’s parameters.)
- GenAI exacerbates the explainability problem because:
  - The breadth and diversity of data used make it exceedingly difficult to map output to inputs, including in the extreme case of hallucination.
  - GenAI architecture and decision-making processes are highly opaque: multiple neural network layers and numerous parameters calculate probabilities for each part of outputs.
- Research is ongoing (see, for example, Ullah and others 2020), but current techniques to improve GenAI explainability remain unsatisfactory.
- Implication for financial sector adoption:
  - GenAI output consists of texts (recommendations, advice, analysis) rather than decisions; human actors should make decisions and assume responsibility.
  - Financial institutions need to understand reasons for their actions; when based on GenAI outputs, institutions should be able to understand the generative process and its limitations.

### Cybersecurity
- GenAI poses new cybersecurity challenges:
  - Can be exploited to generate more sophisticated phishing messages and emails.
  - Facilitates impersonation of individuals or organizations, increasing identity theft and fraud risks.
  - Proliferation of deepfakes (videos, audios, images) could inflict serious damage on organizations and individuals.
- GenAI models could be vulnerable to:
  - Data poisoning attacks: influence AI models at training stage by adding special elements to training data to undermine accuracy or hide malicious actions awaiting special inputs.
  - Input attacks: influence AI models during operation.
- Tools like SEO or GenAI-generated content could be used to manipulate the GenAI data environment for malicious purposes.
- Present risk context:
  - Current GenAI models are trained and operate on pre-2021 internet scraped data; risk may not be material now but could change rapidly with wider awareness and adoption.
  - Enterprise-level GenAI applications could be particularly vulnerable because they use focused data sets that could be targeted by purpose-built cyberhacking tools.
- Increasingly successful “jailbreaking” attacks:
  - Attacks use carefully designed prompts to bypass GenAI rules and filters or insert malicious data/instructions (“prompt injection attack”).
  - Such attacks could corrupt GenAI operations or siphon out sensitive data.
- Full scale of GenAI vulnerability to cyberattacks is not yet comprehensively understood; early signs indicate potentially substantial issues, warranting careful contemplation for large-scale adoption in sensitive and heavily regulated sectors such as finance.

### Financial stability
- AI/ML could bring new sources and transmission channels of systemic risks (as highlighted in the IMF 2021 paper):
  - Greater homogeneity in risk assessments and credit decisions could arise.
  - Out-of-sample risk, coupled with rising interconnectedness, could create conditions for systemic risk buildup.
  - AI/ML may automate and accelerate procyclicality of financial conditions (for example, automating risk assessments and credit underwriting decisions that are inherently procyclical).
  - In a tail risk event, AI/ML could quickly amplify and spread shocks and complicate policy responses.
- GenAI likely brings systemic risks similar to AI/ML and additional concerns:
  - Ease and cost-effectiveness of generating GenAI reports, and lack of effective regulatory regimes, could increase excessive reliance on GenAI and thus contagion and systemic risks.
  - Specific concerns:
    - Decisions based on GenAI-generated economic, market, or risk reports could be susceptible to herd mentality bias and mispricing risk if reports reflect public sentiments captured from the data sets used by the GenAI system, particularly at times of market euphoria. (Footnote: GenAI applications could contribute to liquidity risk if their algorithms inadvertently promote among market participants herd behavior resulting in simultaneous buying or selling decisions; large-scale market dislocations could result.)
    - GenAI hallucination could become systemically important if misleading information spreads in the financial system, aided by concentration of GenAI service providers and challenges in interpreting and identifying sources and counterparties. (Footnote: Financial institutions' reliance on GenAI technologies from a small number of providers could result in concentration risk and lead to making those providers vulnerable to various operational risks or disruptions. The high dependence of financial business and the technological concentration could create a “too-big-to-fail” problem.)
    - GenAI could generate solvency and liquidity risks if AI-driven trades take higher-credit or market risks to maximize profit when models are not trained about risk management properly; herding behavior of GenAI investment advisors could affect market liquidity, and rumors propagated by GenAI could trigger bank runs.
    - Cybersecurity vulnerabilities and data-manipulation attacks by GenAI could create false and malicious content that generates public panic and, in financial services, could result in bank runs.

### Conclusion and regulatory recommendations
- GenAI holds promise for financial sector applications: could drive significant efficiency, improve customer experience, and strengthen risk management and compliance.
- However, intrinsic GenAI risks could pose material threats to financial sector reputation and soundness and could undermine public trust.
- Enterprise-level GenAI applications could help mitigate some risks inherent in public GAIs, but may not be cost efficient for smaller financial institutions.
- Interim regulatory and supervisory actions needed:
  - GenAI use needs close human supervision commensurate with the risks that could materialize from employing the technology in financial institutions’ operations (for example, the use of AI for analysis or recommendations vs. implementation of AI systems that can make and execute decisions).
  - Prudential oversight authorities should strengthen institutional capacity and intensify monitoring and surveillance of the technology’s evolution, paying close attention to how it is applied in the financial sector.
  - Authorities should improve communication with public and private sector stakeholders and collaborate with jurisdictions at regional and international levels.

### Appendix highlights: Comparisons of main Large Language Models (selected entries)
- BERT — Company: Google AI; Parameters: 340 million; Release Date: October 2018; GLUE score: 86.5; Main application: NLP, including question answering and natural language inference.
- Turing-NLG — Company: Microsoft; Parameters: 17 billion; Release Date: February 2020; GLUE score: 92.8; Main application: NLP, including question answering, natural language inference, and text summarization.
- Megatron-Turing NLG — Company: Google AI; Parameters: 530 billion; Release Date: October 2021; GLUE score: 92.6; Main application: NLP, including natural language understanding and natural language generation.
- LaMDA — Company: Google AI; Parameters: 137 billion; Release Date: May 2022; GLUE score: 93.4; Main application: Conversational AI, including question answering and generating different creative formats of text content.
- Blender — Company: Blender Institute, Netherlands; Parameters: 137 billion; Release Date: May 2022; GLUE score: 92.9; Main application: NLP, including question answering, natural language inference, and creative writing.
- Jurassic-1 Jumbo — Company: Google AI; Parameters: 1.75 trillion; Release Date: June 2022; GLUE score: 94; Main application: NLP, including natural language understanding and natural language generation.
- WuDao 2.0 — Company: Beijing Academy of Artificial Intelligence; Parameters: 1.75 trillion; Release Date: June 2022; GLUE score: 94.2; Main application: NLP, including machine translation and question answering.
- GPT-3 — Company: OpenAI; Parameters: 175 billion; Release Date: November 2022; GLUE score: 80.3; Main application: NLP, including machine translation, text summarization, and question answering.
- GPT-4 — Company: OpenAI; Parameters: 175 billion; Release Date: March 2023; GLUE score: 93.3; Note: Better than GPT-3 on complex tasks, though slower and more expensive; Main application: Similar to GPT-3.
- Claude — Company: Anthropic; Parameters: 137 billion; Release Date: March 2023; GLUE score: 92.2; Main application: Good on both broad and complex tasks, more balanced than GPT-4.
- Pi — Company: Inflection; Parameters: 137 billion; Release Date: June 2023; GLUE score: 92.8; Main application: Excellent on both broad and complex tasks, comparable with GPT-4.

### Appendix highlights: Generative AI / Transformer architecture and limitations
- Key transformer steps:
  1. Input embedding: tokens converted into vectors using an embedding layer, plus positional encoding.
  2. Self-attention mechanism: weighs relevance of each word to capture context.
  3. Layer normalization: aids faster and more stable training.
  4. Feed-forward neural network (FFNN): transforms contextualized vectors.
  5. Stacking of layers: repeat steps 2–4 multiple times; number of stacks is a model parameter.
  6. Output layer: final linear layer and softmax activation produce a probability distribution over the vocabulary.
- While powerful, GenAI limitations include:
  1. Understanding context: models do not truly "understand" text as humans do; they are pattern-matching algorithms predicting next tokens.
  2. Lack of common sense: models do not possess innate human knowledge or common sense unless present in training data (example: may not inherently know that an elephant cannot fit inside a car unless seen in training data).

*FINTECH NOTES  Generative Artificial Intelligence in Finance: Risk Considerations — INTERNATIONAL MONETARY FUND*

### 3. Dependence on training data: The quality and scope of the training data greatly affect the

### 3. Dependence on training data: The quality and scope of the training data greatly affect the

### Key limitations identified
- Dependence on training data:
  - The quality and scope of the training data greatly affect the performance of generative AI models.
  - If the training data are biased, the model's output will likely also be biased.
  - If the training data lack certain information, the model won't be able to generate that information accurately.
- Control and safety:
  - It can be challenging to control the output of generative models.
  - Models might create content that is inappropriate, offensive, or misleading.
  - This is a significant area of ongoing research in AI safety.
- Resource intensive:
  - Training generative AI models typically requires a lot of computational resources and data.
  - This makes training inaccessible for individual researchers or small organizations.
- Inability to verify facts:
  - Generative models like GPT-3 don't have the ability to access real-time or current information and can't verify the truth of the information they generate.
  - They can only draw on the knowledge that was available up until the point they were last trained.
  - Applications on top of the models are being developed to perform web searches to look up facts.
- Hallucination:
  - Hallucination refers to the model "imagining" or "making up" details that were not in the input and do not accurately reflect reality.
  - Hallucination is a major issue in tasks where factual accuracy is important, such as news generation or question answering.

### Mitigation options and mechanisms
- Integration with knowledge graphs:
  - Knowledge graphs store structured information as entities and relationships, enabling precise, straightforward queries and linking related information.
  - Search engines (example given: Google's Knowledge Graph) use knowledge graphs to enhance search results with semantic-search information from a variety of sources.
  - Proposed complementarity:
    - The LLM can interpret and generate natural language inquiries and responses.
    - The knowledge graph can deliver factual and consistent information that informs those responses.
  - Potential task-level benefits:
    - Question answering: The LLM can comprehend the inquiry, while the knowledge graph can look up the accurate response.
    - Semantic search: The LLM can interpret the natural language search query and transform it into a structured query for the knowledge graph.
    - Information extraction: The LLM can parse unstructured text to extract entities and relationships, while the knowledge graph can store and query this extracted information.
  - Limitations:
    - Seamless integration of LLMs and knowledge graphs presents a complex challenge.
    - Integration remains an active area of investigation within the field of AI.
- Fine-tuning:
  - Fine-tuning supplements broad pretraining (for example, models like GPT-3) with more specialized or enterprise-specific data.
  - After fine-tuning, the model will produce more focused and accurate output, mitigating the likelihood of spurious or nonmeaningful text.
  - Fine-tuning facilitates tailoring the model's output in alignment with newly incorporated training data.
- Prompt engineering:
  - Prompt engineering involves deliberate configuration of input structure and content to guide and shape the model's output qualitatively.
  - Benefits include enhancing precision of generated text, exercising some degree of control over output, and mitigating inherent bias.

*International Monetary Fund — FINTECH NOTES: Generative Artificial Intelligence in Finance: Risk Considerations (excerpt).*

---


_Source: https://www.imf.org/-/media/files/publications/ftn063/2023/english/ftnea2023006.pdf_