{
  "title": "Big Data: Potential, Challenges and Statistical Implications",
  "publication": "Staff Discussion Notes, September 13, 2017",
  "sourceUrl": "https://www.imf.org/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106",
  "canonical": "https://www.imf.org/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106",
  "overlayPath": "/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106/index.md",
  "summary": "Big data are part of a paradigm shift that is significantly transforming statistical agencies, processes, and data analysis.",
  "sections": [
    {
      "heading": "Summary and framing",
      "content": "- Big data are part of a paradigm shift that is significantly transforming statistical agencies, processes, and data analysis.\n- Administrative and satellite data are already well established; the statistical community is now experimenting with:\n  - structured and unstructured human-sourced data,\n  - process-mediated data,\n  - machine-generated big data.\n- The Staff Discussion Note (SDN) sets out a typology of big data for statistics and highlights that opportunities to exploit big data for official statistics will vary across countries and statistical domains.\n- The SDN provides examples from a diverse set of countries to illustrate opportunities.\n- The SDN discusses key challenges associated with proprietary data from the private sector regarding accessibility, representativeness, and sustainability.\n- The SDN concludes by discussing implications for the statistical community going forward."
    },
    {
      "heading": "Typology and data types (as presented)",
      "content": "- Structured human-sourced data\n- Unstructured human-sourced data\n- Process-mediated data\n- Machine-generated data\n- Administrative data (well established)\n- Satellite data (well established)"
    },
    {
      "heading": "Opportunities and illustrative examples",
      "content": "- Opportunities to exploit big data for official statistics differ across:\n  - countries,\n  - statistical domains.\n- The SDN presents examples from a diverse set of countries to illustrate how opportunities vary (examples summarized qualitatively in the SDN)."
    },
    {
      "heading": "Key challenges identified",
      "content": "- Accessibility\n  - Proprietary data from the private sector may be difficult to access for statistical agencies.\n- Representativeness\n  - Big data sources may not be representative of the population or economic activity of interest.\n- Sustainability\n  - Reliance on private-sector data sources raises concerns about long-term availability and continuity."
    },
    {
      "heading": "Implications for the statistical community",
      "content": "- The SDN discusses implications going forward for statistical agencies, processes, and data analysis in light of the paradigm shift toward big data.\n- The discussion emphasizes adapting statistical practices and institutional arrangements to:\n  - evaluate and integrate new data sources,\n  - address issues of access, representativeness, and sustainability,\n  - exploit machine-generated and human-sourced data where appropriate.\n\n---\n\n Content in this bundle\n\n- Staff Discussion Note\n  - Staff Discussion Note (Markdown version){rel=\"alternate\" type=\"text/markdown\"}\n  - Staff Discussion Note (PDF){rel=\"external\" type=\"application/pdf\"}\n\n---\n\nSource: https://www.imf.org/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106"
    }
  ],
  "bullets": [
    "[Markdown version](/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106/index.md)",
    "[Structured JSON version](/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106/index.json)",
    "[Bundle manifest](/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106/bundle-manifest.json)",
    "Authors: Cornelia Hammer, Diane C Kostroch, Gabriel Quiros-Romero",
    "Published: September 13, 2017",
    "Series: Staff Discussion Notes",
    "DOI: https://doi.org/10.5089/9781484310908.006",
    "Big data are part of a paradigm shift that is significantly transforming statistical agencies, processes, and data analysis.",
    "Administrative and satellite data are already well established; the statistical community is now experimenting with:",
    "The Staff Discussion Note (SDN) sets out a typology of big data for statistics and highlights that opportunities to exploit big data for official statistics will vary across countries and statistical domains.",
    "The SDN provides examples from a diverse set of countries to illustrate opportunities.",
    "The SDN discusses key challenges associated with proprietary data from the private sector regarding accessibility, representativeness, and sustainability.",
    "The SDN concludes by discussing implications for the statistical community going forward.",
    "Structured human-sourced data",
    "Unstructured human-sourced data",
    "Process-mediated data",
    "Machine-generated data",
    "Administrative data (well established)",
    "Satellite data (well established)",
    "Opportunities to exploit big data for official statistics differ across:",
    "The SDN presents examples from a diverse set of countries to illustrate how opportunities vary (examples summarized qualitatively in the SDN).",
    "Accessibility",
    "Representativeness",
    "Sustainability",
    "The SDN discusses implications going forward for statistical agencies, processes, and data analysis in light of the paradigm shift toward big data.",
    "The discussion emphasizes adapting statistical practices and institutional arrangements to:",
    "**Staff Discussion Note**"
  ],
  "related": [
    {
      "title": "Staff Discussion Note",
      "role": "note",
      "sourceUrl": "https://www.imf.org/-/media/files/publications/sdn/2017/sdn1706-bigdata.pdf",
      "summary": {
        "path": "/-/media/files/publications/sdn/2017/sdn1706-bigdata.pdf.md",
        "mime": "text/markdown"
      },
      "binary": {
        "path": "/-/media/files/publications/sdn/2017/sdn1706-bigdata.pdf",
        "mime": "application/pdf"
      }
    }
  ],
  "alternates": {
    "markdown": "/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106/index.md",
    "json": "/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106/index.json",
    "bundleManifest": "/en/publications/staff-discussion-notes/issues/2017/09/13/big-data-potential-challenges-and-statistical-implications-45106/bundle-manifest.json"
  },
  "generatedAtUtc": "2026-09-21T00:27:02.529Z"
}