From Intuition to Insights: Building a Data-Driven Organization
By Andrew Ritter
Organizations across industries are increasingly aspiring to become more data-driven, and for good reason. Recent studies have shown that organizations that use data and advanced analytical tools to streamline operation processes and drive strategic initiatives are more productive and profitable than their non-data-driven competitors.[i] By continuously monitoring and analyzing data, organizations can identify bottlenecks, allocate resources more effectively, set performance targets that align with business goals, and adjust their strategies and operations in real-time. Additionally, pairing data with advanced tools like AI and machine learning can automate routine tasks, freeing up time for decision-makers to focus on the high-value strategic decisions that cannot be programmed. However, the path to realizing these data-driven benefits is fraught with obstacles.
Aside from the financial investment that an organization must make to establish an infrastructure capable of supporting big data and advanced analytics, an organization must overcome resistance from within its workforce as part of its data-driven transformation. Many employees in traditionally non-data-centric fields lack formal training in data analysis and struggle with basic analytical tools. Upgrading to sophisticated tools with greater computational abilities, coupled with insufficient training, can induce anxiety and resistance. Likewise, senior leadership must change decades-old decision-making behavior by substantiating their intuitive decisions with evidence, opening them to potential scrutiny, especially when their choices contradict analytical insights. These challenges are significant but not insurmountable.
Organizations cannot overcome workforce resistance through hiring alone. Nearly every major company is actively seeking data science talent, causing demand to far outpace supply. The United States alone is projected to face a shortfall of approximately 250,000 data scientists in 2024.[ii] The scarcity of data science talent, combined with high competition for these individuals, makes it impractical for organizations to rely solely on hiring to infuse data-savvy talent. Instead, they must focus on investing in continuous training for their existing workforce and cultivating a data-driven culture.
Data-driven transformations often fail, not due to the organization’s inability to acquire data tools and systems but because leadership overlooks the need to establish a data-driven culture to fully realize the benefits of these investments. A data-driven culture exhibits several characteristics:
Data analysis is combined with human intuition to drive informed decisions.
Management provides the necessary motivation, support, and authority to shift from intuition-based decisions to data-driven ones.
Employees receive continuous training in data management, visualization, and analysis.
Data teams ensure data quality by maintaining accuracy, timeliness, completeness, consistency, and relevance.
IT makes data accessible by creating infrastructure and governance policies that allow for secure, seamless data access across the organization.
Establishing a data-driven culture is undoubtedly challenging. A study by NewVantage Partners found that only 26.5% of organizations have successfully established such a culture.[iii] As a result, data-resistant organizations will find it increasingly difficult to compete, leading to a decline in relevance and viability. To build a data-driven culture before committing significant time and financial resources to a full-scale transformation, organizations can start by establishing small-scale pilot projects that deliver quick wins and showcase potential use cases.
Piloting data-driven initiatives is an effective way to achieve quick wins in a safe environment for experimentation, where the results do not impact the organization's bottom line. The goal is to demonstrate the value of data-driven decision-making to all personnel, allowing them to gain hands-on experience and training in a low-stakes setting. A pilot can demonstrate the practical value of data-driven decision-making, build trust among employees, and reduce resistance to change.
How to Design a Data-Driven Decision-Making Pilot Program
A well-designed pilot is essential to an organization’s successful implementation of, and workforce buy-in for, its data-driven transformation. For example, the U.S. Food and Drug Administration (FDA) launched a three-phased AI pilot in 2019 to improve the agency’s ability to identify products that pose a threat to public health. By the end of the pilot, the FDA successfully demonstrated that machine learning algorithms can enhance field operations and improve the identification of hazardous products, leading to better decision-making and resource targeting.
Below is a high-level overview of the key phases and considerations necessary for executing a successful pilot program.
Phase 1: Define Success
Reflect on the regular decisions your organization makes and identify which could benefit from being more data-driven. Establish what success looks like by defining specific, measurable goals and outcomes. These might include increased efficiency, reduced costs, improved customer satisfaction, or enhanced employee engagement.
Phase 2: Establish Metrics and Define Data Collection Methods
Select key performance indicators (KPIs) that are both informational and actionable to accurately measure progress toward the defined goals. Then, determine the best methods for data collection, such as surveys, system logs, or transaction records, and establish the frequency at which your organization should collect it.
Phase 3: Collect, Clean, and Analyze Data
Gather data from identified sources using the defined methods, removing any errors, duplicates, or inconsistencies in the data. Leverage analytical tools, including AI, to identify patterns and insights that can inform decision-making. Regularly assess the metrics’ effectiveness and make adjustments to ensure they remain relevant.
Phase 4: Act and Adapt
Make decisions based on a combination of data analysis and intuition-based judgment. Monitor changes in behavior toward the desired outcomes and adapt strategies based on insights.
Phase 5: Share the Success Story
Record the pilot's outcomes, including successes and lessons learned. Develop a strategy for sharing these results with the broader organization. Identify and empower employees who participated in the pilot to act as champions for data-driven decision-making across the organization. Use success stories to build trust and encourage the wider adoption of data-driven decision-making.
Embrace the Data-Driven Future
The ability to harness data effectively will determine the future success and viability of organizations across all industries. Those that embrace data-driven transformation will find themselves better equipped to compete, innovate, and thrive, while those that resist will likely face a steady decline in relevance.
Establishing a data-driven culture is not just about adopting new technologies but fundamentally reshaping how an organization operates. This transformation requires strong leadership, continuous employee training, and a strategic approach to integrating data into every aspect of the business. By starting with small-scale pilot projects, leadership can help build trust, provide practical experience, and demonstrate the tangible benefits of data-driven decision-making.
Sources
Anton, Eduard, et al. “Beyond Digital Data and Information Technology: Conceptualizing Data-Driven Culture.” https://doi.org/10.17705/1pais.15301.
Berndtsson, Mikael, and Stefan Ekman. “Assessing Maturity in Data-Driven Culture.” https://doi.org/10.4018/IJBIR.332813.
Davenport, Thomas H., and Randy Bean. “The Quest to Achieve Data-Driven Leadership A Progress Report on the State of Corporate Data Initiatives.” https://c6abb8db-514c-4f5b-b5a1-fc710f1e464e.filesusr.com/ugd/e5361a_2f859f3457f24cff9b2f8a2bf54f82b7.pdf.
De Medeiros, Mauricius Munhoz, et al. “Data Science for Business: Benefits, Challenges and Opportunities.” https://doi.org/10.1108/BL-12-2019-0132.
“FDA Moves into Second Phase of AI Imported Seafood Pilot Program.” https://www.fda.gov/food/cfsan-constituent-updates/fda-moves-second-phase-ai-imported-seafood-pilot-program
Gressel, Simone. “Management Decision Making in The Age Of Big Data: An Exploration of The Roles of Analytics And Human Judgment.” https://mro.massey.ac.nz/items/a917b423-b67a-4479-89f8-e64c20f90f0b.
Hahn, Stephen M. “Import Screening Pilot Unleashes the Power of Data and Leverages Artificial Intelligence.” https://www.fda.gov/news-events/fda-voices/import-screening-pilot-unleashes-power-data-and-leverages-artificial-intelligence
Schatsky, David, et al. “Democratizing Data Science to Bridge the Talent Gap.” https://www2.deloitte.com/content/www/us/en/insights/focus/signals-for-strategists/democratization-of-data-science-talent-gap.html.
Szukits, Ágnes, and Péter Móricz. “Towards Data-Driven Decision Making: The Role of Analytical Culture and Centralization Efforts.” https://doi.org/10.1007/s11846-023-00694-1.
“The FDA Moves into Third Phase of Artificial Intelligence Imported Seafood Pilot Program.” https://www.fda.gov/food/cfsan-constituent-updates/fda-moves-third-phase-artificial-intelligence-imported-seafood-pilot-program