Over the past decade, we’ve heard a lot about Big Data and how technological advancements have been driving data generation year after year. I recall an IBM study published in 2013 that made the curious observation that 90% of all the data in human history had been generated in the two years prior to its release.
The term Big Data, initially introduced by then-Gartner analyst Doug Laney in the early 2000s, refers to a vast set of data that cannot be analyzed using traditional Business Intelligence tools. It is characterized by the "3 V's":
Volume: Refers to the massive amount of data generated and stored, which can exceed exabytes (1 billion gigabytes) of data.
Velocity: Indicates the high speed at which data is generated from various sources, including sensors, mobile devices, streaming platforms, social networks, IoT, hardware, and more.
Variety: Highlights the diversity of data formats, which can range from structured data, like traditional tabular databases, to unstructured formats, such as images, audio, video, and text.
As discussions around Big Data expanded and evolved, additional dimensions were added to the original "3 V's," including Veracity, Variability, Visualization, and Value. Moreover, the advent of distributed computing and technologies like Hadoop enabled significant advancements in analyzing these massive volumes of data.
However, few organizations manage to structure themselves effectively and truly reap the benefits of this practice. High investments in expensive databases that often fail to add value to analyses, countless hours of highly qualified professionals spent on repetitive tasks such as standardization and data cleaning to build models that yield no practical results—these are some of the factors that have raised red flags for organizations to rethink their data strategies.
A 2021 Gartner study indicated that by 2025, 70% of organizations will redirect their focus from Big Data to Small and Wide Data.
What is Small Data?
Small Data can be defined as data organized in a format and volume that is easily understandable by humans, without requiring complex analysis, and which can be stored on a standard server or personal computer. The focus here shifts to the quality of collected data, rather than the volume. This doesn’t mean, however, that this strategy cannot provide organizations with a broad view of all aspects of their business. By systematically, automatically, and methodically planning the collection of key information from each area of the organization, it is possible to gather data from virtually any department.
Although Small Data doesn’t require advanced expertise in cloud tools, distributed computing, or machine learning, some level of specialization is still needed to handle it effectively. A common practice observed in various organizations is managers and analysts downloading numerous spreadsheets and CSV files from different systems and manually integrating them through laborious, repetitive processes. This approach is prone to errors, which can compromise the reliability of the final indicators.
The Role of Data Engineering
The identification of data sources, along with the automation of data extraction, cleaning, integration, and updates, can be considered the foundation that supports the entire Business Intelligence structure of an organization—processes that are part of data engineering.
Once this foundation is solidified, it’s time to define the best strategy to fully harness this hidden treasure. Understanding the business context and the activities of each department, knowing the profile of the solution’s end user, and applying programming skills tailored for data analysis come into play to develop custom Business Intelligence solutions such as dashboards, push notifications, and reports.
These solutions, in turn, must be embraced and promoted by the organization’s people to build confidence in the business community. Some of the applications of this information include:
Supporting Data-Driven Management: With reliable, secure, and easily accessible data, decision-making based on assumptions is replaced by decisions grounded in concrete facts and evidence. This type of management enables the implementation of dynamic strategies, the creation of action plans, and the tracking of their results through carefully designed indicators, fostering learning by identifying the initiatives that yield the best outcomes.
Defining and Monitoring Key Indicators: From strategic dashboards for senior management that encompass macro-level indicators from all departments to simpler models supporting operational processes.
Achieving a 360º View of the Business: Integrating data from the organization’s various systems and departments, such as Sales, Marketing, Logistics, Operations, Finance, Human Resources, Procurement, S&OP, and others.
Analyzing and Improving Processes: Identifying opportunities to enhance processes, measuring and monitoring their efficiency, automating them, and establishing new management practices.
By combining structured foundations and a clear strategy, organizations can extract actionable insights and foster a culture of innovation and continuous improvement.
According to reports from companies like McKinsey & Company and Forbes, we are approaching a new era focused on verified, well-defined data that is relevant to the business and ready to be analyzed and incorporated into predictive models: the era of Smart Data.
Here, the emphasis is on optimizing the generation of insights and information to achieve business results, whether by efficiently using Small Data or refining Big Data to extract what truly adds value.
We observe a shift from a strategy that attempts to embrace a world of data without clear objectives (in a "boiling the ocean" style) to one focused on generating meaningful results with what is already available. The phrase "tidying up the house before changing the world" also applies to the business context.
This begins with identifying the main systems and digital data sources a company has, analyzing how the data is structured and presented—whether by exporting it to Excel or accessing a database (even basic SQL knowledge makes this possible). After this initial evaluation, the challenge becomes automating the process of extracting, transforming, and loading (ETL) this data and integrating it in a highly organized, secure, and practical way. Once this foundational structure is established, it’s time to strategize how to leverage this valuable resource.
From this point, a wide range of unique solutions can be created, influenced by the different perspectives, needs, people, management models, cultures, and other unique characteristics of each organization.
Some companies may start with a "Small & Wide" strategy to achieve a 360º view of their organization with the main indicators.
Others may prefer to focus on monitoring and improving processes in specific areas, eliminating costly rework caused by inconsistent reports and fragmented information.
In certain cases, companies may even pursue large volumes of data to apply a predictive model to solve a well-defined problem.
Unsure how to begin this journey? Count on Equal | BI & Data Engineering to support your organization in this transformation.
ความคิดเห็น