Discovery, Learning and the Science of Data

Discovery

The goal of any investigation is to discover something about the world. We begin our lives as investigators. We discover the source of our first meal. As we grow, we discover what we like and don't like. We crawl around the house, bump into things, briefly investigate them, then move on. All the while, we are implicitly asking the questions, what is it, how does it work, and why does it work that way. 

The universe of things to discover is beyond large. If we had to independently discover every aspect of it, it would take 1,000 lifetimes. Our solution was to build knowledge as a community and stand on the shoulders of each other in an endless learning cycle.   

Discovering through investigation and learning from others is one of the fundamental and observable behaviors of humans. It helps us understand the world we inhabit and aids us as we navigate and make choices in life.  

Understanding

Having these skills is great but insufficient. Why? Because understanding the world is not easy, and we often fall short of the goal. We beat our heads on this wall of understanding only to be reminded that life and the world we inhabit are profoundly complex. 

If we get understanding, we get clarity; with clarity, we get precision over concepts, and with precision, we advance and progress faster than without. 

Discovery and Learning

Discovery and learning are different but interrelated and permanently intertwined. Discovery involves finding something new that was previously unknown, while learning involves the acquisition of something known.  

There are many things in life we can discover, but to be effective, we need to learn and acquire skills. Our native language is not discovered but learned through sensory absorption and a deliberate learning process. Mastery comes through the acquisition of requisite skills. Both take time.  

Observation and Data Collection

Observation is a key tool in our discovery toolbox. No significant discovery was made without the aid of this tool, and every human is born with it. Our observations of life and the world we inhabit are made directly using our senses or some proxy. When we cannot leverage our senses, the proxy we most rely on is data. 

However, gathering or collecting data in a haphazard or undisciplined way makes this proxy unreliable. The early discoveries of light, gravity, and planetary motions were backed by a disciplined collection of large quantities of data, which enabled the analysis to yield invaluable insights.  

By the standards of its day (circa 1600), the data collected was large. By the standards of today, that data would fit in a teaspoon.  

Scientific Investigation

From birth, we informally engage in discovery and investigation. An extension of that natural behavior is scientific investigation, which is a specific and formal method of discovery. While its methods and principles have evolved and become more rigorous, its core has not changed. Why? Because its core simply reflects how humans work. We ask questions, we make observations, we propose explanations, we perform endless experiments, and we collect and analyze information and data. All done in an endless cycle of iterations. The informal use of the scientific method predates its formal development by many thousands of years. 

Data as Proxy 

One of the principal intellectual tools used during an investigation is that of observation. No significant discovery of any kind was made without this. 

One’s observations of life can be direct via the senses or by proxy through data. Not all of life can be directly observed through the senses. Therefore data captured away from our senses is the next best proxy. 

As the volume of observations grow, the science of observing and inferring patterns through the proxy of data also grows. 

Data and Decision Making

Data, in all its multifaceted forms, has played a role in our decision-making process throughout human history. Its importance has always been recognized, but its adoption, growth, and use have been hindered by our ability to gather it, our capacity to process it for insights, and our willingness to incorporate the insights into our decision-making. 

While a great many human decisions are made without the aid of data insights, the act of collecting them to inform our choices has remained constant.

Science of Data 

The science of data is the science of observing, analyzing, and learning about the world through the proxy of data, where data represents anything in the world that has been observed, described, or measured. It does not matter whether humans or systems captured the data so long as the data represents some observed behavior of interest. 

The emerging science of data as a discipline is in its infancy. The volume of data we can now collect has grown from a teaspoon to an ocean. The underlying mathematics applied to the discipline is extensive and growing. The techniques to learn from data, explore, and surface insights have improved. And the ambition and funding to apply this to a wide range of human problems is rising faster than a spaceship in orbit. 

The vast quantity of data we now collect can't be subjected to direct sensory observation. For example, you can't physically park yourself on your website in order to observe how your customers behave. Nor can you travel 14 billion miles into space to observe the planets directly through your senses. As a consequence, data becomes the proxy. But insights, whether from human observation or its proxy, can be fallible. To protect from any fallout, both need to be subjected to the same level of scrutiny, controls, and oversight. 

Full Speed or Controlled Momentum

This ocean of data gives us a fighting chance to solve some of our most complex and previously unsolvable problems, and the controls we place around it should not act as a brake or a weight to slow us down. The choice between speeding irresponsibly forward or exercising deliberate and controlled momentum confronts us all. 

When technology collides with a genuine human need, something magical happens. Those who truly understand its significance, are able to investigate its capabilities, and rapidly implement any learnings usually reap the rewards.