The Humanity of Big Data
It may seem that technical terms such as software, data, or computers come up in our discourse only as a direct consequence of the modern era of breakthroughs in information technology. Nevertheless, the seeds of their creation go as far back as the dawn of man.
Since time immemorial, humans sought to discover and learn the world around them. Their curiosity was the leading factor of each invention that revolutionized our civilization, and this curiosity is directly connected to data. Our brains crave data and learning new information. Never has it been the case that man preferred ignorance over enlightenment because it’s inscribed in our very nature.
Survival. Whether you like it or not, all our actions are reflections of our drive to survive, and it’s not specific to us humans since all living beings develop in their own ways to have an edge in the wilderness arena, however, what’s specific to us humans is the path we chose to achieve the optimal odds of survival. Intelligence has been the defining factor that separated us from other animals and that’s where lies all the magic.
Data. The mere knowledge of certain elements of your environment can make all the difference in a life-or-death scenario. Intelligence is an unequivocal consequence of having data. It always has been that way. This is exactly the mode of operation of every company in modern times. Gather data and then use it for analytics. It’s the same process that has been applied by humans for millennia.
What makes the difference between the old era and contemporary times is the sheer volume of data that we are capable of processing. Big data has always existed, but the limits of the old folks were very evident. The tools that were used before were just too time-consuming and very error-prone. Books can store information, but the biggest downside is that they exist. Storing big data in books would likely entail cities of books in size and that’s just optimistic speculation, not to mention the time it would take to compile all of that. The virtual nature of modern tools made it astronomically efficient to store and process data on a large scale.
Big data represents the next evolution of human nature. Its advent transformed the world and continues to shape it to this day. As we continue to yearn for more data, this realm will never cease to grow and mature.
You can also notice the beauty in its architecture. The solutions that make big data efficient are distributed storage and parallel processing. Away from the technical jargon, the distributed storage part means that instead of storing data as a monolith in a single big monster of a computer, we substitute it for many machines that work together to store blocks or individual fragments of that data. This way we can add as many workers or helpers we want to the mix and scale exponentially. Distributed storage symbolizes the social nature of man. We cannot live alone, and we can only achieve survival if we stick in groups together helping each other. This metaphor is not meant to signal the way distributed storage was conceived, but it’s just an intuitive way of understanding the concept and its power. The second element which is parallel computing joins the first one in that each worker processes the individual fragment of data that it has before aggregating all the results together. This is way better than processing the big lump of data that we had before the fragmentation.
As I continue to learn about big data and the possibilities that it opens up, I cannot help but marvel at the beauty and intuitive aspect of the ecosystem it has grown to be. Big data signifies how much we have come along in the journey of man within the shades of the vast hollow universe where we endure.
