Generating insight from data takes an assembly of skills and roles–often in the form of an integrated team. We display the main areas as an arch. The red keystone, which is the business domain knowledge, holds the arch together. The left side of the arch is more technical; the right side less so. The narrative is very important in terms of explaining how the analysis of data has led to insight.
Ideas expressed in this blog were inspired by Gillian Tett’s* book, The Silo Effect: The Peril of Expertise and the Promise of Breaking Down Barriers. Learnings were added from launching a data analytics firm in the middle of an economic downturn.
- silos create barriers in organizations, sectors, and industries
- silos interfere with organizational effectiveness
- silos can be interpreted as narrow, specialist groups*
- silos can also be comprised of systems and data
- silos exist in all forms of organizations (profit, non-profit, gov. etc)
- experts contribute to creating and reinforcing silos
- experts can become vulnerable to job loss in economic downturns
- digital data is double-edged, as it both creates and breaks down, silos
- data analytics platforms can bridge the gaps across silos
- applied data science is an interdisciplinary field that can bridge silos
Silos emerge naturally in all cultures and social settings. The main reason is that we as humans need to classify and group everything around us, but we are not wired to do it the same way. * So we come up with our own interpretation of the world, our work, and our personal lives. Unique interpretations create barriers.
And when we organize people to get things done, like in business, we’re inclined to create functional silos (such as finance, marketing, operations, IT etc.) because it’s efficient to do so. Functions are areas of specialty and are sometimes necessary because of regulation (e.g. professional designations for engineering, law, accounting). But they can be loosely formed too and have no designations per se; as in the case of individuals or teams that simply hoard data for their own benefit.
Whether they emerge organically or by design, the more specialized the silos, the more rigid they become. We see this in functions and disciplines that require deep expertise that takes years to accumulate. The silos lead to barriers when their concepts, jargon, and beliefs become inaccessible to other groups.
The proliferation of silos leads to difficulties among groups trying to work together. People may not understand each others’ roles or ideas; processes may break; opportunities may not be realized. This is problematic in a variety of situations where coordination is essential such as disaster response; product development & launch; construction; healthcare; customer relationship management; energy development etc. Tett* has numerous interesting examples of silos and overcoming their limitations.
Specialists within silos are vulnerable to job loss when the demand for their skills weakens. A good example of this is in the Canadian oil and gas sector. As the economic conditions drop, there is less demand for the associated experts. This chart shows the dip in Alberta-based energy & resource-related jobs as those sectors declined during 2 recessions. There are numerous accounts of how tradespeople and professionals alike lost jobs and have had difficulty transitioning to other sectors.
Alberta, Canada Job Loss In Two Sectors – Indicated by Blue Arrows
(see reference at bottom of blog **)
Silos are not limited to technical sectors. The vast array of digital data in all aspects of society propagates silos. Data amplifies who we are as it contains the language and the values that we use to describe the world around us. Imagine your own computer files (pictures, social media, email, documents, spreadsheets etc.) and how they are organized across multiple devices (computer, phone etc.)–are you in control of all that data? Do you know where to find what you need when you need it? Now imagine what it’s like across a team, an organization, a sector, an industry, a jurisdiction etc.
When organizational and data silos grow unabated, they tend to control us through unnecessary complexity. But there are ways to reduce the negative impact and create value. Drawing on Tett’s advice*, here are a number of suggestions:
- first and foremost, acknowledge that silos exist and emerge naturally
- counter silo-thinking by actively learning beyond your area
- make organizations more fluid (such as rotating staff to different areas)
- create situations for social collisions such as conferences, co-working spaces
- incentivize sharing of information beyond silos
- encourage a culture of interpreting information
- develop some “cultural translators” who are adept at moving across silos
- use computers and data to break down barriers
On the subject of using computers and data, there are at least two key areas to address: Data Analytics Platforms and Data Science (both of which are interpreted broadly in this article).
The use of a Data Analytics Platforms enables data to be integrated from different sources and to be simplified and analyzed–in short, this type of platform helps translate data into information/insight–regardless of the silos from which it came. This idea is not new, but it’s getting easier of late with advancements in cloud, self-serve data analytics, flexible databases, big data technologies, availability of data etc. Those familiar with business intelligence (BI) understand how traditional approaches have been giving way to new capabilities.
Applied Data Science is interpreted as a broad, interdisciplinary field*** that seeks to derive meaning and understanding from digital data. Some of the competencies you might see at play are identified below. Often teams are needed to bring these skills to bear, as it is too difficult to have individuals with such breadth. In this model, the Functional Knowledge pertains to any area being analyzed (e.g. finance, marketing, manufacturing, operations). And the Narrative is the story that pulls it all together, offering an explanation of what the data are saying.
The promise of data analytics platforms and the application of data science is linked their ability to answer questions. To do this responsibly, of course, ethics, policy and the law are essential boundaries for the work. Solving problems and finding opportunities won’t happen by accident; one has to be purposeful and careful to do it right.
As Tett* says, “Data does not reorganize itself, or break down silos by itself; somebody needs to program the computers. What is needed above all is a big dose of human imagination.”*
Applied Data Science is a field that we can look to for those skills. And the interdisciplinary nature of data science may offer new opportunities for specialists (such as engineers, geoscientists, lawyers, accountants etc.) who are dislodged during economic downturns.
** Fletcher, R (2018) Jobs and wages: The winners and losers through 2 Alberta recessions
*** The use of Applied Data Science is similar but a bit broader than the definition offered here Data Science Definition. “Applied” is used to suggest the practical aspects of what one might see in the workplace.
We demonstrate below how one can proto-type some statistical analysis in the form of an interactive dashboard. In doing so, multiple sources of public data can be simplified, visualized and made more insightful. Using two off-the-shelf products from Microsoft, Excel 2016 and Power BI, we scraped data from nearly 20 public web pages on two sites and produced an interactive display.
Using Excel 2016 (and its Query Editor) we were able to extract data from tables on Web pages and create a data workflow to improve it for analysis. The aggregated data was then modeled into several tables within Power BI. The Report building feature of Power BI was then used to create visuals and enable interactivity.
The resulting interactive dashboard for the 2017 Canadian U Sport Soccer Championship is embedded below.
Canadian U Sport is the governing body for Canadian university sports. Their national soccer championship in Nov 2017 awarded Gold to the Cape Breton University Men’s team and Gold to the Université de Montréal Women’s team.
- Of the 6 medals awarded in the finals (Gold, Silver, Bronze for both Men and Women), 4 of the medals went to teams from relatively small schools.
- Two universities garnered two medals each.
- All of the medal winners (with the exception of one team) had strong records in their respective divisions going back 8 years.
Where would you find that information, and which other schools competed and received medals?
According to U Sport, there are 56 universities in Canada with about 12,000 student-athletes competing in various sports. U Sport has its own site for publishing statistics, but there are numerous other ones too, often reflecting the geographic divisions across the country. And then the universities have their own sites to promote their teams, their results and to build esprit de corps on campus.
- You can see the soccer medal summary for both men or women.
- You can see the size of their respective schools.
- You can see the historical goal differential by selecting each team.
Note: Use the bi-directional arrow in the bottom right of the dashboard to make it full screen.
This link will demonstrate how Canadian Scholarship/AFA data can be represented.
Fuzeium’s dashboard platform and expertise are largely based on Microsoft’s Power BI and Microsoft Azure Cloud. The capabilities are applicable to any industry. Please contact us if you like more information.
We are pleased to add Calgary census data as one of three case studies on our site. The two other cases include Alberta Geothermal Potential and Alberta Inactive and Orphan Wells.
You can access all three case studies from the home page.
These cases illustrate how we tackle key questions and generate insights through data dashboards.
They also serve as examples of business intelligence, data analytics, and visualization, while leveraging the Microsoft Power BI and Microsoft Azure platforms.
Whether you are new to Calgary, or a long-time resident, you may be interested in knowing how the city’s population changed over time. And how the population is distributed among the communities.
The following analysis was derived from Fuzeium’s interactive dashboards, which analyze Calgary’s 2017 civic census data.
Feel free to interact directly with the
Calgary census Fuzeium dashboards.
Calgary’s population has grown every year since 1995. Although in 2010 and in 2016, that growth declined due to a net migration out of the city. In total, the population has increased by 89,000 people in the past 5 years.
Calgary is a young city with a median age (i.e the middle age) of 36.4 years (according to Calgary Economic Development). The age category of 25-44 years old, is the largest group, with over 440,000 people. The two communities with the highest number of those individuals are The Beltline and Panorama Hills.
As Calgary continued to grow, we see the biggest increase in population changes in the outer communities, which are newer subdivisions.
The green areas on the map below illustrate which communities had an increase in population (dark green with the highest), whereas the pink-red indicates communities that declined in population.
The five communities with the largest growth over 5 years, grew in total by more than 32,000 people. Not surprisingly, these are on the edges of the city in the north and the south.
They are pinpointed on the map below.
I have written about digital data as an unnatural resource and its importance to Canada’s Innovation Agenda. In this article, I want to draw attention to how digital data can enable a burgeoning, renewable energy industry in Canada: geothermal.
The Canadian Federal Government recently acknowledged in its 2017 budget that “Geothermal energy is one renewable energy source with the potential to reliably meet a portion of Canada’s heating and electricity generation needs….” Coupled with incentives now offered by the government to encourage investment, the Candian geothermal industry is hopeful it can successfully develop the resource on behalf of Canadians.
According to the industry association, CanGEA, “Geothermal energy is a clean source of reliable electricity and large scale direct use of the hot water derived from the earth that can help solve some of Canada’s greatest challenges, namely providing energy security, economic growth and reducing our CO2 emissions.”
“In its simplest terms, geothermal means earth-heat. It is related to the thermal energy of Earth’s interior. On a large scale, the intensity of this thermal energy increases with depth, that is, the temperature of the Earth increases as we travel closer to its centre.”
This means that in order to tap into that “earth heat” one has to drill into the earth. But isn’t that what the oil and gas industry has been doing for decades, albeit for hydrocarbon extraction? In the province of Alberta alone, there are about 430,000 physical surface locations where wells have been drilled for oil and gas. Not all of them function anymore and in some cases, the equipment is gone and the land reclaimed. But the landscape is very much dotted with thousands of structures like these, some working away, some sitting inactive.
Can we tell if these wells are suitable for geothermal energy? By looking at these pictures alone, unfortunately, there is no way to know. That’s because there are many factors that influence decision making. These include but are not limited to, the location of the well; depth of the well; the temperature at the bottom of the well; the physical characteristics and conditions of the well; the status of the well license; the owner of the well license; the well’s age; its proximity to electrical infrastructure and communities; the land leaseholder; existing well liabilities; regulations; the economics to build and operate, and so on.
Complicating matters, there is no one repository of all that data, so multiple sources have to be integrated in a meaningful way and analyzed. Fortunately, we are at a point in time where capabilities are coalescing to enable such decision-making. Much data is available from government agencies; geological data is in abundance; commercial oil and gas data is more prevalent; data integration and analytical tools are more sophisticated. These capabilities, when effectively combined, help reduce large data sets through filters, so that candidate wells can be identified. The following is a concept illustration of the approach for Alberta, but the method would be applicable to other jurisdictions.
Many wells have potential, regardless of their stage of life (e.g. whether they are actively producing oil or gas, they are inactive, or decommissioned etc). But there are cautionary flags raised by some who say there are numerous challenges that will impede progress, such as the condition of the well and its associated liabilities for decommissioning. This perspective was captured in a recent CBC article on the subject. But even if a particular well is not suitable for geothermal, the data stemming from the well and its area (surface and subsurface) might inform prospective developers and investors on where to drill new wells. In other words, digital data that’s accumulated in and around a well has value on its own for identifying geothermal potential.
Innovation has always stood at a crossroads; this is no different for exploiting geothermal in Canada. What’s needed are guide books that set the direction and identify what to look for along the way. CanGEA has been building those resources for over a decade and it will continue to do so on behalf of the industry. And increasingly, we will see digital capabilities emerge as an enabler to finding and exploiting “earth heat” as a renewable energy source.
We agree with much of what’s said in Canada’s Innovation Agenda because it sets the right direction, tone and commits to investment in a digital world. The premise for needing an innovation agenda of this nature is sound. As stated: “our country is blessed with an abundance of natural resources. But we can no longer prosper only from the output that comes from our natural resources. We must also prosper from the energy and ingenuity that comes from our people. That’s how Canada’s economy will outperform the economies of the rest of the world.”
We concur because there is no doubt that digital capabilities are transforming industries and as noted in the Innovation Agenda, a 4th industrial revolution is underway. Consequently, workers must have awareness, interest, aptitude, training and opportunity to work in such a world. What’s at stake for Canada, is its ability to compete globally.
But where the Innovation Agenda comes up a “bit” short (so to speak) is in reference to the importance of digital data. The Agenda makes only basic references to data (e.g. big and open), without explaining the nature of data and its value in a digital world.
In a previous article, we proposed the idea that digital data should be framed as a resource, albeit an unnatural one. Others have suggested that data can be considered the fourth factor of production, in addition to land labor and capital. These ideas underscore the notion that data is an input that creates economic value (just like natural resources have done). To be treated as a resource, however, digital data also requires adequate recognition, investment, and management.
The human skills necessary for exploiting digital data are quite varied. For the purpose of this article, we loosely categorize them under data science, an interdisciplinary field intent on finding meaning and insight in data. The skills can pertain to data management, data analytics, information science, statistics, computer programming, data visualization and storytelling. While many tools and techniques are common across industries, data science also requires knowledge of the area to which it is applied. For example, data science is applicable to both medicine and agriculture, but at some point in the process, data analysis must take on the language and meaning that’s unique to those sectors. And to pull it all together, project/program management techniques are necessary.
Data Science is a great opportunity for helping Canada compete more effectively and for creating jobs for Canadians. While Canada needs to attract skilled workers from abroad, it should also turn to slumping sectors that have professionals with such skills. Oil and Gas is a good example, which has been in a downturn since 2014. Many have been out of work in fields such as geoscience (e.g. geologists, geophysicists, petrophysics), engineering (of all types), and business management (e.g. operations management, finance, accounting etc.).
Many oil and gas professionals have a natural aptitude for data science because they’ve come through rigorous programs that rely on data analysis, use of formulae, systematic thinking etc. They are good candidates for pivoting to other sectors that require those kinds of skills.
In summary, Canada’s Innovation Agenda is a positive step forward in that it acknowledges the importance of the global, digital transformation underway and why Canada needs to adapt. But the Agenda should be enhanced to put a clearer focus on the role of digital data as a resource to create economic value. In particular, outlining the kind of skills within data science would articulate specific development needs of workers. And those needs would guide the right investments, so Canada may benefit from “from the energy and ingenuity that comes from our people.”
Digital data is a resource, but it’s unnatural because it’s not directly from nature but is created and used by computer technology. It is comprised of the facts, logs and sensor readings recorded by digital circuitry, which increasingly imbues the fabric of modern life (e.g. our phones, tablets, computers, cars, fitness trackers, weather, shipping and transportation etc.).
Digital data is now so vast and diverse, its size is beyond comprehension.
We as humans don’t perceive digital data (as it flows through a circuit board or rests on a hard drive) unless it is interfaced to match our senses.
We need ways to perceive data through visual, auditory or tactile ways. When we can sense the data, we gain a better understanding of the information hidden within.
We are used to thinking of natural resources as those which come from nature such as “sunlight, atmosphere, water, land (includes all minerals) along with all vegetation and animal life…” And in various ways, those resources are inputs to the processes affecting how we live; agriculture being a great example.
But is it possible to also think of digital data as a resource, albeit an unnatural one?
It helps if we view the question through an economic lens. Land, labor and capital have long been considered the basic inputs for economic productivity. But a study by Capgemini and The Economist Intelligence unit in 2012 suggested that data could now be considered the fourth factor of production. In other words, data can now be considered an input (along with the others) to create productive output.
Consider the following image from agriculture. The grayed-out portion is like a data echo of the activities, which in turn can be listened to improve operations. It is easy to see how land (the farmer’s field), labor (the person driving the tractor) and capital (the tractor) can be combined to create value. But modern agriculture generates digital data, which in turn serves as inputs for optimizing the operations.
Digital data has also been characterized by many as the “new oil”; oil being a type of resource. In other words, like oil, digital data can be explored, discovered, extracted, processed, packaged, distributed and consumed by end-customers. There is an interesting and helpful depiction of this comparison by one of Linkedin’s data scientists. The image below illustrates the use of data in just one step of the vast oil and gas value chain.
Digital data today is not broadly viewed as a resource. Consequently, it’s not commonly integrated in discourse about the development of natural resources. Yes, there’s considerable talk about Big Data and related subjects (like the Internet of Things), but these concepts are often technical and not easily accessible by the layperson.
If we can view digital data as a resource, analogous to that of a natural resource, we can frame it as an economic stimulator. Aspects of data management and data analytics are common across the spectrum of society. When analyzed and transformed into information and insight, data can help us find and prioritize opportunities.
Digital data then requires broader awareness (e.g. using real examples and accessible concepts), specific education (e.g. from grade school through post-secondary), investment (e.g. to spur innovation) and governance (e.g. to ensure its effective use).
But how does one begin? The value derived from digital data occurs within the context of how it is used. To start with, try to use data to understand what is going on in one’s area of interest (i.e. the description). Then use data to ask, why is that happening? (i.e. the diagnoses). And then ask, how are we going to respond to that problem or opportunity? If you get good at the above, you can begin to predict what might happen.