Skip to main content

Whether we talk about startups or about new technology, the hype gets built because of a few big success stories. Something similar is happening with data science. We all know that data is at the heart of any company’s digitalization journey. We all know how powerful it is at providing insights which improve your operations, make customers happy, decide new products or find newer business avenues. Many people are jumping right into it. Although their intent behind starting data science projects is right, several reports show that more than 75% data science projects fail to deliver the desired outcome. I have been watching this space for several years now and have worked on several successful and a few not so successful projects. I can think of a few reasons why this could be happening. Let’s analyze and understand how you can be a part of the few successful ones.

There are several factors that play a role. Unrealistic expectations, insufficient budgets and data governance issues are definitely some of the key aspects. Some of these issues can be avoided if one runs pilot projects to test the waters. You should invest more and go full force only when you see the potential.

The bigger issues revolve around the right team. As you will find soon, you truly need a multidisciplinary team. The nature of the beast calls for a team with different skillsets. As it starts from business and ends with RoI for business, you will need a business expert who brings in domain expertise. Based on whether an enterprise is trying to extract insights from data or an ISV is trying to infuse data science, the entire approach, choice of tools, technologies, architecture and data strategy changes. You need a seasoned chief data scientist for data science consulting. You need data engineers and software engineers to put the IT elements and the code in place. Large volumes of data must be stored and processed really fast while maintaining its security & integrity. Then in some cases, you may need digital signal processing experts to extract right features from the signals. Then come data scientists. They can decide which features of the data to use, understand patterns, take decisions on picking the right algorithms, choose machine learning models, train the models and guide the teams. They need to not only understand all the aspects of the data science process but also the business aspects.

Then comes the issue of MLOps. The models developed need to be deployed in real world scenarios. This is again a specialized field.

As you can see, this team needs to be multidisciplinary. Just because you are a data engineer, doesn’t mean it is fine to not understand ML or the business scenario. Yes, you need not be an expert in it, but you do need to have a broader understanding. The same logic is applicable to every stakeholder. It is hard to build team with deep expertise in the right areas and good understanding in others. There is a common perception that data engineers can be upskilled to become data scientists. While there is a merit in this argument, there is a difference between a mind-set of a scientist and an engineer. Unless the mind-set of performing experiments is adopted, it is an uphill task.

In this field, there is much lesser supply than demand. Subsequently, such a team with niche skills comes at a cost. It took us more than two and a half years to recruit the right candidates and get them trained in these skillsets. What is the end result? Our success ratio hovers above 90%. We have the right team with redundancy in skills as well.

Then, we do something that is obvious. We start small and we make the smaller project successful. With the success, comes success. I mean money. Seeing is believing; Upper management tends to put in more money when they see potential impact. This also gives the team confidence that they are on the right track. Most importantly, the goals become more realistic. There is a reason why data science is at the top of Gartner hype cycle.

But, to get this smaller project to be successful, you need the right team. Data science is the right thing to do, but building teams should not be the first thing on your mind. It is natural to think that you could build your team first and use a consultant as needed. But that seldom works for companies because data science problems are multi-dimensional problems. Outsource the work to the right partner. You should stay involved as a business or domain expert. This not only increases your chances of success but reduces your fixed costs. And during the first run, you learn what kind of a team you need. Slowly, you can build your own team as your process evolves.

Simply collecting data and dash-boarding it is passé. The real power comes when you draw insights using data science. Let the machines do the hard work while you spend your time solving problems of a higher order.


Srinath Krishnamurthy | Principal Architect (TOGAF 9 certified)

  • 15+ years of experience
  • TOGAF 9 certified
  • Expertise in data warehouses, data architecture and scalable cloud-native architectures
Close Menu