Managing applied machine learning organizations – the Magic Quadrant

Introduction

The Gartner Magic Quadrant is a research methodology that provides a graphical competitive positioning of four types of technology providers in fast-growing markets: Leaders, Visionaries, Niche Players and Challengers. I propose a similar framework for applied machine learning organizations. The framework is based on two dimensions: comprehensive modeling and iteration velocity.

The Dimensions

1. Comprehensive modeling is the ability to define problems, utilize data, label resources, engineer features, and apply holistic modeling approaches to completely solve a business problem.
2. Iteration velocity is the ability for an organization to execute end-to-end machine learning processes quickly and efficiently in order to improve the quality of the models and business outcomes.

The framework is based on the idea that the best applied machine learning organizations are those that have a mature model development process and can iterate quickly in addition to a modeling approach that is comprehensive and holistic.

The Quadrants

The four quadrants are:

1. Leaders – These are the organizations that currently lead the market and have a competitive advantage by utilizing new technologies and modeling approaches to evolve their product offering in order to stay ahead of the competition.
2. Rising Stars – These are organizations with a high development velocity, and although they may not have the most complete solution they are quickly iterating and improving their models in order to catch up to the leaders.
3. Legacy Players – These are the legacy enterprises that have been around for a long time and have chipped away at the problem for a long time. They have a good solution but are not iterating quickly because of technical debt and organizational challenges. They might have the market share but are at risk of losing it to new entrants with better technology and processes. Often times it’s not money that is the problem but the organizational structure and culture.
4. Low Performers – You often see low performers at organizations where a leadership team “wants to do AI” but without vision or an execution strategy. You might see endless POCs (proof of concepts) and a lot of churn but limited business impact. Projects can take quarters or even years to complete and the results are often not what was expected or needed.

Balancing solving the problem vs investing in the process

As a leader in this space it is important to balance the need to solve the problem with the need to invest in the process. The best organizations are those that can do both. Lets looks at some management practices that can help you achieve this balance. Something to keep in mind is that if you can iterate the quickest you can eventually solve the problem and become a leader in the space. But if you are not iterating quickly you will fall behind and become a legacy player. Even if you have a good solution today, if you are not iterating quickly you will eventually fall behind.

Swim lanes

The first thing to do is to split the organization into two or more swim lanes. One lane is focused data science and modeling. The other lane is focused on MLOps, platform and tooling. Depending on the maturity organization and the specific details of the problem you are solving you might have a different split. But the idea is to strategically split the investment in the problem and the process. This helps articulate a clear vision and strategy for the organization and is a resource when communicating with critical stakeholders.

Checklist for prioritizing projects

When you are deciding what projects to work on it is important to have a checklist that helps you prioritize projects. The checklist should include things like:

1. What is the certainty this solution will work?
2. What is the business impact of this solution?
3. How long will it take to complete the project?

Consider projects based on ROI (return on investment) and have a bias for short and high certainty projects. This will help you iterate quickly and build momentum in the organization. If a project has low certainty and a long time to complete then think about how you can break it down into smaller projects or invest in tooling and processes that will help you iterate of these types of projects more quickly in the future.

Machine Learning retrospectives

after deploying projects it is important to have a retrospective. This is a meeting where the team gets together and discusses what went well and what didn’t go well. The goal is to learn from the project and improve the process. This is a great way to build a culture of continuous improvement and to make sure that the organization is iterating and improving.

Machine Learning modeling bottleneck analysis

as part of the retrospective it is important to understand where the bottlenecks are in the process. you can do this by having the lead engineer/scientist on the project create a spreadsheet or other document that outlines the complete time investment in the project. Go over this document in the retrospective with both the data science and MLOps teams and discuss how you can improve the process or introduce tooling to help you iterate more quickly in the future.

Conclusion

In conclusion, the best applied machine learning organizations are those that can balance solving the problem with investing in the process. The Magic Quadrant framework is a useful tool for understanding where your organization is and what you need to do to improve. Some management practices that can help you achieve this balance include splitting the organization into swim lanes, having a checklist for prioritizing projects, having machine learning retrospectives, and doing machine learning modeling bottleneck analysis. By following these practices you can build a culture of continuous improvement and iterate quickly in order to solve the problem and become a leader in the space.

Thanks for reading. I hope you found this article useful. If you have any questions or comments please leave them below.

You can also find me on linkedin at linkedin.com/in/mordechai-w.