Thesis
With the rise of enterprise machine learning applications, there’s been an increase in demand for MLOps. With this trend continuing, there has surfaced a need for more specialized software built for both managing the flow of machine learning models, as well as monitoring them. Machine learning isn’t simply about building models, it’s an intricate and iterative process that needs building, tweaking, re-training, etc. I believe the industry will slowly move away from end to end software that aims to solve the entire lifecycle of ML and move towards specialized platforms that integrate well with the current tech stack. More specifically, post deployment monitoring tools are gaining popularity as they help bridge the gap between data science and business operation teams.
What is MLOps?
Put simply, machine learning operations (MLOPs) is the process of connecting ML with the practice of continuous development. This takes into account the actions/software used to design, deploy and operationalize machine learning. This is similar to the more mature development operations (DevOps) that aims to shorten the software development life cycle. Cristiano Breuel has an excellent article that further explains the importance of MLOps.
Current State of MLOps
The current state of MLOps is focused on open source projects with Kubeflow, MLFlow and TensorFlow Extended (TFX) all used in industry and has a highly active development community. Though they all have their unique distinctions, they all aim to ultimately enable data scientists to deploy machine learning models at a more efficient rate and scale. For example, Kubeflow’s main functionality is to make deployment on Kubernetes simpler and scalable. Similarly, MLFlow aims to make the ML lifecycle easier by tracking experiments, tracking code, leading to faster model deployment. It’s worth noting that these platforms were developed through open source projects by data scientists and engineers, not necessarily with the goal in mind to accelerate enterprise ML adoption. To ultimately have widespread enterprise adoption, it’s important to recognize the tools that will bring data science teams closer to the business.
To understand where MLOps is going, it’s important to make the parallel to DevOps. Many DevOps tools also started out as open source projects and have continued to stay that way. For example, Kubernetes and Docker are both open source projects that are used in today’s software methodology to scale operations across platforms. Additionally, even with the rise of dominant cloud players like AWS, Azure, and GCP, we have yet to see a true end to end DevOps platform that has become the sole player in the software engineering space. Given the complexities of machine learning, it’s not a stretch to say we could see more specialized software emerge within the ML ecosystem that follows a similar methodology to that of DevOps.
Every enterprise will ultimately need a different ML tech stack. Big tech companies that have the budget to hire data scientists, engineers, and product managers will have the talent to produce more tailored models that can be tuned to the specificity of the project. Additionally, these firms often build out internal ML tools such as Facebook’s FBLearner to scale adoption across all business units. On the other hand, enterprises that don’t have the technical talent may look for more end to end and auto ML platforms that aim to simplify the process. Regardless of the MLOps software that an enterprise ultimately selects, there comes a point where post deployment monitoring becomes important. AI assurance and monitoring tools are needed across all enterprises, and startups are beginning to emerge in this space whose sole mission is to better monitor and manage post production models.
Industry Trends and Outlook
Outlook
According to Cognilytica, the market for MLOps solutions is expected to grow to $4B by 2025. Additionally, IDC recognizes the entire AI and ML industry to be worth around $500 billion by that same timeline; this assumes a five year compounded annual growth rate (CAGR) of 17.5%. As the industry continues to accelerate YoY, the rise of applications within the industry will also continue to develop and mature. MLOps will help facilitate enterprise ML adoption, and will ultimately help companies ship faster, better and more scalable models.
Industry trends
Algorithmia 2021 Enterprise AI Report
- 76% of organizations say they prioritize AI/ML over other IT initiatives.
- 64% say the priority of AI/ML has increased relative to other IT initiatives in the last 12 months.
- 85% of machine learning models never make it to production.
- 83% of organizations have increased AI/ML budgets year-on-year
- Governance is by far the top challenge for AI/ML deployment, with more than half of all organizations ranking it as a concern.
HP: Operationalize Machine Learning
- Only 6% of surveyed companies believe their MLOps capabilities are mature or very mature. Firms’ inability to operationalize ML is prohibiting them from scaling machine learning across the business.
It’s no secret that big tech companies have been implementing ML solutions for the past decade. Ad targeting and product recommendations are fairly common place among the big tech giants and has been implemented for quite some time now. That being said, a study from Wired showed that in 2020 only 8.9% of surveyed enterprises were using any sort of AI. Despite the large amounts of money poured into this space, adoption has still been difficult at the macro level.
Hiring trends
2021 Data Science Hiring Survey
- Growth in data science interviews plateaued in 2020. Data science interviews only grew by 10% after previously growing by 80% year over year.
- Data engineering specific interviews increased by 40% in the past year. The second fastest position growth within data science roles went to business and data analysts which increased by 20%.
Companies are understanding the importance of data engineering in the overall data science workflow. The rise in data engineering positions is a testament to the focus surrounding the entire workflow of machine learning and not only focusing on model building. As mentioned previously, deploying enterprise models requires a lot more than just the model building process, and data engineers do a lot of the upfront work to build data pipelines that make data scientist’s work more efficient. I believe similar trends will occur in the post model deployment area, focusing on model governance with data product managers taking a big chunk of this day to day work. Similar to DevOps, industry standards will converge on a few software's that will help facilitate each section in the ML lifecycle.
Superwise.ai
Superwise.ai is an AI assurance platform that provides insights to machine learning models. They currently focus on post production monitoring and have recently raised a $4.3M seed round led by Capri Ventures and F2 Capital. The firm was founded in early 2019 in Tel Aviv, Israel. Tel Aviv is a growing hotspot for tech, as Israeli startups have raised a total of $9.93 billion in 2020, which is up 27% YoY.
Use Case
Let’s say that a company ultimately decides to invest in it’s data infrastructure and pipelines. They then hire a team of data scientists and engineers. At some point, maybe a month later, but more likely a year later, this team ships a model into production. The key question that now arises is, now what? The entire process of shipping a model is extremely difficult. If 85% of all machine learning models never make it into production, then it becomes ever more important to have good governance and assurance practices on the models that are shipped. This is exactly what Superwise.ai aims to do. They are an all in one machine learning monitoring platform that provides insight into what is more often than not a black box algorithm. Through their solution, a broader group of stakeholders will have greater visibility into production models which will ultimately help scale the company’s AI operations and tie it back to business impacts.
All Encompassing
It’s important to remember how each model is tied back to the business’s goals. Throughout the tech world, companies have been adopting objectives and key results (OKRs) as a goal setting framework to highlight the fact software ultimately drives business value. What Superwise.ai understands is that post production model assurance isn’t just for the data science team. The dashboard and monitoring tools should be comprehensible across other teams and stakeholders. Some other important teams that may want to understand the current model performance could be marketing stakeholders, internal product managers, and even sales teams to better understand what they may be pitching. They solve this issue by its distinct easy to understand monitoring features and clean UI.
- Model health analyzer (comparison versus regular model KPIs)
- Smart alerts (integration with slack)
- Automated weak spot alerts (detects deviations from all subchannel outputs)
- Rich model analytics (compare model performance against other iterations of it)
- Model outputs (understand the why behind a certain model decision)
The case for AI monitoring
Explainability is one of the main factors that will ultimately drive widespread enterprise adoption. There are a multitude of variables that need to be considered once a model is deployed, and many of these factors can have immediate business ramifications. Monitoring is a step in the overall ML lifecycle that is often overlooked, but that doesn’t make it any less important.
Superwise.ai isn’t the only startup in this space as the growth in the sector has created similar solutions in this space. Perhaps most prominent is Fiddler.ai that aims to make models more “explainable” in it’s post production procedures. To this date they have raised a total of $13.2M from the Alexa Fund, Lockheed Martin Ventures, Lightspeed and Lux. Both Fiddler and Superwise integrate well with existing platforms like AWS Sage maker, Kubeflow, H20.ai, and others. However, as seen in the industry map, this isn’t necessarily a winner takes all market as ML is highly tailored to a business problem. It’s still unclear which platform will ultimately emerge as the industry standard, but both ventures are focusing on software that is decoupling the latter stages of the lifecycle to ultimately drive explainability, transparency and impact across enterprises.
Final Thoughts
Venture capital firms have been heavily investing in the overall AI and ML space. In fact in 2020 alone, $8.2B invested in this space bringing the total AI startup funding to date to $61.2B. Given the sheer size of the market, when investing in ML startups, it’s crucial to recognize the changing landscape of MLOps. By focusing on AI monitoring tools, I am confident that the company’s product will continue to strive to solve this problem and iterate as the ML industry matures. Think of it as a parallel to PagerDuty and Datadog’s success in the software space. Who knows, maybe end to end ML and auto ML platforms become the norm and software like Fiddler and Superwise die out. However, I like to think of the future of ML as a sum of parts, where each part of MLOps is done through different software's that ultimately drives business value.
