Independent software vendors (ISVs) need to urgently rethink how they distribute their applications to their customers' self-managed environments (i.e. on-premises, VPC, or air-gapped environments). Without more useful metrics to give them insights to their actual holistic performance, they have hidden problems they can't quantify or address, which threaten their success.

Today many vendors are realizing:

1. They don't measure enough things.
2. They can't measure more because it's too hard to get data.
3. They don't have the data to create alignment with all stakeholders.

Real context on commercial software distribution

Traditional on-premises software still dominates the landscape of enterprise IT spending on 3rd-party, commercial applications. While SaaS-only vendors striving for market share often reference these facts like "it's still the first inning for SaaS" or "the legacy vendors that need replacing" the point is that much of IT spend is still going toward self-hosted software and infrastructure. In fact, 75% of ISVs indicate their on-premises software business has grown over the last 5 years, per a study by Dimensional Research. Unfortunately, the narrative often doesn't follow this reality and the teams responsible for enterprise software distribution are dramatically under-staffed and under-resourced. They're often so underwater on the work of distributing their application to customer environments that they rarely have a moment to think about what metrics they should be measuring and what "good" really looks like.

Figure 1. Dimensional Research found that 75% of ISVs surveyed reported growth in their on-premises software business over the past five years.

Getting the right metrics to make good decisions

While application observability (as in open source projects like Prometheus and Grafana) is getting better at reporting specific resource telemetry for operations, it doesn't address the right questions to make business decisions. To date, very few vendors can explicitly or confidently answer fundamental questions about their software distribution outcomes for topics like:

1. How long does it take a customer to successfully install our app? Dimensional Research found that 84% of companies require a week or longer to install and configure software. The answer impacts time-to-value, and takes field engineers away from selling.
2. How often does an install or upgrade fail? This impacts satisfaction and generates support calls, often requiring escalations for engineering help. 72% of companies need to resolve four or more support escalations each month for on-premises software.
3. What specific versions have been adopted for active use by which customers?. If customers are running old versions, they will have an increased risk of running into known CVEs, encounter more support challenges, and will experience delays in the value add of new feature capabilities.
4. What is the uptime of my app in various customer environments? This metric indicates product reliability and resilience (or fragility).
5. How do these metrics vary across instances, apps, customer accounts, or all usage? Knowing this helps the vendor understand where they need to focus their attention.
   

Figure 2. A map of actionable software distribution metrics, which can each be measured per instance, per customer, per release, per channel, and per application.

Not having data-driven answers, or only having anecdotal answers, greatly impairs the ability of software vendors to have a unified understanding and to make strategic improvements. This leaves ISVs unable to address the efficiency of their teams, processes, or product, and likely leads to dissatisfied customers who churn. Yet merely conducting painstakingly researched retrospectives on individual issues doesn't scale and can also be subject to recency and severity bias. For example, when the VP of Operations of an analytics platform was recently asked about his company's upgrade success rate, he said "that's something we don't measure today, but you always hear about the bad ones and not so much about the good ones."

Why this challenge is becoming critical now

In a good economy, the focus is on growth, not efficiency. In a volatile and ultra-lean economy, commercial software vendors need to focus on two goals: to be as efficient as possible in their operations, and to continuously delight their customers. Efficiency matters because budgets are tight and people are stretched thin. Customer delight matters because every purchase or renewal decision is under intense scrutiny now. Failing to address these issues will impact the ISVs revenue and profitability (measured in the time to close opportunities, customer acquisition costs, etc.), and even viability of smaller companies as raising additional rounds of venture capital now is very costly, if it's even possible at all.

Too many stakeholders, too little visibility

A harsh complication is that commercial 3rd-party software distribution has a lot of diverse stakeholders, both at the vendor and customer level. The lifecycle of software has many stages, from planning to development to delivery to support to updates. These various stages are handled by engineers, product managers, field engineers, and support teams within the vendor company. Meanwhile, all purchase decisions are being evaluated with intense scrutiny by end users, IT admins, infrastructure, operations, and security teams among their enterprise customers. Each distinct group has different priorities, limited visibility, and different ideas of what success looks like, and without clear data they can't even align on what problems need to be addressed. As Redmonk analyst Rachel Stephens notes, "There was this big disconnect between what everyone was doing, people didn't have a shared understanding of pain points, and didn't have a shared understanding of ownership."

A new hope for actionable insights

Recently there has been increased discussion on how software vendors can stay alive or even thrive and take market share in a turbulent macro-economy.

A hot new trend is beginning to emerge as demonstrated by diverse companies such as Replicated (with the first open software distribution platform designed for ISVs), GitLab (who is building their own internal practices for measuring metrics), and Palantir's Apollo (which is tied to their other offerings.) For example, Senior Product Manager Dilan Orrino recently said, "Our main performance indicator that the distribution team is working against is our upgrade rate [...] which is the percentage of instances on the latest three versions of GitLab."

What these all have in common is they are modern, experimental approaches to tackle both Day 1 (installation) and Day 2 (ongoing operations) for the commercial software industry.

With new capabilities to better understand the metrics which matter most in delivering commercial apps to their customers, ISVs can now begin to make insightful, focused improvements to their development, packaging, distribution, and support operations. They can start cross-functional discussions about which metrics matter most, then set goals, measure performance, and make adjustments to improve the overall health of their business.

To learn more, visit www.replicated.com

##

To learn more about the transformative nature of cloud native applications and open source software, join us at KubeCon + CloudNativeCon Europe 2023,  hosted by the Cloud Native Computing Foundation, which takes place from April 18-21.

ABOUT THE AUTHOR

Nikki Rouda VP, Marketing, Replicated

Nikki Rouda has deep experience leading enterprise modern apps, containers, big data, analytics, and data center infrastructure initiatives. Before his current role at Replicated, Nikki held senior positions at Amazon Web Services (AWS), Cloudera, Enterprise Strategy Group (ESG), Riverbed, and Veritas. Nikki has an MBA from Cambridge's Judge Business School, and a ScB in geophysics from Brown University.