Independent software vendors (ISVs) need to
urgently rethink how they distribute their applications to their customers'
self-managed environments (i.e. on-premises, VPC, or air-gapped environments).
Without more useful metrics to give them insights to their actual holistic
performance, they have hidden problems they can't quantify or address, which
threaten their success.
Today many vendors are realizing:
- They don't measure enough things.
- They can't measure more because it's too hard to get data.
- They don't have the data to create alignment with all stakeholders.
Real context on commercial
software distribution
Traditional on-premises software still
dominates the landscape of enterprise IT spending on 3rd-party, commercial
applications. While SaaS-only vendors striving for market share often reference
these facts like "it's still the first inning for SaaS" or "the
legacy vendors that need replacing" the point is that much of IT spend is
still going toward self-hosted software and infrastructure. In fact, 75% of
ISVs indicate their on-premises software business has grown over the last 5
years, per a study by Dimensional Research. Unfortunately,
the narrative often doesn't follow this reality and the teams responsible for
enterprise software distribution are dramatically under-staffed and
under-resourced. They're often so underwater on the work of distributing their
application to customer environments that they rarely have a moment to think
about what metrics they should be measuring and what "good" really
looks like.
Figure 1. Dimensional Research found that 75%
of ISVs surveyed reported growth in their on-premises software business over
the past five years.
Getting the right metrics to make
good decisions
While application observability (as in open
source projects like Prometheus and Grafana) is getting better at reporting
specific resource telemetry for operations, it doesn't address the right
questions to make business decisions. To date, very few vendors can explicitly
or confidently answer fundamental questions about their software distribution
outcomes for topics like:
- How long
does it take a customer to successfully install our app? Dimensional
Research found that 84% of companies require a week or longer to install
and configure software. The answer impacts time-to-value, and takes field
engineers away from selling.
- How often
does an install or upgrade fail? This impacts satisfaction and generates
support calls, often requiring escalations for engineering help. 72% of
companies need to resolve four or more support escalations each month for
on-premises software.
- What
specific versions have been adopted for active use by which customers?. If
customers are running old versions, they will have an increased risk of
running into known CVEs, encounter more support challenges, and will
experience delays in the value add of new feature capabilities.
- What is the
uptime of my app in various customer environments? This metric indicates
product reliability and resilience (or fragility).
- How do these
metrics vary across instances, apps, customer accounts, or all usage?
Knowing this helps the vendor understand where they need to focus their attention.
Figure 2. A map of actionable software
distribution metrics, which can each be measured per instance, per customer,
per release, per channel, and per application.
Not having data-driven answers, or only having
anecdotal answers, greatly impairs the ability of software vendors to have a
unified understanding and to make strategic improvements. This leaves ISVs
unable to address the efficiency of their teams, processes, or product, and
likely leads to dissatisfied customers who churn. Yet merely conducting
painstakingly researched retrospectives on individual issues doesn't scale and
can also be subject to recency and severity bias. For example, when the VP of
Operations of an analytics platform was recently asked about his company's
upgrade success rate, he said "that's something we don't measure today, but you
always hear about the bad ones and not so much about the good ones."
Why this challenge is becoming
critical now
In a good economy, the focus is on growth, not
efficiency. In a volatile and ultra-lean economy, commercial software vendors
need to focus on two goals: to be as efficient as possible in their operations,
and to continuously delight their customers. Efficiency matters because budgets
are tight and people are stretched thin. Customer delight matters because every
purchase or renewal decision is under intense scrutiny now. Failing to address
these issues will impact the ISVs revenue and profitability (measured in the
time to close opportunities, customer acquisition costs, etc.), and even
viability of smaller companies as raising additional rounds of venture capital
now is very costly, if it's even possible at all.
Too many stakeholders, too little
visibility
A harsh complication is that commercial
3rd-party software distribution has a lot of diverse stakeholders, both at the
vendor and customer level. The lifecycle of software has many stages, from
planning to development to delivery to support to updates. These various stages
are handled by engineers, product managers, field engineers, and support teams
within the vendor company. Meanwhile, all purchase decisions are being
evaluated with intense scrutiny by end users, IT admins, infrastructure,
operations, and security teams among their enterprise customers. Each distinct
group has different priorities, limited visibility, and different ideas of what
success looks like, and without clear data they can't even align on what
problems need to be addressed. As Redmonk analyst Rachel Stephens notes, "There was this big disconnect between what
everyone was doing, people didn't have a shared understanding of pain points,
and didn't have a shared understanding of ownership."
A new hope for actionable
insights
Recently there has been increased discussion
on how software vendors can stay alive or even thrive and take market share in
a turbulent macro-economy.
A hot new trend is beginning to emerge as
demonstrated by diverse companies such as Replicated (with the first open software
distribution platform designed for ISVs), GitLab (who is building their own internal
practices for measuring metrics), and Palantir's
Apollo (which is tied to their other offerings.) For example, Senior
Product Manager Dilan Orrino recently said, "Our main
performance indicator that the distribution team is working against is our
upgrade rate [...] which is the percentage of instances on the latest three
versions of GitLab."
What these all have in common is they are
modern, experimental approaches to tackle both Day 1 (installation) and Day 2
(ongoing operations) for the commercial software industry.
With new capabilities to better understand the
metrics which matter most in delivering commercial apps to their customers,
ISVs can now begin to make insightful, focused improvements to their
development, packaging, distribution, and support operations. They can start
cross-functional discussions about which metrics matter most, then set goals,
measure performance, and make adjustments to improve the overall health of
their business.
To
learn more, visit www.replicated.com
##
To learn more about the transformative nature of cloud native applications and open source software, join us at KubeCon + CloudNativeCon Europe 2023, hosted by the Cloud Native Computing Foundation, which takes place from April 18-21.
ABOUT THE AUTHOR
Nikki Rouda VP, Marketing, Replicated

Nikki Rouda has deep experience leading enterprise modern apps, containers, big data, analytics, and data center infrastructure initiatives. Before his current role at Replicated, Nikki held senior positions at Amazon Web Services (AWS), Cloudera, Enterprise Strategy Group (ESG), Riverbed, and Veritas. Nikki has an MBA from Cambridge's Judge Business School, and a ScB in geophysics from Brown University.