Without a system of cost controls in place to incentivize efficiency, application owners within your organization might be treating Kubernetes cloud infrastructure resources as an open buffet. They consume and consume without much concern for the costs they're racking up behind the scenes. And why not? Someone will pay the bill for expensive overprovisioning of underutilized resources.

The issue grows as Kubernetes infrastructure scales. To combat it, some organizations implement a chargeback cost management model that brings both transparency and accountability to application owners by requiring them to foot the bill for the resources they use. However, this is often easier said than done because of the complexity of applying a chargeback model to Kubernetes use. The multi-tenant platform hosts an endless string of containers with brief existences and with no internal cost tracking features. To solve this challenge and make chargebacks work, organizations must be sure to provide teams with granular cost insights.

Four pillars of an effective chargeback program

Putting technology and cost management mechanisms aside, a successful chargeback program is built on trust. Software teams that own responsibility for applications must agree that the costs they must cover from their own budgets are correct and acceptable. Ultimately, building a chargeback program isn't about changing how your organization pays its bills or arranges its budgets. It's about creating a culture that values infrastructure spending efficiency at every level.

These four pillars enable the trust necessary to reach that goal. A chargeback program should be:

• Accurate. Avoid estimates. Deliver precise-to-the-penny cost reporting across all Kubernetes infrastructure cost categories; without that detail, engineering teams won't trust the data and won't be able to act on it.
• Timely. Cost reporting is knowledge in application owners' hands. The timelier the reporting, the faster teams can address provisioning issues. In this way, a chargeback program can transform stressful monthly billing surprises into swiftly identified issues that might be resolved in a matter of hours.
• Fair. Implementing solutions addressing complex cost allocation issues requires understanding, collaboration, and fair judgment. For example, if a particular cluster's tenants require a premium support service, it may be appropriate to share that cost of doing business across multiple teams. Stakeholders' confidence in-and support for-the chargeback program can hinge on getting these judgments right.
• Complete. Chargeback reports must fully account for all Kubernetes costs. That includes shared Kubernetes resource costs across cluster tenants, as well as charges external to the cluster such as volume storage, blob storage, backup, and data transfer or networking expenses.
  

Implementing chargebacks

Organizations should carefully ramp up to a complete chargeback strategy by first achieving cost allocation aligned within the four pillars, and then introducing showback reports that provide visibility into spending without attaching responsibility for those costs. Once teams have time to gain trust with the cost data, optimize for cost efficiency guided by that data, and acclimate to the budgeting process, it's then time to move ahead with the chargeback program and enforcement.

First stage: Cost allocation

Use Kubernetes namespaces, services, and labels to delineate cost responsibilities, assigning them to each application team or cost center. The three types of Kubernetes costs to account for are: the "in-cluster costs" of cluster nodes (including CPU, memory, and storage), "external costs" of services dedicated to a particular Kubernetes namespace (such as blob storage), and "shared costs" shared by all Kubernetes namespaces (such as monitoring tools). Accurate, timely, fair, and complete cost allocation means closely tracking resource usage second by second and providing teams with real-time insights into the costs they incur.

Second stage: Provide showback reports

Showback reports allow teams and organizations to take a dry run at practicing a chargeback strategy, ironing out any wrinkles before adding real stakes. By providing each team with the delineated costs they will be responsible for, team leaders and other stakeholders can first ensure that cost allocation methods do in fact adhere to the four pillars. Teams can also then investigate any overprovisioning issues that these reports bring to their attention, without immediately facing crisis mitigation scenarios where time is money out of their budgets. A few months of showback reporting will give teams and the organization, as a whole, time to develop trust in the system, mature cost controls and monitoring practices, and foster a culture of responsibility before real consequences come into play.

Third stage: Chargeback program launch and enforcement

Building on a foundation of appropriate and trusted cost allocation and practice with showbacks,

introducing chargebacks puts muscle behind your accountability measures. Application owners, the finance team, and other stakeholders should be well prepared for the transition at this point, and they should have a clear and realistic idea about what to expect from a budgeting perspective going forward. By now, your organization should be well-versed in Kubernetes infrastructure consumption, and have developed an intrinsic cultural dedication to keeping costs in check.

Introduce Kubernetes cost visibility to rein in expenses at scale

The simple act of making application teams aware of their Kubernetes cost responsibilities can have a profound impact on the care with which those teams approach provisioning. Adding chargebacks makes sure everyone takes those costs seriously. When organizations take the time to implement a fair and thoughtful Kubernetes chargeback strategy where each team directly benefits from the efficiency they achieve, successful results and savings follow.

##

ABOUT THE AUTHOR

Rob Faraj is a co-founder at Kubecost, a company focused on Kubernetes cost monitoring and management. Previously, he held director roles at Duda, Automox, and Weebly.