Sparsity Management: What is it and why does it matter?

Context

Customers want flexibility and agility - they expect their systems to adapt to changing business needs and processes, not the other way around. We often see customers sacrificing on flexibility due to system limitations and constraints which ultimately creates frustration among end-users. One of the topics that we see come up most often as it relates to planning systems (EPM / CPM) is data sparsity - which on the surface, might not sound like an exciting topic but I assure you this might be one of the most important factors to consider when evaluating your next EPM system. Let’s explore what sparsity is and why it matters.

Definition and Example

A dataset is considered to be sparse if a large portion of it contains missing / unpopulated values. This becomes quickly relevant when we combine this topic with multi-dimensionality. In case this is becoming too complex, let’s break it down further with an example.

Let’s say your Finance team wants to analyze the Income Statement to understand actual / forecasted expenses compared to the budget. In order to do this across the business, they will need to see various cuts of data to understand who is spending the $s, where, how, when etc. Each of these cuts forms a dimension. This will enable the Finance team to drill down into as much detail as possible but also roll that information up to summarize. Here are some dimensions that might be required as part of this exercise:

1. Cost Centers: 100

2. Entities / Subsidiaries: 5

3. Channels: 50

4. GL Accounts: 100

5. Vendors: 1000

6. Months: 36 (2 historical years + 1 forecast)

7. Scenarios: 5

Note: Each business looks different and the dimensions needed to be reported will vary

If we put all of this together in a report and aggregate data across each of these dimensions, the result is around 450 Billion cells! However, a large portion of the intersections within this multi-dimensional report may not have any values which makes it extremely sparse. At the same time, having all of these dimensions in one report enables Finance to quickly analyze the information they need. This is why sparsity is relevant - the ability of your EPM system to manage sparsity well determines how many dimensions can be added. This can make the difference between whether users get what they need or if they are required to change their requirements and sacrifice on flexibility.

Treatment of Sparsity in EPM systems

In some legacy EPM systems, each cell contributes to the overall model size - meaning that the 450 Billion cells from our example above would actually occupy space even though most of the dataset might be sparse. Additionally, these legacy EPM systems are also running calculations across each of these intersections which further causes performance degradation. As you can imagine, this is not efficient and often leads to system admins incorporating workarounds into their model which does not fix the sparsity issue permanently and instead causes the accumulation of tech debt over time. Add to this the fact that these systems might even charge you for the extra space being occupied as a result of sparsity.

It can be argued that users might not need all of these dimensions to perform the required analysis but having optionality is key to ensuring a good user experience. In fact, in some cases, having more dimensions can actually reveal new insights which would not have been possible without having the right EPM system in place that can effectively tackle sparsity. This is why sparsity management can be the achilles heel of an EPM system architecture - it is a fundamental requirement and can make the difference between a successful and subpar implementation. 

Sparsity Management in Pigment

On the other hand, Pigment is able to manage sparsity extremely well by treating sparse intersections as blank or null effectively cutting down the number of cells that need to be calculated. This enables users to create reports with ease without having to sacrifice on business requirements. Pigment's calculation engine enables users to add as many dimensions as required giving it unparalleled flexibility compared to other EPM solutions. 

With Pigment, sparsity is not something your team needs to worry about. This can enable you to:

Generate new insights by looking at your data in a way that was not possible before

Create limitless multidimensional reports with ease and flexibility

Elevate your scenario planning by creating as many scenarios as needed across your multidimensional model

Minimize technical debt and workarounds

Check out some of our other blogs to see why Pigment might be the right choice for your organization’s planning needs. Interested in learning more? Chat with us!