White Projects

How to Build a Data-Driven Culture in Your Organization

whitepro_wp_admin — Thu, 22 Aug 2024 14:17:43 +0000

How to Build a Data-Driven Culture in Your Organization

Data is the new oil. It’s valuable, but if unrefined it cannot really be used. It has to be processed and analyzed to have value.

Clive Humby

In today's competitive landscape, the ability to leverage data effectively is more than just an advantage—it's a necessity. Building a data-driven culture within your organization can transform decision-making, streamline operations, and drive strategic growth. However, fostering a culture that prioritizes data-driven decision-making at all levels requires a deliberate and strategic approach. Here are essential strategies to help you cultivate a data-centric environment in your organization.

Lead by Example

The foundation of any successful data-driven culture starts at the top. Leadership must champion the use of data, demonstrating a commitment to evidence-based decision-making. Executives should use data to drive strategic decisions, share insights transparently, and celebrate data-driven successes. As the renowned data scientist Dr. DJ Patil aptly puts it, "Data is a precious thing and will last longer than the systems themselves." When leaders model data-centric behavior, they set a precedent that encourages employees throughout the organization to follow suit.

Invest in Data Literacy and Training

A data-driven culture thrives on the ability of employees to understand and utilize data effectively. Investing in comprehensive data literacy programs is crucial. Offer training sessions, workshops, and resources that help employees at all levels develop their data skills. Ensure that team members know how to interpret data, use analytics tools, and apply insights to their specific roles. The more proficient your team becomes in handling data, the more integrated data-driven decision-making will be in your organizational processes.

Foster Cross-Departmental Collaboration

Data should be seen as a shared resource, not confined to isolated departments. Encourage collaboration between teams to break down data silos and promote a holistic view of the organization’s information. Implement tools and platforms that facilitate data sharing and ensure that departments have access to relevant data from other areas. This cross-departmental approach enhances data visibility and enables more informed, collaborative decision-making.

Establish Clear Data Governance Policies

For data-driven decision-making to be effective, it’s essential to have robust data governance policies in place. Define clear guidelines for data quality, accessibility, and security. Implement protocols for data management, ensuring that data is accurate, up-to-date, and accessible to those who need it. Good data governance not only supports reliable decision-making but also builds trust in the data among employees.

Encourage a Culture of Continuous Improvement

A data-driven culture is one that evolves and adapts. Foster a mindset of continuous improvement where data is used not just for reporting but as a tool for ongoing optimization. Encourage employees to experiment with data-driven strategies, analyze results, and iterate based on insights. Celebrate innovations and learn from failures to refine approaches and enhance overall performance.

Conclusion

Building a data-driven culture is a strategic endeavor that involves commitment from all levels of the organization. By leading by example, investing in data literacy, fostering collaboration, establishing solid governance, and promoting continuous improvement, you create an environment where data is seamlessly integrated into decision-making processes. Embrace these strategies, and you’ll unlock the full potential of your organization’s data, driving better decisions, greater efficiency, and sustained success.

The Art of Data-Driven Decision Making

whitepro_wp_admin — Tue, 26 Mar 2024 13:28:50 +0000

The Art of Data-Driven Decision Making

Introduction

In today's fast-paced digital landscape, data is no longer just a buzzword; it's a valuable asset that can drive businesses to success. Data-driven decision making has emerged as a crucial strategy for organizations aiming to stay competitive and relevant in their respective industries. In this article, we'll explore the significance of data-driven decision making, its benefits, challenges, and best practices for implementation.

Understanding the Foundation

But first, let us get the words straight. Data is not meant to contain decisions directly, however when properly transformed, explained and shown – it’s very good at making business much easier to understand – and this understanding is then driving us toward “good” decisions. So when using the word “data-driven decisions” we’re aiming exactly at this understanding to be achieved.

At the core of data-driven decision making lies the ability to leverage data to inform strategic choices. This process involves collecting, analyzing, and interpreting relevant data to gain insights into various aspects of business operations. Whether it's customer behavior, market trends, or operational efficiency, data provides valuable insights that can guide decision-making processes.

The Benefits of Data-Driven Decision Making

One of the primary advantages of data-driven decision making is its ability to minimize guesswork and uncertainty. By basing decisions on factual data rather than intuition or assumptions, organizations can mitigate risks and make more informed choices. Additionally, data-driven approaches enable businesses to identify opportunities for growth, optimize processes, and enhance overall performance.

Overcoming Challenges

Despite its numerous benefits, implementing a data-driven decision-making culture comes with its challenges. A common obstacle lies in the availability and/or quality of data. Organizations may struggle to access relevant data or encounter issues related to data accuracy and consistency. Additionally, navigating complex data sets and interpreting results can be daunting tasks for individuals without the necessary skills and expertise.

Best Practices for Implementation

To successfully adopt data-driven decision making, organizations must focus on several key principles. First and foremost, they need to prioritize data literacy across all levels of the organization, ensuring that employees understand how and why to collect, analyze, and interpret data effectively. This involves providing comprehensive training programs and resources to empower employees with the skills and knowledge needed to navigate the complexities of data analysis. Additionally, investing in robust data infrastructure and analytics tools is essential for streamlining the data analysis process and ensuring that accurate and timely insights are readily available. By leveraging advanced technologies and modern analytics platforms, organizations can enhance their ability to extract actionable insights from data and drive informed decision-making across the board.

The Role of Business Intelligence in Data-Driven Decision Making

Business intelligence plays a pivotal role in facilitating data-driven decision making within organizations. Business intelligence tools empower organizations to gather, arrange, and analyze extensive datasets sourced from diverse channels, thereby furnishing invaluable insights into key performance indicators (KPIs), trends, and patterns. By leveraging BI platforms, organizations can create intuitive dashboards, reports, and visualizations that empower decision-makers to understand complex data sets and make informed choices quickly. As businesses continue to recognize the importance of data-driven decision making, the role of BI in facilitating access to actionable insights grows in significance.

The Role of People and Management in Data-Driven Decision Making

While advanced technologies and sophisticated analytics tools are essential components of data-driven decision making, the role of people and effective management cannot be overstated. It is crucial for organizations to foster a data-driven culture where employees at all levels understand the value of data and actively participate in the decision-making process. Management plays a critical role in promoting this culture by providing the necessary resources, training, and support to empower employees to leverage data effectively. Additionally, effective communication and collaboration among teams are vital for aligning business objectives with data insights and ensuring that decisions are based on accurate and relevant information. Furthermore, managers need to lead by example, demonstrating a commitment to data-driven practices and encouraging a mindset of continuous learning and improvement. By prioritizing the development of data literacy skills and fostering a collaborative environment, organizations can harness the full potential of data-driven decision making to drive innovation, growth, and competitive advantage.

Qlik Sense's associative engine enables users to explore data dynamically, fostering a more intuitive and creative approach to data analysis. By allowing users to make instant connections between different data points, Qlik Sense encourages a deeper understanding of the data and facilitates more informed decision-making processes.

Conclusion

Data-driven decision making is not just a trend; it's a fundamental strategy for success in today's data-driven world. By embracing data as a strategic asset and leveraging it to inform decisions, organizations can gain a competitive edge, drive innovation, and achieve sustainable growth. As we look to the future, the organizations that prioritize data-driven approaches will undoubtedly thrive in an increasingly complex and dynamic business environment.

Why 2 + 2 is not always = 4

whitepro_wp_admin — Tue, 20 Feb 2024 14:56:00 +0000

Why 2 + 2 is not always = 4

And why it makes perfect sense

"Things are not always what they seem; the first appearance deceives many."

Phaedrus / Plato

The what and why

It may become confusing at first sight, but it's actually pretty common in the realm of Business Intelligence. The whole root cause is coming from how is the data structured - when data is organized in a "flat" table, the aggregation of information across multiple tables can yield unexpected results.

The Complexity Unveiled

Let's consider a simple scenario: a company selling three different products to three different customers. Pretty simple, right?

Initially, this data seems straightforward. We can visualize such sales in multiple ways, lets choose to do a simple pivot table, which shows the Customers in sequential rows, and the Products they purchased in the columns.

Up until this point everything makes perfect sense, and there's no issue whatsoever.

Now lets complicate things ever so slightly

Now to spice things up, we'll add each customer a tag. Each customer can receive multiple tags - similarly as they can purchase from a variety of products.

Complications arise since each customer is assigned multiple tags, introducing a one-to-many relationship between tags and customers. Integrating sales data with customer tags may present a challenge, resulting in duplicated values and unexpected totals—a phenomenon akin to the pitfalls of a Left Join operation in traditional SQL.

As we now have a relation of one to many in the context of Tags to customers - if we would want to connect the sales with the customers and the tags at the same time - we'd occur a problem, as the values would get duplicated if not properly taken care of, as shown in the table below - we can see that the Sales are being repeated, which we may or may not want to happen.

Wait a minute... 25+68+82+68=107?! Why the total at the top doesn't add up to the sum of rows?

This is a regular question from anybody coming from excel, as in the face of these challenges, Qlik's brilliance shines through which may be surprising at first sight. Unlike conventional spreadsheets or query languages, Qlik doesn't naively sum rows to derive totals. Instead, it provides a nuanced understanding of the data's intricacies.

Consider the case of "Starlight Book Emporium," the lone customer with multiple tags. We know that objectively just by attaching multiple Tags to a customer, it doesn't magically increase the amount of business that we have with them (we can only dream for it to be as be simple as that in real life). Qlik recognizes that duplicating sales figures across tags doesn't accurately reflect the true business volume. Instead, it offers various approaches to handle such scenarios, ranging from equitable distribution of sales to selective tagging strategies.

So which is true, the rows, or the summary at the top?

Actually - both are true in their own way, we just need to understand the data and how to look at it.

The beauty of Qlik lies in its adaptability to diverse use cases. Whether it's maintaining data integrity, creating hierarchies, or exploring alternative tagging strategies, Qlik empowers users to tailor their analyses to specific business needs. With each change in dimensions or data drill-down, Qlik dynamically recalculates values, ensuring accuracy and consistency across the board.

The only customer that had multiple tags is "Starlight Book Emporium". We know that it had total Sales of 68, and since this customer is occurring in three different tags, his sale of 68 is repeated in all of the tags, as we didn't "tell" the data model to do something else. Depending on what we want to achieve we could:

Leave it as is
Divide the sales equally between the tags - the customer with multiple tags would have their Sales divided of 68 divided equally to 22.66(6) between the three tags respectively.
Show the sales only at the first tag of any given customer
Create a hierarchy of tags, in which one would be worth more than the other and receive the sales respectively
Tag the products instead of the customers
etc etc.

It all depends on the specific use case, but the possibilites are there, and Qlik does the underlying math whenever you change the dimensions or drill down the data, adjusting the values at every change on the grid to reflect the changes.

Conclusion

In the intricate landscape of BI, Qlik stands as a beacon of clarity amidst complexity. By unraveling the mysteries of data aggregation and interpretation, Qlik empowers businesses to make informed decisions, transforming raw data into actionable insights. So, the next time 2 + 2 doesn't equal 4 in your BI reports, remember—it's not a mistake; it's a revelation waiting to be explored with Qlik.

Qlik publishes new major releases quarterly naming them by the month and year of release (e.g. February 2024). Asides for that, patches and bug fixes are provided on a regular basis.

Business Intelligence as a Gain, not a Cost

whitepro_wp_admin — Fri, 24 Feb 2023 13:35:00 +0000

Business Intelligence as a Gain, not a Cost

The tip of the iceberg

When assessing a potential purchase of a tool, initially, everybody especially sees the cost behind the purchase. As with any investment, it's key to weigh the total cost with the benefits it generates.

While the costs are easily assessable, it's worth to remiind about the concept of a "total cost of ownership", which may consist not only of an obvious monthly subscription, but as well as maintenance fees, investing in cloud computing or your machines, cost of development, training etc.

Although the implementation costs may be high, they are just the tip of the iceberg when compared to the benefits that are waiting below the water.

The benefits are much less obvious... until you get to know them.

A business intelligence tool can demonstrate its value in many ways, the simplest of which are generating insights that lead to increased revenue or cost savings. To name a few:

Identify and prioritize business problems: Identify key business problems that can be solved using the BI tool. Prioritize the problems based on their potential impact on revenue, cost savings, or operational efficiency.
Improved decision-making: BI tools provide valuable insights into business operations, customer behavior, and market trends. With this information, businesses can make more informed decisions and take action based on data rather than intuition. This can lead to better performance, increased productivity, and higher profits.
Save time and money: BI tools can automate data collection, analysis, and reporting, saving businesses time and resources. By reducing manual processes and providing real-time data, businesses can react quickly to changing conditions and make better use of their resources. Simply put - the hours that used to be spent on updating and checking the data in spreadsheets can be invested in other, more fruitful activities.
Competitive advantage: By leveraging data and insights, businesses can identify new opportunities, optimize their operations, and stay ahead of competitors. BI tools can help businesses analyze their strengths and weaknesses, identify areas for improvement, and develop strategies to stay ahead of the curve.

In summary, a BI tool can be treated as a gain by providing valuable insights, improving efficiency, enhancing the customer experience, and providing a competitive advantage. By investing in a BI tool, businesses can unlock the full potential of their data and make more informed decisions that drive growth and success.

How to convert the gains into monetary value? What's the ROI/ROC?

The answer is an evasive "it depends". There's no secret mathematical equation, as the valuation of gains is individual for each company. The possibilities are endless, but just for the sake of not ending in a sad tone, the general idea and thinking pattern behind a valuation would be:

Increased Revenue: Calculate the increase in revenue that results from the BI tool. For example, if the BI tool helps you identify a new market opportunity that generates $100,000 in additional revenue, you can attribute that gain directly to the tool.
Cost Reduction: Calculate the cost savings resulting from the BI tool. For example, if the BI tool helps you identify inefficiencies that lead to a 10% reduction in expenses, you can calculate the monetary value of that cost reduction.
Improved Efficiency: Calculate the time and resource savings resulting from the BI tool. For example, if the BI tool automates a process that used to take 10 hours a week, you can calculate the monetary value of those 10 hours in terms of labor costs saved.
Competitive Advantage: Calculate the impact of the BI tool on market share or customer acquisition. For example, if the BI tool helps you gain 10% more customers than your competitors, you can calculate the monetary value of that increased market share.
Better Resource Allocation: Calculate the impact of the BI tool on resource allocation. For example, if the BI tool helps you allocate resources more effectively, resulting in a 5% increase in productivity, you can calculate the monetary value of that productivity gain.

In summary, to convert gains of implementing a business intelligence tool to a monetary value, you need to quantify the impact of the tool on revenue, cost savings, efficiency gains, competitive advantage, and resource allocation. Once you have calculated the monetary value of these gains, you can compare them to the cost of implementing and maintaining the BI tool to determine whether it is a worthwhile investment for the company.

Searching for a business intelligence tool, why not Qlik Sense?

When searching for a business intelligence tool, there are many options to choose from, but Qlik Sense is definitely one to consider. Qlik Sense is a powerful BI tool that allows users to easily create and share interactive visualizations, reports, and dashboards. It offers a wide range of data connectors, enabling users to pull in data from multiple sources, and has an intuitive interface that makes data analysis and exploration a breeze.

Additionally, Qlik Sense offers a flexible and scalable platform, allowing businesses to easily expand and customize their BI capabilities as their needs evolve. Overall, Qlik Sense is a top contender when it comes to finding a powerful and user-friendly BI tool that can provide a competitive edge in today's data-driven business environment.

The first 1.0 version of Qlik Sense was originally released in 2014, as a successor to QlikView, which was initially released in 1994.

The Dunning-Kruger Effect

whitepro_wp_admin — Thu, 23 Feb 2023 17:01:47 +0000

The Dunning-Kruger Effect

What is it and how does it impact your business?

One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision

- Bertrand Russell (1951)

The chart you've probably seen

The Dunning-Kruger effect is a cognitive bias where people with low ability in a particular task tend to overestimate their skill level, while those with higher ability tend to underestimate their skill level. This bias occurs because individuals with low ability often lack the knowledge and experience necessary to recognize their own mistakes or limitations, leading them to have a false sense of confidence in their abilities. On the other hand, those with high ability tend to have a better understanding of the complexity of the task and therefore are more aware of the gaps in their knowledge and skills. The Dunning-Kruger effect can have significant implications in various areas, such as education, work, and public policy.

The Dunning-Kruger effect is often shown accompanied by a chart that strongly overexaggerates the effect, it's a chart you've probably seen but can't remember its name, something in the likes of the below:

While this chart has nothing to do with the actual study that is behind the forementioned effect, and got washed out when being posted all over social media, it perfectly gets the gist of the topic: people with low knowledge, think that they know more, and/or are more skilled and/or have better performance etc. since they do now know the full spectrum of the topic. On the other hand, when people get to know the topic a little bit better, they don't overestimate their abilities as much.

The chart you probably haven't, but should have seen

That above chart is a fun depiction of the topic, but for the sake of science, lets take a look at the actual charts from the study:

source: Ehrlinger J, Johnson K, Banner M, Dunning D, Kruger J. Why the Unskilled Are Unaware: Further Explorations of (Absent) Self-Insight Among the Incompetent. Organ Behav Hum Decis Process. 2008 Jan 1;105(1):98-121. doi: 10.1016/j.obhdp.2007.05.002. PMID: 19568317; PMCID: PMC2702783.

How can this impact your business?

So now we know that people tend to overestimate their abilities, how can this impact your business?

Firstly, if individuals within the organization are affected by the Dunning-Kruger effect, they may overestimate their abilities in certain areas, which could lead to poor decision-making and suboptimal outcomes. For example, a manager who lacks the necessary skills and experience in a particular area may believe that they are fully capable of making decisions in that area, leading them to make poor decisions and potentially harm the business.

Secondly, the Dunning-Kruger effect can also affect the behavior of employees and their interactions with each other. Individuals who overestimate their abilities may be less likely to seek feedback or input from others, which could lead to a lack of collaboration and cooperation within the organization. This could ultimately impact the overall performance of the business.

Qlik Sense has a community of over 200,000 users worldwide who actively share their knowledge and expertise with each other. Qlik Community is an online platform where users can connect with each other, ask and answer questions, share tips and tricks, and even contribute to the development of Qlik Sense through open-source projects.

Fact-based decision-making instead of opinion-based

So if people tend to overestimate basing on their own knowledge, how can you minimize this effect?

In general, having more data can help mitigate the impact of the Dunning-Kruger effect, as it provides a more objective basis for decision-making and can help individuals better recognize their limitations and areas for improvement. However, even with limited data, it's important for businesses to foster a culture of self-awareness and open communication, which can help individuals better recognize and address their own biases and limitations.

Data can provide an objective basis for decision-making. By using data to inform decisions, individuals can rely on more objective and reliable information rather than their subjective beliefs or assumptions. This can help minimize the impact of the Dunning-Kruger effect by providing a reality check for individuals who may be overconfident in their own abilities.

Data can also reveal gaps in knowledge and skills. When individuals are confronted with data that contradicts their assumptions or beliefs, it can highlight areas where they may be lacking in knowledge or skills. This can help individuals recognize their own limitations and work to improve their skills or seek out additional resources.

Fact-based decision-making can encourage collaboration and feedback. By relying on data and objective information, individuals may be more open to collaboration and feedback from others. This can help individuals recognize and address their own biases and limitations and improve their decision-making skills over time.

Qlik Sense as a fact-based decision tool

Qlik Sense is a business intelligence and data analytics software that can help organizations make fact-based decisions by providing powerful data visualization, analysis, and reporting capabilities. Here are some ways that Qlik Sense can help in fact-based decision-making:

Data visualization: Qlik Sense offers an intuitive and interactive interface for visualizing data, allowing users to easily explore and analyze complex datasets. By representing data visually, users can identify patterns and trends that may not be immediately apparent from raw data, which can help inform fact-based decision-making.
Data analysis: Qlik Sense provides powerful data analysis capabilities, including data modeling, data blending, and data discovery. Users can quickly explore data from multiple sources, and Qlik Sense's built-in machine learning algorithms can help identify correlations and outliers in the data, which can provide valuable insights for decision-making.
Collaboration: Qlik Sense allows users to share data, visualizations, and insights with others in the organization, which can encourage collaboration and facilitate fact-based decision-making. Users can also create and share custom apps and dashboards, which can help ensure that everyone in the organization is working from the same data and insights.
Real-time data: Qlik Sense can connect to a variety of data sources, including real-time data streams. This allows users to monitor key metrics in real-time, which can help inform quick and fact-based decision-making.

Overall, Qlik Sense can help organizations make fact-based decisions by providing powerful data visualization, analysis, and collaboration capabilities that can help users explore and understand complex data.

Qlik Sense vs Power BI

whitepro_wp_admin — Wed, 28 Dec 2022 12:59:00 +0000

Qlik Sense vs Power BI

A brief comparison of the most popular Business Intelligence tools

Introduction

Two very similar tools, with the same purpose in mind, but with seemingly small differences that may turn out to be crucial when creating a whole reporting environment for an organization.

The market is definitely overrun by Power BI, which has been a very popular solution in the recent years. Geographically speaking, Qlik is more popular in Europe (especially in the Nordic Countries as it originates in Sweden), while Power BI has a stronger customer base in the United States and Asia. The global business intelligence tools market share percentage varies strongly from source to source, but in most of them we can see, that Power BI has at least double the market in comparison to Qlik.

Business Intelligence market share in 2021:

source: https://www.trustradius.com/vendor-blog/business-intelligence-statistics-and-trends

Report Development cycle

Starting of with the first large difference, the development cycle for both applications is a little different.

Power BI requires a somewhat two step process, where you first develop a report inside of Desktop application (Power BI Desktop) and when you're finished you later publish the application into Power BI Service. While the Power BI Service covers a large portion of the functionality as the desktop application, it's not fully the same. It gives a possibility to modify part of the visuals, but not the full process of creating a complicated report from scratch. Whenever creating a new report, or editing a report which requires change in the data source or more complicated formulas - you first download the report file to the desktop application, make the changes there and later publish it when finished. It's not a huge inconvenience, however it can be a pain e.g. when working on multiple reports requiring small changes.

Qlik Sense does also offer a desktop application, but its fully optional in the development cycle - all of the functionality is in one place on the server running the Qlik Sense environment. While the desktop app has all the functionality, since the server-side covers all the needs, the desktop app mostly is used for tests.

The difference in running everything on server-side vs partially on the desktop-side also means, that for Power BI you need to have a machine capable of running the desktop application at a reasonable speed. This will come as more important in the "Speed" and "Costs" sections.

Speed

Here is where Qlik truly shines above all other Business Intelligence applications.

We've prepared a simple scenario in which there are 10 million records in a "flat" table stored inside a CSV file. To make the test fair, in both cases we're loading the data via a desktop application, on the exact same machine. It takes Power BI desktop 1:50 minutes to load the data, while Qlik Sense required 9 seconds that's 12x faster.

This difference in speed further increases when the data requires some further transformation on the way. What's crucial - the current workflow for Power BI requires to use the desktop application, which means running it on older machines will be significantly requiring more patience.

What's important though, the difference in speed is not noticeable when the data is already loaded - making selections and filters is running very similar in both applications, so from the end-user point of view it doesn't really matter (especially as they'll mostly run it from the server-side anyway). So the difference in speed regards mostly to people creating and modifying the reports.

Ok so Qlik is 12x faster when loading 10m records, but that's only a minute here and there, that doesn't really matter, right? Wrong. Imagine how many times you need to extract, load and transform the data before reaching the desired output. The example above uses just 10 million records of a flat datasource, the difference is further highlighted when working with 100 million records, or e.g. joining, and transforming a couple tables. In a relatively simple example, if you load the application 15 times before the data looks like you'd expect, in the same scenario of 10 million records, we're talking about a difference of 25 minutes. For 100 million records that'd well above 4 hours.

This difference in speed changes the way you work. Imagine a excel file, that whenever you make a change in a formula, you have to wait 2 minutes before the sheet recalculates, you instantly disable the dynamic calculations into on-demand, to lessen the pain, but whenever you need to hit F9 to calculate, you go and do something else since the machine freezes. That's what's working with a large dataset on Power BI feels like. With Qlik you rather work fully dynamic.

As a side-note, the speed for Qlik can be further increased by storing the data in a format native for Qlik - .qvd files take around 20-30% less time to load in comparison to .csv files.

Costs

Nowadays, software companies try to squeeze out the dollars out of customers in multiple complicated ways, so when talking about the costs, we'll be taking into consideration what's called a "Total Cost of Ownership".

The licensing model:

Straight of the bat, Power BI Desktop is a free application downloadable from the Microsoft Store. That's a great marketing move that wins a lot of points to Microsoft (wouldn't it be great if other Microsoft applications had a similar price-model?).

Then we get a free version of Power BI Service for one user (which is also a great marketing move) and after we've tested everything out for free, when we're ready we can choose from a couple of pricing options:

Asides for some marketing differences, the core thing between Power BI Pro and Premium is Power BI Pro has a limit of a data model of 1GB and 8 refreshes per day, so it's a valid option for very small reports and/or companies. The "Per Capacity" option requires to also buy Power BI Pro separately, so it's reasonable from above 500 users. So for most use cases we'll be talking about the 20$/user/month Power BI Premium version.

On the other hand Qlik Sense doesn't have any free version (just a trial, but after contacting sales), and shares just one option on their website, which costs basically 50% more than Power BI Premium.

Ok so Power BI wins this round? Not necessarily. The 30$ option in Qlik is just a basic price for a small/individual customer. The Power BI Services offered above run only on SaaS in the Microsoft cloud, while Qlik gives you an option to choose between SaaS and on-premises. In Qlik Sense you can also get two different types of licenses - one for Developers ("Professional" License) and a much cheaper one for report viewers ("Analyzer" License). While the costs are hidden behind a contact us button, it's well worth to compare, as the price for Qlik is negotiable and is usually related to the amount and type of licenses bought in one package. Microsoft also offers an on-premises option but they do much to make you not want it, requiring a couple more products for it to run, so it's mostly an option for larger corporations.

The machines:

Mentioned a couple of times above, for Power BI as part of the process you need to run a desktop application, and therefore you cannot go with a decade-old laptop and need to invest in a machine with a lot of ram and computing power. You don't have to calculate this step for Qlik, as it may 100% run on the server-side.

The additional services:

Qlik is prepared as an independent tool, created to use multiple-different data sources, Power BI as any other Microsoft product, is created to sell you more Microsoft products. Love it or hate it, you'll most likely have to go with Microsoft Azure at some stage, since the whole Power BI infrastructure is built around it. For Qlik - choose any service you like, or don't choose any at all - since you can build a data warehouse inside of Qlik Sense and don't get any additional costs. To be fair, what's worth noting - a data warehouse in Qlik makes while being easy to set up and maintain, it's not preparing to being shared across multiple different applications but rather for the purposes of Qlik itself.

Total Cost of Ownership

As a brief summary of this section - when calculating the costs, don't take the marketing hook and just the standard pricing, think about the product and its infrastructure as a whole. The Qlik sales team knowing that Power BI is a direct competitor has a couple of documents and tools comparing the details, they're even confident enough as to state that in a 100 users scenario, the total costs of Qlik are 30% lower than Power BI. And honestly, even if it would come down to the costs itself, it would be easier to negotiate with Qlik than with Microsoft.

Design and visuals

The key difference the visuals logic is that Power BI gives you the possibility to change font types and sizes, as well as place objects in a pixel-perfect manner. Colors and other design elements will be familiar to people used to Microsoft Office tools. Qlik on the other hand sticks to a grid-based design, and is taking control of the objects sizing later on.

Asides for the above all of the rest honestly boils down to individual taste of the user interface. Graphically speaking, both of the tools give similar possibilities and components of visualizations, filters etc. As for other customizations, Qlik has some possibility of Themes which can be shared in between multiple reports, and are modifiable via CSS. Power BI has settings similar as in Power Point, where you can download some predefined templates and customize them to your needs.

Example of the same basic report with two Charts and 4 filter panes below, both are using standard settings for the components coming out of the box:

Power BI

Qlik Sense

Report Developer Experience

Both tools have some learning curve. Power BI is easier to catch up as its interface will be familiar for anyone who has some familiarity in Excel (and especially its Power Query tool) so it's more beginner-friendly. Qlik while having a harder start, is more rewarding on the long run.

Data loading and transformation:

Basically, Qlik is much more centered around scripting, and Power BI is more centered arount point-and-click giving you all sorts of different menus to choose from, which have their options hidden, but are friendly after reaching some familiarity with them. However when you need to do anything that Microsoft didn't find as a "basic" use case, you need to do some scripting anyways. Scripting in its nature is usually much faster than going through menus, which are just a interface to do the Scripting for you (e.g. each step in Power Query makes a line of script), However while being faster, it requires to be learnt, and is scaring of people used to only clicking through menus.

Both applications have their own scripting language for transformations and measures, Power BI with Dax, and Qlik with its native language. Below there's a example of a function inside a measure calculating a sum of Quantity for 'Catergory A' in both tools:

Power BI: CategoryA = sumx(FILTER('CSV_10m',RELATED('Categories'[CategoryName])="Category A"),'CSV_10m'[Quantity])

Qlik Sense: Sum({}Quantity)

Both measures are human-readable. Power BI requires nesting of functions (similar to excel) and Qlik has a simpler syntax but with a broad amount of different brackets which are unintuitive at first sight.

Variable expressions

Power BI has some possibility to input values by the end-user by using editable parameters, but it's nowhere near flexible as what's offered by Qlik. Inside of Qlik a variable can reach any value, be any function, or a function with a parameter. Therefore by taking control over one variable, you can e.g. dynamically change the expression of a function, have it change another variables value etc. This helps with creating more dynamic content, instead of having predefined sheets, and in the long run requires much less time and is more friendly to the end users as an outcome.

The "feel" of the application

Power BI is a tools that somewhat derives from the ever increasing needs placed towards Excel. In its basis, Power BI uses Power Query to do the data loading, which is fine for smaller datasets, as its having a nice preview of what's happening with the data step-by-step, but it's a pain in a more complicated data environment, as it's much slower and the syntax is painful and not something to which programmers are used. The interface needs a lot of clicking through, and while the options are mostly where you'd expect them to be, sometimes what would require writing one line of a script, requires a dozen different clicks, which may be somewhat tiresome. Qlik may be scary at first sight as it's using its own language, and basically most things that aren't visual require some scripting, but it's much faster and more rewarding once you get to know it.

End-user Experience

The core of the experience is quite similar. You have sheets, charts, selectable objects, personal bookmarks etc. But while there are many similarities, there's a couple small, but powerful differences.

The filter pane in Qlik Sense is always on the top of the page and you have easy access to what filters are applied, and can easily change them to other values. The Filters in Power BI are shown only after selecting an object - which tells you which filters are impacting this object, but gives you no possibility to change them from there, you need to go to the specific filter and change it from there. To disable all filters, you need to either go through them one by one, or create a special bookmark without any filters applied. Qlik is encouraging the users more to click through the possible options and see what's inside, and gives you a easy "clear this filter" or "clear all filters" option available always on top, which gives a higher chance of gaining more insights just by clicking through with curiosity.

Selections made on a filter inside of Power BI hide the other unrelated options. Selections made in Qlik highlight the selected options, but show you with color-coding the other possibilityble options, and unrelated options. In this scenario, if you filter a specific Catergory, in Power BI you'll see the related Year-Month, in Qlik you'll see both related and unrelated Year-Months. This may seem insignificant, but when e.g. there's sales missing from this Catergory inside one month, inside of Power BI you won't see this month, inside of Qlik you'll see it in a different color, indicating additional information. More on this topic in a separate article on the Qlik associative engine.

As pictured on the screen below, Qlik gives an option to select data directly from any chart, you can select either one value, or a range of them, which after confirming is applied onto other objects. This option is nowhere to be found in Power BI:

Power BI has native commenting functions to each object, engaging users to leave remarks while reading them. Qlik requires buying an additional product to have this function.

Both Qlik and Power BI have an option to export the data visible on an object. However Power BI exports it as a .csv without formatting, when opened in excel it requires further processing before the values are usable. Qlik exports it as a xlsx remaining formatting and table layout. This is probably a negligence from Microsoft which should be changed in future releases as it doesn't make reasonable sense.

Extensions and Integrations

If in the standard objects you're missing any visualization needed, you have an option to download more. Power BI has "AppSource" and Qlik has "Qlik Garden" both filled with tons of additional extensions. However there are two differences:

Power Bi has a special window inside of it to easly download visualizations and test them right away with sample data etc. It's simple and fast to use.
Qlik requires you to download the extensions on a separate webpage and import them to the application and restart it, but gives you not only visualizations, but all sorts of extensions since it allows communication with its engine through an API and supports anything coming through Javasript

As for additional integrations, Power BI easily communicates with other Microsoft products from the Power family, and Azure, but also others. Qlik doesn't link to any other supplier, but gives you development options through its "Dev-Hub" like Mashups, previously mentioned custom extensions (most of which are free, but there are some paid worth noting as well).

Miscellaneous

Updates and future development

Power BI has monthly updates, mostly with bug fixes

Qlik Sense has Quarterly updates, with feature releases, and in the meantime provides patches/build updates with bug fixes

Mobile devices access

Power BI has a dedicated mobile app, and the desktop app gives the possibility to separately design the content of the mobile version of a report giving much flexibility.

Qlik Sense doesn't require any app to be installed, you just need to log in as usual to your Qlik environment via a web browser. On the other side it has a very limited set of options on designing the report specifically to the mobile version.

The commonly agreed full form of the abbreviation QLIK is Quality, Learning, Interaction and Knowledge. In its first years Qlik was Named Quik (standing Quality, Understanding, Information and Knowledge), but later changed its name due to trademark reasons.

Final Thoughts

Both tools have their advantages, but in our opinion Qlik is the winner of this comparison. While Power BI is easier to get into for simple usecases especially since it's offering a free desktop application and has a familiar interface, Qlik is the more refined and specialized tool created and further developed for decades strictly as a Business Intelligence tool and going with it will have much more worthwhile benefits in the long run. As Power BI has a large portion of the market, companies tend to replicate the solution which is more popular, but not necessarily the better choice when giving it more thought and concideration.

Qlik associative engine

whitepro_wp_admin — Thu, 15 Dec 2022 10:47:21 +0000

Qlik associative engine

What differentiates it from others?

The context

The Qlik associative engine is what drives Qlik products under its hood. The way that the engine is constructed, enables fast calculations on large and complicated data models. That sounds great and all, but first of what does it even mean associative in this context?

associative adjective

1. relating to the association of ideas or things

• making associative links

2. (mathematics) giving the same result no matter in what order the parts of a problem are done, for example (a × b) × c = a × (b × c)

Oxford Dictionary

The mechanics of Qlik's engine relate to both meanings of this definition. On one hand we have many interconnected data objects, with their respective indexing to provide the neccesary speed, on the other hand the engine makes sure that the values stay true no matter what is the context they are placed in.

An interconnected microcosmos of data

In the associative model, all data points are interconnected and have their respective relationships between each other. This allows to store data separately, with a guideline between separate tables. Thanks to the provided link between the data, there is a possibility to connect infomation, which is not neccesarily directly related to one another.

An example of a simple interconnected data model of product sales:

In the above example, nothing stops us of making a list of Warehouse inventory by Sales persons date of birth. While this would probably make not much sense, thanks to the associations provided, although this two informations are not connected directly, Qlik "knows" how they relate to one another. This gives us a lot of benefits:

Load multiple data sets at one place - have more information in one report without the neccesity to create a separate report for a separate set of data
Easly filter data - because all data is associated, the engine can filter out unrelated data from across all data
No queries are made - all of the data is available straight away, and is dynamically reached whenever needed
Speed - calculations even on very large data sets are performed fast

How it compares to the traditional data model

The associative model may be hard to understand at first glance especially for people used to excel and databases, as these tools require to have a strictly defined row of data. It can be briefly summed up as:

Query tools - ask, wait, answer. Load a specific subset of data.

Qlik associative engine - load everything, have all needed data at hand, the engine takes care of efficient use of resources to maintain speed.

In order to query across multiple tables inside of a database, you need to join the data on the way to receive it in the desired format. This has worked wonders for decades, however in the ever increasing datasets of the 21st century, it's becoming more and more hard to perform, especially when there isn't a simple way to do a one-to-one connection. The database model requires to do some precalculation and/or preagregation to simplify the data to a unified row.

Inside of Qlik, the preagregation and precalculation is not a must (it's there, just as optional phase). Thanks to that you can work seamleslly on a sum of milion rows, and a second later drill down into specific records of the same data.

Qlik was founded in 1993 in Lund, Sweden. The first product was named "QuikView", which was later renamed to QlikView.

"The power of Grey"

Whenever making selections/filtering inside of Qlik, you don't see only the selected options, but also marked in two shades of grey - alternative values (light grey), and unrelated values (dark grey).

As pictured below, only the 2010s decade is selected, and you can instantly see what are the other possible, and other non-related values. It's logical, that 2010s blong to the 2000s century. Years belonging to the 2010s are possible to select (white) and years unrelated to 2010s are marked in dark grey.

This gives additional insight on values you'd expect, or don't expect in a given range. Imagine a situation when you select the current Year and Month, and a given Sales person is marked as dark grey - that'd mean that there weren't any sales related to that person in this period:

It may be a data higene issue,
It may be that this person no longer works, and therefore it's what you'd expect and everything is ok,
It may be that this person works, everything is ok with the data, and therefore it's a crucial piece of information to look deeper into.

And that's just thanks to making one selection.

Qlik Sense snowflake model

whitepro_wp_admin — Wed, 09 Nov 2022 15:49:55 +0000

Qlik Sense snowflake model

How does it compare with the traditional model?

As winter is just around the corner, and snowflakes are inevitable in most places in the world, lets look into the snowflake model inside of Qlik and its core features.

Traditional SQL-like model

Databases and SQL were first introduced almost 50 years ago in the 70s. No wonder they’re a basis to a lot of applications, which modeled a way around them. This was later-on cultivated by the rise of spreadsheet applications like MS Excel. As a result of this technical background, in most applications its necessary to unify data in a joined table to simplify data treatment. This is either done manually in code, or sometimes in a visual manner connecting the tables by hand. In the latter while the interface shows all of the connections as being separate tables, the application itself does the unification under the hood.

This approach works fine when handling a relatively-simple data model, but can quickly become a bottleneck in matter of performance – doing a lot of joins on a large dataset is requiring a lot of computing power. While this can be overcome either by equipping oneself in patience, or increasing the processing capacity, it can also become a logical issue when trying to data.

Let us imagine a simple real-world example with reporting a budget and actuals data, along with tables of customers, items, and sales people.

Example dataset:

Table	Column count	Row count	Dataset [Column * Rows]
Budget	31	400 000	12 400 000
Actuals	53	1 300 000	68 900 000
Customers	28	1 000	28 000
Items	37	5 000	185 000
Sales People	15	300	4 500
TOTAL	164	1 706 300	81 517 500

In the beginning, in total we’re starting with a dataset of 164 columns and <82milion individual data cells. In a traditional model, you need to repeat the data from customers, items and sales people inside of the Actuals and Budget, and connect them to one another - just to help the program process it through a “flat” table.

The process of pre-calculating all of the data into one table helps with its readability for the machine, but has an downside of artificially increasing the number of records. After joining the data into one table, the amount of data in the set would have tripled from the original < 82 million, up to < 218milion. This results from the fact, that we repeat 80 columns from the dimensions inside of the facts. By this stage we’ve only added three tables in a relatively simple case.

It is worth to note, that as it is with joins in their nature – they can generate duplicates when the data matching is not performed correctly. So to make things an ounce more complicated, lets imagine that the items contain data stating their quantity in the current inventory.

Before Joining:

Actuals
Item	Sales
A	2
A	3
A	56
B	7
B	8
B	15
B	20
TOTAL	111

Items
Item	Inventory Quantity
A	856
B	201
TOTAL	1 057

When this data gets joined inside of the fact tables, it would get repeated for every row inside of the facts, this makes it hell when trying to make a sum of all of the inventory and requires taking extra steps.

After Join:

Actuals + Items
Item	Sales	Inventory Quantity
A	2	856
A	3	856
A	56	856
B	7	201
B	8	201
B	15	201
B	20	201
TOTAL	111	3 372

Qlik was noted on NASDAQ since 2010 untill August 2016, when it was acquired by a private equity firm Thoma Bravo, LLC.

While it’s possible to work around this issue in the traditional model, and this approach has worked wonders for decades, it started back when the datasets were much smaller. The amount of data we work with increases exponentially, and simply speaking – this approach is no longer valid in the modern world of analytics. Instead of working around the limitations, Qlik tackled the issue from the ground up, and the result is the counter-intuitive snowflake.

Qlik Sense snowflake model

First and foremost, in the snowflake approach, you concatenate all fact data into one table. The tables containing dimensions are then stored separately.

This results with the exact same dataset in size as we’ve started with, and all of the calculations are made by Qlik on the fly, whenever needed. This is possible thanks to the Qlik associative engine which handles everything under the hood in a matter of milliseconds, even on much larger datasets.

To be exact, technically speaking by definition at this stage we’re taking about a Star model, not a Snowflake, but from Qlik Sense point of view this definition is just a matter of adding next levels to this model, for example:

And this can go many further levels down.

While the start or snowflake models aren’t exclusive just to Qlik, a crucial component that makes this an excellent solution is the Qlik Associative model [which we’ll talk about in more detail another time], with one of its key features – when loading the data, instead of doing precalculations and joins as in the traditional model, the engine focuses its computing power on the indexing between data, to make accessing and calculations on them much simpler and faster. It also does some optimizations after each selection being made by the end user.

The best of two worlds

While Qlik works great with the snowflake model, it still can easily work with the traditional model alike, however it usually makes sense just to join part of the model inside of one table to simplify the model without repeating data and/or make complicated calculations work easier inside of code. The freedom is in the hands of the developer to choose the best way to tackle each scenario.

Does it somehow matter to the end user?

The first and obvious thing that the end user will notice is the difference in speed – both in selections, and in the time it takes to refresh an application and its data. But it doesn’t stop there.

A less obvious point is a capability to combine multiple datasets into one application. It can be even ones, that aren’t logically connected in any manner. A real-life scenario would be any management report that overviews the whole company. Why not dare to combine data from HR, Finance, Sales and Purchasing in one place, with the ability to drill down to the individual record? Traditionally it would either require a couple stages of pre-calculation, or the application would be divided into separate ones, as being too slow and/or too expensive to develop.

Qlik doesn’t have issues with mixing apples with oranges while simultaneously being able to separate them on the fly back again to insightful data.