When a Fast Query Still Makes a Slow Dashboard: A Superset Performance Retrospective

Performance problems rarely have a single cause.

A dashboard can be slow because the database is slow. It can be slow because queries are waiting in a queue. It can be slow because results are being delivered inefficiently. It can even be slow because the system has enough capacity for one request but not enough capacity for thirty requests arriving at once.

Recently, I worked through a performance problem on an Apache Superset deployment where dashboards with dozens of charts were taking far too long to become usable.

What initially looked like a database optimization exercise turned into something much broader: data architecture, asynchronous query delivery, Celery concurrency, caching, infrastructure capacity, configuration management, testing methodology—and, importantly, learning how easy it is to draw the wrong conclusion from a bad performance test.

This is the retrospective I wish I had before starting.

The Problem: “The Dashboard Is Slow”

The environment consisted of several Superset dashboards containing a large number of charts.

Individual charts were taking several seconds to load, and the complete dashboard experience could be considerably worse because many charts were being requested at roughly the same time.

The first instinct was obvious:

The database queries are too slow.

And there was evidence to support that conclusion.

The primary dataset was backed by a complex live join against an enterprise database. Individual chart queries could take multiple seconds, depending on the filters being applied.

But as the investigation progressed, it became clear that query execution time was only one layer of the problem.

The performance bottlenecks turned out to exist at several different stages:

Query execution — the underlying data model was expensive.
Result delivery — asynchronous query results were not always being pushed correctly to the browser.
Query scheduling — background workers had limited concurrency.
Infrastructure capacity — once the software bottlenecks were removed, CPU became the next constraint.

The important lesson was that “the query takes five seconds” and “the user waits five seconds” are not necessarily the same thing.

Step 1: Stop Making the Dashboard Query the Production Database

The biggest improvement came from changing the data architecture.

Instead of having every dashboard chart repeatedly execute an expensive multi-table join against the source system, a separate analytical table was created specifically for reporting.

The reporting dataset was:

pre-joined,
pre-aggregated where appropriate,
structured at the required reporting grain,
refreshed periodically by an independent ETL process.

The dashboard could then query a relatively simple analytical table rather than repeatedly reconstructing the same business logic.

This produced a dramatic improvement.

Chart queries that had previously taken several seconds were reduced to roughly one second on average.

More importantly, the improvement applied across the dashboard rather than requiring individual optimization of every chart.

Why this worked

There is a fundamental architectural difference between:

“Run the business logic every time someone opens a dashboard”

and:

“Perform the expensive work periodically and make the dashboard read from the resulting analytical model.”

The second approach is much more appropriate when the data does not need to be real-time to the second.

The lesson isn't that every Superset deployment needs a data warehouse.

The lesson is:

If dozens of charts repeatedly perform the same expensive transformation, the problem may be the data architecture rather than the individual SQL queries.

A Dangerous Optimization: When Faster Became Incorrect

During the investigation, another optimization was tested: reducing result-set limits in an attempt to reduce the amount of data being processed and returned.

It sounded reasonable.

It was also wrong.

The change caused some charts to silently under-report their totals.

There were no obvious application errors. The dashboard still loaded. The numbers simply weren't always complete.

That is arguably more dangerous than a slow dashboard.

A slow dashboard tells you there is a problem.

A dashboard showing incorrect business numbers may not.

The issue was eventually identified by comparing dashboard results against direct SQL results.

After reverting and correcting the limits, the affected totals matched the underlying data again.

This was one of the most important lessons from the entire exercise:

Performance experiments on reporting systems must include correctness checks.

A useful performance optimization that changes a business number isn't an optimization.

It's a data integrity incident.

Step 2: The Database Wasn't the Only Bottleneck

After improving query execution, something interesting happened.

Some users were still experiencing surprisingly long chart-load times.

The database said the queries were executing in a couple of seconds.

Superset's logs, however, showed some charts taking considerably longer from the application's perspective.

That suggested something important:

The query wasn't necessarily spending all that time executing.

Some of the time was being spent waiting.

The culprit was the asynchronous task queue.

Step 3: Celery Was Quietly Queuing the Dashboard

Superset uses Celery for background task processing.

In this deployment, the worker configuration provided only a small number of concurrent execution slots.

Imagine opening a dashboard containing thirty-plus charts.

If the system can only process four background jobs simultaneously, the first four jobs execute while the remaining jobs wait.

Once the first batch completes, another batch starts.

The database might be perfectly capable of handling the workload, but the application is deliberately feeding it only a few queries at a time.

This created a useful distinction:

Query time

How long does the database take to execute the query?

Queue time

How long does the query wait before a worker starts processing it?

End-to-end time

How long does the user actually wait before seeing the result?

These numbers can be dramatically different.

In our case, application logs revealed charts that appeared to take tens of seconds even though the underlying SQL execution was much shorter.

The difference was largely queueing.

The Fix: Increase Worker Concurrency

Worker concurrency was increased progressively rather than changing everything at once.

The deployment eventually moved from a single worker with limited concurrency to multiple workers, providing enough execution slots to accommodate a large dashboard burst.

This was much more effective than simply making individual queries faster.

It also highlighted an important principle for dashboard systems:

Optimize for bursts, not just averages.

A dashboard isn't a normal web page.

Opening one dashboard can generate dozens of requests almost simultaneously.

A system that comfortably handles ten isolated queries per minute may still struggle when thirty queries arrive within a few seconds.

Step 4: The Websocket Problem

There was another layer that initially wasn't obvious.

Superset's asynchronous query system relies on websocket communication to efficiently deliver results back to the browser.

In this environment, websocket delivery had stopped working correctly.

The result was particularly deceptive because the application wasn't completely broken.

Queries could still execute.

Results could still become available.

The browser simply wasn't receiving them through the intended push mechanism and instead relied on polling.

This made the dashboard feel slower even when the underlying query performance had improved.

The root cause turned out to be a configuration mismatch introduced during a secrets/configuration change.

One component had been updated while another component responsible for validating the websocket token had not received the corresponding configuration.

The failure was effectively silent.

No dramatic database error.

No obvious application crash.

Just a feature that didn't work as intended.

The Lesson: Configuration Changes Have Dependency Graphs

One of the easiest mistakes to make in a distributed application is assuming that a configuration value belongs to one service.

In reality, a secret or configuration value may have several consumers.

Changing it requires identifying all of them.

A better process is:

Identify the configuration value.
Identify every service that consumes it.
Update all consumers.
Recreate services where necessary.
Test the complete path end-to-end.

That last step matters.

A configuration change isn't complete because the containers are running.

It is complete when the feature that depends on the configuration has been tested.

Another Surprise: Restart Isn't Always Restart

One operational lesson came up repeatedly.

Changing an environment file and running:

docker compose restart

doesn't necessarily cause a container to receive the updated environment.

The container may simply be restarted using the environment it was created with.

For configuration changes, recreating the affected container was necessary.

This distinction seems small, but it can produce extremely confusing debugging sessions.

You change the configuration.

You restart the service.

The service still behaves exactly as before.

You assume your configuration is wrong.

It isn't.

The container simply never received the change.

A useful operational rule emerged:

Configuration change → recreate the affected service.

And when possible, scope the recreation carefully rather than unnecessarily touching unrelated services.

Step 5: Then We Hit the CPU Ceiling

Once the query architecture, websocket delivery, and worker throughput had been addressed, the next bottleneck became visible.

CPU.

This is an important pattern in performance engineering:

Removing one bottleneck often reveals another.

Initially, database query cost dominated.

After the analytical dataset was introduced, it didn't.

Then queueing became more visible.

After increasing worker capacity, infrastructure utilization became more relevant.

Eventually, CPU became the actual limiting resource.

The Superset application host was scaled up, and the analytical database host was also given additional CPU headroom.

But this brings us to one of the most important lessons from the entire exercise.

The Performance Test That Lied

At one point, a load test suggested that the analytical database degraded sharply once concurrency reached a certain level.

The conclusion seemed straightforward:

The database needs more CPU.

So the infrastructure was upgraded.

The upgrade was harmless and provided additional headroom—but the conclusion behind it was wrong.

The problem wasn't the database.

It was the test.

The load test was implemented using Python threads in a single process.

Some of the work involved CPU-heavy Python operations, including processing and serialization.

Because of Python's Global Interpreter Lock, the test harness itself introduced serialization.

In other words, the benchmark was partly measuring the benchmarker's ability to process results, rather than the database's ability to execute queries concurrently.

The database looked overloaded from the perspective of the test.

But when database-side resource usage was checked, it was barely working.

A cleaner test using independent database connections showed substantially better concurrency.

That completely changed the conclusion.

The Benchmark Was the Bottleneck

This was probably the most valuable lesson of the project.

When a performance result looks surprising, don't immediately assume the system being tested is at fault.

Ask:

What else could be limiting the measurement?

For a database benchmark, inspect:

database CPU,
database memory,
I/O,
network throughput,
connection counts,
client CPU,
client memory,
serialization time,
application-side processing,
the benchmark implementation itself.

A benchmark is software too.

It can have bugs.

It can introduce contention.

It can serialize work.

It can produce misleading measurements.

And a bad benchmark can lead directly to an expensive infrastructure decision.

Production Load Testing Is a Different Problem

Another experiment attempted to generate a large amount of synthetic load directly from the production host.

It was a bad idea.

The test itself consumed enough resources to destabilize the environment.

The lesson was simple:

Don't use a production machine as your stress-testing laboratory unless you have deliberately designed it for that purpose.

A benchmark that takes down the environment doesn't prove the environment is incapable of handling the workload.

It may prove that the benchmark was poorly placed.

For serious load testing, use a separate environment or generate load from dedicated clients.

Don't Trust Your Eyes: Analyze the Dependency Graph

Another smaller but interesting problem involved dashboard filters.

Several filters were configured to depend on one another.

Some dependencies provided little practical narrowing of the available values, while others created circular relationships.

The initial review was done manually by looking at the configuration.

That found some of the problems.

It didn't find all of them.

A simple graph traversal algorithm did.

This is another broader engineering lesson:

When the problem is a graph, use graph algorithms.

Don't rely on visual inspection when a deterministic algorithm can identify cycles and dependencies.

The same principle applies to:

service dependencies,
database relationships,
CI/CD pipelines,
network topology,
package dependencies,
permission inheritance.

If the structure is a graph, treat it like one.

What the Final Architecture Looked Like

After the changes, the system had a much clearer separation of responsibilities.

Conceptually:

                    ┌──────────────────┐
                    │      Browser     │
                    └────────┬─────────┘
                             │
                        Reverse Proxy
                             │
                    ┌────────▼─────────┐
                    │     Superset     │
                    └───────┬──────────┘
                            │
                 ┌──────────┴──────────┐
                 │                     │
          Async task queue       Websocket delivery
                 │                     │
        ┌────────▼────────┐            │
        │ Celery Workers  │            │
        │  Multiple slots │            │
        └────────┬────────┘            │
                 │                     │
                 └──────────┬──────────┘
                            │
                    ┌───────▼────────┐
                    │ Reporting DB   │
                    │ Pre-processed   │
                    │ analytical data │
                    └───────▲────────┘
                            │
                         ETL job
                            │
                    ┌───────┴────────┐
                    │ Source Systems │
                    └────────────────┘

Each layer now has a more clearly defined role.

The source system doesn't need to repeatedly perform expensive reporting joins.

The reporting database serves dashboard workloads.

Celery provides parallel background execution.

Websockets provide efficient result delivery.

The application host has sufficient capacity for the expected burst.

The Numbers Improved—But the Numbers Were Not the Whole Story

The final improvements were significant.

At a high level:

| Area | Before | After | | -------------------- | --------------------------------------- | ---------------------------------------------- | | Dashboard query cost | Several seconds per chart | Around one second on average | | Query queueing | Significant under bursts | Substantially reduced | | Worker capacity | Very limited | Multiple workers with substantially more slots | | Websocket delivery | Not functioning correctly | Working reliably | | Data accuracy | One experiment caused silent truncation | Results validated against source SQL | | Infrastructure | Undersized for peak concurrency | Additional CPU headroom | | Monitoring | Primarily manual investigation | Repeatable performance checks |

But the most important improvement wasn't a particular number.

It was understanding where the time was actually going.

What I Would Do Differently Next Time

The retrospective exposed several things that should have happened much earlier.

1. Validate the measurement before changing infrastructure

Before buying CPU, changing database parameters, or redesigning anything:

Validate the benchmark.

Check the system being measured.

Check the client generating the load.

Check whether the benchmark introduces its own bottleneck.

2. Separate query time from queue time

Don't look only at database execution time. Superset dashboard load waterfall Measure:

Request received
      ↓
Queue wait
      ↓
Worker starts
      ↓
Database execution
      ↓
Result processing
      ↓
Result delivery
      ↓
Browser rendering

Every stage can add latency.

3. Treat correctness as a performance metric

Every optimization affecting a reporting query should answer two questions:

Did it become faster?

and:

Did the result remain correct?

If you only measure the first one, you can accidentally optimize your way into incorrect reporting.

4. Change one major variable at a time

Performance tuning becomes much easier when every change has a clear hypothesis.

For example:

Increasing worker concurrency should reduce queue wait.

Change it.

Measure it.

Compare against the previous baseline.

Then decide what to do next.

This is far more reliable than changing ten configuration values simultaneously and trying to determine which one helped.

5. Build the monitoring tool before the investigation ends

One of the most useful outputs of the work wasn't a configuration change.

It was a small repeatable performance-checking script.

It combined application timing information with infrastructure statistics so future investigations wouldn't have to start from scratch.

That's an important distinction:

Fixing today's performance problem is useful.

Making tomorrow's performance problem easier to diagnose is even better.

The Bigger Lesson

The biggest takeaway wasn't “add more workers.”

It wasn't “use PostgreSQL.”

It wasn't even “pre-aggregate your dashboard data.”

It was this:

Performance is a systems problem.

A dashboard is the end of a chain.

Data model
    ↓
SQL
    ↓
Database
    ↓
Connection pool
    ↓
Task queue
    ↓
Worker concurrency
    ↓
Application
    ↓
Websocket / HTTP
    ↓
Browser
    ↓
User

Optimizing one layer while ignoring the others can produce surprisingly little improvement.

Worse, it can lead you in the wrong direction.

The database may be fast while the queue is slow.

The query may be fast while the result delivery is broken.

The application may have enough capacity for ten charts but not thirty.

The infrastructure may appear overloaded because the benchmark itself is inefficient.

And an optimization that improves performance can still be a failure if it changes the numbers being reported.

Final Takeaway

The most useful mindset I've found for performance engineering is:

Measure → hypothesize → change → measure again → validate correctness → repeat.

Not:

Guess → change everything → hope.

The first approach is slower at the beginning, but much faster in the long run.

And perhaps the most humbling lesson from this particular exercise was that some of the biggest problems weren't caused by the technology at all.

They were caused by our assumptions about the technology.

The system eventually became faster not because we found one magical Superset setting, but because we stopped treating “the dashboard is slow” as a single problem.

We broke the request into its constituent stages, measured each one, challenged our own hypotheses, removed bottlenecks one at a time, and—perhaps most importantly—kept checking that faster still meant correct.

That's the real performance engineering lesson.