Incidents | Telemetrydeck

Query Issue

Mon, 27 Jul 2026 14:47:00 -0000

We've implemented a fix and will continue monitoring the situation.

Query Issue

Mon, 27 Jul 2026 12:02:00 -0000

We've isolated the issue to our long-term storage servers, which have been problematic a lot these last few days.

Query Issue

Mon, 27 Jul 2026 11:55:00 -0000

We're having an issue with out query servers and are looking into it.

Queries not running

Wed, 22 Jul 2026 12:06:00 -0000

We're back online, but were still working on improved stability in the affected areas

Queries not running

Wed, 22 Jul 2026 10:57:00 -0000

A fix is implemented and we're monitoring the deployment.

Queries not running

Wed, 22 Jul 2026 10:40:00 -0000

We've found a misconfiguration and are working on a fix.

Queries not running

Wed, 22 Jul 2026 10:15:00 -0000

We're investigating issues with query running. Data ingestion is not affected.

Queries not working

Tue, 21 Jul 2026 21:46:00 -0000

All back online, all data is available, ingestion has not been affected, and data processing has caught up.

Queries not working

Tue, 21 Jul 2026 21:38:00 -0000

Current Data Stores have recovered, still waiting on long-term stores. Ingestion was unaffected, but processing incoming data is delayed slightly.

Queries not working

Tue, 21 Jul 2026 21:37:00 -0000

We've found the issue and deployed a fix. It seems our clusters are recovering... standing by.

Queries not working

Tue, 21 Jul 2026 21:27:00 -0000

We're investigating an issue with queries failing. Data ingestion is not affected and all data is safe.

Older data temporarily unavailable

Mon, 20 Jul 2026 08:59:00 -0000

We've fixed the issue and added more monitoring to prevent this from happening in the future!

Older data temporarily unavailable

Mon, 20 Jul 2026 07:36:00 -0000

We're working on the servers that hold data from between 24 months ago and 3 months ago. Data in that time period will not be available for queries during that time. Thanks for your understanding. No data is lost.

Older Data Temporarily Unavailable

Mon, 13 Jul 2026 11:58:00 -0000

All back online!

Older Data Temporarily Unavailable

Mon, 13 Jul 2026 09:23:00 -0000

API Slowdowns

Tue, 23 Jun 2026 13:07:00 -0000

We've implemented a root cause and have deployed a fix. The fix is affecting the deprecated V3/Query/Async API, so if you're using that, we strongly recommend to use the v4/query/tql API instead.

API Slowdowns

Mon, 22 Jun 2026 08:18:00 -0000

We are aware of our APIs being periodically slow and/or unavailable for individual requests. This seems to be because our number of users have increased considerably. We're investigating and working on various fixes. Thanks for your patience <3

Infrastructure Update

Mon, 15 Jun 2026 07:15:37 -0000

We're updating our infrastructure. Please allow for some delays or failed queries.

Planned Maintenance

Fri, 01 May 2026 09:45:15 -0000

We're updating our systems and there might be short blips of unavailability.

Ingest API DNS Change

Tue, 21 Apr 2026 16:55:32 -0000

We're updating the DNS servers for nom.telemetrydeck.com. We're not expecting any downtime, and will be monitoring during the changeover. In case the servers are unavailable, the TelemetryDeck SDK should automatically retry sending undeliverable events.

Dashboard Issue

Mon, 20 Apr 2026 20:35:00 -0000

We'r marking the issue as resolved but will continue monitoring.

Dashboard Issue

Mon, 20 Apr 2026 20:25:00 -0000

Our network is back up, but we're continuing to monitor the situation.

Dashboard Issue

Mon, 20 Apr 2026 19:43:00 -0000

We're experiencing a problem with our network backbone that is resulting in flaky connectivity an errors in the query API. We're investigating.

Short maintenance

Fri, 17 Apr 2026 13:46:46 -0000

We're updating an internal component, which will make the API unavailable for a few minutes

Read-only Mode

Tue, 14 Apr 2026 07:00:00 -0000

The TelemetryDeck Dashboard will switch to read-only mode for an hour for a planned database migration.

Data Duplication Incident

Thu, 26 Mar 2026 23:59:00 -0000

The incident has been resolved.

Data Duplication Incident

Thu, 26 Mar 2026 09:12:00 -0000

The issue has been resolved for all paid customers.

Data Duplication Incident

Wed, 25 Mar 2026 17:10:00 -0000

Backups are being installed successfully. Most customers should have their clean data back by now, we're working on some stragglers.

Data Duplication Incident

Wed, 25 Mar 2026 13:17:00 -0000

We've begun replaying backups for affected customers.

Data Duplication Incident

Wed, 25 Mar 2026 11:24:00 -0000

The affected time range is March 16 through March 24, and we've identified affected namespaces. We'll start reloading those customers from backup after some additional checks.

Data Duplication Incident

Wed, 25 Mar 2026 10:07:00 -0000

We're investigating reports of duplicated events for some or all customers. The affected time range seems to be the last 7 days. We're exploring restoring that data from backups.

Some queries not running

Thu, 12 Feb 2026 11:58:00 -0000

The issue has been identified and resolved. Some servers were running out of RAM at the same time as a large number of big queries came in. We've updated the configuration so that this is less likely to happen in the future.

Some queries not running

Thu, 12 Feb 2026 09:28:00 -0000

We're investigating reports that some queries are timing out.

Scheduled Query Server Maintenance

Mon, 09 Feb 2026 17:00:03 -0000

We're updating our query servers. We're expecting no downtime but slightly degraded performance.

Delayed Data

Fri, 23 Jan 2026 17:00:00 -0000

We can now provide at least once-an-hour updates to paid customers again, and will probably also include free customers who regularly open the dashboard to this as well, while working on restoring real-time for as many of our customers as possible. In the meantime, we'll communicate the current ingestion mode for your namespace directly in the dashboard.

Delayed Data

Fri, 16 Jan 2026 14:24:00 -0000

We're slowly returning faster updates. If you're a free customer, you should see your data update once a day. If you're a paid customer, you should see your data update once per hour.

Delayed Data

Thu, 15 Jan 2026 15:33:00 -0000

We have identified workarounds and are starting to implement them.

Delayed Data

Thu, 15 Jan 2026 01:39:00 -0000

Due to the outage, we're currently only able to update data sets every 24 hours. We apologize for the inconvenience and are working hard to fix this as soon as possible.

Delayed Data

Wed, 14 Jan 2026 21:38:00 -0000

Due to a slow down at our hosting provider, events are arriving slower than normal for some customers. All data is safe, and we're monitoring the situation and deploying mitigations.

Dashboard Queries unreliable

Fri, 09 Jan 2026 18:00:00 -0000

All data is back available

Dashboard Queries unreliable

Fri, 09 Jan 2026 16:33:00 -0000

The last 4 months of data are now available.

Dashboard Queries unreliable

Fri, 09 Jan 2026 12:50:00 -0000

Currently, only the most recent data is available. Data will become available from newest to oldest in the next few hours.

Dashboard Queries unreliable

Fri, 09 Jan 2026 11:23:00 -0000

We're still working on the root cause of the problem and apologize for the inconvenience.

Dashboard Queries unreliable

Fri, 09 Jan 2026 09:58:00 -0000

We've isolate the incident to just our legacy data source, so querying a namespace should work fine.

Dashboard Queries unreliable

Fri, 09 Jan 2026 09:42:00 -0000

We're investigating an issue where queries are not always running.

Scheduled Maintenance

Thu, 08 Jan 2026 20:10:00 -0000

We're updating our systems on the query server, so individual queries might time out occasionally.

Dashboard Queries unreliable

Mon, 15 Dec 2025 14:44:00 -0000

We've fixed a bug that caused some queries to not run properly for users with multiple organizations.

Dashboard Queries unreliable

Mon, 15 Dec 2025 14:22:00 -0000

We're seeing reports that some customers are getting empty results in insights and charts, and are investigating.

Query Queue Service is Flaky

Tue, 09 Dec 2025 15:51:00 -0000

We've fixed the cache server that caused the query queue to be flaky.

Query Queue Service is Flaky

Tue, 09 Dec 2025 15:41:00 -0000

Our query queue is not working correctly and queries are running slow and queueing up. We're actively working on it and we should be back to normal shortly. In the meantime, we recommend turning on "Sync Queries" at https://dashboard.telemetrydeck.com/namespace to basically skip the queue

Monday Maintenance

Mon, 24 Nov 2025 08:30:16 -0000

We're updating our servers and some of them might have to reboot. We expect no full downtime, but please allow for some slowdownn.

Monday Maintenance

Mon, 17 Nov 2025 09:55:24 -0000

We're updating our servers and some of them might have to reboot. We expect no full downtime, but please allow for some slowdownn.

Queueing System Maintenance

Sun, 16 Nov 2025 10:00:31 -0000

We're updating parts of our queue system. Queries might not run or run delayed.

Monday Maintenance

Mon, 03 Nov 2025 11:00:09 -0000

Every Monday we update various services and servers to keep them safe and secure.

Dashboard is unavailable

Wed, 29 Oct 2025 19:17:00 -0000

The outage has been fixed and dashboard.telemetrydeck.com is available again.

Dashboard is unavailable

Wed, 29 Oct 2025 18:28:00 -0000

The TelemetryDeck dashboard is currently unavailable. There is a DNS outage concerning Microsoft Azure, which we are using to host the static files for the dashboard. We apologize for the inconvenience and hope to be back online as soon as possible.

Login not working

Tue, 14 Oct 2025 15:31:00 -0000

Login not working

Tue, 14 Oct 2025 14:31:00 -0000

Our authentication system is having a bug where all users are marked as unauthorized. Sorry about that, should be back in a few minutes!

Frontend API Problems

Sun, 21 Sep 2025 19:02:00 -0000

We've fixed a problem with our caching server and things should be back to working order again.

Frontend API Problems

Sun, 21 Sep 2025 18:52:00 -0000

Our caching and database servers are reporting errors and we're investigating.

Query Server Maintenance

Thu, 18 Sep 2025 09:15:03 -0000

We're updating query servers for slow and old datasources. If you look at data older than one year, or if you're not using a namespace yet, you might encounter slowdowns or missing data until the end of the maintenance window.

Query Server Maintenance

Mon, 15 Sep 2025 11:20:16 -0000

Our query servers are restarting for security updates and won't be able to respond to queries for a few minutes.

Query Servers Maintenance

Thu, 21 Aug 2025 13:00:53 -0000

We're updating our query servers. Queries might not run, or run slowly.

Problems Managing Apps

Fri, 15 Aug 2025 11:14:00 -0000

We're deploying a possible fix to chaching and organization permissions management and will monitor the reported errors, .

Problems Managing Apps

Fri, 15 Aug 2025 10:41:00 -0000

Some users are reporting issues when creating apps, dashboards, or insights. We're investigating.

Data Lagging Behind

Wed, 13 Aug 2025 10:49:00 -0000

Our Namespaced datasets have caught up and data is real-time again

Data Lagging Behind

Wed, 13 Aug 2025 07:55:00 -0000

Our servers are currently catching up on analytics data, and are still a few hours behind. They're catching up fast and should reach live within a few hours.

Cache Servers Maintenance

Thu, 07 Aug 2025 09:00:00 -0000

We're updating our cache servers, which might impact performance and the query queue.

Data Lagging Behind

Wed, 30 Jul 2025 21:21:00 -0000

The issue has been resolved and data is being received as normal again.

Data Lagging Behind

Wed, 30 Jul 2025 19:53:00 -0000

The main dataset has caught up with the current time and our servers are now updating namespaces to catch up as well.

Data Lagging Behind

Wed, 30 Jul 2025 09:02:00 -0000

Our servers are currently catching up on analytics data, and are still a few hours behind. They're catching up fast and should reach live within a few hours.

Queries not working

Wed, 16 Jul 2025 13:41:00 -0000

We've resolved the caching issue and queries should now be back to working order.

Queries not working

Wed, 16 Jul 2025 13:38:00 -0000

Due to an issue with our caching servers, queries are running very slowly or not at all. We're on it and hope to resolve this shortly.

Datasource telemetry-signals temporarily not available

Tue, 08 Jul 2025 07:49:00 -0000

We've identified the root cause for the issue and have begun reloading data. Queries are working again and we're viewing the issue as resolved.

Datasource telemetry-signals temporarily not available

Tue, 08 Jul 2025 03:20:00 -0000

Due to an error at our hosting provider, detail data is not available for customers using the dashboard without the namespaces features. We apologize for the inconvenience and will resolve this as soon as possible.

API Servers Update

Fri, 13 Jun 2025 19:30:01 -0000

We're updating our API Servers, which might lead to the dashboard not being fully reliable during the process.

Dashboard unavailable

Tue, 10 Dec 2024 15:28:00 -0000

We've updated our infrastructure such that we can sustain the traffic surge. It has also stopped for now and we're back to regular operations.

Dashboard unavailable

Tue, 10 Dec 2024 11:00:00 -0000

We’re experiencing a large traffic spike and our performance is degraded. Investigation mitigations.

Servers Updating

Sun, 05 May 2024 05:00:00 -0000

Maintenance completed

Servers Updating

Sun, 05 May 2024 00:00:00 -0000

We're updating the installed version of Kubernetes on our clusters, which will result in services sporadically not being available for a few minutes at a time.