System Status
ClawBotCloud runs entirely on Fly.io infrastructure. If Fly.io is experiencing issues, your bots may be affected.
Partially Degraded Service
Last updated 3 minutes ago
Active Incidents
Most networking is largely healthy between primary North American regions. Some Machines may see ongoing packet loss and higher latency communicating with other Machines on certain routes. We're continuing to monitor the backbone health upstream.
We are currently investigating degraded network performance between sites in NA due to an upstream incident
Components
Core Infrastructure
Key Regions
Recent Resolved Incidents
Network issues in SIN, NRT1 day agomajor
This incident has been resolved.
Our upstream provider is continuing to experience network issues in SIN and NRT regions. Apps running in those regions may be unreachable or experience high packet loss at this time.
SIN, NRT network issues1 day agomajor
This incident has been resolved.
We are continuing to experience network issues with an upstream provider in SIN and NRT regions.
A fix has been implemented and we are monitoring the results.
Our upstream provider has identified the issue and is working on a fix. Apps running in NRT (Tokyo) region may also have issues reaching certain destinations at this time.
We are investigating an upstream network issue in the SIN (Singapore) region. Apps may be unreachable or have higher packet loss.
Log search unavailable4 days agominor
Most queued historical logs have been ingested and should now be available through log search. Log ingestion rates have returned to normal levels.
We've applied a fix for this issue. Historical logs are currently backfilling. We will post an update once logs have finished backfilling and current logs are being ingested normally.
Log search is available; however, new app logs since ~1 hour ago are missing and new logs are not being ingested. We are continuing to investigate.
We are investigating an issue causing application log search to be unavailable. This is affecting the Fly Metrics log search panels, and historical application logs initially returned from the `fly logs` command. Streaming logs using `fly logs`, the Live Logs page in the dashboard, and Fly Log Shipper services continue to work as expected.
Network Issues in SIN7 days agomajor
This incident has been resolved.
Network connectivity in SIN has been fully restored. We're continuing to monitor.
We are seeing recovery of network connectivity between SIN and most destinations. We're continuing to work with our upstream provider to resolve the remaining issues.
Some machines in SIN are unreachable. A few Managed Postgres clusters may fail to fail-over or update. We are in the process of fixing this with our upstream provider.
We are currently investigating network connectivity issues in the SIN region. Hosted apps may be unavailable.
Macaroon Auth + Machines API Issues8 days agocritical
This incident has been resolved and we are seeing all platform functions operate normally.
A fix has been implemented and we are monitoring the results.
We have deployed another change and are seeing wider improvements in platform stability across all regions. Performance is trending to normal, though users may still see some degradation at this time. We are continuing to closely monitor to ensure full, stable recovery. We will provide another update in 15m.
We are seeing elevated cluster errors with Managed Postgres clusters as the MPG control plane recovers from the API outage. MPG Users may see elevated rates of failing or slow connections, as well as increased primary/replica failovers. The managed postgres team is addressing any degraded clusters. We will provide a further update within 15m.
We continue to seeing degraded performance and increased errors with the Machines API and other platform features at this time. We are continuing to work on fully restoring service.
An initial fix has been deployed and we are starting to see platform features recover. Users may still see degraded performance and intermittent failures at this time. We are continuing to address the issue to ensure a full stable recovery.
We have identified the cause of the issue and are working on deploying a fix. Impacted features remain unavailable or degraded at this time. Already running customer applications/machines remain available. MPG clusters remain generally reachable and healthy, however new clusters cannot be provisioned and failovers may not complete. We will provide another update within 15 minutes.
We are continuing to address this issue. Platform authentication with macaroon based tokens is currently failing. Platform features that authenticate with macaroons including Machines API operations, Dashboard logins, some flyctl commands, fly-metrics.net Grafana, and deployments are failing at this time. Existing, running customer applications and machines remain reachable and running. We will provide another update within 15 minutes
We are investigating issues with Macaroon based authentication. This is impacting parts of the Machines API, Fly.io Dashboard, some flyctl operations and other platform features that rely on this.