WarpBuild - Notice history

Third Party: Cloudflare → Cloudflare Sites and Services → R2 experiencing partial outage

Dashboard - Operational

100% - uptime
Mar 2024
Apr 2024
May 2024

API - Operational

100% - uptime
Mar 2024
Apr 2024
May 2024

GitHub Runners - Operational

100% - uptime
Mar 2024
Apr 2024
May 2024

Docker Builders - Operational

100% - uptime
Mar 2024
Apr 2024
May 2024
Operational

Third Party: Github → Actions - Operational

Third Party: Github → API Requests - Operational

Third Party: Github → Webhooks - Operational

Third Party: Github → Git Operations - Operational

Third Party: Github → Pull Requests - Operational

Notice history

May 2024

Jobs stuck in queued state
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    GitHub is currently facing an issue with their actions API, where the status of newly created jobs is stuck in the queued state.

Runner delays in getting picked up
  • Resolved
    UTC
    Resolved

    This incident has been resolved. All pending jobs have been processed.

    Rootcause: Major service disruption to GCP in the America regions across VPC, GKE, and other services.

  • Monitoring
    UTC
    Monitoring

    The issue is because of a GCP outage. We are seeing recovery now.

  • Investigating
    UTC
    Investigating

    We are seeing a spike in runner pick up delays. Currently investigating this incident.

Runners experiencing delays
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Investigating
    UTC
    Investigating

    There was a Github actions downtime which led to some of WarpBuild runners seeing delays. We are working on a fix.

Apr 2024

Intermittent networking slowdown issue with arm64 and amd64 runners
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    The network interruption was because of a faulty configuration of network MTU pushed to the runner instances. This caused some workloads to drop packets or hang.

ARM64 runners are timing out
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Recovery is in progress.

  • Identified
    UTC
    Identified
    We are continuing to work on a fix for this incident.
Monitoring issues with github.com
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    Monitoring the incident and waiting for the resolution from GitHub.com

Mar 2024

Dockerhub rate limits being hit because the local cache is down
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    the fix has been implemented and we are seeing improvement

  • Identified
    UTC
    Identified
    We are continuing to work on a fix for this incident.
Github.com has partial outage with actions.
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    We are currently monitoring the impact of this incident to WarpBuild users. https://www.githubstatus.com/incidents/7x5z8plb48t6

Github.com is unable to schedule actions
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    WarpBuild systems are catching up with the huge backlog of jobs that have been waiting since the Github outage. Monitoring the situation for improvement now.

  • Identified
    UTC
    Identified

    We are continuing to monitor this incident - https://www.githubstatus.com/incidents/wcl1sw4mzg60

Mar 2024 to May 2024

Next