WarpBuild Status - Incident history https://warpbuild.instatus.com WarpBuild Mon, 18 May 2026 06:03:49 +0000 WarpBuild status page migrated to a new home: https://status.warpbuild.com/ Type: Incident May 18, 06:03:49 GMT+0 - Resolved - WarpBuild status page migrated to a new home: <https://status.warpbuild.com/> Please resubscribe for email/slack updates. Type: Incident

<p><small>May <var data-var='date'> 18</var>, <var data-var='time'>06:03:49</var> GMT+0</small><br><strong>Resolved</strong> - WarpBuild status page migrated to a new home: <https://status.warpbuild.com/> Please resubscribe for email/slack updates..</p> ]]> Mon, 18 May 2026 06:03:49 +0000 https://warpbuild.instatus.com/incident/cmpastefm08gmqjjg9gh14z40 https://warpbuild.instatus.com/incident/cmpastefm08gmqjjg9gh14z40 Scheduled maintenance for infrastructure upgrades Type: Maintenance Duration: 1 hour Affected Components: Dashboard, API, GitHub Runners, Docker Builders May 9, 05:30:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. There may be \~5m delays in scheduling during this time. May 9, 05:30:01 GMT+0 - Identified - Maintenance is now in progress May 9, 06:30:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 1 hour

Affected Components: , , ,

<p><small>May <var data-var='date'> 9</var>, <var data-var='time'>05:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time. There may be \~5m delays in scheduling during this time..</p> <p><small>May <var data-var='date'> 9</var>, <var data-var='time'>05:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>May <var data-var='date'> 9</var>, <var data-var='time'>06:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Sat, 9 May 2026 05:30:00 +0000 https://warpbuild.instatus.com/maintenance/cmowce0mx03bysn35yultzgol https://warpbuild.instatus.com/maintenance/cmowce0mx03bysn35yultzgol Linux runners on x86-64 arch scheduling is delayed Type: Incident Duration: 19 minutes Affected Components: GitHub Runners Mar 5, 10:45:00 GMT+0 - Identified - We are seeing delays in job pick up times of up to 10 mins. Mar 5, 11:03:31 GMT+0 - Resolved - This incident has been resolved. The queues are now back to normal. Type: Incident

Duration: 19 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>10:45:00</var> GMT+0</small><br><strong>Identified</strong> - We are seeing delays in job pick up times of up to 10 mins..</p> <p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>11:03:31</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. The queues are now back to normal..</p> ]]> Thu, 5 Mar 2026 10:45:00 +0000 https://warpbuild.instatus.com/incident/cmmdcx5bp0024snpg8m68ivx2 https://warpbuild.instatus.com/incident/cmmdcx5bp0024snpg8m68ivx2 MacOS runners are seeing delays in scheduling Type: Incident Duration: 1 hour and 12 minutes Affected Components: Dashboard, API, GitHub Runners, Docker Builders Feb 13, 12:00:54 GMT+0 - Identified - We have identified the problem and are continuing to work on a fix for this incident. Feb 13, 12:33:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Feb 13, 13:13:20 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 12 minutes

Affected Components: , , ,

<p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>12:00:54</var> GMT+0</small><br><strong>Identified</strong> - We have identified the problem and are continuing to work on a fix for this incident..</p> <p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>12:33:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Feb <var data-var='date'> 13</var>, <var data-var='time'>13:13:20</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 13 Feb 2026 12:00:54 +0000 https://warpbuild.instatus.com/incident/cmlku6i9y00pa13evljv1rbb8 https://warpbuild.instatus.com/incident/cmlku6i9y00pa13evljv1rbb8 Degradation in upstream GitHub services causing jobs to not get picked up / timeout Type: Incident Duration: 22 minutes Affected Components: Dashboard, API, GitHub Runners, Docker Builders Jan 28, 23:37:25 GMT+0 - Investigating - We are currently investigating this incident. Jan 28, 23:59:40 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 22 minutes

Affected Components: , , ,

<p><small>Jan <var data-var='date'> 28</var>, <var data-var='time'>23:37:25</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jan <var data-var='date'> 28</var>, <var data-var='time'>23:59:40</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 28 Jan 2026 23:37:25 +0000 https://warpbuild.instatus.com/incident/cmkyo0m7800yuppzbcal5wztt https://warpbuild.instatus.com/incident/cmkyo0m7800yuppzbcal5wztt Upstream GitHub issue - API and actions degraded Type: Incident Duration: 34 minutes Affected Components: Third Party: Github → Actions, Third Party: Github → API Requests Dec 11, 16:40:01 GMT+0 - Resolved - The GitHub incident doesn't seem to have any real impact. Dec 11, 16:06:00 GMT+0 - Identified - API Requests and Actions are experiencing degraded performance. <https://www.githubstatus.com/incidents/xntfc1fz5rfb> Type: Incident

Duration: 34 minutes

Affected Components: ,

<p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>16:40:01</var> GMT+0</small><br><strong>Resolved</strong> - The GitHub incident doesn't seem to have any real impact..</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>16:06:00</var> GMT+0</small><br><strong>Identified</strong> - API Requests and Actions are experiencing degraded performance. <https://www.githubstatus.com/incidents/xntfc1fz5rfb>.</p> ]]> Thu, 11 Dec 2025 16:06:00 +0000 https://warpbuild.instatus.com/incident/cmj1mrucu00xcbf5g13fw92h6 https://warpbuild.instatus.com/incident/cmj1mrucu00xcbf5g13fw92h6 Runners are not allocated or are failing to start Type: Incident Duration: 1 hour and 19 minutes Affected Components: Dashboard, API, GitHub Runners, Docker Builders Dec 11, 14:00:39 GMT+0 - Investigating - Runners were either not starting or failing to start with a 503 after allocation. Dec 11, 14:30:39 GMT+0 - Identified - We are working on a fix for this Dec 11, 14:50:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Runners are working as usual now. Dec 11, 15:20:00 GMT+0 - Resolved - This incident has been resolved. A full post-incident report will be published shortly along with mitigation steps. Dec 11, 15:21:00 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 19 minutes

Affected Components: , , ,

<p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>14:00:39</var> GMT+0</small><br><strong>Investigating</strong> - Runners were either not starting or failing to start with a 503 after allocation..</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>14:30:39</var> GMT+0</small><br><strong>Identified</strong> - We are working on a fix for this.</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>14:50:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result. Runners are working as usual now..</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>15:20:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. A full post-incident report will be published shortly along with mitigation steps..</p> <p><small>Dec <var data-var='date'> 11</var>, <var data-var='time'>15:21:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Thu, 11 Dec 2025 14:00:39 +0000 https://warpbuild.instatus.com/incident/cmj1lilnm00o7ws65rkaamur1 https://warpbuild.instatus.com/incident/cmj1lilnm00o7ws65rkaamur1 WarpBuild Runners are not starting with a "failed to fetch" error Type: Incident Duration: 7 minutes Affected Components: GitHub Runners Dec 2, 07:16:59 GMT+0 - Investigating - We are currently investigating this incident. Dec 2, 07:23:47 GMT+0 - Resolved - This incident has been resolved. Dec 2, 07:48:34 GMT+0 - Postmortem - The issue was caused because of a faulty change to the `auth` service during a refactor. This was not expected to have an impact on the runner allocation flow. This was reverted to fix the error. We are also going to add tests to ensure issues of this nature do not creep through. Type: Incident

Duration: 7 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>07:16:59</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>07:23:47</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>07:48:34</var> GMT+0</small><br><strong>Postmortem</strong> - The issue was caused because of a faulty change to the `auth` service during a refactor. This was not expected to have an impact on the runner allocation flow. This was reverted to fix the error. We are also going to add tests to ensure issues of this nature do not creep through..</p> ]]> Tue, 2 Dec 2025 07:16:59 +0000 https://warpbuild.instatus.com/incident/cmio8w7f1015udkq5tdrw2dc0 https://warpbuild.instatus.com/incident/cmio8w7f1015udkq5tdrw2dc0 Github reporting degradation for Actions Type: Incident Duration: 17 minutes Affected Components: Third Party: Github → Actions, GitHub Runners Nov 19, 17:52:12 GMT+0 - Identified - GitHub is investigating an actions related outage - [https://www.githubstatus.com/incidents/zs5ccnvqv64m.](https://www.githubstatus.com/incidents/zs5ccnvqv64m) This is leading to queued jobs not getting picked up. Nov 19, 18:02:35 GMT+0 - Monitoring - GitHub applied a mitigation. We are seeing recovery and monitoring the situation. Nov 19, 18:09:34 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 17 minutes

Affected Components: ,

<p><small>Nov <var data-var='date'> 19</var>, <var data-var='time'>17:52:12</var> GMT+0</small><br><strong>Identified</strong> - GitHub is investigating an actions related outage - [https://www.githubstatus.com/incidents/zs5ccnvqv64m.](https://www.githubstatus.com/incidents/zs5ccnvqv64m) This is leading to queued jobs not getting picked up..</p> <p><small>Nov <var data-var='date'> 19</var>, <var data-var='time'>18:02:35</var> GMT+0</small><br><strong>Monitoring</strong> - GitHub applied a mitigation. We are seeing recovery and monitoring the situation..</p> <p><small>Nov <var data-var='date'> 19</var>, <var data-var='time'>18:09:34</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 19 Nov 2025 17:52:12 +0000 https://warpbuild.instatus.com/incident/cmi6av0rk008x104t5u0pvkzd https://warpbuild.instatus.com/incident/cmi6av0rk008x104t5u0pvkzd GitHub reporting failures with Git operations Type: Incident Duration: 1 hour and 19 minutes Affected Components: Third Party: Github → Git Operations, Third Party: Github → Webhooks, Third Party: Github → Actions, Third Party: Github → API Requests Nov 18, 20:41:00 GMT+0 - Identified - There is an increase in job failures because of an incident with Github <https://www.githubstatus.com/incidents/5q7nmlxz30sk>. We are monitoring the situation. Nov 18, 22:00:17 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 19 minutes

Affected Components: , , ,

<p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>20:41:00</var> GMT+0</small><br><strong>Identified</strong> - There is an increase in job failures because of an incident with Github <https://www.githubstatus.com/incidents/5q7nmlxz30sk>. We are monitoring the situation..</p> <p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>22:00:17</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 18 Nov 2025 20:41:00 +0000 https://warpbuild.instatus.com/incident/cmi52uico004zy41owmyv1h2g https://warpbuild.instatus.com/incident/cmi52uico004zy41owmyv1h2g WarpBuild services are down because of cloudflare global outage Type: Incident Duration: 2 hours and 35 minutes Affected Components: API, Dashboard, , Third Party: Cloudflare → Cloudflare Sites and Services → R2, Third Party: Cloudflare → Cloudflare Sites and Services → API, GitHub Runners, Docker Builders, Cloudflare → Nov 18, 11:40:00 GMT+0 - Investigating - This issue is being added retroactively because the status page was also down. <https://www.cloudflarestatus.com/incidents/8gmgl950y3h7> Nov 18, 13:30:00 GMT+0 - Resolved - WarpBuild is seeing full recovery. Type: Incident

Duration: 2 hours and 35 minutes

Affected Components: , , , , , ,

<p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>11:40:00</var> GMT+0</small><br><strong>Investigating</strong> - This issue is being added retroactively because the status page was also down. <https://www.cloudflarestatus.com/incidents/8gmgl950y3h7>.</p> <p><small>Nov <var data-var='date'> 18</var>, <var data-var='time'>13:30:00</var> GMT+0</small><br><strong>Resolved</strong> - WarpBuild is seeing full recovery. .</p> ]]> Tue, 18 Nov 2025 11:40:00 +0000 https://warpbuild.instatus.com/incident/cmi4qxbc7007wcest7dltfl3y https://warpbuild.instatus.com/incident/cmi4qxbc7007wcest7dltfl3y Remote docker builders and arm64 runners are not functional because of the AWS outage Type: Incident Duration: 13 hours Affected Components: GitHub Runners, Docker Builders Oct 20, 09:00:00 GMT+0 - Identified - Remote docker builders and arm64 runners are not functional because of the AWS outage There may be some delay in jobs getting picked up because of queueing and image related delays. Windows, custom runners, and snapshot runners are also affected. Oct 20, 18:19:25 GMT+0 - Identified - Windows runners are also impacted because of the AWS outage. Oct 20, 21:54:50 GMT+0 - Monitoring - We are seeing full recovery in the impacted services: * Docker builders * arm64 runners * custom runners * Snapshot runners * Windows runners x64 runners were not impacted and are continuing to work as usual. Oct 20, 22:00:00 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 13 hours

Affected Components: ,

<p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>09:00:00</var> GMT+0</small><br><strong>Identified</strong> - Remote docker builders and arm64 runners are not functional because of the AWS outage There may be some delay in jobs getting picked up because of queueing and image related delays. Windows, custom runners, and snapshot runners are also affected..</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>18:19:25</var> GMT+0</small><br><strong>Identified</strong> - Windows runners are also impacted because of the AWS outage..</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>21:54:50</var> GMT+0</small><br><strong>Monitoring</strong> - We are seeing full recovery in the impacted services: * Docker builders * arm64 runners * custom runners * Snapshot runners * Windows runners x64 runners were not impacted and are continuing to work as usual..</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 20 Oct 2025 09:00:00 +0000 https://warpbuild.instatus.com/incident/cmgzgiz8303is55kx8hbmfs2w https://warpbuild.instatus.com/incident/cmgzgiz8303is55kx8hbmfs2w [No customer impact] Investigating a potential issue impacting some runners Type: Incident Duration: 19 minutes Affected Components: API, GitHub Runners Sep 16, 03:46:05 GMT+0 - Investigating - Tailscale has an ongoing incident that may affect some runner types in WarpBuild. <https://status.tailscale.com/incidents/01K589GKH7AGJ2VBY131HM4ZD1> Sep 16, 03:52:48 GMT+0 - Monitoring - There was no user impact because of the tailscale outage and systems are working as expected. Continuing to monitor the situation. Sep 16, 04:04:44 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 19 minutes

Affected Components: ,

<p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>03:46:05</var> GMT+0</small><br><strong>Investigating</strong> - Tailscale has an ongoing incident that may affect some runner types in WarpBuild. <https://status.tailscale.com/incidents/01K589GKH7AGJ2VBY131HM4ZD1>.</p> <p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>03:52:48</var> GMT+0</small><br><strong>Monitoring</strong> - There was no user impact because of the tailscale outage and systems are working as expected. Continuing to monitor the situation..</p> <p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>04:04:44</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 16 Sep 2025 03:46:05 +0000 https://warpbuild.instatus.com/incident/cmfm0getv0262v7mp2xtdu0kf https://warpbuild.instatus.com/incident/cmfm0getv0262v7mp2xtdu0kf Intermittent issues with network speeds Type: Incident Duration: 9 days and 51 minutes Affected Components: GitHub Runners Jul 14, 10:00:00 GMT+0 - Identified - We received sporadic notifications of low network throughput. Jul 23, 10:51:21 GMT+0 - Resolved - A small set of servers had network performance issues. These have been replaced and the network related issues are resolved. Type: Incident

Duration: 9 days and 51 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 14</var>, <var data-var='time'>10:00:00</var> GMT+0</small><br><strong>Identified</strong> - We received sporadic notifications of low network throughput..</p> <p><small>Jul <var data-var='date'> 23</var>, <var data-var='time'>10:51:21</var> GMT+0</small><br><strong>Resolved</strong> - A small set of servers had network performance issues. These have been replaced and the network related issues are resolved..</p> ]]> Mon, 14 Jul 2025 10:00:00 +0000 https://warpbuild.instatus.com/incident/cmdfvodwx00u4j4w4klvnuk9w https://warpbuild.instatus.com/incident/cmdfvodwx00u4j4w4klvnuk9w Service interruption for github runners and docker builders Type: Incident Duration: 17 minutes Affected Components: API, Dashboard, GitHub Runners, Docker Builders Jul 4, 12:32:15 GMT+0 - Resolved - This incident has been resolved. Jul 4, 12:15:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Type: Incident

Duration: 17 minutes

Affected Components: , , ,

<p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>12:32:15</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>12:15:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> ]]> Fri, 4 Jul 2025 12:15:00 +0000 https://warpbuild.instatus.com/incident/cmcosnqbg000sjzwd8561g28h https://warpbuild.instatus.com/incident/cmcosnqbg000sjzwd8561g28h Usage alerts and limits will not work for 24 hours Type: Maintenance Duration: 19 hours and 37 minutes Affected Components: GitHub Runners, Docker Builders Jun 24, 09:37:22 GMT+0 - Completed - Maintenance has completed successfully. Jun 23, 14:00:01 GMT+0 - Identified - Maintenance is now in progress Jun 23, 14:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Type: Maintenance

Duration: 19 hours and 37 minutes

Affected Components: ,

<p><small>Jun <var data-var='date'> 24</var>, <var data-var='time'>09:37:22</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Jun <var data-var='date'> 23</var>, <var data-var='time'>14:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jun <var data-var='date'> 23</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> ]]> Mon, 23 Jun 2025 14:00:00 +0000 https://warpbuild.instatus.com/maintenance/cmc94vr4y0006hbf5ehozu8d3 https://warpbuild.instatus.com/maintenance/cmc94vr4y0006hbf5ehozu8d3 Docker builders using arm64 seeing timeouts Type: Incident Duration: 1 hour and 4 minutes Affected Components: Docker Builders May 23, 08:52:00 GMT+0 - Identified - This was caused by a misconfiguration during testing. We have implemented the fix and are monitoring the situation. May 23, 09:56:18 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 4 minutes

Affected Components:

<p><small>May <var data-var='date'> 23</var>, <var data-var='time'>08:52:00</var> GMT+0</small><br><strong>Identified</strong> - This was caused by a misconfiguration during testing. We have implemented the fix and are monitoring the situation..</p> <p><small>May <var data-var='date'> 23</var>, <var data-var='time'>09:56:18</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 23 May 2025 08:52:00 +0000 https://warpbuild.instatus.com/incident/cmb0mi3o5001t2pmefcx3q5xo https://warpbuild.instatus.com/incident/cmb0mi3o5001t2pmefcx3q5xo 32x instances are seeing delays in startup for some users Type: Incident Duration: 44 minutes Affected Components: GitHub Runners Mar 26, 17:27:33 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 26, 18:11:58 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 44 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 26</var>, <var data-var='time'>17:27:33</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Mar <var data-var='date'> 26</var>, <var data-var='time'>18:11:58</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 26 Mar 2025 17:27:33 +0000 https://warpbuild.instatus.com/incident/cm8q76kt1005z74oe5sh1kkxg https://warpbuild.instatus.com/incident/cm8q76kt1005z74oe5sh1kkxg Scheduled maintenance Type: Maintenance Duration: 35 minutes Affected Components: GitHub Runners Jan 25, 05:00:01 GMT+0 - Identified - Maintenance is now in progress Jan 25, 05:35:13 GMT+0 - Completed - Maintenance has completed successfully. Jan 25, 05:00:00 GMT+0 - Identified - There may be some delays to processing runners during the maintenance window. Type: Maintenance

Duration: 35 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 25</var>, <var data-var='time'>05:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jan <var data-var='date'> 25</var>, <var data-var='time'>05:35:13</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Jan <var data-var='date'> 25</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - There may be some delays to processing runners during the maintenance window..</p> ]]> Sat, 25 Jan 2025 05:00:00 +0000 https://warpbuild.instatus.com/maintenance/cm6aby513005lyb8tszyza2qf https://warpbuild.instatus.com/maintenance/cm6aby513005lyb8tszyza2qf Runners are not starting Type: Incident Duration: 3 hours and 45 minutes Affected Components: GitHub Runners Jan 13, 03:29:51 GMT+0 - Resolved - This incident has been resolved. Jan 12, 23:45:00 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 3 hours and 45 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 13</var>, <var data-var='time'>03:29:51</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jan <var data-var='date'> 12</var>, <var data-var='time'>23:45:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Sun, 12 Jan 2025 23:45:00 +0000 https://warpbuild.instatus.com/incident/cm5ue3qz700hs1276stvjsc3m https://warpbuild.instatus.com/incident/cm5ue3qz700hs1276stvjsc3m Runner misconfiguration causing some jobs and snapshots to fail Type: Incident Duration: 19 days Affected Components: GitHub Runners Dec 3, 10:30:00 GMT+0 - Identified - This incident was because configs for setting up the `runner` user were not applied during a routine runner binary upgrade. This issue affected \~5% of our users. The resolution required a rebuild and rollout of the affected images. This incident has been resolved. Dec 4, 06:30:00 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 19 days

Affected Components:

<p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>10:30:00</var> GMT+0</small><br><strong>Identified</strong> - This incident was because configs for setting up the `runner` user were not applied during a routine runner binary upgrade. This issue affected \~5% of our users. The resolution required a rebuild and rollout of the affected images. This incident has been resolved..</p> <p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>06:30:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 3 Dec 2024 10:30:00 +0000 https://warpbuild.instatus.com/incident/cm4bs3zol001xafcumum8848v https://warpbuild.instatus.com/incident/cm4bs3zol001xafcumum8848v WarpBuild jobs are seeing delays in start Type: Incident Duration: 1 hour and 13 minutes Affected Components: GitHub Runners Jul 12, 14:30:00 GMT+0 - Investigating - We are currently investigating this incident. Jul 12, 14:50:49 GMT+0 - Identified - We are continuing to work on a fix for this incident. Jul 12, 15:12:04 GMT+0 - Identified - AWS EC2 API has an ongoing incident. This is causing instance capacity to run out. Jul 12, 15:17:43 GMT+0 - Monitoring - We're starting to see recovery in the job delays. The backlog should be processed shortly. Jul 12, 15:42:35 GMT+0 - Resolved - AWS had an issue which was causing capacity issues for WarpBuild instances. <https://health.aws.amazon.com/health/status#multipleservices-us-east-1%5F1720795596> This has now been resolved and we have seen full recovery of all WarpBuild services. Type: Incident

Duration: 1 hour and 13 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>14:30:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>14:50:49</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>15:12:04</var> GMT+0</small><br><strong>Identified</strong> - AWS EC2 API has an ongoing incident. This is causing instance capacity to run out..</p> <p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>15:17:43</var> GMT+0</small><br><strong>Monitoring</strong> - We're starting to see recovery in the job delays. The backlog should be processed shortly..</p> <p><small>Jul <var data-var='date'> 12</var>, <var data-var='time'>15:42:35</var> GMT+0</small><br><strong>Resolved</strong> - AWS had an issue which was causing capacity issues for WarpBuild instances. <https://health.aws.amazon.com/health/status#multipleservices-us-east-1%5F1720795596> This has now been resolved and we have seen full recovery of all WarpBuild services..</p> ]]> Fri, 12 Jul 2024 14:30:00 +0000 https://warpbuild.instatus.com/incident/clyitbu6m214706j3o7y6p80yxs https://warpbuild.instatus.com/incident/clyitbu6m214706j3o7y6p80yxs Github is not delivering webhooks Type: Incident Duration: 1 hour and 29 minutes Jul 5, 18:19:00 GMT+0 - Resolved - This incident has been resolved. Jul 5, 16:50:00 GMT+0 - Identified - We are continuing to monitor Github for a fix - <https://www.githubstatus.com/> Type: Incident

Duration: 1 hour and 29 minutes

<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>18:19:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>16:50:00</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to monitor Github for a fix - <https://www.githubstatus.com/>.</p> ]]> Fri, 5 Jul 2024 16:50:00 +0000 https://warpbuild.instatus.com/incident/cly8yd02y132695iclr0wh8ahfi https://warpbuild.instatus.com/incident/cly8yd02y132695iclr0wh8ahfi Jobs stuck in queued state Type: Incident Duration: 6 hours and 17 minutes Affected Components: Dashboard, API, GitHub Runners May 21, 12:49:06 GMT+0 - Monitoring - GitHub is currently facing an issue with their actions API, where the status of newly created jobs is stuck in the queued state. May 21, 19:05:45 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 6 hours and 17 minutes

Affected Components: , ,

<p><small>May <var data-var='date'> 21</var>, <var data-var='time'>12:49:06</var> GMT+0</small><br><strong>Monitoring</strong> - GitHub is currently facing an issue with their actions API, where the status of newly created jobs is stuck in the queued state..</p> <p><small>May <var data-var='date'> 21</var>, <var data-var='time'>19:05:45</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 21 May 2024 12:49:06 +0000 https://warpbuild.instatus.com/incident/clwge6aem7441bsopkqxgl3gd https://warpbuild.instatus.com/incident/clwge6aem7441bsopkqxgl3gd Runner delays in getting picked up Type: Incident Duration: 34 minutes Affected Components: GitHub Runners May 16, 23:51:05 GMT+0 - Investigating - We are seeing a spike in runner pick up delays. Currently investigating this incident. May 17, 00:19:18 GMT+0 - Monitoring - The issue is because of a GCP outage. We are seeing recovery now. May 17, 00:25:15 GMT+0 - Resolved - This incident has been resolved. All pending jobs have been processed. Rootcause: Major service disruption to GCP in the America regions across VPC, GKE, and other services. Type: Incident

Duration: 34 minutes

Affected Components:

<p><small>May <var data-var='date'> 16</var>, <var data-var='time'>23:51:05</var> GMT+0</small><br><strong>Investigating</strong> - We are seeing a spike in runner pick up delays. Currently investigating this incident..</p> <p><small>May <var data-var='date'> 17</var>, <var data-var='time'>00:19:18</var> GMT+0</small><br><strong>Monitoring</strong> - The issue is because of a GCP outage. We are seeing recovery now..</p> <p><small>May <var data-var='date'> 17</var>, <var data-var='time'>00:25:15</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. All pending jobs have been processed. Rootcause: Major service disruption to GCP in the America regions across VPC, GKE, and other services..</p> ]]> Thu, 16 May 2024 23:51:05 +0000 https://warpbuild.instatus.com/incident/clw9wmd73129841aoopr0ep47oe https://warpbuild.instatus.com/incident/clw9wmd73129841aoopr0ep47oe Runners experiencing delays Type: Incident Duration: 40 minutes Affected Components: GitHub Runners May 13, 22:57:44 GMT+0 - Investigating - There was a Github actions downtime which led to some of WarpBuild runners seeing delays. We are working on a fix. May 13, 23:37:23 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 40 minutes

Affected Components:

<p><small>May <var data-var='date'> 13</var>, <var data-var='time'>22:57:44</var> GMT+0</small><br><strong>Investigating</strong> - There was a Github actions downtime which led to some of WarpBuild runners seeing delays. We are working on a fix..</p> <p><small>May <var data-var='date'> 13</var>, <var data-var='time'>23:37:23</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 13 May 2024 22:57:44 +0000 https://warpbuild.instatus.com/incident/clw5ke8mu37255bmo6ye97z6ey https://warpbuild.instatus.com/incident/clw5ke8mu37255bmo6ye97z6ey Intermittent networking slowdown issue with arm64 and amd64 runners Type: Incident Duration: 3 hours Affected Components: GitHub Runners Apr 26, 15:00:00 GMT+0 - Resolved - This incident has been resolved. Apr 26, 12:00:00 GMT+0 - Identified - The network interruption was because of a faulty configuration of network MTU pushed to the runner instances. This caused some workloads to drop packets or hang. Type: Incident

Duration: 3 hours

Affected Components:

<p><small>Apr <var data-var='date'> 26</var>, <var data-var='time'>15:00:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 26</var>, <var data-var='time'>12:00:00</var> GMT+0</small><br><strong>Identified</strong> - The network interruption was because of a faulty configuration of network MTU pushed to the runner instances. This caused some workloads to drop packets or hang..</p> ]]> Fri, 26 Apr 2024 12:00:00 +0000 https://warpbuild.instatus.com/incident/clvjlri9i0721cgo8ekixjxqc https://warpbuild.instatus.com/incident/clvjlri9i0721cgo8ekixjxqc ARM64 runners are timing out Type: Incident Duration: 2 hours and 59 minutes Affected Components: GitHub Runners Apr 19, 12:41:13 GMT+0 - Identified - We are continuing to work on a fix for this incident. Apr 19, 15:40:21 GMT+0 - Resolved - This incident has been resolved. Apr 19, 13:03:43 GMT+0 - Monitoring - Recovery is in progress. Type: Incident

Duration: 2 hours and 59 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 19</var>, <var data-var='time'>12:41:13</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Apr <var data-var='date'> 19</var>, <var data-var='time'>15:40:21</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 19</var>, <var data-var='time'>13:03:43</var> GMT+0</small><br><strong>Monitoring</strong> - Recovery is in progress..</p> ]]> Fri, 19 Apr 2024 12:41:13 +0000 https://warpbuild.instatus.com/incident/clv6nsvmn239259ben4o5z54p54 https://warpbuild.instatus.com/incident/clv6nsvmn239259ben4o5z54p54