Codestin Search App

Thanks to visit codestin.com
Credit goes to status.improvmx.com

Codestin Search App Incidents reported on status page for ImprovMX https://status.improvmx.com/ https://d1lppblt9t2x15.cloudfront.net/logos/3c44d4675f92e08a1a2a6a200ef64282.svg Codestin Search App https://status.improvmx.com/ en Codestin Search App https://status.improvmx.com/incident/896437 Fri, 15 May 2026 17:19:00 -0000 https://status.improvmx.com/incident/896437#19225d2c89a6dc4eeef4ffb85981e9b954a80821b2ead33c3cd03525686b181d Postmortem posted here: https://improvmx.com/blog/2026-05-15-postmortem-smtp-tls-cert-expiry/ Codestin Search App https://status.improvmx.com/incident/896437 Fri, 15 May 2026 05:16:00 -0000 https://status.improvmx.com/incident/896437#358e374ad97a5092ab80e509cf06df160a8501a9affd8c06321877ee9c8ff4b0 We have pushed a fresh certificate, and SMTP sending functionality has been restored. We will investigate and post a full postmortem tomorrow. Codestin Search App https://status.improvmx.com/incident/896437 Fri, 15 May 2026 04:42:00 -0000 https://status.improvmx.com/incident/896437#15e6dab43f14a276d473830103230bb6c6c0aea868530f97bff9e6981eac897f Examining reports of expired SMTP certificate affecting ImprovMX SMTP sending. Codestin Search App https://status.improvmx.com/incident/874738 Sat, 18 Apr 2026 22:37:00 -0000 https://status.improvmx.com/incident/874738#fe84b3ad7ef23cf710410abe4c9bd853a12dc0c1928caba60e35ead5e197ccf4 We've identified the problem as isolated to our alerting. No actual downtime for our customers. Codestin Search App https://status.improvmx.com/incident/874738 Sat, 18 Apr 2026 22:16:00 -0000 https://status.improvmx.com/incident/874738#5541c5f2e1611cda5389a9b36cc88f1c254b8de1a1ec35f00559a6e87f02c5ed We're investigating machine reports that SMTP sending is not working. Codestin Search App https://status.improvmx.com/incident/868415 Fri, 10 Apr 2026 02:09:00 -0000 https://status.improvmx.com/incident/868415#8af7f254884fa45b0070fbb2f1123f254214626c99fddf659f3a321058199d17 Deliveries seem to be back to normal, we will root cause this and post a postmortem. Codestin Search App https://status.improvmx.com/incident/868415 Fri, 10 Apr 2026 00:10:00 -0000 https://status.improvmx.com/incident/868415#71afc0b33c8a73fc15b6de141957650fe99d26443b10b957bc172d50488d27d2 We're seeing reports of deliverability issues for all email providers. Codestin Search App https://status.improvmx.com/incident/867268 Thu, 09 Apr 2026 13:02:00 -0000 https://status.improvmx.com/incident/867268#62a942dc8fe91573163d1305e713cd71b6be624284f579d4df5c1678e0c3a780 Postmortem posted to our website: https://improvmx.com/blog/2026-04-09-postmortem-database-outage/ Codestin Search App https://status.improvmx.com/incident/867268 Wed, 08 Apr 2026 21:58:00 -0000 https://status.improvmx.com/incident/867268#91a15c218684a8c50adfa0790c77f612da044c16d2b6d6f242efa732cfe02d03 api and app have recovered, we will continue monitoring. Apologies for the downtime, a full postmortem will be posted tomorrow. Codestin Search App https://status.improvmx.com/ Wed, 08 Apr 2026 21:57:14 +0000 https://status.improvmx.com/#b83e2380b51c72102e8bab204fd5c6accecdbe31a50320c2719c6ea924ab2fd1 api.improvmx.com recovered Codestin Search App https://status.improvmx.com/incident/867268 Wed, 08 Apr 2026 21:55:00 -0000 https://status.improvmx.com/incident/867268#744225e22205773a090e3adf7dc43d631bbc58b2500a7b713b70fb0fc9cb51f8 Mail delivery has recovered. API servers are still degrading and we're investigating Codestin Search App https://status.improvmx.com/incident/867268 Wed, 08 Apr 2026 21:50:00 -0000 https://status.improvmx.com/incident/867268#0b2a7d66ba811af8a4f296d6d3691eb356a6d7501175bc06b47ab24c2c92b4f4 We've killed a bad query on the primary database, and services have recovered. We will continue monitoring. Codestin Search App https://status.improvmx.com/incident/867268 Wed, 08 Apr 2026 21:37:00 -0000 https://status.improvmx.com/incident/867268#252c2f82cbca5f6b428d467a4c0e152f0dd0aa7db5501b9109f6c5f4f66338f9 recurrence of issue, we're investigating Codestin Search App https://status.improvmx.com/ Wed, 08 Apr 2026 21:25:22 +0000 https://status.improvmx.com/#b83e2380b51c72102e8bab204fd5c6accecdbe31a50320c2719c6ea924ab2fd1 api.improvmx.com went down Codestin Search App https://status.improvmx.com/incident/867268 Wed, 08 Apr 2026 17:32:00 -0000 https://status.improvmx.com/incident/867268#425505a9bae597b430299c16514eafb250c17bcab251501ecc07061411d90dfa app.improvmx.com and api have returned to normal. All services have recovered. Postmortem will be posted tomorrow. Codestin Search App https://status.improvmx.com/ Wed, 08 Apr 2026 17:31:53 +0000 https://status.improvmx.com/#f6fd20fb893f3961a6a3f9b6f6c407495a47d2a0027db9d3e5ab5704ae40bc64 api.improvmx.com recovered Codestin Search App https://status.improvmx.com/incident/867268 Wed, 08 Apr 2026 17:27:00 -0000 https://status.improvmx.com/incident/867268#8fdb8c85b80ef817483993aa8f9da042452a7d61cdf8735f7ac5de563754733f Mail delivery has recovered. We are still investigating app/api. Codestin Search App https://status.improvmx.com/incident/867268 Wed, 08 Apr 2026 17:18:00 -0000 https://status.improvmx.com/incident/867268#62a61a83610927191a89fa3795a599654759ca874b4a6fe69018bf1fc49d6083 We've upgraded our primary database, and are seeing services recover. Codestin Search App https://status.improvmx.com/incident/867268 Wed, 08 Apr 2026 17:02:00 -0000 https://status.improvmx.com/incident/867268#a02264ec49bdcfe9ad14f343abeafa5496bc04d4bf2c6c89a79618032622d768 We are having issues with our primary database, affecting all services. Codestin Search App https://status.improvmx.com/ Wed, 08 Apr 2026 16:53:40 +0000 https://status.improvmx.com/#f6fd20fb893f3961a6a3f9b6f6c407495a47d2a0027db9d3e5ab5704ae40bc64 api.improvmx.com went down Codestin Search App https://status.improvmx.com/incident/865133 Sun, 05 Apr 2026 17:02:00 -0000 https://status.improvmx.com/incident/865133#502dd6a9869766eff331a002fcddda0c1fecbf07cc23dd3a852554b941ace7ec Delivery to apple email addresses has returned to normal. Codestin Search App https://status.improvmx.com/incident/865133 Sun, 05 Apr 2026 14:30:00 -0000 https://status.improvmx.com/incident/865133#bbdb1139b4a736b7d8be339bbafa0ea22b7b9c3ebe31414f4082a6267fe60fa0 We're investigating delivery issues to apple emails (me.com, icloud.com, etc.) Codestin Search App https://status.improvmx.com/incident/862711 Wed, 01 Apr 2026 20:53:00 -0000 https://status.improvmx.com/incident/862711#2ad3e4dbc4a5c438fd45948697eab654f881e3d30fc44d7c9c1de18e18112492 We had a brief spike in emails forwarded through our system. Our system auto-scaled to handle the spike, and services are back to normal. We will continue to monitor our systems, and implement better auto-scaling policies to prevent this in the future. Codestin Search App https://status.improvmx.com/incident/862711 Wed, 01 Apr 2026 20:41:00 -0000 https://status.improvmx.com/incident/862711#b2aaa572988278d0a77281a7466fb15c4af250a31af777be2fc181ab1da4588f We're investigating delayed email forwardings. Codestin Search App https://status.improvmx.com/incident/861015 Mon, 30 Mar 2026 18:37:00 -0000 https://status.improvmx.com/incident/861015#cd64b54593a1ec5099bdfc7ead192b569fd4d809352a407599a1d2db3b08d79e This is a similar pattern to the incident a few days ago. We've applied the same mitigation methods (a lot faster this time), and services are returning to normal. We will continue monitoring our systems, and implementing greater resilience to prevent these incidents in the future. Codestin Search App https://status.improvmx.com/incident/861015 Mon, 30 Mar 2026 17:46:00 -0000 https://status.improvmx.com/incident/861015#5a1258d418aa401f6619ac8b2dfe9a9c56f70f6e9c668f843cca0ee98460ec6a We're investigating delayed email forwardings. Codestin Search App https://status.improvmx.com/incident/859148 Sun, 29 Mar 2026 03:35:00 -0000 https://status.improvmx.com/incident/859148#1f426a7f040350f385dfea1249d39889ca679a3e2297fad2591185f57f3d61d1 Postmortem posted to https://improvmx.com/blog/2026-03-29-postmortem Codestin Search App https://status.improvmx.com/incident/859148 Fri, 27 Mar 2026 20:42:00 -0000 https://status.improvmx.com/incident/859148#07058c87683c05b659401f0b10f5127999cfd18f1d8a6a06b43da8d9bf41df39 We mitigated a prolonged DDOS attack with many spam senders overloading our mail servers. All mail deliveries are back to normal now. PostMortem with more detailed information will be posted tomorrow. Codestin Search App https://status.improvmx.com/incident/859148 Fri, 27 Mar 2026 15:42:00 -0000 https://status.improvmx.com/incident/859148#6be592032a5115fe2a50001ec0f0de123f12df3a3b3bc7fb8ccc462f99a79c73 We're investigating delayed email forwardings. Codestin Search App https://status.improvmx.com/ Wed, 25 Mar 2026 10:41:55 +0000 https://status.improvmx.com/#9ec1daa6d247da78dfa43310d80e60847dd78a15ebfab4c193720b7125314a9b api.improvmx.com recovered Codestin Search App https://status.improvmx.com/ Wed, 25 Mar 2026 10:35:50 +0000 https://status.improvmx.com/#9ec1daa6d247da78dfa43310d80e60847dd78a15ebfab4c193720b7125314a9b api.improvmx.com went down Codestin Search App https://status.improvmx.com/incident/842325 Fri, 06 Mar 2026 10:16:00 -0000 https://status.improvmx.com/incident/842325#716b867bd9aedb1c62f9046d21ec7e4e9a0b98cfcf5be4eb37a62783f1601aa3 We had a broken deployment go out, and we've reverted it. Systems are returning to normal. Postmortem to follow. Codestin Search App https://status.improvmx.com/incident/842325 Fri, 06 Mar 2026 10:10:00 -0000 https://status.improvmx.com/incident/842325#5d1296feb5512a26b06fb16d845a7d88b30cd39898c18c6e82f6f402e2567a18 We're investigating downtime on our API servers. Codestin Search App https://status.improvmx.com/incident/831550 Mon, 23 Feb 2026 20:30:00 -0000 https://status.improvmx.com/incident/831550#eadb76f5fd10b70306cb5e2a37fec119784ded277c9b53fdb2689a729e940805 Microsoft inbox delivery continues to recover, we will continue to monitor the situation. Codestin Search App https://status.improvmx.com/incident/831550 Mon, 23 Feb 2026 20:17:00 -0000 https://status.improvmx.com/incident/831550#070ab9886785cf59b6d70d0b5d5882c6d1620c90d80ad8775c81072eefb3507a We have deployed a mitigation and are seeing recovery for microsoft inbox delivery (hotmail.com and live.com). (outlook.com is unaffected) Codestin Search App https://status.improvmx.com/incident/831550 Mon, 23 Feb 2026 19:26:00 -0000 https://status.improvmx.com/incident/831550#3b0d4049ee6488766d17ca698de20f4e867987fecba5b42fd9083da21503fd4b We're seeing an uptick in email deliverability issues to live.com and outlook.com Codestin Search App https://status.improvmx.com/incident/739965 Wed, 08 Oct 2025 17:10:00 -0000 https://status.improvmx.com/incident/739965#6422686d7e765f81b82c1521edc1fae078400b0a3dbe6bab5e4ddc43985ce3e5 PostMortem posted to our blog: https://improvmx.com/blog/post-mortem-2025-10-08 Codestin Search App https://status.improvmx.com/incident/739965 Wed, 08 Oct 2025 15:23:00 -0000 https://status.improvmx.com/incident/739965#9352dccd82745a1912debc5eceb2e30ca0f47bdff0f413680101409f569138db Delivery times are back to normal. PostMortem will be posted later today with our findings. Codestin Search App https://status.improvmx.com/incident/739965 Wed, 08 Oct 2025 15:11:00 -0000 https://status.improvmx.com/incident/739965#4fb31c193804fb5fe81eb9e8b0ebd20edb950ab2096c55865829ae6189548fca Mitigation seems successful, delivery times are returning to normal. Codestin Search App https://status.improvmx.com/incident/739965 Wed, 08 Oct 2025 15:04:00 -0000 https://status.improvmx.com/incident/739965#b83adb5cc203c338d2245ebed19b1fdacef1aa260f1aae5363155d2bd37a82e9 We've identified a SQL performance issue, and are attempting a mitigation. Codestin Search App https://status.improvmx.com/incident/739965 Wed, 08 Oct 2025 14:28:00 -0000 https://status.improvmx.com/incident/739965#1a3361544a23949d6bc0c7a7969cf81bbee116ddf97de7a7630a3142838e38da Currently investigating some reports of delayed delivery times. Codestin Search App https://status.improvmx.com/incident/737891 Sun, 05 Oct 2025 14:36:00 -0000 https://status.improvmx.com/incident/737891#d558d5593da015fca2824b7e0cefeab3e23026554d9933b9af8ae49096204554 Post Mortem has been posted to [improvmx.com/blog/2025-10-05-post-mortem-sql-servers-overloaded](https://improvmx.com/blog/2025-10-05-post-mortem-sql-servers-overloaded) Codestin Search App https://status.improvmx.com/incident/737891 Sun, 05 Oct 2025 13:14:00 -0000 https://status.improvmx.com/incident/737891#38c09c0ff01359f3e34438313e32f7efa56eed68e9537c4d0dce9a277da709b9 We upgraded our SQL servers to handle more capacity, and service has returned to normal. Post Mortem to be posted later. Codestin Search App https://status.improvmx.com/incident/737891 Sun, 05 Oct 2025 12:22:00 -0000 https://status.improvmx.com/incident/737891#470ea5d4afe142562cc74b4c3e60c4a68284a87519546eb02a4d336b3f5f5914 This is affecting forwarding as well. Codestin Search App https://status.improvmx.com/incident/737891 Sun, 05 Oct 2025 12:18:00 -0000 https://status.improvmx.com/incident/737891#91e2b48eda0eec93ae97944e8dd07fff4f2cd0d29503cfb31405a88e366dfa7d We're seeing increased error rates on api.improvmx.com Codestin Search App https://status.improvmx.com/incident/737454 Sat, 04 Oct 2025 21:54:00 -0000 https://status.improvmx.com/incident/737454#50fb39f057351b8a5c8d3a8039f3516a082479e6badf2c9900c6108921c2fbf1 Post Mortem has been posted to [improvmx.com/blog/post-mortem-backed-up-delivery-queue](https://improvmx.com/blog/post-mortem-backed-up-delivery-queue) We sincerely apologize for the downtime here, this doesn't meet our reliability bar. We will diligently make improvements to prevent this kind of issue going forward. Codestin Search App https://status.improvmx.com/incident/737454 Sat, 04 Oct 2025 12:54:00 -0000 https://status.improvmx.com/incident/737454#6ddbeba4ed11ce875cee9d77d6223fd858112a988c329bb2e2b405b8e4aa9514 We've churned through our backlog of emails. Email delivery is back to normal now. Full PostMortem will be posted later today to this status page and to https://improvmx.com/blog Codestin Search App https://status.improvmx.com/incident/737454 Sat, 04 Oct 2025 12:09:00 -0000 https://status.improvmx.com/incident/737454#54447c6e2af5da0e1495fef1ee3b19e155e76e5c3ad9b62da8ba34eadc8ec977 NOTE: OpenSRS/hostedemail.com mail servers seem down, any customers with those inboxes will not be receiving emails until they recover. Codestin Search App https://status.improvmx.com/incident/737454 Sat, 04 Oct 2025 11:55:00 -0000 https://status.improvmx.com/incident/737454#ed0b240b0a390849080adef3846b4712afe696aba9dc789e95a58a9baa388582 We have mitigated the issue, our mail servers are making their way through the backlog of queued emails. We will post again when the queue is complete and email delivery is back to normal. A full postmortem will be posted tomorrow. Codestin Search App https://status.improvmx.com/incident/737454 Sat, 04 Oct 2025 11:18:00 -0000 https://status.improvmx.com/incident/737454#a7549201d6eddd9b92ea45bb024f3643575706eec1a8bbf2f944333b613041d0 We have identified the root cause, and are mitigating the issue now. Emails queued for delivery are beginning to come through. We will post another status update soon. Codestin Search App https://status.improvmx.com/incident/737454 Sat, 04 Oct 2025 09:33:00 -0000 https://status.improvmx.com/incident/737454#78712982417a6944e369b5dd7df685f9704d144c6d0191a93e126a35f33b7114 We are currently investigating increased error rates on email deliveries. Codestin Search App https://status.improvmx.com/incident/730110 Wed, 24 Sep 2025 15:27:00 -0000 https://status.improvmx.com/incident/730110#da061d93de72b423989158ccc416d0b38b58b70656d5f3c4d210558fc7676f21 # Post Mortem (also posted to [improvmx.com/blog/2025-09-23-post-mortem](https://improvmx.com/blog/2025-09-23-post-mortem)) ## Summary Spike in connections overloaded our SQL server and triggered a latent performance degradation in SQL lookups, causing delayed delivery until performance mitigation. ##Timeline 11:30AM EST: Incoming mail connections spike to 30x steady state, causing increased load on our servers 11:49AM [*Downtime Begins*]: Emails begin being delayed/backed up. 11:58AM [*First Alert Fired*]: Automated Alerting notified Matthew Tse on call that there was a deliverability issue with Microsoft 11:59AM [*First Responder Signs On*]: Matthew Tse signs on and begins investigating 12:08PM [*Customers Alerted*]: Matthew Tse posts an incident to [status page](https://status.improvmx.com/incident/730110) 12:42PM [*Mitigation Attempted*]: Matthew Tse pushes code that increases logging to track offending users, and also increases the connection limit on front door mail requests 01:34PM [*Mitigation Attempted*]: Matthew Tse unlocks the max SMTP autoscaled server limit 02:00PM [*Recovery Begins*]: The SQL servers begin to handle the load, and emails begin being delivered again, but delayed. 02:10PM [*Mitigation Attempted*]: Matthew Tse adds additional logging to root cause the dropped front door messages. We discover that there is a SQL connection pool overload error being emitted constantly. 04:27PM [*Mitigation Attempted*]: Matthew Tse pushes async connection pool optimizations to decrease load on SQL servers 07:10PM [*Mitigation Attempted*][*Recovery Complete*]: Matthew Tse pushes further async connection pool optimizations, that fully eliminate the SQL error. ##Action Items IMX-1337: Audit all python SQL connection pool logic across all clients ensuring the thundering herd issue doesn't happen again. IMX-1338: Audit/persist all SQL limit changes made during the incident, ensuring they persist past server reboot. IMX-1339: Add Metrics Tracking the number of Connections made to our SQL database, ensuring this issue surfaces immediately in the future. IMX-1340: Add Metrics Tracking number of mail rejections due to unhandled SQL connection issues. This should further improve our speed and reliability during delivery. We apologize for the downtime in our services. But this incident has brought us several key learnings which will improve our reliability going forward. If you have any questions, feel free to reach out to me at [[email protected]](mailto:[email protected]) Codestin Search App https://status.improvmx.com/incident/730110 Tue, 23 Sep 2025 23:13:00 -0000 https://status.improvmx.com/incident/730110#4a672c65c177a174658d26b64d84d7b9ab4eca7a1a0d8b3ac97f799349708958 We have fully root caused and mitigated the issue. Services should be returning to 100%. We will post a full postmortem and RCA tomorrow. Codestin Search App https://status.improvmx.com/incident/730110 Tue, 23 Sep 2025 18:08:00 -0000 https://status.improvmx.com/incident/730110#c96368762badc6b20ae167dbff452d631ba745b3acba5c4fb5b098481716dee7 We have found an initial mitigation to the issue. And are bringing services back up now. Email forwarding and SMTP sending should begin recovering. We will continue to post updates. Codestin Search App https://status.improvmx.com/incident/730110 Tue, 23 Sep 2025 15:42:00 -0000 https://status.improvmx.com/incident/730110#faeb97ba1472086a75de5054b4ea408d097c8d9d438aa539f234b4c89c372465 Mail delivery may be delayed while we mitigate the issue. Codestin Search App https://status.improvmx.com/incident/625532 Fri, 25 Jul 2025 06:58:00 -0000 https://status.improvmx.com/incident/625532#0d8d9ff25a5e4c11536ca0d6ebecdd2304e4ef4e72ce99741db0fa442e2691a1 We have rotated in a new set of sending IPs, and we're seeing Microsoft deliverability rates return to normal. We will continue monitoring this over the coming days. Codestin Search App https://status.improvmx.com/incident/625532 Thu, 24 Jul 2025 18:00:00 -0000 https://status.improvmx.com/incident/625532#d1cde7cef63a5d238485d0b317aeee80de0c0d9cad2653c895b930ab236f2439 We're currently experiencing degraded deliverability to Microsoft email servers (outlook, live, hotmail, etc.). We're working on rotating in a new set of IPs to resolve the situation. Codestin Search App https://status.improvmx.com/incident/619603 Mon, 14 Jul 2025 23:06:00 -0000 https://status.improvmx.com/incident/619603#f1cd4f297b85e51079cc0b8516b6093ca4730aa429e53deed64e5cb35bf0feea PostMortem: OVH DNS Outage Issue Summary: DNS resolvers on our OVH data center provider went down. Our mail sending infrastructure was unable to pull new mail tasks from the queue, and unable to deliver them due to inability to lookup MX records. Timeline: 7/14/2025 17:54 [Downtime Begins] OVH DNS Outage, Cargo cannot pull messages off SQS queue, the few messages it has cannot be delivered because it cannot lookup MX records 18:01 [First Alert] BetterStack pings Matthew for "Gmail Deliverability" 18:01 [Acknowledge Alert] Matthew acknowledges alert, begins investigating Matthew at first thinks it's the Gmail Deliverabilities Alert being too sensitive. 18:09 Matthew realizes it's a system wide outage, and posts to #ops in slack 18:11 [Public Status Posted] Matthew posts to status page: https://status.improvmx.com/incident/619603?mp=true 18:21 [Recovery] Matthew sees messages come in again, posts to #ops 18:23 [Public Status Resolution] Matthew posts resolution to status page. Root Cause: We don't have backup DNS servers on our OVH infrastructure. It would have failed over automatically if we had Google or Cloudflare set as backups. Resolution and Recovery: OVH's DNS servers recovered on their own, and the mail queue immediately began completing. Corrective and Preventative Measures: - install dnsutils on all servers, to ensure DNS query tools like dig are immediately available - setup backup DNS resolvers on OVH Servers Codestin Search App https://status.improvmx.com/incident/619603 Mon, 14 Jul 2025 22:23:00 -0000 https://status.improvmx.com/incident/619603#0e5763e431d5d387c50667fdd4dc217338af4e116a2f1a933b7e39815250ab04 Deliverability issues have been mitigated, we are continuing to investigate, and will post a postmortem within 24 hours. Codestin Search App https://status.improvmx.com/incident/619603 Mon, 14 Jul 2025 22:11:00 -0000 https://status.improvmx.com/incident/619603#dc2fb4494e163428c27e3ae8a5de7cb3db5a0dbd94e71140e0b880368a10ca3c We are examining reports of deliverability issues across email providers. Codestin Search App https://status.improvmx.com/incident/600656 Wed, 11 Jun 2025 10:46:00 -0000 https://status.improvmx.com/incident/600656#b079a744dd7bc33c3af0e6847fcb7c6522d1f6020dcfd15993f478797981ca23 We have rotated out affected sending IPs, and microsoft deliverability is returning to normal. Codestin Search App https://status.improvmx.com/incident/600656 Tue, 10 Jun 2025 17:42:00 -0000 https://status.improvmx.com/incident/600656#b657903453cc72416c27c40a78b0039813f6368d1a3127f2f4b77d5e8f9b79c5 We're currently investigating impacted email forwarding to Microsoft mailboxes. Codestin Search App https://status.improvmx.com/incident/564179 Wed, 14 May 2025 12:42:00 -0000 https://status.improvmx.com/incident/564179#5a46d66351fed4ea4eb2aab7e16bd30d9c4bee647109d6296f9e46df9078067c Magic-link user based logins have been fixed. Codestin Search App https://status.improvmx.com/incident/564179 Wed, 14 May 2025 05:09:00 -0000 https://status.improvmx.com/incident/564179#bd350394244303469b60244297335455a5ceef6e8abcd03b4ff02cce46d42acb At 1AM EST we pushed a bug out that affected users trying to login to their accounts with email magic link (password login was unaffected). The fix has been identified and resolved as of 8AM EST. Codestin Search App https://status.improvmx.com/incident/555535 Fri, 02 May 2025 15:26:00 -0000 https://status.improvmx.com/incident/555535#5de91ca0ca0a4e036bca8d419edc31c7db3652546882bb1c7c3c08e5f0ce0341 The SPF record issue has been resolved. Codestin Search App https://status.improvmx.com/incident/555535 Fri, 02 May 2025 05:00:00 -0000 https://status.improvmx.com/incident/555535#9afcecead26e45a6e7c66592ce1b71feafbb20b0748fae59dc388c0209f94324 We pushed out a bugged SPF record change last evening (05/02/2025) at 11:30PM Eastern US Time. It was designed to improve email deliverability by reducing the number of SPF lookups, but had an issue that temporarily caused SPF lookup failures. We have pushed corrected SPF records as of 05/02/2025 11AM Eastern US Time. This will require a few hours to fully propagate. Codestin Search App https://status.improvmx.com/incident/524452 Fri, 07 Mar 2025 23:27:00 -0000 https://status.improvmx.com/incident/524452#ee31be37e882171e0cc23100b1903a4241afb5bd387b0e11bed6cdd746be624f We have moved the affected servers out of rotation, so message deliverability to microsoft & outlook addresses should be returning to normal. Codestin Search App https://status.improvmx.com/incident/524452 Fri, 07 Mar 2025 17:39:00 -0000 https://status.improvmx.com/incident/524452#45e192688bcfb632d1eac3235e71de190e4092fa0631900263bdb512c7a2838d A subset of our sending servers experiencing degraded deliverability to Microsoft & Outlook mail servers. We are actively working on replacing these servers to restore high deliverability standards. Codestin Search App https://status.improvmx.com/incident/514862 Tue, 18 Feb 2025 13:53:00 -0000 https://status.improvmx.com/incident/514862#cbeff20dca845b95c18d732c93869939e37e8f8d93a4a72125ac84b81289b331 PostMortem: We have identified actors spamming our load balancers with traffic which affected a portion of users. We have banned the bad actors, added enhanced logging and alerts to the affected load balancers, and will act accordingly the next time this happens. Codestin Search App https://status.improvmx.com/incident/514862 Tue, 18 Feb 2025 12:07:00 -0000 https://status.improvmx.com/incident/514862#f48b9c69bd6be8f8aaad3417f770bc82635beeaf1e2a89215fc2e5d769bf527a Email deliverability has returned to normal for most users. We are continuing to investigate. Codestin Search App https://status.improvmx.com/incident/390824 Fri, 28 Jun 2024 18:09:00 -0000 https://status.improvmx.com/incident/390824#7227a164ed2a8cd0847fe11d665ad25a7e22da85c1ddd3b2b9ab01025f1fd510 The queue has gone back to normal levels and we're seeing delivery times sub 10 seconds again. This is the all clear, and there will be no further updates. Codestin Search App https://status.improvmx.com/incident/390824 Fri, 28 Jun 2024 17:40:00 -0000 https://status.improvmx.com/incident/390824#04129269b8b45e0953e2f01a70320d2e8fe5a45b67061cdd61df5c5dc5f39598 We have identified the root cause of the problem. OpenDNS, a DNS provider we used for the delivery part of our infrastructure, has decided to ban all French-based traffic, impacting half of our MX servers. This ban follows a court order mandated by the French sport code to crack down on services illegally streaming Formula 1. OpenDNS has decided to retaliate by completely shutting down all French traffic, as well as some areas in Portugal — without notice. We have pushed all our traffic to our alternate DNS provider for the impacted services, and email delivery is resuming normally. We'll keep this status update open and hopefully provide the all-clear in the next 30 minutes. Codestin Search App https://status.improvmx.com/incident/390824 Fri, 28 Jun 2024 17:08:00 -0000 https://status.improvmx.com/incident/390824#f643a1ad1b55804e9870f845e4d3cd341022947f92c09ce56a6d9a3427490e38 We are currently seeing elevated delivery times and investigating an incident within our infrastructure. We are able to confirm that emails are queued for delivery and saved properly in our system, so no emails are lost at this point. Next update will follow in 30 minutes. Codestin Search App https://status.improvmx.com/incident/253811 Tue, 05 Sep 2023 23:00:00 -0000 https://status.improvmx.com/incident/253811#47439a2b1bc365a849c1ce759b41fa2ea3c22dcd7c856a1d0ee0d7fca03ea832 Maintenance completed Codestin Search App https://status.improvmx.com/incident/253811 Tue, 05 Sep 2023 21:00:00 -0000 https://status.improvmx.com/incident/253811#18fb7a659b85dc0779e9e516ca5a8c84a6757c6ec196e7a2f8124a7eed412b1e We have scheduled a maintenance window that will specifically affect the testing of email aliases. During this time, test emails won't be sent in real-time but will be dispatched as soon as maintenance concludes. Please be assured that this **will not** impact the main ImprovMX infrastructure or your regular email service. Your understanding and patience are greatly appreciated as we continue to enhance our service. Thank you for your trust in ImprovMX. Codestin Search App https://status.improvmx.com/incident/246907 Fri, 18 Aug 2023 12:37:00 -0000 https://status.improvmx.com/incident/246907#deaba345743315ddf3c5a102be76898b3924e94e3a3f140d1e01bcefca8e3a90 #### **Post Mortem** Recently at ImprovMX, we encountered a challenge within our email forwarding service. An unexpected update to a server OS package on AWS disrupted our system's ability to automatically scale instances as demand increased. ##### **Resolution and Recovery:** Our dedicated team responded quickly with a two-step plan: 1. **Quick Fix:** We redeployed our servers with a targeted fix to the issue caused by the unexpected update, enabling us to resume our service quickly. 2. **Final Resolution:** To ensure that such an incident does not recur, we revamped our deployment strategy. Now, instead of deploying new server instances with a fresh OS, we deploy them pre-installed. This approach not only eliminates nearly all deployment issues but also boosts deployment speed, resulting in much faster auto-scaling whenever there is increased demand or usage of our service. ##### **Moving Forward:** We have learned from this incident and made significant improvements to our systems. We are more robust and resilient than ever before, and we remain committed to offering seamless service. ##### **Transparency and Trust:** At ImprovMX, we believe in honest communication and accountability. We understand the importance of trust, and we assure you that we have taken all necessary steps to prevent such an occurrence in the future. Thank you for your continued support and trust in ImprovMX. Codestin Search App https://status.improvmx.com/incident/246907 Thu, 17 Aug 2023 08:56:00 -0000 https://status.improvmx.com/incident/246907#a4e22b76ce4718c49a5fc5ed5314de297d3576eef1912d273d5157799546cb2e Email forwarding is now resuming. Because of the massive amount of servers re-trying sending we are forwarding slower than usual, but we are actively monitoring our system to ensure things get back to normal levels. We'll be sharing the post mortem here when it is ready. Codestin Search App https://status.improvmx.com/incident/246907 Thu, 17 Aug 2023 08:25:00 -0000 https://status.improvmx.com/incident/246907#22a02a25f4499e23bfd6215b674e2d9b08530af6157db654ba3930cf9333bd27 A fix has been implemented and we are monitoring the results. Codestin Search App https://status.improvmx.com/incident/246907 Thu, 17 Aug 2023 07:20:00 -0000 https://status.improvmx.com/incident/246907#81f88086842df77b774daceb356defd7e98cdeac79737f7e61502b3591afa1c5 The issue has been identified and we are working on fixing it. Codestin Search App https://status.improvmx.com/incident/246907 Thu, 17 Aug 2023 04:09:00 -0000 https://status.improvmx.com/incident/246907#00959f972f8f9536d5a4e2b6cd1c9add6908e19a2b3af61cc51115de10448239 We are experiencing a temporary degradation of email delivery. Our team is currently conducting a detailed investigation to identify and resolve this matter. Please be assured that this situation is being treated with the highest priority, and we're committed to restoring optimal performance as quickly as possible. **UPDATE:** The issue has been resolved, and the post mortem is now available. Codestin Search App https://status.improvmx.com/incident/229939 Fri, 07 Jul 2023 11:36:00 -0000 https://status.improvmx.com/incident/229939#6f9de8ebb686823f3e8ab75e7f322d0e00fdd60825758fb77b02f4fd46dbaaee We have identified the root cause of the issue and have implemented a fix. We will be monitoring this to ensure that it remains stable. You should start seeing your emails being delivered as the backlogs are gradually cleared from our infrastructure. Codestin Search App https://status.improvmx.com/incident/229939 Fri, 07 Jul 2023 11:28:00 -0000 https://status.improvmx.com/incident/229939#bec70fde47c68a41fa879c354bdf143f7b60e1cfdf50d35c05e3f973add37fcb We are experiencing a temporary degradation of email delivery. Our team is currently conducting a detailed investigation to identify and resolve this matter. Please be assured that this situation is being treated with the highest priority, and we're committed to restoring optimal performance as quickly as possible. **UPDATE:** This has been resolved. Codestin Search App https://status.improvmx.com/incident/138750 Thu, 24 Nov 2022 12:50:00 -0000 https://status.improvmx.com/incident/138750#cacf11538f45d80fc5f977db2f20edcc3fc0fb836997630a62e48acf12244a41 Deliverability and forwarding from Outlook have resumed as usual. We're still actively monitoring the situation to prevent any reignition, but this is the all-clear for now. We'll provide a detailed post-mortem along with action items for this incident in the coming days. Codestin Search App https://status.improvmx.com/incident/138750 Thu, 24 Nov 2022 09:37:00 -0000 https://status.improvmx.com/incident/138750#1419b01cb0a9e5c86c6a7f66c976c7bd85082aeb3c4ad44c8e477fec808db21b We've been able to manage the issue on the human level, with the abusive customer removing approx 800 DNS entries from their setup to avoid ImprovMX being the destination server of bounce reports. We are already seeing connections dropping and a more manageable environment overall. We will continue monitoring in the next few hours, but emails should gradually get more and more accepted on the first try. Codestin Search App https://status.improvmx.com/incident/138750 Thu, 24 Nov 2022 08:11:00 -0000 https://status.improvmx.com/incident/138750#0a9b5c5024fd3f4aa5b164aaf27626f3dd5efc8b97083f66996f52f3cb2013ca We're currently battling an attack on our infrastructure which is leveraging Microsoft email servers (specifically Outlook) to send up to 100k connections per minute our way. This means that emails originating from Microsoft servers and trying to hit a redirection alias created on ImprovMX are soft-bouncing. Outbound traffic to outlook isn't affected, nor is any other email provider. Unfortunately, there is no quick way for us to block the incoming traffic, but we've identified the cause of the problem and are working on multiple levels to get this resolved as soon as possible. 1) Human - if the ESP block the bad actor, the problem is solved in a minute. We've been very insistent, but if it doesn't work, we'll start legal action to get movement 2) At the Microsoft level. We've lodged complaints through several channels so that they stop sending all the bounce reports back for that specific IP range. Unfortunately, this is usually very slow and doesn't get solved quickly. 3) At our technical level - We've increased AWS instances to the maximum we can afford (we're 10xing costs for this month - erasing all profits basically). It's like increasing the size of a leaky bucket, but it works temporarily. Codestin Search App https://status.improvmx.com/incident/90811 Fri, 27 May 2022 09:58:00 -0000 https://status.improvmx.com/incident/90811#a14aa2884f52d8d6033afcc174047627ee2b1da156955722c87688e9e3e0cd8e We've identified the issue and it's been automatically resolved by our redundancy plan. There will be no further updates. Codestin Search App https://status.improvmx.com/incident/90811 Fri, 27 May 2022 09:51:00 -0000 https://status.improvmx.com/incident/90811#e825c54cb63a125c5b277ee8222727a13bbce2fe7e2fd6d354ee376d4ddfd443 The dashboard is currently unavailable for all users. Email forwarding isn't impacted. We will update in 15 minutes with more information. Codestin Search App https://status.improvmx.com/incident/85352 Mon, 02 May 2022 08:07:00 -0000 https://status.improvmx.com/incident/85352#ab96486d55d140f598a7460e4c630ef0cfe8bebfb47a9f68845891f52312439a Netlify, our hosting provider for the marketing site and app.improvmx.com is currently experiencing an outage. https://www.netlifystatus.com/incidents/m1d4vwctxsrs?u=vy7qn93y8qbp Codestin Search App https://status.improvmx.com/incident/78009 Fri, 25 Mar 2022 16:12:00 -0000 https://status.improvmx.com/incident/78009#bef8a70e2a2de8585012e939132c57649c5f125a9cd0dc268676756526a8ce0e We have managed to identify and exclude the range of IP that was flagged at Gmail. We're able to confirm that we're not seeing soft/hard bounce issues, but will continue to monitor this throughout the rest of the day. Codestin Search App https://status.improvmx.com/incident/78009 Fri, 25 Mar 2022 13:55:00 -0000 https://status.improvmx.com/incident/78009#ed299d9df2da3b1d685d34e6e202fd455f3fd1b1aeab42c776b8b61795c41cd8 We're currently seeing elevated soft/hard bounces to redirection email addresses set on Gmail. This is sparse and seems to affect only a portion of our IP range. From our initial investigation, it shows that in the small subset of messages that are hard-bounced, senders are still able to receive a bounce notification back. We'll update the report once we know more.