On October 26th at 7:29 AM UTC we were alerted by our notification service about increased error rates and latency across all Customily services.
We quickly started investigating the issue and by 7:40 AM UTC most of the services were back to normal. However, the service that provides the API endpoints for Customily Shopify and WooCommerce stores was still experiencing high error rates.
During this time, options and previews were not loading on some Shopify and WooCommerce stores. Customers were still able to add to cart but without any personalization. At 8:30 AM UTC we identified a memory leak in the service that was preventing it from recovering. By 10:50 AM UTC we were able to fully diagnose the root cause, and pushed a fix to the service.
We know how many businesses rely on Customily and we are deeply sorry for the issues this may have caused today. Although things are already back up and running the entire team is already implementing measures to improve our monitoring services and our service recovery system, not only to prevent this from happening again, but to also react faster to any future events.
We appreciate your patience and support as we continue to improve our service each day. If you’re experiencing any problem or need further information don’t hesitate to contact us right away.
Was this article helpful?
Have more questions? Submit a request