01/19/2021 |
TrendSpider DNS Issue Post Mortem and Update
Hello Everyone,
I know a lot of customers have had some issues accessing the TrendSpider system today. This post is to briefly explain what happened and why it won’t happen again, as well as provide workaround options in case anyone still has issues.
What Happened
Last night, at about 3 AM CST, one of our service providers that hosts our main website and our DNS system, migrated servers to a new platform. While we were aware of this change, we did not anticipate their failure to copy the DNS zones for charts.trendspider.com to their new system.
When they did this, it caused the DNS for charts.trendspider.com to be overwritten with the incorrect server IP address.
We noticed this right away and mobilized, but by then the DNS change had somehow propagated to many ISPs around the world, and these incorrect settings were cached locally at those ISPs.
To complicate the matter, some ISPs ignored our programmed “TTL” (or Time To Live) instruction for the DNS. We had it set to 1 minute, but some providers defaulted this to 24 hours instead.
We recognize now that this was a mistake on our part. Some ISPs automatically default to 24 hours when the TTL is below 5 or 15 minutes.
What We Did About It
We immediately corrected the DNS settings. However, due to the TTL propagation issue, some internet service providers will still advertise the incorrect information for the rest of today before refreshing their DNS in about 15 hours from the time of this post (3 AM CST tonight.)
In the meanwhile, we have set up a new mirror address that will work if you have issues accessing the platform. This address is https://mirror.trendspider.com
If you cannot access TrendSpider normally, please try this address!
You will be able to access the system with the mirror, all your settings and charts exist on the mirror. At about 3 AM tonight, the Charts.TrendSpider.com main address will refresh DNS worldwide to correct settings with updated TTL settings so that this cannot happen again in this way
What We Are Doing to Prevent This From Happening Again
We are in the process of migrating our entire DNS setup to a specialist company that focuses only on managed DNS services. This will give us more control and flexibility to correct mistakes quicker as they come up. While we do not anticipate this ever happening again, we will take steps to make sure that is the case.
