Incident Report[]
- Jun 22, 2021 - 12:35 EDT: UPDATE: The Hypixel Server is now open in a limited maintenance mode, which allows ranked players to join. More info > https://hypixel.net
- Jun 20, 2021 - 13:46 EDT: UPDATE: We have the new hardware configured, but we still need more time for set up and for our external partners to do what they need to do. But you can read more about it on the website > https://hypixel.net/
- Jun 19, 2021 - 11:44 EDT: UPDATE: A progress bar has been added to the website to show how much more needs to be done before we can get up and running again. Additionally, we posted an update regarding what's going on with the maintenance, so be sure to check it out > https://hypixel.net/ If you are having trouble seeing this new update, make sure to press CTRL and F5 at the same time while on the page to refresh the cache.
- Jun 17, 2021 - 20:33 EDT: UPDATE: The Hypixel Network will be down for emergency maintenance starting now. We are sorry for any inconvenience and thank you for being patient with us. You can read more at https://hypixel.net
- Jun 16, 2021 - 19:31 EDT: We are continuing to work with our upstream provider to resolve this. We have resolved the issue of some players being unable to connect. We appreciate your patience and support as we work to resolve these issues.
- Jun 16, 2021 - 07:30 EDT: IDENTIFIED: We are aware of the issue still existing and are working on a larger solution with our provider. This may take some time and we'll keep this incident updated.
- Jun 15, 2021 - 21:55 EDT: MONITORING: Upstream has implemented a change and we are monitoring the results.
- Jun 15, 2021 - 20:38 EDT: IDENTIFIED: We are aware of the connection issues occurring again and are working with our upstream provider.
- Jun 15, 2021 - 19:38 EDT: MONITORING: Upstream has implemented a change and we are monitoring the results.
- Jun 15, 2021 - 15:33 EDT: UPDATE: We are continuing to work with the upstream provider to resolve the issues as soon as possible
- Jun 15, 2021 - 15:03 EDT: IDENTIFIED: We have identified the issue with an upstream provider and are working with them to resolve it as soon as possible.
- Jun 15, 2021 - 15:00 EDT: INVESTIGATING: We are currently investigating an issue with intermittent connection issues to the Minecraft Server and Forums.
Admin Note and Updates[]
Hey guys!
We will be taking down the Hypixel Network for emergency maintenance. This maintenance begins now and will last for at least 24 hours. Should we choose to extend this downtime, we will post an update on this news post.
During this time we will be reworking the network to ensure the issues we have been having these last few days don't happen again.
We would like to apologize for the inconvenience the last few days have caused and we hope that our community understands the difficulty of the situation.
The situation is: Our host is currently under large-scale DDoS attacks and they are actively working to resolve the situation. We are assisting them the best we can, which is why we have to put the server on maintenance during this process.
We have already done many network improvements that have lessened the impact of the attacks that are either directed at us or not, but it has proven not to be enough. Our top priority is to mitigate the impact of these attacks as quickly as possible while working on permanent solutions that will shield us better from future attacks and isolate our network.
The good news is that we have dealt with DDoS attacks for well over 8 years now and we have a great understanding of the situation. Each of these attacks is helping us better understand weak points that need to be improved from our side of things and also for our host. As much as this situation is bad for everyone at this time, it is giving us very good data to further understand and lessen the impacts of future attacks.
In reality, about a year ago, these attacks would have had no impact at all, but some recent changes at our host caused a flaw in our setup. so the host is working with us to resolve the situation. We all know what to do to resolve the situation because we have done it before, but it requires quite a bit of work to get to that place again. We’re hoping 24 hours is enough for this upgrade, but please understand it may take more than that.
To be fair, we haven't been as alert or ready as we should have been as it's been about 2 years since our last DDoS attacks. This is definitely a good opportunity for us to set some improvements in motion and deal with this once and for all.
We made sure any time-based services on the network are taken care of. This means pausing boosters, pausing the SkyBlock Auction House and Bazaar timers, pausing timers on MVP++ and daily rewards, etc. We’re hoping not to forget anything, but if we do, we are always open to a chat on our Help Desk.
Furthermore, when we bring the server back online, we will be turning on Triple Coins and Triple XP for at least a week.
As the situation progresses, we will be posting any updates to our status page. Once we are ready to go live again, we will make an update with more details and share as much as we possibly can with our community.
As always, we are extremely thankful for everyone's support in these difficult moments and apologies once again for the issues. We want to reassure our community that the current situation is our top priority.
────────────────────────────────────────────────────────────────
Update #1 (June 18th - 3:35PM ET): There are still several ongoing large-scale DDoS attacks going on in the Minecraft server scene at this time and we are not quite done with our changes before we can come back online safely. We do not have a precise ETA at this moment, as we are relying on several external partners to help us finalize the DDoS protection and major changes to our network infrastructure. We will do our best to give an ETA by tomorrow morning, but overall we aim to get back online at some time this weekend if all goes well. Once again, we appreciate your support during these difficult times, we will keep you guys updated the best we can.
────────────────────────────────────────────────────────────────
Update #2 (June 19th - 11:15AM ET): We’ve added a progress bar to give you a better understanding of where we are in regards to bringing the Hypixel Server back to all players. Please note that this is not representative of time, and will be manually updated, so don’t completely rely on it. This will, however, give you all at least some transparency on where we are with the work. It's currently too difficult to give an exact ETA as several external teams are involved. As we are passed updates, we will quantify the% and add it to the progress bar.
This morning, the on-site team moved most of the core hardware to be on the Cloudflare network, and our servers will no longer be on any network of our host. We will be effectively 100% Cloudflare, with full DDoS protection from them. Unfortunately, in order to do this, we have had to move (and still are) quite a lot of hardware from the current host’s network; this has taken a fair bit of time due to the large number of servers and racks we have. However, all critical machines required to begin initial setup and hold a decent player count have been moved.
Today, external network teams are being dispatched to assemble and configure the new network. This new setup will be on Cloudflare, and your ping should technically be somewhat the same. However, because this is the first time we will be using full Cloudflare, there’s a possibility that we may have some issues at first. Despite that, their team is assisting us and we have worked with them for years with great success.
Note that the forums, API and some other things might be broken at first, but will be sorted later on.
We will try to give another update as soon as possible. Keep an eye on the progress bar - we will do our best to update it whenever possible!
Thank you for your support!
────────────────────────────────────────────────────────────────
Update #3 (June 20th - 12:45PM ET): All of the new hardware has been configured to the new network and all the core servers and minigame servers are working great. We are now entering the final steps of setting up and configuring the last bits of the network and DDoS protection, this is the more tricky part because we heavily rely on external partners to do their parts so we have no clear vision on progress at this time.
We are still seeing continuous large-scale DDoS attacks across the board in the Minecraft server scene, which means our decision to make this major change was the right path forward.
Tonight, if all goes well of course, we will be doing tests assuming our partners meet their deadlines. Please note that these tests will require reboots and downtime - the player count will be severely limited. As soon as the situation is fully back in our hands, you will see more frequent updates from us and then slowly increase the player count to allow all of you in.
Our team has done everything humanly possible, spared no expense and worked with incredible partners to make this happen as quickly as possible for our amazing community that we are grateful for.
────────────────────────────────────────────────────────────────
Update #4 (June 21st - 1:00PM ET): Quick update, we have successfully finalized and tested our new network, including all minigames.
We are still in the final steps of configuring the DDoS protection with our external partners and making good progress. Given the new network configuration, we don’t know how much time this will take for our partners to complete the setup, but for us the hardest part is done and we should be testing with players soon.
We truly appreciate the support, rest assured we are working night and day to come back online as soon as possible.
────────────────────────────────────────────────────────────────
Update #5 (June 22nd - 1:00AM ET): The Hypixel server is ready for a testing session later TODAY! Our current ETA for this session is some time around the evening, Eastern Time (ET).
We will provide more details about the testing session a few minutes beforehand, including information on the actual public opening timeline that will be shared here and on our Discord.
Due to the limited nature of these tests, please understand that we will have to restrict access to ranked players, as our capacity will be restricted to ensure a controlled testing environment.
────────────────────────────────────────────────────────────────
Update #6 (June 22nd - 12:00PM ET): The Hypixel server is now in limited maintenance mode which allows ranked players to join. Please be aware that we will potentially require partial and full reboots until we can nail down the settings to scale to our original capacity and complete the configuration of the new network.
As such, we may end up going offline for longer periods of time to change some deeper network settings. These downtimes could take an hour or more. We would recommend avoiding activities that shouldn't be interrupted, like competitive game modes.
If you encounter any connection bugs that seem related to this new network, please tweet us at twitter.com/HypixelNetwork and we will do our best to monitor.
Quick notes for time based items/features:
- If you had/have MVP++ active during the maintenance, you will be given extra days to compensate - even if it expired. This will come later during the week.
- If you had/have booster Cookies active in Hypixel SkyBlock during the maintenance, you will be given brand new untradeable cookies to compensate. Like the above, these cookies will come later during the week.
- All network boosters were paused at the start of maintenance and will be reactivated only when we can open to full capacity. Booster multipliers are still active during the pause
- Purchases made during the maintenance have worked fine. However, if you have issues, please contact our support team and they will sort it out as soon as possible. The link is at the bottom of the website.
- Daily Reward Streaks have been extended from 2 days to 7 days
- Website and API will remain offline for a few more days while we assess the situation
- The server will actually be functioning normally, stats are saved and there will be no rollbacks at the end of the testing session.
────────────────────────────────────────────────────────────────
Update #7 (June 22nd - 13:00PM ET): The Hypixel.net website is back online! You're looking right at it.
