There are some ongoing iwstack issues, mainly with the storage, one node crashed and the storage remained allocated, we are working on this. If you have a VM stuck, please shut it down and boot it (reboot will not work). This will make it start on another node. We are aware there are some VMs which cannot be shutdown due to a reboot instead of shutdown, they will be reset when the storage problem will be solved.
Only VMs on one node were affected. However, during this issue, it would be best to not reboot your instances as a precaution, no matter on the node they are located on, if you absolutely must do it, use shutdown and then start.
In short: we are working on it. Once a solution has been found and applied will update the topic with the action(s) (if any) you should take if this affected you.
IWStack issues
Re: IWStack issues
We are sorry to announce that there is no ETA yet.
We hope there will be a backlog of migrations cleared, but for the stuck VMs this will not work.
Whoever experienced this issue, please apply for credit. We will offer it proportionally with the effect it had on you more for a bigger VM, less for a small one, but no less than 10 Eur.
A quick fix would be very disruptive for all the customers, not just a few with stuck VMs as such we prefer to not rush things and try to do something that will:
1. Preserve all data;
2. Have a minimal or no impact on people not already affected.
We hope there will be a backlog of migrations cleared, but for the stuck VMs this will not work.
Whoever experienced this issue, please apply for credit. We will offer it proportionally with the effect it had on you more for a bigger VM, less for a small one, but no less than 10 Eur.
A quick fix would be very disruptive for all the customers, not just a few with stuck VMs as such we prefer to not rush things and try to do something that will:
1. Preserve all data;
2. Have a minimal or no impact on people not already affected.
Re: IWStack issues
More data came in.
A clvm cluster is broken and Salvatore tries to restore it.
This is a delicate operation which may fail so will take a long time to investigate all possible ways to do it, maybe days.
If your instance is one of the few dozen stuck please recreate it and restore data from a snapshot.
Unfortunately the IP will change. We can restore the data which is not lost, just inaccessible, but we try to do it without rebooting the cluster which would affect a lot of people.
A clvm cluster is broken and Salvatore tries to restore it.
This is a delicate operation which may fail so will take a long time to investigate all possible ways to do it, maybe days.
If your instance is one of the few dozen stuck please recreate it and restore data from a snapshot.
Unfortunately the IP will change. We can restore the data which is not lost, just inaccessible, but we try to do it without rebooting the cluster which would affect a lot of people.
Re: IWStack issues
Since problems were spreading had to do an emergency restart of the whole system. This will also apply some patches.
All VMs will be briefly down. Should be back in about 10 minutes in average.
All VMs will be briefly down. Should be back in about 10 minutes in average.
Re: IWStack issues
Everything seems to be running stable for a while now.
People affected by the "stuck instance" bug can apply for credits.
We are very sorry for this issue and still investigating what might have caused it in order to avoid it in the future.
People affected by the "stuck instance" bug can apply for credits.
We are very sorry for this issue and still investigating what might have caused it in order to avoid it in the future.
Who is online
Users browsing this forum: No registered users and 17 guests