We had some problem from Nov 16, 2017 15:10 to Nov 16, 2017 15:35, and from Nov 17, 2017 09:10 to Nov 17, 2017 13:50 (KST)
There was a problem in our cloud database that processes guest(user)'s online job, because part of our logic did not operate properly due to traffic increase on our service.
The problem we found is as follows:
The API that brings user chat information from Desk app failed had delay when receiving online user status. Thus the loading failed.
There was a delay in sending the messages to users, because there was a problem sending SMS based on the user's online status when managers send messages on the Desk app.
We responded as follows:
Nov 16, 2017 15:10 : We confirmed kernel panic on our cloud server that runs database. We quickly responded to reboot the server to fix the problem
Nov 16, 2017 15:35 : However, we confirmed that the problem still remained as unfixed and the server was not able to handle online features
Nov 17, 2017 09:50 : The problem still remained as traffic increased, and we attempted to fix the problem by patching the server without online feature around 12:00.
Nov 17, 2017 13:00 : We fixed the problem by increasing our physical database and configured the server properly
It took longer than we expected to recover, because we had to change the settings while the service was life during the morning on Nov. 17th. We have changed multiple backend server settings and increased the number of new database servers.
We have checked if there was any loss in information, and found there was no security problem or information loss.
We deeply appologize for your inconvenience, and appreciate your patience.
We will do our best for this to not occur again.
Posted Nov 17, 2017 - 16:24 KST
Monitoring
The issue has been solved but may have some delay during urgent fix update. Thank you for your understanding.
Posted Nov 17, 2017 - 12:58 KST
Investigating
User Chat view action is slow so we are figuring out why. Log in may not work during the fix update. Sincere apologies for your inconvenience.