Igor Pashev,
Judging by the logs, you have more problems than just binlog index corruption, and they appear to be related.
Do you have some kind of a wrapper around mysqld / mysqld_safe which starts/stops/restarts the server? For example, where these records in the error log come from?
and more importantly
The obvious problem (that might or might not be the reason for the binlog issue, but there is a good chance it is) is that your server does not shut down properly upon restart, so during restart you have a concurrent use of essential files. Maybe binlog index is one of those, then, if both the "old" and the "new" instances write into it at once, it's not surprising it ends up with a garbage.
There are visible indications that the concurrency problem does exist.
First, look at the shutdown/restart (a relatively normal one, without a crash or anything):
So, the normal shutdown just started, it's not completed (it would have said "Shutdown complete"). But somehow, something decides it's "ready to mysql" and starts the new one. Apparently, it does not even SIGKILL the previous server before doing so (it would have been a bad idea, but starting a new one without stopping the old one is even worse):
As the log says, something is still using the aria control file. Given the above, it's apparently the old process which is still shutting down.
There are other weird artefacts which might be caused by the same issue:
etc.
I suggest to get rid of this problem (e.g. fix the wrapper so it waits for the shutdown to finish, with all proper checks on the PID), to start fresh and see if the binlog index corruption re-appears. If the wrapper is something we provide that I'm not aware of, please let me know.
We'll be waiting for your feedback.
Igor Pashev,
Judging by the logs, you have more problems than just binlog index corruption, and they appear to be related.
Do you have some kind of a wrapper around mysqld / mysqld_safe which starts/stops/restarts the server? For example, where these records in the error log come from?
and more importantly
The obvious problem (that might or might not be the reason for the binlog issue, but there is a good chance it is) is that your server does not shut down properly upon restart, so during restart you have a concurrent use of essential files. Maybe binlog index is one of those, then, if both the "old" and the "new" instances write into it at once, it's not surprising it ends up with a garbage.
There are visible indications that the concurrency problem does exist.
First, look at the shutdown/restart (a relatively normal one, without a crash or anything):
So, the normal shutdown just started, it's not completed (it would have said "Shutdown complete"). But somehow, something decides it's "ready to mysql" and starts the new one. Apparently, it does not even SIGKILL the previous server before doing so (it would have been a bad idea, but starting a new one without stopping the old one is even worse):
As the log says, something is still using the aria control file. Given the above, it's apparently the old process which is still shutting down.
There are other weird artefacts which might be caused by the same issue:
etc.
I suggest to get rid of this problem (e.g. fix the wrapper so it waits for the shutdown to finish, with all proper checks on the PID), to start fresh and see if the binlog index corruption re-appears. If the wrapper is something we provide that I'm not aware of, please let me know.
We'll be waiting for your feedback.